One of the professors in my department, Arnold Rosenberg wrote a paper "The hardest natural languages" which appeared in Linguisticae Investigationes III (1979) 323-339.
Speakers of English use the phrase "It's Greek to Me" to indicate something they don't understand. Rosenberg hypothesizes that this makes Greek a harder language than English. Through extensive research, Rosenberg learned that speakers of Greek use a phrase that means "It appears Chinese to me" making Chinese harder than Greek. He also documents an interesting cycle in which Arabic is harder than Turkish, Turkish is harder than Persian, and Persian is harder than Arabic!
So what is the hardest natural language? According to Rosenberg, it is Chinese, because of their saying "It's Heavenly script to me."
Feel free to read the paper for more information.