Well, all the languages has their problems. I find French more difficult than English. But in English there's many things that are hard to remember (not hard to understand). English is more straight forward to learn, even tho, it has so much exceptions in grammar. French has more complex grammar in general. And it ain't helping that it has feminine and masculine words, which you simply gotta remember which is which.
Out of all languages that I have learned, Japanese is the easiest to speak. Without a doubt. It took 6 months to become almost fluent in speaking in Japanese. After 15 years of English studies, I'm still not fluent in it. Tho, my Japanese is rusty now since I haven't practiced it at all in year or so. And grammar is very easy. Well, atleast for me... It reminds bit like Finnish with it's flexible word order. Writing and reading is of course somewhat more difficult than European languages. But you quickly catch it, too. Katakanas and hiragana (kanas are syllables) you can learn in couple months and be able to read and write with them. It then just takes rest of the life time to learn all the kanjis (characters that means whole words)...
Swedish I never learned. 6 years of studies for nothing... (well, I didn't even want to learn it but we had to study it...). It's said to be easier than English, tho. But I do not agree with that. Altho, I can understand bit of Swedish, thanks to English. But I could not survive if my life would depend on Swedish skills...
But the little I know about it, helps me to understand Dutch. Altho, my vocabulary is so small that I couldn't write one sentence in it... But I can understand little bit it when reading. Dutch is like mixture of French, English and Swedish.
Spanish looks kinda like French, to me, and it has about the same learning curve, I think. French might be harder tho, at long run. Italian seems to be the easiest of romance languages. At least it's much easier to pronounce for me than anything else (especially French is hard), and I can actually hear what they are saying ! Tho, I don't know anything about Italian grammar... Translating from Italy to English is very hard, tho. I'd say that it's even harder than from French to English. Altho, I'm not very good at either one.