The difficulty is pretty comparable. French grammar and vocab is much easier, especially the more formal or complicated the subject matter. But, because the German rhythm is timed to the syllable-stress on the word rather than the whole phrase, like French and most kinds of Spanish, it's quicker for a lot English-speakers to learn to understand spoken German and to get a more natural-sounding accent. But then, French is extremely standardized such that outside of parts of Canada, you'll rarely meet a French speaker you can't communicate easily with in French, while real-world spoken German has an enormous variety of dialects, some of which can get quite weird... especially in Switzerland, where people also tend to get annoyed if they have to speak the standard language.
Not being good with accents myself but having spent several years in French-speaking countries, I speak it confidently but with an immediately noticeable accent. Perversely, despite having spent almost no time in Germany, I find it relatively easy to have a convincing accent in German.