There's a big difference in perspective.
First of all, English is weird. It 75% loan words on a lower estimate. It also has over 20 vowel sound but only 5 vowel letters, and that works because you don't really need to pronounce the word, especially the vowels, all that well. English speakers have to learn that two Japanese words that sound the same to them are actually pronounce differently.
English has 3.2 billion people who natively speak a language related to it. Japanese is a language isolate.
In Japan, there is a main dialect people are expected to speak in formal situation. There's no main dialect of English, at least not worldwide.
There's also the big fact that English is the world language, which means that native English speakers are exposed to so many people who are not good at speaking English. Then you combine that with all the immigrant the USA receive, which probably includes your parents or grand parents. There's a reason I said that Americans speak broken English on a native level.
Compare all that to Japan. It makes perfect sense to an English speaker that being able to speak a language and communicate vs being fluent in a language are two very different things, but not to a Japanese person.