Learned some Spanish in high school and went beyond curriculum because I really like linguistics. Started learning Japanese in middle school, fell off, and picked back up on it a few months ago.

Japanese is definitely more difficult for the average native English speaker.

With virtually no languages beyond dialect distinction can you take an idea from one and directly translate it to the other and be fully fluent—however your rate of success doing so is going to be higher between English and Spanish than it'd be between English and Japanese.

I will say, however, as a native American English speaker, I have an easier time listening to Japanese than I do Latin American Spanish.