I believe it's mostly illegal for both parties, but in practice less often enforced for the downloading party, as this enforcement would require too much resources for the enforcing side.

To give concrete examples, downloading pirated material is illegal in both the U.S and in Sweden, and afaik the latter is on par with the rest of the EU.