The main oversimplification is where browsers "just visit websites", SSH can be really powerful. You can send/receive files with scp, or even port forward with the right flags on ssh. If you stick to ssh user@host without extra flags, the only thing you're telling SSH to do is set up a text connection where your keyboard input gets sent, and some text is received (usually command output, like from a shell).

As long as you understand what you're asking SSH to do, there's little risk in connecting to a random server. If you scp a private document from your computer to another server, you've willingly sent it. If you ssh -R to port forward, you've initiated that. The server cannot simply tell your client to do anything it wants, you have to do this yourself.