You probably already do, but if you need SSH, use crowdsec, SSHGuard or fail2ban to help filter bot nets.

I have IPs hitting from all over the world, trying logins all the time. Like several per minute, I can only imagine what it would be like if I wasn't blocking IPs with multiple failed login attempts.

No, there are other ways to get access to your machine without needing it. In general you can classify vulnerabilities as either code execution or privilege escalation, a code execution vulnerability allows an attacker to execute code on your machine, a privilege escalation allows him to break barriers that you might have imposed on him.

For example, if you're running service X as root, and someone manages to find a way to use something on service X to execute code, they might get a reverse shell to your box and run anything there. So you might set service X to run as your user instead of root, now that vulnerability is less important because it only compromises your user, but the attacker could use this one in conjunction with the other one to gain control of your user, then escalate to become root.

If this is something you're interested in, there's a cool website called hackthebox where you have to do these sort of things for real. If you want to have an idea on how it looks, there are some excellent videos here showing walkthroughs for many of them he boxes, I recommend checking something labeled easy since these boxes can get quite complex, but it will give you a good idea of the steps attackers need to take to compromise your system

Yes, or SSH keys or any other means of user authentication. The cool thing in this technique is that it's twofold and you (as an attacker) can cherry-pick the info given. If you walk up locally to someone's running system, you could skip the first half and go with the 'hey, can you resize this XFS image for me' bit.