So it's my first time setting up a VPS. Is it to be expected to ban 54 IPs over a 12h timespan? The real question for me is whether this is normal or too much.

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  |- Total failed:     586
|  `- Journal matches:  _SYSTEMD_UNIT=ssh.service + _COMM=sshd
`- Actions
   |- Currently banned: 51
   |- Total banned:     54
   `- Banned IP list:   [list of IPs]

fail2ban sshd.conf

$ sudo cat /etc/fail2ban/jail.d/sshd.conf 
[sshd]
enabled = true
mode = aggressive
port = ssh
backend = systemd
maxretry = 3
findtime = 600
bantime = 86400

I have disabled SSH login via password. And only allow it over an SSH key.

$ sudo sshd -T | grep -E -i 'ChallengeResponseAuthentication|PasswordAuthentication|UsePAM|PermitRootLogin'
usepam no
permitrootlogin no
passwordauthentication no