Do pull backups instead of push backups: Backup server connects to local machine.

Encrypt before send, and if you want to have protection against deletions of the data have a cold backup offline other than during the copy.

Restic has quite a good solution for this: https://restic.readthedocs.io/en/stable/060_forget.html#security-considerations-in-append-only-mode

Append only mode.

I think you could do it somewhat like hetzner does for their storage boxes. You get an account that has read and write access to a directory and nothing outside. The accound can only run a limited set of commands, like ls, cat, nano, rsync etc. but has no access to commands that modify the filesystem.

Then you can use a copy on write fs like btrfs and make scheduled staggered snapshots.

I usually do 1x per year, 1x per month of current year, 4 per week of current montg, 7 per day in current week.

I have no clue what they use to limit the user accounts like that btw. but maybe that gives you a new jump off point for further research.

The suggestion I have heard is to have the remote machine connect to the machine on a schedule and pull the backups onto itself. Then your local machine doesn't have direct access to the backups, making it harder to compromise the backups if hacked. But this also assumes the backup machine is locked down and isolated so it is lower risk than the local machine.

Append only, like others are saying.

What you are talking about is Immutability and an append only backup. The s3 file system and some others has Immutability built in. Not all backups can do append only.

I have solved that by giving the distant machine the credentials to connect to the local machine. And the distant machine can't be accessed from the outside.

Scheduled snapshots (btrfs or zfs). If the compromised account deletes or modifies files, they're still there in the past snapshots

Filesystem-level snapshots are quite space-efficient because they don't make copies of all the files or even whole files; just the blocks that changed.

Personally as some extra spice as I worry about ransomeare, I have a few key files I check across my array that should never change. If any of their hashes are off, I abort immediately.

You've got the right idea with the permission change... the key is that you can have code executing on the remote side with different permissions. So the writer process has permission to write in one directory, and the turnsyle procees (often the root super-user) rotates the files or directories at a different time (or on a signal, sometimes).

I mean depends on the solution you are using, but you can have multiple accounts on the remote backup. IE so upon completion of the backup. The remote machine moves the backup to an offline or read only share (depending if you need those credentials to access the data again later),

Obviously most important thing is your credentials that make the backup... should be very limited in scope to just doing those backups.

WORM: write once, read many. Any good backup software supports this.

You could also keep offline backups. You can't compromise what you can't reach.

Yep offline backups are useful, although it does require remembering & making the effort to do it each time

Yes that seems like one way to go. Although I am using rsync so maybe keeping the files in place and changing owner could be an option

For doing snapshots did that means the local system identifies the changes? Or it all gets copied each time?

Certainly sounds relevant, although overall it is quite a different approach than in currently using

Are you connecting to the victim to push a backup into storage? If so, there's SO much you should improve on that.