Depending what you find acceptable there are things you can do. For example at work we proxy through a mailgun account to do simple smtp notifications from bash scripts and webapps.I would imagine you could also proxy from an email server.

And you can at least manage your own email domain accounts via various webhosts. This may not meet your criteria for self hosted but its more independent then using gmail.

Instead of trying to build reputation, you can buy it. Pick a reputable spam solution that offers accounts for like $3-5/m and route all mail through it.

Still your server, your mail, but you're buying their business reputation, and getting some spam filtering back. Plus you might get some protection from outages or maintenance windows if they can cache some mail for delivery for you

So it boils down to it's doable but it requires a lot of effort and dedication.