That's how taxation works, but it doesn't solve government finance by itself.

The US government has always paid interest on the debt. It's never missed a payment. That makes it a very stable investment, and until recently it was considered the most stable, predictable investment that could be made.

That also means the US could have stupidly-low interest on its debts. And because the investment is so safe, it also creates a bottom for interest rates. No other investment is safer, so any time the fed rate goes up, all other loan rates go up as well - otherwise investors would just put their money in the US instead of on a home or business loan.

That and other factors result in inflation generally being higher than the interest on loans made to the US, which results in a situation where paying cash up front is actually more expensive than getting a loan and paying with future tax revenue, because future tax revenue grows with inflation that outpaces the interest on the loan.