Like daq mentioned, reduced battery life is one downside if you're on a laptop. RAM usage is also higher, usually 50-100MB more per instance than traditional terminals (sometimes more depending on the terminal and your config).
In terms of Ghostty specifically, it's still a fairly young project, so the chance of hitting an edge case issue is higher than if you were using a more mature GPU-accelerated terminal.