You could say there is actually no such thing as a Matrix server either, because it's a protocol.

A Matrix server is a server that uses the Matrix protocol. You could be more precise and say Synapse but that excludes community-developed forks like Continuity, Contunuwuity, Tuwunnel, etc. etc. So yes, I maintain that is the "correct" terminology.