They store your phone number, and have to route all the messages you created to the other phone numbers / user IDs in their database. This means anyone with access to signal's centralized database has social network graphs: who talked to who, and when.

If your threat model is "I just trust them", then its not a good one.

Privacy advocates have been raising the alarms about signal forever, but like apple, their fanbase just feels the security "in their gut", and think that because it has a shiny interface, it must be secure.

The only people who know what the server stores are the people running it.