I don't really understand why you think this, can you explain? Signal stores, and has access to, no message metadata. They don't know who your contacts are, which group chats you're in, when you're sending messages, or who you're talking to.
To be convinced of this, take a look at the client source code, and compile the app yourself. None of this information ever leaves your phone without being encrypted or otherwise masked. No analysis of their server code is required to be convinced of this.