Signal stores, and has access to, no message metadata.
Phone numbers are the most important metadata you can give them, far more important than message content. It means your real identity / name and address. With phone numbers you can build social networking graphs: who talked to who, and when.
To be convinced of this, take a look at the client source code, and compile the app yourself.
Client source code is irrelevant here. Signal is a centralized service, you can't verify what their US-based server is actually running (although they did go a full year without publishing any server updates at one point, until they received a lot of backlash for it).
None of this information ever leaves your phone without being encrypted or otherwise masked.
You gave them your phone number / real identity when you signed up. The most important piece of info they could possibly give them, you already did.