When messages are sent and the traffic does have to physically get to the recipent. That is to say you would need to schedule sends (which it can do) and use something obfuscate the network trafic (i.e. Tor, a VPN) and the reciptent would still be exposed in that case.

So if Signal's server was compromised and network traffic could be analyzed you determine who was sending to who and at what times. This could be leveraged for more targeted attacks or simple censorship.