You know, I've been thinking...
Signal's end to end encrypted, yes... But we do the key exchange process through Signal's servers, don't we? How do we know they don't store copies of the keys? Does the client have a mechanism in place to make sure the man in the middle doesn't do anything funny? I haven't actually delved very deep into the code, but it sounds like I should.
And... Sure, their server code may be open source too, but nobody guarantees that that's the code actually running on their servers.