Matrix is only really enjoyable with unencrypted chats.

I have fairly recently switched phones. Now, a good number of older messenges fail to decrypt. Even when I don't switch devices, a group chat for a three day long event with a very small group already gave us lots of fun decryption errors. Oh, and I have this other fun group chat where Fluffy Chat constantly reminds me that people whose devices I haven't verified will be able to read my messenges every time I send something. Also, Matrix has coutless clients with different feature sets and I heard calls are a pain to ßet up.

Signal, on the other hand, just works. Federation and decentralization is obviously nice but a functional product is more importmant. I haven't had a chance to try XMPP unfortunately.