That's all FUD. Matrix is as secure as Signal if you - like Signal - rely on a single centralized server. Actually, since you can host it yourself, it would be even more secure since you don't need to trust Signal.
(I defend infrastructure and perform hacks against cryptograph & protocols for a living)