Even if you fixed the issue with drivers...

...your modem runs it's own firmware with a lot of extremely shady behavior, and you can't touch that regardless of which OS you install. Even your SIM card can arbitrarily execute Java applets and fetch from the network without your command, but at least it's somewhat contained. Your modem though, it can do a lot without your control and people like Qualcomm have been caught doing nasty stuff with it (plus, of course, giving the US the data whenever they ask for it).

This is why people like Stallman and Snowden often talk about teaching users how to use libre software on their computers, but rather than pushing for the same with smartphones, they tell you to not touch these at all instead. They're fundamentally anti-privacy devices, built this way.

Of course I carry one, it's fairly hard to live without a phone nowadays, but we must be aware of the impossibility of fully containing the data harvesting they do.