In the US, copyright is limited by Fair Use. It is still IP. Eventually, you'd just be changing how Fair Use works. Not all for the better, I think.
Maybe one could compare it to a right of way over someone's physical property. The public may use it for a certain purpose, in a limited way, which lowers its value. But what value it has, belongs to the owner.