Firstly it is the texture, I have ADHD and liked auhd, and texture play a huge role in what I can and cannot eat.
I have tried more than the white button mushrooms in the grocery store, but living in a small rural community that is all that is there, so it is free and far between.
Secondly, the ones I have tried tasted like dirt and sadness. A few years back Jersey Mike's removed their vegetarian wrap from the menu and replaced it with a grilled portabelo sandwich. In preparation for the change I ordered a pizza with mushrooms, I ate one piece with them, the second piece I picked them off, after starting the 3rd piece I tossed it.