Well, one could argue that it is the other way around. We consider most mushrooms as feasibly edible when it is easy enough to distinguish them from non-edible ones. There are thousands of mushroom species out there that are probably edible but that are just not worth anyone's time. And we also focus more on certain groups of mushrooms that contain a higher percentage of edible ones, like boletes (and also champignons).

All mushrooms are edible. At least once.

There are edible mushrooms which are really hard to ID, but you just don't try to eat those.

Most people don't go after the tasty Sheathed woodtuft (Kuhernomyces mutablis) because the risk of confusing it with the deadly funeral bell (Galerina marginata)

Check out the wikipedia page for destroying angels. Members of this family can look very similar to button mushrooms or puffballs when they're in the early stages of growth.