There already was a company called Brandless that tried to do the same thing with basic groceries like ketchup and paper towels. Looks like they're in the process of getting back to market, but seems like they had the same mindset.

Would be really nice to have something similar for clothes.