Is the stiff part of the brim cardboard or plastic? If cardboard, you have to be careful of warping it if it gets saturated. I would try and enzyme-based cleaner like oxyclean. Maybe a string solution of it and soak it if not cardboard. You might also try putting it in the top rack of the dishwasher to machine wash it without it getting bent.