I've bought a wool cap blank from Etsy.

If you look how industrial hat production works, say, how they do cowboy hats, they basically do blanks separately and have dedicated craftsmen that have steam tools shape the hat, polish it and attach ribbons & stuff.

So to match the wool base I've bought wool felt - the wool fabric itself doesn't really retain shape. Just beware that some of them look more like plastic while some look more wool-ish (and bit more soft & bendy). I've ordered black & purple felt and black was like the first and purple like the later, so I just went with all purple.