I also have a red/white hat that looks absolutely nothing like a maga hat, and I also stopped wearing it after a dude laughed and told me "man I thought you were wearing a trump hat I was about to kick your ass" (as if that was comforting lmao), so I understand the sentiment.
But also there's still a bunch of sports teams that do use red fabric as well, red fabric isn't exclusive to trump, and imo we should all learn to read instead of needing to avoid red fabric like a crip. It's really not that hard to tell a maga guy from some swiftie that bought "one of those Travis Kelce sports hats" tbh, just read the dang hat, y'know?