Idk if I'd say they "fuck everyone" given how much of the market is already heavily concentrated in a handful of hedge funds and investment banks. Most people don't have real exposure to the stock market. And of those who do, most don't indulge in active trading - they have savings in a 401k that maps to an index fund or other basket of blue chips, updating on daily or quarterly cycle.
The folks the HFT really fuck over are the day traders and investment bankers who are, themselves, trying to rapidly reposition ahead of market data. Warren Buffet's Berkshire team loses more to HFTers in a day than any lay citizen would lose in a lifetime.