With modern battery tech, that "waiting around for it to charge" would take 5-10 minutes. Let's assume that that 100 mile range car has a 30kWh battery (my 250 mile range car has 63ish I think). If it runs 800V architecture like Hyundai/Kia, it can charge at 350kW, but let's round down to 300 for easy math. Even if you got down to 0% as you got to the charger (which you wouldn't assume for a gas car getting to a gas pump, so that's a stretch), it'll be so fast to charge that it doesn't even make sense to walk away to buy a coffee.
30kWh/300kW=(1/10)h=6 minutes.
I rounded in unfavorable directions, which probably accounts for losses, more or less.
"Sounds expensive," you might be thinking. Nope. I've never seen it above $0.65/kWh, but again let's round for easier math and assuming bad luck. At $0.75/kWh, that 6 minute 0-100% 30kWh charge costs $22.50. Slightly more expensive than gas, but this is presumably for uncommon trips, so charging at home where your rate is probably closer to $0.15/kWh the vast majority of the time will more than offset those day trips since that refuel costs $4.50. Even if every other charge were at that high rate, that's still an average of $13.50 for a full tank. Assuming a gas car with 25mpg, that's like paying $3.38/gallon on average. Not great, but better than reality right now.
If it's more like every tenth charge is in the city at that high rate, then your average charge is $6.30, which is like paying an average of $1.58/gallon.

So it doesn't take longer to refuel this EV than a standard gas car, and it's cheaper than a standard gas car if you're refueling at home at least half the time. The only real variable here is whether there is a charging station at the optimal location for your journey, and those are popping up everywhere.

In reality, I agree with you, but for a different reason. For battery longevity, it's best to keep the state of charge at 20-80% most of the time. So I'd rather have a 200+ mile range to do my 100 mile trips. And that's what I did. I bought a used Bolt EUV with a range of about 240 miles (down from 250 when new) and I charge to about 75% for my daily 90ish mile commute, and I get home and plug in at like 30%.

I'm actually out of town with my EV right now. I charged to 100% at home the night before, drove a little over 2 hours to this hotel, a bit over an hour round trip for dinner last night, probably about 30 minutes round trip today, and I'll probably stop for a short charge on the way home tomorrow just for peace of mind. It's nowhere near as big a headache as most people make it out to be. For most people, if you need to buy a car and you have the ability to charge at home while you're sleeping, it doesn't make sense to not buy an EV. Used ones are still cheaper than they should be.