Yeah, that can certainly happen. It really depends on how the router handles contention, it's overall capability and performance, and what loaf balancing algorithms, if any, have been implemented. QoS basically guarantees that some form of traffic management is happening. With it off, it's really just a guessing game whether something will work well or fall over, as you've discovered.

Have a great day.