What is the client situation?
The easiest fix is to find a client that can direct play all of your files and take transcoding out of the equation. Ugoos am6b+ as an example but if you don’t need Dolby vision there are cheaper options that are easier to configure with native jellyfin clients (instead of coreelec/kodi). Or if you need av1. But this needs to be done per user and costs money
Alternatively what is your hardware? Do you have intel quick sync video? If so do you have hardware transcoding setup? Like if you have Jellyfin setup in a docker are you passing through the igpu to the container? And if you’ve done that have you set up the hardware transcoding in Jellyfin? What gen cpu and what kind of files?
If you have the transcoding happening on the CPU and not the iGPU (assuming you don’t have a discrete gpu in a mini pc and frankly with quick sync you don’t need one unless your cpu is ancient, save the power usage) transcoding will crawl. But if you have quick sync video it should be fine with <4-5 users