If you aren't transcoding, and the player is taking too long to cache the video before starting, you might be having some sort of storage issue. You would need to try a couple of different things to figure out what, specifically, is taking so long to send the video out.

The first thing that comes to mind is that your storage is on an SSD, and it is nearly full. An SSD that is nearly full will usually perform much much worse than it would if it had more space to work with. https://pureinfotech.com/why-solid-state-drive-ssd-performance-slows-down/

The next thing that comes to mind is that your files are stored on the same drive that jellyfin transcodes onto, and it is not using an SSD. If you have jellyfin reading from a single drive, jellyfin encoding to that same drive, and also everything else also running, you might be causing your hard drive to seek a lot in order to get everything up and running. You could test this by changing the jellyfin transcode location to a different storage device.

I've also found that page and video loading times tend to be directly affected by the storage medium's seek times. If you had jellyfin installed on the same hard drive as your videos, it will be slower than if you had installed jellyfin on a ssd separate from the drive you store your videos on. This one wouldn't likely result in minute loading times though.

Are you streaming to a Chromecast, by any chance? Or a older Galaxy device?
The 4k Chromecast with Google TV does not support AV1, but the 1080p version does. Jellyfin tried Direct Playing AV1 files, which obviously went poorly.

I run the same CPU in my NUC as you do, and all data is on a NAS shared with NFS. It's been absolutely bulletproof for about a year, so I'm confident you should be able to make this work.

Any other containers running at the same time?
No cooling issues?

Just asking because mine dropped massively in temp when repasted.

Check the log then, it'll tell you the speed. 1x speed is exactly realtime. 0.5x is unable to keep up, and playing at half speed. 2.0x is streaming at double playback speed.

Do any videos play well? What's the difference between those and others?

Check your disk access usage maybe? Like if you store things on an external drive with USB2 or something you're gonna have a bad time with multiple videos/high bitrate stuff because you're saturating the bandwidth of the connection