535
Improve very slow library scans on Jellyfin 10.11 / 12 on spinning media (lemmy.ca)
submitted 3 months ago by to c/selfhosted@lemmy.world
82 comments fedilink hide all child comments
 

I've been trying to upgrade from 10.10 to 10.11 for a while now, as the Android TV app keeps nagging me, and every attempt ended with impossibly long library scan times.

After some thorough investigation, it appeared that a Home Videos type collection causes unending scan (yet to be solved), but also that Jellyfin does a lot of writes to the config directory (either database or metadata or both). Mine's on spinning media part of a ZFS pool. I tried a few performance tuning options, such as testing the config dir with recordsize (similar to block size) of 4K, 8K, 64K, 128K and library scans fell from 30-40 minutes down to 8-13min with 4K-64K. The ZFS tuning wiki suggest 64K recordsize with LZ4 compression for SQLite workloads such as Jellyfin. That seems to work as well as 4K and 8K but likely is faster when reading thumbnails and such.

Note that upgrading to 12-rc3, which is supposed to speed up library scans did not improve scan times for me. Optimizing config/database write speed did. I cross-checked the culprit by experimenting with moving the config dir to NVMe and RAM. Both of those got the scan times down to 8-9 minutes compared to the optimized spinning media's 12-13.

So if you had upgraded (or about to) to 10.11 your library scans are (about to get) dog slow and your Jellyfin's config dir resides on spinning media, optimize its write performance for SQLite.

you are viewing a single comment's thread
view the rest of the comments
[–] 14 points 3 months ago (20 children)

Yes you do. It just does it for you including handling SSL.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 20 child comments
    •
  • [–] -1 points 3 months ago (19 children)

    Inside the network you do, but not for remote access outside the network.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 19 child comments
    •
  • [–] 21 points 3 months ago (17 children)

    Outside the network you still have to open your server to allow plex cloud to access it and potentially tunnel your traffic through plex.tv if a direct connection doesn’t work. This happens automatically but it does happen

    With Jellyfin you just have to supply your own relay, like Tailscale or wireguard, which does require more setup but is completely free, means that you can stop your media traffic from being funneled through services that harvest your data (which plex absolutely does), and doesn’t go to shit if plex.tv goes down

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 17 child comments
    •
  • [–] 5 points 3 months ago (1 child)

    You can do the exact same thing with Plex, you just don't need to since automatic remote streaming is built in. In fact, the article's solution is literally Tailscale.

    You can also bypass plex authentication on any ip range you choose, so if you add your local network and plex's auth service goes down, it's no big deal.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 11 points 3 months ago

    Yeah obviously, you can also use Tailscale to tunnel to whatever service you want (including just to the server itself). But if you’re bothering to override remote plex access with all this I don’t know why you wouldn’t just pick the option that doesn’t harvest your data and increasingly treat its customers as hostile

  • permalink
  • fedilink
  • source
  • parent
  • [–] -1 points 3 months ago* (last edited 3 months ago) (14 children)

    This happens automatically but it does happen

    And that's the key, it happens automatically and it just works. With jellyfin you have to expose the web server to the internet, or point an app to something else exposed on the internet. With plex you don't. If you don't want to (or can't) a direct connection from your server to their servers, then their servers to your device is established. No VPNs, no reverse proxies, no port forwarding, nothing exposed to the rest of the internet.

    What I'm praying for is for Jellyfin to add a tailscale like direct peer to peer system. They can skip the backup funneling traffic through their servers. But just have something that coordinates a direct connection with just basic NAT.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 14 child comments
    •
  • [–] 10 points 3 months ago (10 children)

    They won’t do that because it’s expensive and free software generally doesn’t have the budget to do this

    I don’t know why you think the “just works” is any different from tunneling with a service. Your server is still opened to the internet through upnp with direct connections and through a tunnel to plex.tv when a direct connection is not possible. In fact plex is inherently less secure because their infra is both the encryption endpoint (as opposed to your client with Jellyfin and Tailscale or whatever) and their infra has been vulnerable in the past (like the massive breach in 2022).

    Jellyfin with something like nginx and a vpn is open to the internet, yes, but a service that tunnels (like wireguard, Tailscale) bypasses this issue and it’s up to you to set it up as to your level of comfort

    Plex is just easier but as with all things tech (especially those infected with VC dollars) “ease” translates to less secure and far more likely to exploit your data

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] [S] 1 point 3 months ago (9 children)

    I came up with a funny strategy I use to lock it down a bit. What's exposed to the internet for me is Apache2 reverse proxy. The proxy is locked down to reject all connections EXCEPT for the ones coming from a special subdomain which is something like a 64-character long random string. This prevents pretty any unwanted connections. Obviously the special subdomain must remain as secret as a shared password among the Jellyfin users. It works for trusted users.

    What I want ideally is an "authenticated firewall." OpenWrt rejecting all connections on the open port except for an allowlist of IPs. Then there must be a system where users can authenticate and their IP is added to the allowlist. I haven't found an off-the-shelf solution like this but I'll make it some day. Too bad I figured this random string subdomain trick cause it seems good enough for now. :D

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 1 point 3 months ago (7 children)

    How is your DNS set up for that subdomain? Is it on a wildcard DNS record?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] [S] 1 point 3 months ago* (last edited 3 months ago) (6 children)

    Do you mean the SSL cert? Yes, that's wildcard on *.mydomain.com. Then the subdomain is kvtn4ftxfreurdcw7qtr21mcywxaqqm.mydomain.com.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 1 point 3 months ago (5 children)

    In this case I mean the DNS entry for the random string. The thing pointing that subdomain at your IP/proxy.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] [S] 1 point 3 months ago (4 children)

    It's just an A record pointing to my IP. IP's updated from my router via the DNS provider API.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 3 months ago (3 children)

    If that A record isn’t a wildcard, anyone can see it, is what I’m getting at.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 1 point 3 months ago* (2 children)

    Hm. As far as I know that can only be done if AXFR is enabled (it's not), if the domain has entered some search engine that has remembered it, through SSL cert that's not wildcard, or through brute force lookup. Am I missing something?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 months ago (1 child)

    Give it a look on https://dnsdumpster.com/ or similar. Your dns records are inherently public, so anyone that works out the domain can work out the subdomain. In fact there are plenty of tools that just scan every domain/subdomain they can find.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 3 months ago

    Thanks for the pointer. Checked, tried a couple others - they don't know about it. They have some other non-random DNS records. AFAIK they can absolutely find it if they scan for all domains, but it'll take forever if the name is randomly-generated and sufficiently long. Someone has to be determined to spend the resources. This doesn't guard against that but against bots trying to fuck with the service at the port. I could move it to a wildcard though. There's an overlap with another subdomain (they're actually sub-subdomains and the first sub is common) but I could move that.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 3 months ago (1 child)

    It is literally a cloudflare tunnel/ tailscale type thing but contained in the app. The only real difference is that it is bundled in the app, so less setup and in exchange, Plex itself is free to harvest all of your data.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 3 months ago

    Let'scrypt and port forward. Sprinkle on some free ddns with a sync script/job to keep ddns pointing to your real public IP. Can even roll in some headscale if you're feeling adventurous

  • permalink
  • fedilink
  • source
  • parent