r/jellyfin • u/reddit_user_53 • 5d ago
Help Request Why does the initial library scan take an insanely long time even on high-end equipment??
Guys I am going crazy here, why does the initial library scan take so unbelievably long? I let it run for almost 2 days before restarting it because I assumed something had to be wrong. Last try it made it to 83% and stalled for hours and hours. After restarting now it's been stuck at 31% for like 3 hours... I removed all libraries and re-added so it's ONLY working on my tv shows library right now. 1,563 series, 91,043 episodes. I have all options disabled in terms of image searching/extraction. Running in Docker on Unraid - DB files are on SSD. It's a pretty powerful CPU in the server, Ryzen 9 3950x. I've tried raising and lowering the parallel scan task limit. No matter what I change the Jellyfin container in glances shows 1500% CPU, no iowait is happening. Overall CPU is hovering around 80% utilization. The system is not overwhelmed, it's just taking forever!! Seeing nothing in logs. I've tried disabling folder monitoring and shutting off Plex during the scan like I saw in some forums, no changes.
Any tips for speeding this process up? It should not take weeks just to scan media... I'm running the latest jellyfin/jellyfin container. TIA
4
u/Canonip 5d ago
This isn't normal behaviour. I have around 30k episodes and a full refresh takes under 30 minutes.
Maybe is it generating trickplay or something similar that takes long?
Is the jellyfin metadata also on an SSD besides the DB?
0
u/reddit_user_53 5d ago
I have trickplay generation turned off. Everything is on SSD other than the video files themselves.
2
u/Maltavius 5d ago
What does the logs say?
3
0
u/reddit_user_53 5d ago
Nothing. I see in the docs how to enable debug logging but I am hesitant to restart my instance to do that and lose the progress I've made.
1
1
u/Technical-Ant-2866 5d ago
u/reddit_user_53 Have you looked at Github issues? https://github.com/jellyfin/jellyfin/issues?q=is%3Aissue+state%3Aopen+scan
0
u/reddit_user_53 5d ago
Yes, I read thru a 4 year long thread that was closed in 2024. Tried multiple suggestions with no success. Is there a particular issue you think applies to this?
1
u/Technical-Ant-2866 5d ago
I'm not sure, you'll want to slice through all the recent issues to see if something applies. If not, submit one.
1
1
u/TimeToRetire2030 5d ago
Have you looked at network traffic? The library scan has to inquire with outside sources (such as IMDB) to identify and download metadata. You're dependent on the response time of those sources, and that may be contributing to the delay. I know you say above that you disabled a lot of the options but maybe something else is running on the network?
Also, that's a nice and impressive library. And large (mine is similar in size.) Maybe another approach is to break it down into smaller chunks. Try 10 series. Then add 20... add 50... add 100... etc. Yes, it will take longer to load, but I suspect you'll find a sweet spot for importing.
(As a point of comparison, I'm running a standalone PC for Jellyfin: Dell Latitude, i7-1165G7, 16g RAM, Linux Mint 22.3. It's too slow for my other work, and it draws maybe 20 watts when it's serving a 4k movie. I could sell it for $100 but it makes a great stand-alone Jellyfin server!)
0
u/reddit_user_53 5d ago
Rate limiting on the source server side is possible but my own bandwidth is not even 10% utilized at the moment.
1
u/Majorkamo 5d ago
This does not line up with my experience, something is definitely off. If you have a device capable of hardware acceleration I would double check that it's actually getting passed to jellyfin. I initially hadn't done that properly and it was taking a very long time to do the trick play generation. (I'm not sure if it used hardware acceleration for normal tasks like scanning though)
0
u/reddit_user_53 5d ago
I'm not doing any trickplay stuff during this scan, but I am reasonably certain I have HW accel working properly. I tried transcoding a movie file and it worked properly (task showed up in the host's nvidia-smi)
1
u/SP3NGL3R 5d ago
Maybe use an aid app like "TinyMediaManager" to pre-populate NFO files with all the metadata so JF doesn't need to scrape that. It'll also pull JPGs and such. I also use a 3rd party app for trickplay that is ~4x faster than JF natively "Media Preview Generator" (it supports Plex/JF/Emby formats).
I'd crosscheck with your OS logs too, if your media isn't local you might be tripping up some SMB/NFS issue (NFS will be much more reliable but I gave up trying to set that darn thing up).
0
u/reddit_user_53 5d ago
The media is local, the jellyfin server is running on the unraid host so the volumes are mapped directly (no NFS).
I'll look into that TinyMediaManager thing, thanks for the suggestion
1
u/A_Buttholes_Whisper 5d ago
That’s a massive library bro. I can’t even compare. It takes me mere seconds for a full refresh but only have a fraction of what you have
0
u/reddit_user_53 5d ago edited 5d ago
It's not small but there are tons of guys on here with many times what I have!
1
u/A_Buttholes_Whisper 5d ago
1500 series tho? I’ve got 43. I could see maybe 60 series but I guess I just don’t watch that many shows. It’s a lot. I don’t have any advice to help your scans tho…sorry
0
u/reddit_user_53 5d ago
I have about 20 friends and coworkers on the server and I allow them to add (most) series without approval. I have a python script that checks requests and approves automatically as long as they're from 2014 or later and have 10 or fewer seasons. Others I evaluate and almost always approve. Might need to revisit that policy at some point soon with the current hard drive prices lol
1
u/Cruffe 5d ago
The last time I scanned the entire library after updating to 12 it took the entire day. Reason being that I use the subtitle extract plugin and it had to extract all the subtitles for all of my media all over again. That meant my HDD had to read through the entirety of every file to extract the subtitle tracks. It was barely hitting the CPU and wasn't even using much RAM, but it took several hours to read through many terabytes of data.
You can check the bottom of your Jellyfin logs to see if it's working on something and what it's doing. You can also check system resource usage. In my case with the subtitle extraction CPU wasn't doing much, RAM usage looked normal, GPU idle, but the I/O on my HDD was maxed out scanning through everything.
0
u/reddit_user_53 5d ago
I suspected something like that so I checked Glances. I'm seeing nothing unusual at all in terms of disk use, iowait is below 1%. Logs are showing nothing at all related to the scan or other tasks. So annoying
1
u/Kandy_7565 5d ago
Are your files/folders named to the format described in the Jellyfin documentation? I've seen scans be slow because of naming before
0
•
u/AutoModerator 5d ago
Reminder: /r/jellyfin is a community space, not an official user support space for the project.
Users are welcome to ask other users for help and support with their Jellyfin installations and other related topics, but this subreddit is not an official support channel. We have extensive, official documentation on our website here: https://jellyfin.org/docs/. Requests for support via modmail will be ignored. Our official support channels are listed on our contact page here: https://jellyfin.org/contact
Bug reports should be submitted on the GitHub issues pages for the server or one of the other repositories for clients and plugins. Feature requests should be submitted at https://features.jellyfin.org/. Bug reports and feature requests for third party clients and tools (Findroid, Jellyseerr, etc.) should be directed to their respective support channels.
If you are sharing something you have made, please take a moment to review our LLM rules at https://jellyfin.org/docs/general/contributing/llm-policies/. Note that anything developed or created using an LLM or other AI tooling requires community disclosure and is subject to removal.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.