#33 Manual non-destructive database refresh

[Eric Sauvageau]

MiniDLNA fails to pickup changes done to networked filesystems (no inotify support for these) or done to local filesystems while it's not running. I would recommend providing a way to refresh the database content that doesn't involve rebuilding it from scratch. This could be done either through a command line switch, or on a configurable timer (db_refresh=14400 would refresh the DB every 24 hours)

[Teddy]

I too would like MiniDLNA to be able to do a "refresh on demand" - some way to simulate the "inotify" event.

For example, here's the simplistic solution that if implemented, would work for me perfectly... If MiniDLNA gets SIGHUP signal, it can look for a certain text file (the file name can be configurable), and then read from it the list of directories and files that have to be added to (or refreshed, if it already exists) or removed from the database. Each line in the file will start from "+" or "-" sign (to indicate "add/refresh" or "remove") followed by the directory/file name, and can be handled the same way as corresponding inotify event.

[Ric]

I've been looking for the same feature. At least a smart rescan option should be provided at minidlna startup:
- take a fast fingerprint of each file (i.e.: use file size, file name and crc32 of a bunch of data at the end of file)
- if fingerprint already in database, use metadata from previous entry
- if fingerprint not in database scan for new metadata

Thank you

[Julio Sánchez Fernández]

The logic behind inotify seems to be easy to leverage for incremental scans. It is possible to scan all dirs and call inotify_insert_file for every file, its logic will ignore files whose timestamps have not changed and skip them. This is efficient, but will not detect modified files with unchanged time. Maybe add a force flag or something. Detecting deleted files is slightly harder. Best approach is to add a "present" column to the object table, make it default to "true". When scanning, set "present" on all rows to "false", as every object is seen, set the value to "true". At the end of the scan, call inotify_remove_file for each file where "present" is "false". Add a global lock so you don't get into this routine twice or a real mess will be the result.

[Will]

Sounds good to me, personally I like how ushare handles it.

[Frederic Bourgeois]

Unfortunately creating a index for a larger database can be very time consuming, for this i think non-destructive database refresh can be a good improvement

Thank you

[Lars Rönnbäck]

I really would like to see this feature as well. On a fast machine scanning and indexing all my media files still take 20 minutes.

It would also be nice if it was documented somewhere in the files provided in the distribution that the following setting must be larger than the number of media files:

echo 65536 >/proc/sys/fs/inotify/max_user_watches

[Frederic Bourgeois]

Yes war59312 I tried with ushare and it is incredibly faster. For 700 Go of files - Ushare some minutes, Minidlna hours - (Nas DNS 323)

But Ushare use a trick, it work only in memory not with sqlite

[tehsu]

I think this would be a good idea as for OS X I don't see anyway to use inotify.

[James Pharaoh]

I also think this is a very important feature. jsanchez and cionci77 have the right idea - it should be a light refresh which only examines the file meta data unless it looks like something has changed.

I don't think handling deleted files is hard as jsanchez suggests. You just compare the list of each directory you scan to the list in the database as you go. No need to do it all in one batch at the end.

I think cionci77's idea to crc the end of the file is a bad idea. This will slow things a lot. We can simply use the timestamp (and other meta data) to detect probably changes.

This should probably be the default on every startup, probably at the same time as the inotify watches are placed. I guess some people might want to disable it for efficiency but I think the "least surprising option" is for it to perform the scan.

I might have a go at this some time soon. It would be interesting to hear from the project maintainers on it?

[James Hewitt]

I'm also tempted to try writing this up.

[wedok]

7 years later and it still doesnt work :)
linux sucks !!!

why aren't you developers so intelligent and let minidlna work like vsftpd?
it doesnt scan overall!
u just browse through the folders and when a file with a compatible extension (.avi, .mp3, .mkv) is found it is shown and can be selected for playback?

[Shrimpkin]

why aren't you developers so intelligent and let minidlna work like vsftpd?

Because of UPnP-AV/DLNA specifications requires more data than the filesystem alone can provide. Take a look at the specifications available online (easily found using a search engine) to become "intelligent" too!

[Shrimpkin]

Patch available for testing here.