#33 Manual non-destructive database refresh

open
nobody
None
5
5 days ago
2010-11-21
Eric Sauvageau
No

MiniDLNA fails to pickup changes done to networked filesystems (no inotify support for these) or done to local filesystems while it's not running. I would recommend providing a way to refresh the database content that doesn't involve rebuilding it from scratch. This could be done either through a command line switch, or on a configurable timer (db_refresh=14400 would refresh the DB every 24 hours)

Related

Feature Requests: #79

Discussion

  • Teddy
    Teddy
    2010-11-23

    I too would like MiniDLNA to be able to do a "refresh on demand" - some way to simulate the "inotify" event.

    For example, here's the simplistic solution that if implemented, would work for me perfectly... If MiniDLNA gets SIGHUP signal, it can look for a certain text file (the file name can be configurable), and then read from it the list of directories and files that have to be added to (or refreshed, if it already exists) or removed from the database. Each line in the file will start from "+" or "-" sign (to indicate "add/refresh" or "remove") followed by the directory/file name, and can be handled the same way as corresponding inotify event.

     
  • Ric
    Ric
    2010-12-12

    I've been looking for the same feature. At least a smart rescan option should be provided at minidlna startup:
    - take a fast fingerprint of each file (i.e.: use file size, file name and crc32 of a bunch of data at the end of file)
    - if fingerprint already in database, use metadata from previous entry
    - if fingerprint not in database scan for new metadata

    Thank you

     
  • The logic behind inotify seems to be easy to leverage for incremental scans. It is possible to scan all dirs and call inotify_insert_file for every file, its logic will ignore files whose timestamps have not changed and skip them. This is efficient, but will not detect modified files with unchanged time. Maybe add a force flag or something. Detecting deleted files is slightly harder. Best approach is to add a "present" column to the object table, make it default to "true". When scanning, set "present" on all rows to "false", as every object is seen, set the value to "true". At the end of the scan, call inotify_remove_file for each file where "present" is "false". Add a global lock so you don't get into this routine twice or a real mess will be the result.

     
  • Will
    Will
    2011-02-20

    Sounds good to me, personally I like how ushare handles it.

     
  • Unfortunately creating a index for a larger database can be very time consuming, for this i think non-destructive database refresh can be a good improvement

    Thank you

     
  • I really would like to see this feature as well. On a fast machine scanning and indexing all my media files still take 20 minutes.

    It would also be nice if it was documented somewhere in the files provided in the distribution that the following setting must be larger than the number of media files:

    echo 65536 >/proc/sys/fs/inotify/max_user_watches

     
  • Yes war59312 I tried with ushare and it is incredibly faster. For 700 Go of files - Ushare some minutes, Minidlna hours - (Nas DNS 323)

    But Ushare use a trick, it work only in memory not with sqlite

     
  • tehsu
    tehsu
    2012-01-12

    I think this would be a good idea as for OS X I don't see anyway to use inotify.

     
  • James Pharaoh
    James Pharaoh
    2012-04-12

    I also think this is a very important feature. jsanchez and cionci77 have the right idea - it should be a light refresh which only examines the file meta data unless it looks like something has changed.

    I don't think handling deleted files is hard as jsanchez suggests. You just compare the list of each directory you scan to the list in the database as you go. No need to do it all in one batch at the end.

    I think cionci77's idea to crc the end of the file is a bad idea. This will slow things a lot. We can simply use the timestamp (and other meta data) to detect probably changes.

    This should probably be the default on every startup, probably at the same time as the inotify watches are placed. I guess some people might want to disable it for efficiency but I think the "least surprising option" is for it to perform the scan.

    I might have a go at this some time soon. It would be interesting to hear from the project maintainers on it?

     
  • James Hewitt
    James Hewitt
    5 days ago

    I'm also tempted to try writing this up.