Feature suggestion: Exclude open files based on timestamp

[Toby Gerenger]

Hello. I've been using SnapRAID for my HTPC server, which includes a SageTV recording server.

I occasionally run into an issue where the server is recording a new video file at the same time a "sync" operation is scheduled, which causes the log file to include a few hundred errors, and eventually end with a "fatal" error. The sync runs fine if I later rerun the command when the system is idle.

It occurred to me that this issue might be dealt with automatically if there was a way to exclude recently-modified files based on a timestamp, so that I could exclude any files that were less than 5 minutes old (for example). That way, any files that were currently being modified could be automatically excluded (until the next scheduled sync).

Just a suggestion (and great software, BTW!)

Cheers,
Toby G.

[Manuel Succo]

I haven't encounterd this problem, yet, but I see it. Especially for larger SnapRAID volumes (for which the software has been designed), the sync process can take many hours. Is it impossible to move files to the drives in that time without causing an error? Or is the answer yes, and the problem lies with the recording server writing to the file over longer periods of time? If that is the case, you should record to a temporary drive and copy after the recording is finished.

[Linus Vikstrom]

Instead of a separate drive you can also create a temp-folder to record to that you exclude from Snapraid, then just do a filemove to the correct folder once recording stops (can probably even script it in SageTV).

edit Or simple record to a temp file extension and just rename the file in the folder once done. You can have Snapraid ignore e.g. .mkv.recording and then rename it to .mkv (or whatever format it records to)

[Leifi Plomeros]

A little quote from Microsoft on the topic:

Time stamps are updated at various times and for various reasons. The only guarantee about a file time stamp is that the file time is correctly reflected when the handle that makes the change is closed.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx

In other words, at least for NTFS, as long as the recording/downloading/copying/whatever, is ongoing the last modified time stamp will remain unchanged even if the file has been continously modified for days.

[Andrea Mazzoleni]

Hi Toby,

More than the timestamp, it's more relevant if the file is open by some application.

We could exclude new files that are still open when the sync process is started.

I'll add it in the TODO list.

Ciao,
Andrea

[Toby Gerenger]

Thanks, all. These are all good thoughts. Note that I don't have control over the file location or file name, as that is handled by the SageTV software, so I don't think I can use an excluded temporary directory for active recordings or a temporary file name as the exclusion criteria.

The file is generally open for the duration of the recording (e.g., if the show being recorded is an hour long, then the file is open for an hour). I run the SnapRAID sync process in the middle of the night when there isn't typically any show being recorded, but it does occasionally happen. The issue is then normally resolved by the next sync (which happens the next night), but it seems to leave things partly unsynced for that 24 hour period.

The ability to exclude open files as of the start of the sync process would be a useful feature to reduce conflicts with open files.

Appreciate all the feedback!

[Leifi Plomeros]

Maybe a good enough alternative would be to add an optional sync parameter where user can select a minimum last modified age for new files?

Something like this: Snapraid sync --min-age [minutes]

[Andrea Mazzoleni]

I've not yet found a reliable way to check if a file is opened by another application,
and checking the age of it seems a good compromise.

Added in the TODO list also the --min-age possibility

Ciao,
Andrea