ENTRY_RENAME_FROM_TO or Rename Cookie

Hey Uwe,

thank you very much for the great library!

I have a question regarding a combined rename/from-to event: I know that inotify (Linux) offers a cookie to map the two rename events (FROM + TO). Is there a way to access this cookie using jpathwatch, or is there another way to map the two events?

Regards,
Philipp

Hey again,

I looked at the code and saw that you're receiving the cookie from inotify, but you're not using it.

Here's a fix that works great on Linux:

1. Create a class "LinuxMovePathWatchEvent":

public class LinuxMovePathWatchEvent extends PathWatchEvent {
    private int cookie;
   
    public LinuxMovePathWatchEvent(Kind<Path> kind, Path path, int count, int cookie) {
        super(kind, path, count);
        this.cookie = cookie;
    }

    public int getCookie() {
        return cookie;
    }
}

2. Change two lines in the "LinuxPathWatchService", method "inotifyEventHandler" in the if-blocks "IN_MOVED_TO" and "IN_MOVED_FROM":

In both cases (line 183 + 193 or revision 130), replace the following line:
- Remove: key.addWatchEvent(new PathWatchEvent(kind, path, 1));
- Add: key.addWatchEvent(new LinuxMovePathWatchEvent(kind, path, 1, cookie));

et voila, you can access the cookie like this:

if (event instanceof LinuxMovePathWatchEvent) {
    LinuxMovePathWatchEvent moveEvent = (LinuxMovePathWatchEvent) event;
    int cookie = moveEvent.getCookie();
    …
}

Could you maybe integrate this?
Is this also possible for Windows/Mac? Do they have "cookies"?
Regards,
Philipp

Hi Philipp,

The basic assumption for RENAME_FROM and RENAME_TO is that they always appear in pairs in the queue, so that you can assume that no two RENAME_FROM will appear without a RENAME_TO in between.
However, I see that the current implementation may have an issue here, because as you rightfully point out the cookie is simply ignored in LinuxPathWatchService.inotifyEventHandler(), and nothing is done to ensure the order of events.

I think I probably overlooked this because on Windows this order is guaranteed by the kernel (so there is no cookie). On OSX/FreeBSD there is no way to track renaming for files in a directory.

The bottom line is: This could be a bug in LinuxPathWatchService, but I'm surprised noone complained about this earlier. A solution would not be to add a new event or to provide the inotify event cookie, but instead to maintain a consistent API across platforms by fixing the issue at hand (if there is one).
The idea is to batch up RENAME_FROM events until a matching RENAME_TO appears, and then to release both at once into the queue.

It'll take some time until I can look at this though, because my Linux PC is currently on a container heading for Europe (sea freight is taking weeks…)

What you can do to assist is to provide a unit test that shows the problem (look at http://jpathwatch.svn.sourceforge.net/viewvc/jpathwatch/trunk/jpathwatch/test/name/pachler/jpathwatch/JDK7Test.java?revision=130&view=markup for examples; these are the Unit tests for jpathwatch)

Cheers,

Uwe

Hello Uwe,

I understand that you want to keep the API's behavior identical on all platforms. That makes sense :-)

However, I do not see a problem in making the small changes I proposed, since they 1) do not change the behavior and 2) only apply to the LinuxWatchService anyway. So in my point of view, there is no harm since it's all backwards compatible and only adds information for the Linux version …

Regarding the order of the RENAME_FROM / RENAME_TO events on a Linux box I can say the following: unlike Windows, the order is definitely not guaranteed. When I move a folder with many subfolders, I often receive a lot of RENAME_FROM entries, and then the corresponding RENAME_TO entries.

As I understood it, you suggested to map the FROM and TO events only according to the order in which they occur (at least for Windows). I don't think that's gonna work, since there are sometimes single FROM or TO events that do not belong together

Example: Assume the watched Directory is /tmp.
- Rename file /tmp/one.txt to /one.txt
- Rename file /two.txt to /tmp/two.txt

What you'll receive is the following; and that is definitely not what happend :-D
RENAME_FROM one.txt
RENAME_TO two.txt

Do you have any other ideas?

Also take a look at the code I posted here . It implements a BufferedWatcher, i.e. a watch service that delays events to map FROM and TO events together. It can be used like this:

BufferedWatcher w = new BufferedWatcher();

WatchKey rootKey = w.addWatch(new File("/tmp/some-folder"), true, new WatchListener() {
    @Override
    public void watchEventOccurred(WatchKey rootKey, WatchEvent event) {
        System.out.println("root =" +rootKey+", event = "+event);
    }
});

w.start();

It doesn't work perfectly yet, but I'll keep on working on it.
Tell me what you think ;-)

Regards,
Philipp

http://bazaar.launchpad.net/~binwiederhier/syncany/trunk/files/head:/syncany/src/name/pachler/nio/file/contrib/

Hi Phillip,

I understand that you want to keep the API's behavior identical on all platforms. That makes sense :-)

Yes, and what you're saying is that the behaviour is not the same on Linux, because RENAME_FROM and RENAME_TO don't form brackets, so you can't piece them back together without the cookie. In my book that's simply a bug, and won't require a new feature that's Linux-specific - it simply needs fixing so that the behaviour is the same as on Windows (there are no other platforms supporting rename)

Regarding your example: Assuming that you're watching /tmp, I'd say what you're seeing there is quite normal: You are moving one.txt out of the watched directory, so the watcher can't see it any more (hence no event). Same goes in reverse for two.txt. As said earlier I haven't got a Linux box ATM to test, but I'm sure you won't see events that happen outside the watched directory. In fact, cross-folder moves would only work on Windows because that's the only platform which supports FILE_TREE.

Regarding the BufferedWatcher: I haven't had time to look at the code yet, but piecing RENAME_FROM and RENAME_TO together from the cookies is exactly what I proposed as fix in my previous message.

Cheers,

Uwe

Hey there again,

In my book that's simply a bug, and won't require a new feature that's Linux-specific - it simply needs fixing so that the behaviour is the same as on Windows (there are no other platforms supporting rename)

If the WatchService API specifies that the order of the events has to be guaranteed, than this could definitely be categorized as a bug. So getting the events in the correct order is certainly a good idea. However, from a user's perspective, I don't think that's enough to map FROM+TO events together.

If a user application only receives a list of events in a particular order, there is always some level of ambiguity to whether two FROM+TO events belong together or not (as my example from above shows).

Assume a user application is watching a particular directory. In most cases, the application might want to watch the following things:
1. a file is created (or moved in the watched directory !!)
2. a file is moved from A to B (within the watched directory)
3. a file is deleted (or moved out of the watched directory !!)

I suspect that for most applications it does not matter whether a file is a) created or moved to the watched directory as in case 1 or b) is actually deleted or moved to some other location as in case 3. Fact is that it just appeared or disappeared in/from the watched directory. In my point of view, it should be possible to detect that. With the Linux cookies, it is possible because you can map FROM+TO events together. With just the order (as Windows provides it), this is not possible, or is it?!

Regards,
Philipp

Hi there,

moving files into and out of watched directory is a separate problem. The simplest solution is to make them show up as create and delete events.

So what's the value of the cookie if you move a file into/out of the watched directory? I'd expect it to be zero or some other kind of special value - so you can then simply map it to CREATE and DELETE events.

In conclusion, there are potentially two issues here (but I don't believe it until I have seen a unit test that checks specifically for them failing):
* On Linux, RENAME_TO and RENAME_FROM don't appear in pairs (likely, there is no code to check for this case)
* On Linux, for files moved into and out of a watched directory are believed to appear as RENAME_FROM and RENAME_TO; in essence IMHO they should show up as CREATE/DELETE events

Uwe

Hey again,

seems to me like an endless discussion :-D

* On Linux, RENAME_TO and RENAME_FROM don't appear in pairs (likely, there is no code to check for this case)
* On Linux, for files moved into and out of a watched directory are believed to appear as RENAME_FROM and RENAME_TO; in essence IMHO they should show up as CREATE/DELETE events

The first point is correct, and to be consistent with the bahavior of the Windows version, it might need fixing.
With the second point, I honestly don't know if that's a good idea, because it would throw events that are not there…

I feel the need to repeat what the actual purpose of this thread is, because I think we're drifting off :-) I think the Linux cookie for FROM+TO events is a _good_ thing, because it even lets you track the movement of a file across different watched directories. So in my opinion,

1)  the Linux version should be adjusted to make the cookie available,
2) and (more importantly) the Windows version should introduce a similar mechanism (if available and/or possible).

As you can see,, 1) is already fixed (in my point of view). The only thing I need is a way to track Windows renames determistically (2). Do you think that's even possible?

Thanks for your support, and thanks again for the great project :-)

Regards,
Philipp

Hi Philip,

The first point is correct, and to be consistent with the bahavior of the Windows version, it might need fixing.

No it is not. I just wrote and ran another unit test which tests bulk renames. And all IN_MOVE_FROM/IN_MOVE_TO are IN ORDER. So there is NO BUG there. This test creates a large number of files in the same directory, registers a WatchKey, and then renames them.

With the second point, I honestly don't know if that's a good idea, because it would throw events that are not there.

Other OS's do report this as ENTRY_CREATE/ENTRY_DELETE, so the expectation will be that this is also the case on Linux (at least by default).
However, no other OS's create cookies, so there is no way to track renames like on Linux. The only way I can see is to provide a special modifier on Linux that changes the behaviour to report these as ENTRY_RENAME_*, and then provide the cookie.

I still wonder what you need this for though. What's your use case?
And if you will be relying on Linux-specific functionality anyways, why not use inotify directly (all of it is exposed in jpathwatch's Linux class)?

Cheers,

Uwe

Hey Uwe,

thank you for your effort. So for me the bottom line is that on other OSs, it's not possible to track renames … That's a little bit disappointing, but not the end of the world.

My use case is my Dropbox-like file sync tool Syncany , for which I need to track where files are being moved to and what their origin was to get a version history of all the changes, renames and delete-events. The inotify cookies make that really easy. If they are not there, I'd have to fall back to heuristics, and try to guess the origin. That'll probably work in most cases, but it's not as efficient as just receiving an event…

Does Windows give you anything that could be transformed into an artificial cookie? Something like filesize or last modified date?

Cheers,
Philipp

www.syncany.org

I have one additional thought: you said that on Windows, a file that is being moved out of a watched directory appears as DELETE event by the path service, right? if that is the case, there should never be any single RENAME_FROM or RENAME_TO events on Windows. Is that right? If that's the case, we could very well map them together as you suggested in the first place…

That's what I've been trying to tell you all along…

The sad news is though that this will be hard to fix: On Linux, if you have requested both ENTRY_CREATE and ENTRY_RENAME_* event kinds on Path.register(), jpathwatch can't easily decide if the IN_RENAME_FROM that just occurred belongs to a rename within the watched directory or to a 'foreign' rename. The only thing I can see so far that it can do (without being able to predict the future) is to wait if a IN_RENAME_TO will occur.

But it can't wait indefinitely, because then you would only see foreign renames if other events occur as well. So the implementation will have to be timeout based, which is not nice at all, and cause havok when you're trying to debug it.

Guess it's time to raise this as a bug :(

HOORAY, added the first bug in the bug tracker (previously there was no need; I was so awesome to fix every bug on the spot…)

https://sourceforge.net/tracker/?func=detail&aid=3309509&group_id=245450&atid=1419016

For some reason I cannot comment on the bug, so I'll do it here:

That's what my buffered watcher does. It waits ~ 200ms (or less) and maps the two events together.
Case 1: File moved within the watched directories, i.e. RENAME_FROM(cookie=123), RENAME_TO(cookie=123) occur. The buffered watcher will discard both events and throw an artificial RenameWatchEvent, that contains both of them: new RenameWatchEvent(movedFromEvent, movedToEvent).

Case 2: File moved in the watched directory, i.e. only a RENAME_FROM(cookie=?!) with an unknown cookie occurs. After 200ms have gone by, the buffered watcher throws the original event.

Case 3: File moved out of the watched directory. Analogue to case 2.

With the "new information" about the Windows events, for Case 2/3, it would be easy to also map two Windows events together to an artificial RENAME_FROM_TO event - i think. Look at the fireDelayedEvents() method.