MinGW - Minimalist GNU for Windows / Issues / #1081 find not reporting all expected matches

Earnie Boyd - 2007-09-10

Logged In: YES
user_id=15438
Originator: NO

I believe this goes back to the issue of the mount point directory not physically existing but will allow the mount to succeed because Windows users are not used to UNIX semantics. If you ``mkdir /d/usr/MSYS-1.0.11/d'' the ``find /d -iname ...'' command will be much happier about finding the things in /d/usr.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2007-09-11

Logged In: YES
user_id=1369729
Originator: NO

I was able to reproduce it, somewhat.

For some reason, ``find'' is stopping right after traversing a path listed in the mount table. For instance:

$ touch /c/aardvark
$ mkdir /c/aatest
$ echo "c:/aatest /xyz" >> /etc/fstab
$ find /c
/c
/c/aardvark
/c/aatest
/c/aatest/a
[ends right here]

Earnie's suggestion of creating a physical mount point directory didn't change the result.

In the following, note that ``find'' appears to traverse the file system in case-insensitive alphabetical order.

In Keith's first case (find /d), it must be stopping after traversing d:/opt (it's in his mount table). Before stopping, it traverses d:/MSYS and finds the msys.bat in there (MSYS comes earlier than opt in traversal order). It doesn't reach d:/usr because d:/opt comes before d:/usr in search order.

I can't explain why Keith's second case succeeds, however. It should stop in d:/usr/local before reaching d:/usr/MSYS-1.0.11, if my theory was correct.

In any case, I think what I have is enough to start debugging.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2007-09-11

Logged In: YES
user_id=1369729
Originator: NO

The test case should read:
$ touch /c/aardvark
$ mkdir /c/aatest
$ mkdir /c/aatest/a # <- forgot this line
$ echo "c:/aatest /xyz" >> /etc/fstab
$ find /c
/c
/c/aardvark
/c/aatest
/c/aatest/a
[stops right here]

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2007-09-12

Logged In: YES
user_id=1369729
Originator: NO

Hi Keith,

Please locate an "oldfind.exe" program in your /bin directory. If you do not have it, grab it from the findutils package in the Snapshot download category on SourceForge.

Then:
1) rename "find.exe" to "ftsfind.exe"
2) rename "oldfind.exe" to "find.exe"
3) retry the failed command.

Does it work?

In findutils-4.3, two versions of "find" are always built and installed. "find.exe" uses a "new" file hierarchy traversal method (FTS), which appears to be incompatible with MSYS. "oldfind.exe" retains the "old" method, which works in MSYS.

The simplest workaround is using the --without-fts at configure time. It causes "find.exe" to use the old traversal method.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-13

Logged In: YES
user_id=823908
Originator: YES

Thanks for following up, Cesar.

> Please locate an "oldfind.exe" program in your /bin directory.

Hmm. It's not there; guess I missed that update :-( ...

> If you do not have it, grab it from the findutils package in the
> Snapshot download category on SourceForge.

Ok, got it now.

> Then:
> 1) rename "find.exe" to "ftsfind.exe"
> 2) rename "oldfind.exe" to "find.exe"
> 3) retry the failed command.
>
> Does it work?

'Fraid not. Seems to search for about 50% longer, but still only finds the one file in /d/MSYS/1.0

Not a rigorous benchmark, I know, but...

$ cp findutils-4.3.0/bin/find.exe /bin

$ find --version
GNU find version 4.3.0
Features enabled: LEAF_OPTIMISATION FTS

$ date +%T-%N; find /d -iname msys.bat 2>/dev/null; date +%T-%N
11:48:56-095000000
/d/MSYS/1.0/msys.bat
11:49:01-048000000

(Repeated 4 times; consistent at ~4.9 sec).

$ cp findutils-4.3.0/bin/oldfind.exe /bin/find.exe

$ find --version
GNU find version 4.3.0
Features enabled: LEAF_OPTIMISATION

$ date +%T-%N; find /d -iname msys.bat 2>/dev/null; date +%T-%N
11:51:20-534000000
/d/MSYS/1.0/msys.bat
11:51:28-097000000

(Repeated 4 times; consistent at ~7.5 sec).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Earnie Boyd - 2007-09-13

Logged In: YES
user_id=15438
Originator: NO

There may be another introduced bug but I do remember that older versions of find would exhibit this situation when the physical directory didn't exist for the mount point.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-13

Logged In: YES
user_id=823908
Originator: YES

> There may be another introduced bug but I do remember that older
> versions of find would exhibit this situation when the physical
> directory didn't exist for the mount point.

Sorry Earnie, but I think this is a red herring.

My MSYS installation root is:

$ (cd /; pwd -W)
D:/usr/MSYS-1.0.11

Note that D: is a real partition, (probably extended; I didn't set the box up, and it's too much hassle to check at the moment), on a local hard drive. D:\usr\MSYS-1.0.11 (obviously) exists as a physical path, in the Woe32 domain. D:\usr\local also exists as a physical path, and is mapped to /usr/local via an /etc/fstab reference, as an MSYS virtual path; (the intention here was to share /usr/local between MSYS-1.0.10 and MSYS-1.0.11 -- not concurrently of course; that did appear to work at one time, but now MSYS-1.0.10 dies on start-up, so it's become rather redundant).

But back on topic: if I now start up cmd.exe, and

D:\> cd \usr\msys-1.0.11

D:\usr\MSYS-1.0.11> mkdir d

D:\usr\MSYS-1.0.11> mkdir d\usr

D:\usr\MSYS-1.0.11> mkdir d\usr\local

D:\usr\MSYS-1.0.11> mkdir usr

D:\usr\MSYS-1.0.11> mkdir usr\local

and back in MSYS, after closing all MSYS sessions and starting anew, it makes not a blind bit of difference:

$ find /d -iname msys.bat 2>/dev/null
/d/MSYS/1.0/msys.bat

$ find /d/usr -iname msys.bat 2>/dev/null
/d/usr/MSYS-1.0.11/home/keith/sandbox/msys/msys/store/noarch/bin/msys.bat
/d/usr/MSYS-1.0.11/msys.bat
/d/usr/tmp/msys/msys.bat

Or, am I missing something?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-13

Proposed `umount' command script for MSYS

umount

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-13

Logged In: YES
user_id=823908
Originator: YES

Lest we become distracted by the /usr/local issue, it too appears to be a red herring:

$ umount /usr/local

$ find /d -iname msys.bat 2>/dev/null
/d/MSYS/1.0/msys.bat

$ find /d/usr -iname msys.bat 2>/dev/null
/d/usr/MSYS-1.0.11/home/keith/sandbox/msys/msys/store/noarch/bin/msys.bat
/d/usr/MSYS-1.0.11/msys.bat
/d/usr/tmp/msys/msys.bat

"`umount'?", did I hear you ask? I'll open a feature request to discuss it in more depth, but I've attached it for completeness; it's a simple script to filter entries out of /etc/fstab, so /usr/local isn't mounted now:

$ mount
D:\usr\MSYS-1.0.11 on / type user (binmode,noumount)
D:\usr\MSYS-1.0.11 on /usr type user (binmode,noumount)
d:\usr\mingw on /mingw type user (binmode)
x:\control on /db type user (binmode)
C:\TEMP on /tmp type user (binmode,noumount)
d:\opt on /opt type user (binmode)
\dev on /dev type user (binmode)
c: on /c type user (binmode,noumount)
d: on /d type user (binmode,noumount)
e: on /usb type user (binmode)
f: on /f type user (binmode,noumount)
g: on /g type user (binmode,noumount)
i: on /i type user (binmode,noumount)
j: on /j type user (binmode,noumount)
l: on /l type user (binmode,noumount)
t: on /t type user (binmode,noumount)
u: on /u type user (binmode,noumount)
v: on /v type user (binmode,noumount)
w: on /w type user (binmode,noumount)
x: on /x type user (binmode,noumount)
z: on /z type user (binmode,noumount)

(And, yes, I do have a corresponding `mount' script, but that's for the feature request).

File Added: umount

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2007-09-13

Logged In: YES
user_id=1369729
Originator: NO

Please try:

find /d > find.txt
grep msys.bat find.txt
tail find.txt

Do not send find.txt, just the output from the commands above.

Let's see where find stops...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-14

Logged In: YES
user_id=823908
Originator: YES

$ find /d 2>find.err >find.log
$
$ grep msys.bat find.log
/d/MSYS/1.0/msys.bat

$ tail find.log
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd7.htm
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd8.kf
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd9.kf
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/desktop.ini
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/INFO2
/d/SYSTEM
/d/System Volume Information
/d/Total
/d/usr

Hmm. Doesn't seem to be traversing into /d/SYSTEM, "/d/System Volume Information" or /d/Total; let's see...

$ ls -a /d/SYSTEM
. ..

Ok, this one's empty...

$ ls -a /d/System\ Volume\ Information/
ls: reading directory /d/System Volume Information/: Permission denied

...this one, I cannot access...

$ ls -a /d/Total/
. .. ENG_SAPGUI62 HelpDeskInfo PIPC Voloview

...but for this one, I should see a traversal! Let's add something elsewhere...

$ touch /opt/packages/msys.bat /d/SYSTEM/msys.bat /d/Total/msys.bat

$ find /d 2>find.err >find.log

$ grep msys.bat find.log
/d/MSYS/1.0/msys.bat
/d/opt/packages/msys.bat
/d/SYSTEM/msys.bat

Ah ha! Now we are seeing something interesting; we've found the additions in /d/opt/packages, (recall that /d/opt is mounted on /opt), and in /d/SYSTEM, but still nothing in /d/Total or in /d/usr...

$ tail find.log
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd7.htm
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd8.kf
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/Dd9.kf
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/desktop.ini
/d/RECYCLER/S-1-5-21-117609710-764733703-839522115-39808/INFO2
/d/SYSTEM
/d/SYSTEM/msys.bat
/d/System Volume Information
/d/Total
/d/usr

...and indeed, we now see a traversal into /d/SYSTEM, but still not into "/d/System Volume Information", (which we wouldn't expect anyway, because of its permissions), nor into /d/Total or /d/usr.

I did wonder if, maybe, the name `usr' might be significant, (because `/usr' is special). I think that's another red herring, (it wouldn't explain the failure to traverse /d/Total), but pursuing it did turn up something interesting:

If I rename it to `kdm'...

$ find /d 2>/dev/null >find.log

$ grep msys.bat find.log
/d/kdm/MSYS-1.0.11/home/rgbl0264/sandbox/msys/msys/store/noarch/bin/msys.bat
/d/kdm/MSYS-1.0.11/msys.bat
/d/kdm/tmp/msys/msys.bat
/d/MSYS/1.0/msys.bat
/d/opt/packages/msys.bat
/d/SYSTEM/msys.bat

Eureka! Those expected files have miraculously appeared. Even if I rename as `opu', (moving it lexically below the /opt mount point...

$ find /d 2>/dev/null >find.log

$ grep msys.bat find.log
/d/MSYS/1.0/msys.bat
/d/opt/packages/msys.bat
/d/opu/MSYS-1.0.11/home/rgbl0264/sandbox/msys/msys/store/noarch/bin/msys.bat
/d/opu/MSYS-1.0.11/msys.bat
/d/opu/tmp/msys/msys.bat
/d/SYSTEM/msys.bat

...but, if I move it to, say `tsr'...

$ find /d 2>/dev/null >find.log

$ grep msys.bat find.log
/d/MSYS/1.0/msys.bat
/d/opt/packages/msys.bat
/d/SYSTEM/msys.bat

...they vanish again!

It seems that the pivotal entity is that "/d/System Volume Information" directory, which I can see, but can't see into; any directory appearing lexically above it is traversed; anything below is not. So what's special about it? Access permissions? Let's see:

$ cat find.err
find: /d/8957ac83a281329dbadd: Permission denied
find: /d/csc: Permission denied
find: /d/System Volume Information: Permission denied

If it were permissions alone, then I'd expect no traversal of any directory appearing lexically below /d/8957ac83a281329dbadd, (which is first of all present, so I'd expect no hits at all).

$ head find.log
/d
/d/8957ac83a281329dbadd
/d/csc
/d/DOCUMENT
/d/lordoc
/d/lordoc/.anin-man.ms.swp
/d/lordoc/81pc251.ms
/d/lordoc/81pc251.pdf
/d/lordoc/91F1-firing.html
/d/lordoc/anin-man.pdf

So, not permissions alone then; what else stands out? Spaces in the path name, perhaps?

$ mkdir "/d/Path With Spaces"

$ touch !$/msys.bat
touch "/d/Path With Spaces"/msys.bat

$ find /d 2>/dev/null > find.log

$ grep msys.bat find.log
/d/MSYS/1.0/msys.bat
/d/opt/packages/msys.bat
/d/Path With Spaces/msys.bat
/d/SYSTEM/msys.bat

Nope. It finds "/d/Path With Spaces/msys.bat" ok. Maybe a combination of the two? I don't know. I'm out of ideas. Over to you.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2007-09-14

Logged In: YES
user_id=823908
Originator: YES

Of course, "System Volume Information" is a special system directory. It's hidden in a Woe32 explorer view, even when "Show Hidden Files and Folders" is enabled; I have to disable "Hide Protected Operating System Files" to see it there. Even if I do that, I can see that it has the "Hidden" attribute set, but I'm not authorised to even inspect its security attributes.

There are two other directories, which are similarly concealed; the "csc" directory, which I can also see bears the hidden attribute, but cannot inspect its security attributes either, and the "RECYCLER" directory, which again bears the hidden attribute, but I *can* see it's security attributes, (inherited, and it appears that I can do anything I like with it). "find" doesn't seem to have a problem with "csc" or "RECYCLER"; there appears to be something different about "System Volume Information", but I can't see what it is.

Just thinking aloud.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Earnie Boyd - 2007-09-14

Logged In: YES
user_id=15438
Originator: NO

I'm wondering if find or msys is getting confused by the previous error? So I am trying ``find /c'' on my existing version of MSYS. It'll be a little while ............ if I ever leave this company they get their laptop back with a new hard drive on it ............. did I say this is going to be a while ............ 40 minutes later and I'm in /c/WINNT. So it is reading past "System Volume Information".

I'll have to download the latest and greatest snapshot to see if that makes a difference.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-02-28

Logged In: YES
user_id=823908
Originator: YES

After the initial flurry of interest, I guess this got moved to the back burner; I'd forgotten about it myself, until it bit me again today.

Cesar, did you gain any further insight?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2008-03-02

Logged In: YES
user_id=1369729
Originator: NO

Hi Keith,

Thanks for the reminder. I did a little bug hunting today, which was fruitful. The next snapshot of the MSYS Runtime will contain a fix. Meanwhile, you can copy your ftsfind.exe back to find.exe and use:

alias find='find -L'

The problem was a subtle bug in the "opendir" implementation of the MSYS runtime. Instead of storing the POSIX path (/d/opt) in the file descriptor, it stored the Win32 path (d:/opt). Later, "fchdir" converted it back to POSIX, using the mount table (/opt), giving inconsistent results. "find -L" works because it avoids using fchdir altoghether.

Regards,
Cesar

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-03-05

Logged In: YES
user_id=823908
Originator: YES

Thanks Cesar,

> I did a little bug hunting today, which was fruitful. The next
> snapshot of the MSYS Runtime will contain a fix.

This *is* good news :-)

> Meanwhile, you can copy your ftsfind.exe back to find.exe and
> use:
>
> alias find='find -L'

How strange, that an option instructing `find' to follow symbolic links should have this useful side effect, on a platform which doesn't support symbolic links. I wondered if it might be the associated implied `-noleaf' predicate that makes the difference, but no, that isn't it; anyway it does appear to provide an effective temporary work around:

$ find /d -iname msys.bat 2> /dev/null

$ find /d -iname msys.bat -noleaf 2> /dev/null

$ find -L /d -iname msys.bat 2> /dev/null
/d/MSYS/1.0/msys.bat
/d/opt/MSYS-1.0.11/msys.bat
/d/opt/MSYS-1.0.11/usr/msys.bat
/d/opt/packages/msys.bat
/d/Path With Spaces/msys.bat
/d/SYSTEM/msys.bat
/d/Total/msys.bat
/d/usr/MSYS-1.0.11/msys.bat
/d/usr/MSYS-1.0.11-old/home/keith/sandbox/msys/msys/store/noarch/bin/msys.bat
/d/usr/MSYS-1.0.11-old/home/keith/sandbox/msys-dvlpr/msys/store/noarch/bin/msys.bat
/d/usr/MSYS-1.0.11-old/msys.bat
/d/usr/tmp/msys/msys.bat

Thanks again, and regards,
Keith.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2008-07-21

Logged In: YES
user_id=1369729
Originator: NO

Hello Keith,

Please try the new MSYS Runtime Snapshot, it should fix this issue.

You can find it as MSYS-1.0.11-20080710-dll.tar.gz, from the SF download page:
http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=24963&release_id=46827

Regards,
Cesar

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2008-07-21

status: open --> pending
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-07-22

status: pending --> open
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-07-22

Logged In: YES
user_id=823908
Originator: YES

Thanks, Cesar.

I've installed this latest update, and I've deleted the "alias find='find -L 2>/dev/null'", which I've had in place since we previously identified this workaround. This does now seem to find all instances of msys.bat, in the file system hierarchy as before. (oldfind.exe however, does not; it still bails out after "System Volume Information", as before, unless the "-L" option is given).

Unfortunately, having removed the stderr redirection, I now see another problem with find.exe, (which isn't apparent with oldfind.exe):

$ find /d -iname msys.bat
(null): /d/8957ac83a281329dbadd: Bad address
(null): /d/csc: Bad address
/d/MSYS/1.0/msys.bat
...

$ oldfind /d -iname msys.bat
oldfind: /d/8957ac83a281329dbadd: Permission denied
oldfind: /d/csc: Permission denied
/d/MSYS/1.0/msys.bat
...

Note the NULL pointer dereference, in the error messages from find.exe, where I would expect to see the program basename from argv[0], stripped of its extension; I guess the "Bad address" error message also refers to this invalid pointer dereference, in place of the expected "Permission denied". FWIW, the behaviour of oldfind.exe is correct, in this respect.

Can you fix this please, before we close this one?

Cheers,
Keith.

BTW, after testing, I've renamed the new mount.exe to msysmnt.exe, to work with my replacement mount/umount scripts, (as provided in https://sourceforge.net/tracker/?func=detail&aid=1818517&group_id=2435&atid=352435\). Do you foresee any problems which that might cause?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2008-08-27

Logged In: YES
user_id=1369729
Originator: NO

Hi Keith

> $ find /d -iname msys.bat
> (null): /d/8957ac83a281329dbadd: Bad address

I just released a new findutils package containing a fix. Please also update your MSYS runtime, it fixes one more issue with `find' I discovered in the meantime. Alternately, try the new msysCORE release.

> BTW, after testing, I've renamed the new mount.exe to msysmnt.exe,
> to work with my replacement mount/umount scripts

I included them in the latest MSYS release, thanks.

Regards,
Cesar

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-08-27

Logged In: YES
user_id=823908
Originator: YES

Hi Cesar,

>> $ find /d -iname msys.bat
>> (null): /d/8957ac83a281329dbadd: Bad address
>
> I just released a new findutils package containing a fix. Please
> also update your MSYS runtime, it fixes one more issue with `find'
> I discovered in the meantime.

I installed:
MSYS-1.0.11-20080821-dll.tar.gz
findutils-4.3.0-MSYS-1.0.11-2-bin.tar.gz

Now I see:
$ find /d -iname msys.bat
find: /d/csc: Bad address
/d/MSYS/1.0/msys.bat
...

so, you seem to have fixed the "(null)" vs. "find" program name bug, but the error message is still "Bad address", (which corresponds to EFAULT, errno == 14), where I expected "Permission denied", (EACCES, errno == 13). Notice that

$ oldfind /d -iname msys.bat
oldfind: /d/csc: Permission denied
/d/MSYS/1.0/msys.bat

gets this right, (but it still terminates it's search prematurely).

> Alternately, try the new msysCORE release.

I didn't try this, but I assume it's just the same, in a more comprehensive package.

Regards,
Keith.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Cesar Strauss - 2008-08-28

Logged In: YES
user_id=1369729
Originator: NO

Hi Keith

> Now I see:
> $ find /d -iname msys.bat
> find: /d/csc: Bad address

Please try the new findutils-4.3.0-MSYS-1.0.11-3-bin.tar.gz. It should give the correct error message now.

There was another instance of a null pointer being passed around, because the return value of a C library function wasn't being checked for failure.

Regards,
Cesar

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-08-28

Logged In: YES
user_id=823908
Originator: YES

Hi Cesar,

> Please try the new findutils-4.3.0-MSYS-1.0.11-3-bin.tar.gz.
> It should give the correct error message now.

Looks good now, thanks. I'll leave it to your discretion to remove, or retain oldfind.exe in the distribution. Otherwise, I see no further reason to keep this open; marking as closed, and fixed.

Regards,
Keith.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Keith Marshall - 2008-08-28

milestone: --> IINR_-_Include_In_Next_Release

status: open --> closed-fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

find not reporting all expected matches

A native Windows port of the GNU Compiler Collection (GCC)

Group

Searches

Help

#1081 find not reporting all expected matches

Discussion