#2950 filesystem-7.8 fails on darwin without NO_REALPATH

obsolete: 8.5a3
closed-fixed
9
2006-03-28
2004-11-11
No

filesystem-7.8 fails on darwin when realpath() is used, and succeeds
otherwise. The failure is that
cd simplefs:/simpledir
succeeds but does not change the current directory

I've tracked this down to differing behaviour in
TclpObjNormalizePath() when normalizing 'simplefs:/simpledir'
depending on NO_REALPATH being defined:

- with NO_REALPATH, the "slow way" of getting the subpath to be
normalized at the start of TclpObjNormalizePath() fails at access()
on the first path component 'simplefs:', which leads to the correct
return value of 'simplefs:/simpledir' as expected.
- without NO_REALPATH, the fast way of using realpath() to get the
subpath to be normalized calls 'realpath("simplefs:")' which returns
the absolute path "[pwd]/simplefs:", this leads to
TclpObjNormalizePath() returning the incorrect value '[pwd]/
simplefs:/simpledir'.

the result of this path normalization is used to identify the
filesystem owning the path in question, this fails to find the
simplefilesystem in the second case which ends up leading to the
test failure

the crux of the issue stems from the behaviour of realpath() on
relative paths. The BSD spec says that realpath() always returns an
absolute path, and that all except the last path component given
must exist (c.f. manpage below). This is exactly what seems to
happen in this case on darwin, so it looks like the design of
TclpObjNormalizePath() didn't take BSD realpath into account?
according to the caveats in the manpage, solaris realpath() doesn't
behave in the same way, this may also be true on other platforms?

as I've proposed in the past, maybe a compat/ implementation of
realpath() is needed that behaves the same on all platforms? this
would also solve the non-thread safeness issue of realpath() on
darwin that requires NO_REALPATH to be defined in the threaded
build.

REALPATH(3) BSD Library Functions Manual
REALPATH(3)

NAME
realpath - returns the canonicalized absolute pathname

LIBRARY
Standard C Library (libc, -lc)

SYNOPSIS
#include <sys/param.h>
#include <stdlib.h>

char *
realpath(const char *pathname, char
resolved_path[PATH_MAX]);

DESCRIPTION
The realpath() function resolves all symbolic links, extra ``/''
charac-
ters and references to /./ and /../ in pathname, and copies the
resulting
absolute pathname into the memory referenced by
resolved_path. The
resolved_path argument must refer to a buffer capable of
storing at least
PATH_MAX characters.

The realpath() function will resolve both absolute and relative
paths and
return the absolute pathname corresponding to pathname. All
but the last
component of pathname must exist when realpath() is called.

RETURN VALUES
The realpath() function returns resolved_path on success. If an
error
occurs, realpath() returns NULL, and resolved_path contains the
pathname
which caused the problem.

ERRORS
The function realpath() may fail and set the external variable
errno for
any of the errors specified for the library functions lstat(2),
readlink(2) and getcwd(3).

CAVEATS
This implementation of realpath() differs slightly from the
Solaris
implementation. The 4.4BSD version always returns absolute
pathnames,
whereas the Solaris implementation will, under certain
circumstances,
return a relative resolved_path when given a relative pathname.

SEE ALSO
getcwd(3)

HISTORY
The realpath() function first appeared in 4.4BSD.

BSD February 16, 1994
BSD

Discussion

  • Daniel A. Steffen

    • priority: 5 --> 7
     
  • Vince Darley

    Vince Darley - 2004-11-19

    Logged In: YES
    user_id=32170

    TclpObjNormalizePath should not be called on a path starting
    with "simplefs:/" (assuming the simplefilesystem is
    registered, of course).

    Why is this being called?

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    maybe it shouldn't be called, but it certainly is, I've forgotten
    why at this point though... just set a breakpoint there and run
    the test snippet.

     
  • Vince Darley

    Vince Darley - 2004-11-19

    Logged In: YES
    user_id=32170

    I could try, but I can't compile cvs head on the compilefarm
    any more, something about an extra ')' in
    unix/configure...(I think from your last commit).

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    just fixed configure, sorry

     
  • Vince Darley

    Vince Darley - 2004-11-19

    Logged In: YES
    user_id=32170

    Thanks -- I'll look into it. Note that Tk's build also
    fails now:

    gcc2: cannot specify -o with -c or -S and multiple compilations
    ...failed CompileC
    /home/users/v/vi/vincentdarley/tk/macosx/../../build/tk/Development.build/Wish.build/TkStubLibrary.build/Objects-normal/ppc/tkStubImg.o
    ...
    gcc2: cannot specify -o with -c or -S and multiple compilations
    ...removing
    /home/users/v/vi/vincentdarley/tk/macosx/../../build/tk/Development.build/Wish.build/TkStubLibrary.build/Objects-normal/ppc/tkStubImg.o
    ** BUILD FAILED **
    make: *** [develop] Error 1

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    Vince,
    don't see this Tk build failure with the HEAD myself, not sure what's going
    on there, does it still happen with current HEAD for both tcl & tk?

     
  • Joe Mistachkin

    Joe Mistachkin - 2004-12-23

    Logged In: YES
    user_id=113501

    Confirmed failure on FreeBSD 4.x STABLE.

    ==== filesystem-7.8 vfs cd FAILED
    ==== Contents of test case:

    set dir [pwd]
    cd [tcltest::temporaryDirectory]
    file delete -force simpledir
    file mkdir simpledir
    testsimplefilesystem 1
    # This can variously cause an infinite loop or simply have
    # no effect at all (before certain bugs were fixed, of
    course).
    cd simplefs:/simpledir
    set res [pwd]
    cd [tcltest::temporaryDirectory]
    testsimplefilesystem 0
    file delete -force simpledir
    cd $dir
    set res

    ---- Result was:
    /home/phantom/tcl/tcl/unix
    ---- Result should have been (exact matching):
    simplefs:/simpledir
    ==== filesystem-7.8 FAILED

     
  • Anonymous - 2005-01-31

    Logged In: YES
    user_id=585068

    This test is also failing in NetBSD 2.0 x86:

    ==== filesystem-7.8 vfs cd FAILED
    ==== Contents of test case:

    set dir [pwd]
    cd [tcltest::temporaryDirectory]
    file delete -force simpledir
    file mkdir simpledir
    testsimplefilesystem 1
    # This can variously cause an infinite loop or simply have
    # no effect at all (before certain bugs were fixed, of
    course).
    cd simplefs:/simpledir
    set res [pwd]
    cd [tcltest::temporaryDirectory]
    testsimplefilesystem 0
    file delete -force simpledir
    cd $dir
    set res

    ---- Result was:
    /usr/home/gps/src/cvs/HEAD/tcl/unix
    ---- Result should have been (exact matching):
    simplefs:/simpledir
    ==== filesystem-7.8 FAILED

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    Vince,

    this is still failing, looking it why once again

    you wrote:
    > TclpObjNormalizePath should not be called on a path starting
    > with "simplefs:/" (assuming the simplefilesystem is
    > registered, of course).
    >
    > Why is this being called?

    the relevant backtrace from current HEAD is

    (gdb) bt
    #0 TclpObjNormalizePath (interp=0x0, pathPtr=0x7a6018,
    nextCheckpoint=0) at unix/tclUnixFCmd.c:1870
    #1 0x000a6854 in TclFSNormalizeToUniquePath (interp=0x0,
    pathPtr=0x7a6018, startAt=0, clientDataPtr=0xbfff9ae0) at generic/
    tclIOUtil.c:1448
    #2 0x000cb6cc in TclFSNormalizeAbsolutePath (interp=0x0,
    pathPtr=0x7a6450, clientDataPtr=0xbfff9b98) at generic/tclPathObj.c:406
    #3 0x000ce5cc in Tcl_FSGetNormalizedPath (interp=0x0,
    pathPtr=0x7a64c8) at generic/tclPathObj.c:1900
    #4 0x000a8c30 in Tcl_FSChdir (pathPtr=0x7a64c8) at generic/
    tclIOUtil.c:2854

    i.e. Tcl_FSGetNormalizedPath calls TclFSNormalizeAbsolutePath with a
    pathPtr of 'simplefs:/simpledir' which in turn calls
    TclFSNormalizeToUniquePath with the same pathPtr, which ends up calling
    the native normalize on that path (c.f. below for another likely bug discovered
    in TclFSNormalizeToUniquePath).

    To fix the issue in TclpObjNormalizePath, I would propose simply checking
    whether realpath has transformed a relative path into an absolute one and if
    yes, falling back to the slow method. I have verified that this situation only
    comes up in a few edge case tests in fileSystem.test when running the
    complete testsuite:

    ---- filesystem-1.33 start
    TclpObjNormalizePath: realpath relative -> absolute: 'C:' -> '/tmp/C:'
    ---- filesystem-7.2 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    ---- filesystem-7.4 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    ---- filesystem-7.5 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    ---- filesystem-7.6 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    ---- filesystem-7.7 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'
    ---- filesystem-7.8 start
    TclpObjNormalizePath: realpath relative -> absolute: 'simplefs:' -> '/
    tmp/simplefs:'

    a patch to tclUnixFCmd.c implementing this workaround is below, it fixes the
    failure in filesystem-7.8 on Darwin.
    If you agree, I would like to commit this for 8.5a4.

    As to the likely bug in TclFSNormalizeAbsolutePath, it appears that the check
    for fsRecPtr == &nativeFilesystemRecord can fail even if fsRecPtr actually
    contains the native filesystem:

    (gdb) p &nativeFilesystemRecord
    $1 = (FilesystemRecord *) 0x146c7c
    (gdb) p *(&nativeFilesystemRecord)
    $2 = {
    clientData = 0x0,
    fsPtr = 0x146c00,
    fileRefCount = 1,
    nextPtr = 0x0,
    prevPtr = 0x7c9068
    }
    (gdb) p fsRecPtr
    $3 = (FilesystemRecord *) 0x7c9008
    (gdb) p *fsRecPtr
    $4 = {
    clientData = 0x0,
    fsPtr = 0x146c00,
    fileRefCount = 1,
    nextPtr = 0x0,
    prevPtr = 0x7c9088
    }

    this results in the normalize function for the native FS to be called last
    instead of first as intended

    /*
    * Call each of the "normalise path" functions in succession. This is a
    * special case, in which if we have a native filesystem handler, we call
    * it first. This is because the root of Tcl's filesystem is always a
    * native filesystem (i.e. '/' on unix is native).
    */

    one way to fix this might be to change tests for the native FS in tclIOUtil.c as
    follows

    - if (fsRecPtr == &nativeFilesystemRecord) {
    + if (fsRecPtr->fsPtr == nativeFilesystemRecord.fsPtr) {

    however, it would probably be better to figure out why the record for the
    native FS in filesystemList can become different from nativeFilesystemRecord
    in the first place, I haven't looked into this.

    Note that this second bug has no impact on the filesystem-7.8 failure, as the
    native normalize is still called on 'simplefs:/simpledir' at some point in
    TclFSNormalizeToUniquePath even if the check is changed as above.

    Index: unix/tclUnixFCmd.c

    ====================
    RCS file: /cvsroot/tcl/tcl/unix/tclUnixFCmd.c,v
    retrieving revision 1.52
    diff -u -p -r1.52 tclUnixFCmd.c
    --- unix/tclUnixFCmd.c 25 Mar 2006 01:32:11 -0000 1.52
    +++ unix/tclUnixFCmd.c 28 Mar 2006 08:14:43 -0000
    @@ -1884,8 +1884,17 @@ TclpObjNormalizePath(
    nativePath = Tcl_UtfToExternalDString(NULL, path,
    lastDir-path, &ds);
    if (Realpath(nativePath, normPath) != NULL) {
    - nextCheckpoint = lastDir - path;
    - goto wholeStringOk;
    + if (*nativePath != '/' && *normPath == '/') {
    + /*
    + * realpath has transformed a relative path into an
    + * absolute path, we do not know how to handle this.
    + */
    + fprintf(stderr, "TclpObjNormalizePath: realpath relative ->
    absolute\n");
    + Tcl_DStringFree(&ds);
    + } else {
    + nextCheckpoint = lastDir - path;
    + goto wholeStringOk;
    + }
    }
    }
    }

     
  • Daniel A. Steffen

    • milestone: 387207 --> obsolete: 8.5a3
    • priority: 7 --> 9
     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    of course the patch should not have the fprintf to stderr anymore, sorry...

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    apropos the issue with the native FS record being different from
    nativeFilesystemRecord:

    it turns out that this is due to the copying of the filesystem list into a thread
    specific cache in FsRecacheFilesystemList, which means that the pointers in
    the list returned by FsGetFirstFilesystem() cannot be compared against
    &nativeFilesystemRecord to check for native FS, as they may point to a copy
    of the nativeFilesystemRecord data...

    the fix is indeed to always compare fsRecPtr->fsPtr against
    &tclNativeFilesystem instead, as is already done in a number of other places
    in tclIOUtil.c.

    patch implementing this on HEAD is below

    if you agree, I will commit this for 8.5a4 as well

    Index: generic/tclIOUtil.c

    ====================
    RCS file: /cvsroot/tcl/tcl/generic/tclIOUtil.c,v
    retrieving revision 1.129
    diff -u -p -r1.129 tclIOUtil.c
    --- generic/tclIOUtil.c 16 Mar 2006 18:22:57 -0000 1.129
    +++ generic/tclIOUtil.c 28 Mar 2006 09:26:33 -0000
    @@ -608,7 +608,7 @@ FsRecacheFilesystemList(void)

    /*
    * Code below operates on shared data. We are already called under mutex
    - * lock so we can safely proceede.
    + * lock so we can safely proceed.
    *
    * Locate tail of the global filesystem list.
    */
    @@ -977,7 +977,7 @@ Tcl_FSUnregister(
    */

    fsRecPtr = filesystemList;
    - while ((retVal == TCL_ERROR) && (fsRecPtr != &nativeFilesystemRecord)) {
    + while ((retVal == TCL_ERROR) && (fsRecPtr->fsPtr !=
    &tclNativeFilesystem)) {
    if (fsRecPtr->fsPtr == fsPtr) {
    if (fsRecPtr->prevPtr) {
    fsRecPtr->prevPtr->nextPtr = fsRecPtr->nextPtr;
    @@ -1426,7 +1426,7 @@ TclFSNormalizeToUniquePath(

    fsRecPtr = firstFsRecPtr;
    while (fsRecPtr != NULL) {
    - if (fsRecPtr == &nativeFilesystemRecord) {
    + if (fsRecPtr->fsPtr == &tclNativeFilesystem) {
    Tcl_FSNormalizePathProc *proc = fsRecPtr->fsPtr-
    >normalizePathProc;
    if (proc != NULL) {
    startAt = (*proc)(interp, pathPtr, startAt);
    @@ -1442,7 +1442,7 @@ TclFSNormalizeToUniquePath(
    * Skip the native system next time through.
    */

    - if (fsRecPtr != &nativeFilesystemRecord) {
    + if (fsRecPtr->fsPtr != &tclNativeFilesystem) {
    Tcl_FSNormalizePathProc *proc = fsRecPtr->fsPtr-
    >normalizePathProc;
    if (proc != NULL) {
    startAt = (*proc)(interp, pathPtr, startAt);
    @@ -3669,7 +3669,7 @@ FsListMounts(

    fsRecPtr = FsGetFirstFilesystem();
    while (fsRecPtr != NULL) {
    - if (fsRecPtr != &nativeFilesystemRecord) {
    + if (fsRecPtr->fsPtr != &tclNativeFilesystem) {
    Tcl_FSMatchInDirectoryProc *proc =
    fsRecPtr->fsPtr->matchInDirectoryProc;
    if (proc != NULL) {

     
  • Nobody/Anonymous

    Logged In: NO

    Please do apply the patch -- (didn't realise you were
    looking into the fsRecord stuff as well, as I've just
    patched that bit myself).

    thanks!

    Vince.

     
  • Daniel A. Steffen

    Logged In: YES
    user_id=90580

    done, committed tclUnixFCmd.c patch to HEAD and both patches to core-8-4-
    branch.

     
  • Daniel A. Steffen

    • status: open --> closed-fixed
     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks