[Tcl-bugs] [ tcl-Bugs-929534 ] OS X: filesystem FAILED link normalisation

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #929534, was opened at 2004-04-05 13:43
Message generated for change (Comment added) made by das
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=929534&group_id=10894

Category: 36. File System
Group: current: 8.4.6
Status: Open
Resolution: None
Priority: 5
Submitted By: tauvan (tauvan)
Assigned to: Daniel A. Steffen (das)
Summary: OS X: filesystem FAILED link normalisation

Initial Comment:
filesystem-1.2 FAILED
filesystem-1.3 FAILED
filesystem-1.4 FAILED
filesystem-1.7 FAILED
filesystem-1.9 FAILED
filesystem-1.10 FAILED
filesystem-1.11 FAILED

these occur both local and root
You can see more detailed description under 20 failures bug

machine: mac ppc G4 (1st PCI 400Mhz) OSX 10.1.5
My solution attempt at trying to be helpfull
attached but needs to be "Tcl compatable"

However, since I'm trying if I have questions "not bugs"
where do I go?

----------------------------------------------------------------------

>Comment By: Daniel A. Steffen (das)
Date: 2004-05-01 01:48

Message:
Logged In: YES 
user_id=90580

Could you provide a cvs diff against tcl HEAD please?
otherwise this is unnecessarily painful to test, e.g. I'm not sure  
what to do with the attached files...
I want to be able to do
cd tcl && patch -Np0 < tauvan.patch && cd unix && make test 
&& cd .. && patch -Rp0 < tauvan.patch

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-05-01 01:44

Message:
Logged In: YES 
user_id=1011552

  Here is a wrapper code for inserting into the current 
TclpObjNormalizePath().

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-05-01 01:38

Message:
Logged In: YES 
user_id=1011552

  Here's the code not a patch for 
TclpObjNormalizePath(). One normalized difference to note is in 
cmdAH.test "home:test" is passed, returned result is "/home/test".
As this is the function pointer for unix and osx, and os9 is being 
dropped, I didn't think a problem, both cases returned 0 as startAt.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2004-04-30 17:09

Message:
Logged In: NO 

I think we're very happy to test a different implementation of 
TclpObjNormalizePath() which doesn't need realpath(), 
especially with the speed improvements you cite, provided it 
works on all Unix systems.

I'm actually away for the next week, so can't test anything, but 
perhaps someone else can give this a go?

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-30 15:31

Message:
Logged In: YES 
user_id=1011552

  The errno for both realpath fails is 13, EACCES. The second thing to 
note is that realpath doesn't necessarily point to the offending path 
segment in its return, sometimes it does and sometimes it doesn't. 
Realpath depends on several unix functions that are access sensitive. 
fchdir, chdir, and getcwd are access sensitive.
  If you are willing to create a patch and try something a little 
different, I've got some code that fixes the fileSystem.test fails, 
normalizes paths, consistently points to the last good path segment, 
always has nextCheckpoint point to the last good separator, runs at 
minimum 10% faster with threads enabled and in the 20s with threads 
disabled. It solves at least 4 realpath issues, removes the security 
issue of access(), etc, etc. And if you wish to keep the two realpaths 
and the slow way, I can provide a wrapper to insert code which only 
reduces speed increases by about 1%.

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-30 08:08

Message:
Logged In: YES 
user_id=32170

First off, thanks for the *very* clear summary!

My first question would be that if '/tmp/tcl.foo.dir' is the first thing 
passed to realpath, it ought to return '/private/tmp/tcl.foo.dir', and 
if it returns NULL then we have a problem (I wonder what 'errno' is 
set to?).

The second issue is that it sounds as if file normalization (in the 
current code) is affected by what permissions are available on the 
sub-directories of the path, in that calls to 'access()' may fail, not 
because the given directory doesn't exist (which would signal that 
normalization is complete), but rather because there are 
inadequate permissions....  Is that true?  I'm not sure what to do 
about that (I'm not particularly Unix-savvy, so it would be helpful 
to get someone else's input here, since what I'm writing might be 
totally wrong).

Daniel, do you know much about this?

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-30 06:08

Message:
Logged In: YES 
user_id=1011552

/tmp is a symlink to /private/tmp.   /tmp -> private/tmp
At the time when
Tcl_EvalEx: script = file exists /tmp/tcl.foo.dir/file]\n\n
/private/tmp/tcl.foo.dir/file exists but can only be read as root.
/private/tmp can be read as local and root.
/tmp can be read as local and root.
The nativePath in TclpObjNormalizePath passed to the first realpath
is /tmp/tcl.foo.dir. It returns NULL with normPath set to /tmp.
The slow way then pieces /tmp/tcl.foo.dir/file with accesses = OK.
The second realpath fails with normPath set to /tmp.
This leaves fsPathPtr->normPathPtr = /tmp/tcl.foo.dir/file.

  The true normalized name is /private/tmp/tcl.foo.dir/file and is not 
being reported.
  The test succeeds but normalization fails as /private/tmp/tcl.foo.dir/
file and /tmp/tcl.foo.dir/file are the same file.

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-30 04:57

Message:
Logged In: YES 
user_id=32170

Can you explain further?  Does /private/tmp exist?  Is /tmp
a symlink to /private/tmp?  If so, then either both:

/tmp/tcl.foo.dir/file and /private/tmp/tcl.foo.dir/file
exist, or neither of them exists.  Which of these cases is
it?  What Tcl command in the test is triggering the issue
you describe?

test cmdAH-19.11 {Tcl_FileObjCmd: exists} {unixOnly notRoot} {
    file delete -force /tmp/tcl.foo.dir/file
    file delete -force /tmp/tcl.foo.dir
    makeDirectory /tmp/tcl.foo.dir
    makeFile 12345 /tmp/tcl.foo.dir/file
    file attributes /tmp/tcl.foo.dir -permissions 0000

    set result [file exists /tmp/tcl.foo.dir/file]

    file attributes /tmp/tcl.foo.dir -permissions 0775
    removeFile /tmp/tcl.foo.dir/file
    removeDirectory /tmp/tcl.foo.dir
    set result
} 0

When you say 'it fails', what is it?  

Sorry, but there's just not enough information here to
really help much....

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-30 04:45

Message:
Logged In: YES 
user_id=1011552

  Not sure if this is a seperate bug, or even a bug, but the only dumb 
question is the one not asked.
  In cmdAH.test, test 19.11, TclpObjNormalizePath is asked to 
normalize "/tmp/tcl.foo.dir/file" from CheckAccess. Currently, 
TclpObjNormalizePath returns "/tmp/tcl.foo.dir/file" because realpath 
fails, assumed due to permission. But, if you do normalize the path to 
"/private/tmp/tcl.foo.dir/file" it fails because TclpObjAccess fails.
  So I'm not sure if TclpObjNormalizePath should be checking access or 
provide a normalized path?

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-22 00:44

Message:
Logged In: YES 
user_id=32170

Ok, in that case you should return the full path (including
/foo), but the 'startAt' that should be returned should
point to the '/' before foo -- thereby saying that the path
was understand (and normalized) up to that point.  In fact
'startAt', as returned, should always point to a '/'.

(I think that's correct - no time to look at the code/docs
right now, sorry).

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-21 23:38

Message:
Logged In: YES 
user_id=1011552

Maybe I phrase wrong?
   The path to normalize entering TclpObjNormalizePath is "/Users/
steven/Desktop/tclmyproj/tcl8.4.6/macosx/build/gorp.file/foo".
At that instant it is passed there is a "/Users/steven/Desktop/
tclmyproj/tcl8.4.6/macosx/build/gorp.file", but my new code errors
on "/Users/steven/Desktop/tclmyproj/tcl8.4.6/macosx/build/gorp.file/
foo" exits with error "ENOTDIR" because gorp.file is not a directory. It 
returns a NULL string with startAt of 53. It is then passed "/Users/
steven/Desktop/tclmyproj/tcl8.4.6/macosx/build/link.file/foo" exiting 
"ENOTDIR" with NULL and startAt of 53.
  So I am trying to be sure of what is being requested a that precise 
instant in the program. Is it supposed to return "/Users/steven/
Desktop/tclmyproj/tcl8.4.6/macosx/build/gorp.file/foo" possibily 
allowing someone to try an operation on that file? If gorp.file was a 
directory and link.file a directory it passes, hence, should I be passing 
a string ignoring the information found and/or what should be the 
startAt?

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-21 17:09

Message:
Logged In: YES 
user_id=32170

As you say, you're better off trying to create a replacement
for realpath, since the rest of the code is known to work ok.

As for: [file normalize [file join gorp.file foo]]

Well, file join will return gorp.file/foo, and that must
then be normalized partly by other functions and partly by
the code you're trying to write.  You probably want to read
the documentation of 'file join'...

I don't see how the code that you write should even notice
whether it's being given:

   file normalize gorp.file/foo
or
   file normalize [file join gorp.file/foo]

Since your code is only inside the 'file normalize' routines....

Anyway, 'file join' is very well documented (more so than we
can explain here is a few lines), so please do read that,
and good luck with the Realpath implementation.

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-21 13:34

Message:
Logged In: YES 
user_id=1011552

Hi
  I was doing some testing, I haven't mention the code I sent you has 
one small bug, but it tests identical to tests with using realpath, except 
for startAt return value.
  Now, I created new code that handles exceptions, a realpath 
replacement, and luckily got some FAILS. So I need some clarifaction 
on what the tests are asking or what join is permitted to do.
  As an example (from test filesystem-1.2):
  [file normalize [file join gorp.file foo]]
questions: should grop.file be turned into a dir to yield gorp.file/foo?
      : if gorp.file is turned into a dir is link still valid?
      : should it be gorp.filefoo?
:am I just loonie?
my return values from TclpObjNormalizePath show ....../gorp.file/foo
Thanks

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-20 23:58

Message:
Logged In: YES 
user_id=1011552

Not sure if this helps, but as tcl is mentioned on a list of realpath users 
BSD issued a notice about a security issue, and the new realpath 
mentions something about thread safe.

----------------------------------------------------------------------

Comment By: Daniel A. Steffen (das)
Date: 2004-04-20 17:14

Message:
Logged In: YES 
user_id=90580

sorry for the delay

this a known problem with threaded tcl on OSX; the OSX realpath is not 
threadsafe (it uses chdir internally and the current dir is process global...), 
so we've had to disable it in the threaded build:

    2003-05-13  Daniel Steffen  <da...@us...>

        * unix/tclUnixPort.h: worked around the issue of realpath() not
        being thread-safe on Mac OS X by defining NO_REALPATH for 
        threaded builds on Mac OS X. [Bug 711232]

see 

        https://sourceforge.net/tracker/?
func=detail&atid=110894&aid=711232&group_id=10894

at the time, Zoran said he'd submitted a bugreport with Apple about this, 
don't know the current status of that though; Jim?

to fix this in tcl, TclpObjNormalizePath() would need to be implemented 
properly for systems without realpath

     /* 
     * We should really now convert this to a canonical path.  We do
     * that with 'realpath' if we have it available.  Otherwise we could
     * step through every single path component, checking whether it is a 
     * symlink, but that would be a lot of work, and most modern OSes 
     * have 'realpath'.
     */

(in fact realpath on OSX does exactly this, but in an thread-unsafe 
manner...)

or we could add a crossplatform realpath() to compat.

tauvan, for questions that are "not bugs" use the tcl-mac mailing list:
        https://lists.sourceforge.net/lists/listinfo/tcl-mac

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-08 19:15

Message:
Logged In: YES 
user_id=32170

This is somewhat beyond me, but I think Daniel will be able
to shed some light (when he gets back from cycling next week
;-).

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-08 17:21

Message:
Logged In: YES 
user_id=1011552

Sorry to do this but I just found so new data that may
or may not just make this a OSX issue.
In trying to gather more info on another bug, I had this
idea of doing a straight unix build, not using macosx makefile nor
my build system. Well I accedently forgot to "--enable-threads".
After running the tests, I noticed there was not 27 failures.
All fileSystem failures disappeared. I then ran "--enable-threads"
they appeared. Then, tried without enabled again, disappeared.
I also verified that the test did not skip because of testthread.
  I only ran this as local user, but thought I should pass this on
immediatly, because it changes things.

----------------------------------------------------------------------

Comment By: tauvan (tauvan)
Date: 2004-04-06 00:38

Message:
Logged In: YES 
user_id=1011552

ac_cv_func_realpath=${ac_cv_func_realpath=yes}

However, in tclUnixPort.h;
#ifdef MAC_OSX_TCL
/* 
 * On Mac OS X, realpath is currently not
 * thread safe, c.f. SF bug # 711232.
 */
#define NO_REALPATH
#endif

----------------------------------------------------------------------

Comment By: Vince Darley (vincentdarley)
Date: 2004-04-05 18:58

Message:
Logged In: YES 
user_id=32170

Thanks for the report -- we'll look into these.  The first
thing that occurs to me is whether 'realpath' is supported
or not on OS X 10.1.5.

Anyway, if you have questions, either comp.lang.tcl or the
mac-tcl mailing list is the place.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=929534&group_id=10894

[Tcl-bugs] [ tcl-Bugs-929534 ] OS X: filesystem FAILED link normalisation

The Tool Command Language implementation

[Tcl-bugs] [ tcl-Bugs-929534 ] OS X: filesystem FAILED link normalisation