#2975 change to tcl_findLibrary breaks freewrap

obsolete: 8.4.8
closed-invalid
Don Porter
None
9
2004-11-26
2004-11-23
Colin McDonald
No

A recent change to tcl_findLibrary in auto.tcl,
released in Tcl 8.4.8, breaks freewrap.

For all platforms freewrap stores its Tcl & Tk
library files in a zip based virtual filesystem,
under directories named /tcl & /tk. These
directories are pointed to by the TCL_LIBRARY &
TK_LIBRARY environment variables set in the
application initialisation.

The recent change to tcl_findLibrary introduced
use of file normalize in the section "# uniqify
$dirs in order". Unfortunately under Windows, file
normalize changes the /tk directory path by
treating it as a volume relative name, and
applying the default device name e.g to become
D:/tk. The result of this is that Tk fails to find
its library files and freewrap doesn't initialise.

Note that freewrap is not using the modern tcl
virtual file system api. Instead it is using the
old TclPro hooks, which may explain why file
normalize manipulates /tk as a Windows path.

Removing the file normalize fixes the problem.

Discussion

  • Don Porter
    Don Porter
    2004-11-23

    • labels: 105676 --> 104242
     
  • Don Porter
    Don Porter
    2004-11-23

    • assigned_to: dgp --> vincentdarley
     
  • Don Porter
    Don Porter
    2004-11-23

    Logged In: YES
    user_id=80530

    sounds like filesystem problems to me.

    if $path and [file normalize $path] do not
    point to the same file, then something
    in the filesystem support code is broken.

     
  • Vince Darley
    Vince Darley
    2004-11-24

    • status: open --> pending
     
  • Vince Darley
    Vince Darley
    2004-11-24

    • status: pending --> pending-invalid
     
  • Vince Darley
    Vince Darley
    2004-11-24

    Logged In: YES
    user_id=32170

    Under Windows paths starting with '/' are _defined_ to be
    volume relative, therefore Tcl's behaviour is correct. Now
    it could be there's some bug you have found (certainly a
    number of volume-relative bugs have existed in Tcl and many
    have been fixed in the last 12 months).

    However, the more general point is that freewrap's use of
    '/tcl' and '/tk' is not correct, and should not have worked
    in the past -- i.e. freewrap relied on Tcl not having
    correct handling of volume relative paths, it seems.

    A possible resolution is for freewrap to claim those paths
    as absolute by mounting a new volume, like this:

    vfs::filesystem mount -volume /tcl "#not sure what goes here"

    (but I see you're not using modern Tcl interfaces).

    Then Tcl will not treat that path as volume relative (but I
    would be quite wary of this approach, since it is in some
    sense fundamentally incompatible with the WinTcl
    filesystem). Better would be to mount a new filesystem:
    freewrap:/ with 'tcl' and 'tk' as subdirectories.

    In summary:

    * freewrap's use of '/tcl' is relying on bugs in old
    versions of Tcl
    * many volume relative bugs have been fixed, it is possible
    there are more, but Tcl's treatment of '/tcl' as volume
    relative is correct.

    Question: does undoing the change in tcl_findLibrary
    actually workaround the problem? (If so, there is a bug in
    Tcl still, in its usage of the old char-based TclPro hooks,
    if not, then it just shows that volume relative paths are
    now treated correctly).

    I hope that helps. Do feel free to file bugs in Tcl's
    volume-relative file treatment, if any still exist, but
    beyond that this seems to be a freewrap problem, really.

    So, don't know what to suggest, except that I don't see us
    making any filesystem changes to accomodate this behaviour.
    Probably using 'freewrap:/' is the best bet.

    Vince.

     
  • Colin McDonald
    Colin McDonald
    2004-11-24

    Logged In: YES
    user_id=883691

    My concern with this problem is that moving
    from 8.4.7 to 8.4.8 in the stable 8.4.* release
    series breaks freewrap, a deployment tool which
    is used by many people.

    Undoing the change in tcl_findLibrary does
    actually workaround the problem. Access to files
    in the freewrap zip file system is expected to be
    pretty simple i.e. read only, no writing, no
    globbing and no file normalize (which did not
    even exist when the TclPro hooks were
    developed). It is not a full-blown virtual file
    system in the (excellent) sense of your modern
    interfaces.

    The change to tcl_findLibrary introduces a
    dependency on file normalize for successful
    access to the tk_library files. As it does not
    behave as expected for the freewrap /tk files
    then freewrap breaks. I think it is a bit unfair
    to say that freewrap is dependant on buggy
    behaviour in Tcl - it is dependant on the
    expected behaviour and use of the TclPro hooks as
    they were originally implemented.

    I can see that your suggestion of freewrap:/ may
    work - but would intoduce platform dependant code
    into freewrap, with different mount points for
    Unix & Windows.

    I will notify the freewrap author, Dennis LaBelle
    of this problem and discussion. Hopefully he
    will comment.

     
  • Colin McDonald
    Colin McDonald
    2004-11-24

    • status: pending-invalid --> open-invalid
     
  • Vince Darley
    Vince Darley
    2004-11-24

    Logged In: YES
    user_id=32170

    Ok, so freewrap depends on being able to intercept a
    particular string representation of a file ('/tcl'), and as
    such init.tcl's new approach of converting that string
    representation into a real path (by normalization) means
    that freewrap can no longer intercept what it wants.

    The root of the problem is that freewrap is trying to
    intercept a path which (on Windows) is not fixed -- the real
    interpretation of '/tcl' changes depending on the current
    working directory.

    So, anyway, it's not a filesystem bug.

     
  • Vince Darley
    Vince Darley
    2004-11-24

    • assigned_to: vincentdarley --> dgp
    • labels: 104242 --> 105676
     
  • Don Porter
    Don Porter
    2004-11-24

    • labels: 105676 -->
    • status: open-invalid --> closed-invalid
     
  • Don Porter
    Don Porter
    2004-11-24

    Logged In: YES
    user_id=80530

    ok, so appears this is really
    a problem in freewrap, and should
    be fixed there. Incompatibilities
    with [file normalize] likely go much
    deeper than [tcl_findLibrary].

     
  • Don Porter
    Don Porter
    2004-11-25

    Logged In: YES
    user_id=80530

    A serious bug (or two?) escaped in Tcl 8.4.8,
    so there will be a Tcl 8.4.9 pretty soon.

    Would it help buy freewrap some
    bug fixing time if line 133 of auto.tcl
    were changed from

    lappend uniqdirs $norm
    to
    lappend uniqdirs $i

    That way, [tcl_findLibrary] would
    only unquify its search list, and not
    also normalize its entries.

    There's still a real bug in freewrap
    if it can't work compatibility with
    [file normalize], and it should be fixed,
    but if this change helps keep freewrap
    functioning in the Tcl 8.4 series, I'm
    willing to make it.

     
  • Don Porter
    Don Porter
    2004-11-25

    • status: closed-invalid --> open-invalid
     
  • Don Porter
    Don Porter
    2004-11-25

    • priority: 5 --> 9
     
  • Colin McDonald
    Colin McDonald
    2004-11-25

    Logged In: YES
    user_id=883691

    I have tested my installation with the following, and it enables
    freewrap to work successfully:

    # uniquify $dirs in order
    array set seen {}
    foreach i $dirs {
    if {[info exists seen($i)]} { continue }
    set seen($i) ""
    lappend uniqdirs $i
    }
    set dirs $uniqdirs

     
  • Don Porter
    Don Porter
    2004-11-26

    • status: open-invalid --> closed-invalid
     
  • Don Porter
    Don Porter
    2004-11-26

    Logged In: YES
    user_id=80530

    requested change made for
    Tcl 8.4.9 and 8.5a2.

    another bug report should be
    filed with freewrap to sort out
    its remaining problems with
    [file normalize].

     
  • Logged In: YES
    user_id=110848

    I'm afraid this bug notice was closed out too soon. There appears to be a
    bug in the C code that supports the [file normalize] command. It looks like
    TCL's C code has a side effect that improperly overwrites memory pointed
    to by a TCL variable.

    The "reasons" put forth so far to explain the bug notice are incorrect. Here
    are the reasons for my conclusion:

    1) The TCLSH only version of freeWrap still builds and runs fine under
    Windows.

    2) The posted problem does not occur until the TCL/TK version of
    freeWrap is built.

    3) Although removing the call to [file normalize] in the auto.tcl file does
    "fix" the problem, placing the following line after the offending line should
    also provide a fix...But it doesn't!

    set norm $i

    Replacing the offending line with:

    file normalize $i
    set norm $i

    still produces the problem. This means that [file normalize $i] is producing
    an unwanted side-effect that is preventing proper TK initilization.
    Comments in the TCL C code seems to indicate that the normalization
    process extends the path string by using the memory pointed to by a
    pointer. The intermittent "failure" of the code is consistent with the
    accidental overwriting of memory (Sometimes the memory overwritten is
    important, sometimes it is not).

     
  • Logged In: YES
    user_id=79902

    Nonetheless, please file that as a separate bug.

     
  • Don Porter
    Don Porter
    2004-11-29

    Logged In: YES
    user_id=80530

    We've previouslty established that
    freewrap insists on making its own
    interpretation of strings beginning
    with /tcl or /tk that is inconsistent
    with Tcl's normal parsing of path
    names on Windows.

    [file normalize], like any operation
    that treats a value as a filesystem
    path, will keep a cache of the
    resolved filesystem name in
    the internal rep of the value.
    Nothing buggy about that.

    Since freewrap insists on interpreting
    the string differently, it should take
    care to extract only the string value
    and act on it, to avoid problems
    with the internal rep that are contrary
    to its desires.

    It must be emphasized that the ability
    to set the internal rep cache of a Tcl_Obj
    is in no way "buggy" or "an overwrite of
    memory". It's Tcl functioning properly.

    It's unfortunate the simple change I made
    was not enough to restore freewrap's
    functioning, but the bug is in freewrap,
    and fixes and/or workarounds really belong there.

     
  • Logged In: YES
    user_id=110848

    The previous explanation by dgp doesn't address why the TCL-only
    version of freeWrap does not crash but the TCL/TK version does. If
    there's a problem with the ZIP virtual file system why doesn't the TCLSH
    version crash?

    Also, using a debugger to step through freeWrap's ZIP VFS shows that the
    TCL/TK version of freeWrap crashes before any "normalized" file name is
    sent to it.

     
  • Don Porter
    Don Porter
    2004-11-30

    Logged In: YES
    user_id=80530

    Sorry, I thought that was understood.

    Tk_Init() calls [tcl_findLibrary] which
    (as of Tcl 8.4.8) calls [file normalize]
    on path values held in $::env(TK_LIBRARY),
    $::auto_path and others.

    Tcl itself never calls [tcl_findLibrary].

    However, [tcl_findLibrary] is just
    the trigger here. Any operation at
    all that used one of these path values
    as a filesystem path would cache
    the resolved value and cause the
    problem described in the original
    report.

    The original report said nothing
    about a "crash" (as I understand
    the term), and I've not duplicated
    this problem myself, so I don't
    know how to answer that.

     
  • Colin McDonald
    Colin McDonald
    2004-11-30

    Logged In: YES
    user_id=883691

    I believe that my analysis of the original
    problem which I saw and reported was correct.
    Tk was failing to find its library files using
    tcl_findLibrary because the path was being
    changed from /tk to device:/tk. I was able to
    verify this with diagnostic statements.

    The change made by Don Porter in response to my
    report removes the problem in my installation - I
    am rebuilding from source with freewrap 5.5. The
    resulting build works fine. Note that the file
    normalize behaviour as described is only a
    problem accessing the zipfs file system, the use
    of which is quite limited i.e. to store tcl/tk
    library files and the application program files.
    file normalize on the rest of the user's Windows
    file system works as expected in freewrap, and
    is not a problem.

    I am not experiencing any memory corruption or
    crashes in my build. I think that must be a
    different problem.

     
  • Don Porter
    Don Porter
    2004-12-01

    Logged In: YES
    user_id=80530

    For the Tcl 8.4.9 release, I've
    disabled the use of [file normalize]
    by [tcl_findLibrary] in hopes that
    will be sufficient to keep freewrap
    at least apparently working.

    For Tcl 8.5, [file normalize] is staying
    in, and freewrap should aim at
    resolving its conflicts with Tcl's
    filesystem interfaces in order to
    support Tcl 8.5 and later.