#4941 file join/normalize bugs violate EIAS

obsolete: 8.5.10

Reproduced on WinXP SP3 with ActiveState 8.4 and 8.6 and Ubuntu 11.04 Tcl 8.5

The following code produces unexpected results :

# Bug #1 : why $foo1 != $foo2 !?
cd /tmp

set foo1 [file join [pwd] foo/]
set foo2 [file join /tmp foo/]

puts "foo1=*$foo1*"
puts "foo2=*$foo2*"

# Bug #2 : why normalize works differently for the same actual string !?

set fooFromStr "/tmp/foo/"
if { $foo1 eq $fooFromStr } {
puts "Yes, I'm not crazy, foo1 eq fooFromStr !"
puts "norm foo1 =*[file normalize $foo1 ]*"
puts "normFormStr=*[file normalize $fooFromStr]*"

Outputs is :

Yes, I'm not crazy, foo1 eq fooFromStr !
norm foo1 =*/tmp/foo/*

There is two problems :
1.There is no reason why [file join] would return different strings when the same parameter is passed two different ways.
2.There is no reason why two identical strings would produce different results when [file normalize] is called with them as parameters.


1 2 > >> (Page 1 of 2)
  • Strykaas

    Strykaas - 2011-09-27
    • priority: 5 --> 9
    • status: open --> open-accepted
  • Andreas Leitgeb

    Andreas Leitgeb - 2011-09-28

    FWIW, with Tcl 8.5.8 I can reproduce only the second (about file normalize), but not the first (both have trailing slash)

  • Donal K. Fellows

    The tip of the trunk can't reproduce Bug#1 either.

  • Don Porter

    Don Porter - 2011-09-28
    • assigned_to: vincentdarley --> dgp
    • priority: 9 --> 5
  • Andreas Leitgeb

    Andreas Leitgeb - 2011-09-28

    Bug#1 is even more strange:
    $ cat testfile.tcl
    cd /tmp
    puts [info patchlevel]
    puts "pwd:[file join [pwd] foo/]"
    puts "/tmp:[file join /tmp foo/]"
    $ tclsh8.5 testfile.tcl
    $ tclsh8.5 < testfile.tcl

    Just in case that SF hoses the formatted text: I make a small script that does cd, and then
    puts for the two file join lines. If I start tclsh8.5 with the script as an argument, then the bug
    shows up, however if I merely redirect tclsh8.5's stdin from the script, then both variants
    of file join do show the trailing slash.

  • Andreas Leitgeb

    Andreas Leitgeb - 2011-09-28

    PS: my previous comment also applies to tclsh8.6b2 produced from the thusly-versioned official tarball.

  • Ralf Fassel

    Ralf Fassel - 2011-09-28

    Seems related:

    $ tclsh8.5
    % info patchlevel
    % cd /tmp
    % set foo1 [file join [pwd] foo/]
    % file normalize $foo1
    % string length $foo1
    % file normalize $foo1

    Note how the result of "file normalize" changes after taking the "string length" of foo1.

  • Don Porter

    Don Porter - 2011-09-28

    1) Testing results on 8.5.8 are not really
    very interesting. 8.5.10 or later please.

    2) Yes, it's clear that [file normalize] operates
    differently on different intreps, and that's a bug
    that needs fixing. We don't need more script
    variants provoking shimmering to demo that.

  • Andreas Leitgeb

    Andreas Leitgeb - 2011-09-28

    1) Did you miss the test report for 8.6b2 which I posted?

    Btw., 8.5.8 is what is installed on Ubuntu 10.10, maybe thus its popularity...

  • Strykaas

    Strykaas - 2011-09-28

    Indeed, the problem depicted here is visible if the provided code is sourced (i.e. code not typed directly within a tcl shell). That is probably why the two first comments state bug #1 is not reproducible.

  • Don Porter

    Don Porter - 2011-10-05

    An experimental branch fixes this by
    eliminating the buggy intrep. See

    This needs testing to determine whether it
    has unacceptable performance impacts. Also,
    it causes these test failures:

    ==== filename-13.1 globbing with brace substitution FAILED
    ==== Contents of test case:

    glob globTest/\{\}

    ---- Result was:
    ---- Result should have been (exact matching):
    ==== filename-13.1 FAILED

    ==== filename-14.15 asterisks, question marks, and brackets FAILED
    ==== Contents of test case:

    lsort [glob globTest/*/]

    ---- Result was:
    globTest/a1 globTest/a2 globTest/a3
    ---- Result should have been (exact matching):
    globTest/a1/ globTest/a2/ globTest/a3/
    ==== filename-14.15 FAILED

    ==== filename-14.30 Bug 2710920 FAILED
    ==== Contents of test case:

    file rootname [lindex [lsort [glob globTest/*/]] 0]

    ---- Result was:
    ---- Result should have been (exact matching):
    ==== filename-14.30 FAILED

    Where the results of a [glob] are actually
    tested to be sure they have the trailing /
    while the bug report here wants it dropped.

  • Don Porter

    Don Porter - 2011-10-05

    A quick trial of this branch vs trunk on
    tclbench's file.bench reveals these
    apparent problems:

    007 FILE glob tmpdir (60 entries) 262.35 137.62

    011 FILE glob / dirname 702.31 255.50

    014 FILE glob / extension 384.40 267.40

    020 FILE glob / rootname 385.44 285.91

    022 FILE glob / tail 565.95 255.27

  • Don Porter

    Don Porter - 2011-10-05

    Not everything is slower though.

    009 FILE glob / atime 718.00 744.44
    012 FILE glob / executable 694.96 714.80
    013 FILE glob / exists 693.88 714.44
    015 FILE glob / isdirectory 719.39 741.74
    016 FILE glob / isfile 720.11 740.94
    017 FILE glob / mtime 717.75 746.52
    018 FILE glob / owned 744.35 768.85
    019 FILE glob / readable 696.89 713.09
    021 FILE glob / size 723.59 746.08
    023 FILE glob / writable 697.59 712.37

  • Don Porter

    Don Porter - 2011-10-06

    experimental branch now passes all tests.
    No significant change to the file.bench results.

    The problematic benchmark, it seems to me,
    is 007 in the number below which suggests that
    a basic

    glob -nocomplain -directory $dir *

    operation on a directory with 60 entries
    (30 file, 30 subdits) is twice as slow
    after this change.

  • Strykaas

    Strykaas - 2011-10-06

    Thanks dgp for trying to fix the issue. What is the actual status of your fix ? It is obviously unacceptable to take such a performance hit, although the language would be more sane. Maybe a more elaborate fix could avoid that ?

  • Don Porter

    Don Porter - 2011-10-12

    Everytime I'm forced to dive into tclPathObj.c, it's
    to construct a "more elaborate patch". And everytime
    I rant that life would be so much easier if we just trashed
    all this complexity. Finally got ticked off enough to try it
    this time. At least now we know what good it does.

    It's not all that hard for me to come up with a "more elaborate"
    patch that will work resolving the reported issue.

    It's much much harder for me to come with the patch that
    fixes the issue that I also have string confidence will not
    introduce a new cluster of bugs at the same time. that's what
    will hold me back. The code is just that tangled.

  • Don Porter

    Don Porter - 2011-10-27

    See branch bug-3414754 for proposed fix.

  • Don Porter

    Don Porter - 2011-10-27

    Test added.

  • Don Porter

    Don Porter - 2011-10-27

    Before I merge to a release branch, I need
    a report of a successful test on Windows to
    be sure I've not broken something terribly there.

  • Don Porter

    Don Porter - 2011-10-28

    Joe, can I trouble you to test this branch
    on Windows? Thanks!

  • Don Porter

    Don Porter - 2011-10-28
    • assigned_to: dgp --> mistachkin
    • status: open-accepted --> pending-accepted
  • Don Porter

    Don Porter - 2011-10-28
    • assigned_to: mistachkin --> nijtmans
  • Don Porter

    Don Porter - 2011-10-28

    Jan, is this a test you can run? I'm holding
    back 8.5.11RC generation until I see whether
    this bug fix is fit for merging into it.

  • Jan Nijtmans

    Jan Nijtmans - 2011-10-29

    Ran the test on Windows 7, with both mingw
    and Visual Studio 10.

    Looks OK to me, yes I wouls merge this.

    The only thing I encoungered was a test
    failure in filesystem-1.3 and filesystem-1.4
    (see bug 3075027, the other failures
    mentioned there are solved already),
    but those two tests are already failing
    for ages, I would not hold 8.5.11 for it.

  • Jan Nijtmans

    Jan Nijtmans - 2011-10-29
    • assigned_to: nijtmans --> dgp
1 2 > >> (Page 1 of 2)

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks