#541 [glob] can return non-existing filenames

[Mark Janssen]

glob doesn't escape filenames starting with ~. This means glob by default can return filenames that 1) don't exists (annoying) or even point to a different spot all together (a user directory for instance, which is outright dangerous).

For example:
create a file name "~test"

% foreach file [glob *] { if {![file exists $file]} {puts "huh $file doesn't exists"}}
huh ~test doesn't exists
% info pa
8.5.4

Interactive session above was with Tcl 8.5.4.
Dangerous situations can occur if a filename matches a valid user directory (~ for instance) and results of glob are used in recursive delete operations (which would lead to removal of the users home dir in this case)

It seems that me that glob should properly escape ~ as "./~test".

[Don Porter]

There's a related regression
in [file tail]:

% set files [glob *oo]
~boo ~foo
% set files2 [glob -directory [pwd] *oo]
/home/dgp/cvs/tcl8.4/unix/~boo /home/dgp/cvs/tcl8.4/unix/~foo
% file tail [lindex $files2 0]
./~boo
% info patch
8.4.19

% set files [glob *oo]
~foo ~boo
% set files2 [glob -directory [pwd] *oo]
/home/dgp/cvs/tcl8.5/unix/~foo /home/dgp/cvs/tcl8.5/unix/~boo
% file tail [lindex $files2 0]
~foo
% file tail /home/dgp/cvs/tcl8.5/unix/~foo
./~foo
% info patch
8.5.6

The symptoms point to inconsistent
handling of values of the "path"
Tcl_ObjType depending on the details
of the intrep. Same string ought
to produce same results, but do not.

[Nobody/Anonymous]

Yes expansion of ~ is the culprit in this scenario. (most of?) Tcl's file commands interpret ~ as a home directory. If [glob] does not escape ~ in a filename it returns it is a bug in [glob]. Rewriting glob results to escape ~ is not an option. The contract of [glob] specifies it returns filenames. If you try to argue that automatic ~ expansion is a bad thing, I am likely to agree (it's a UNIX-ism for one) but that's unfixable without breaking backward compatibility.

[Donal K. Fellows]

The backslashes will have no effect; they're processed at an entirely different level of Tcl.

The only question is what the contract for [glob] in this situation should actually be. (Note that it is pretty clear what [glob -tails] should be doing; it's when that option is not specified that I'm not sure what should be happening.) Also, if things are changed then are we going to be adversely impacting uses of [glob]? That's also important.

(Well, there's also the bizarre bugs in [file tail], but they probably should be their own Bug...)

[Don Porter]

Reported again within 2834453.

[Don Porter]

Note that with VFS, this problem can be more
general than ~ expansion. Any time the tail of
an absolute filename is recognized on its own
also as a different absolute filename, there's going
to be confusion that the pathnames returned by
[glob] do not point back the the same files whose
existence [glob] is acting upon.

Need to dig in to be sure, but it seems the right
solution is some checking within [glob] that
the paths it returns that it expects to be relative
are in fact relative, and applying some fix
(following the ./ prefixing example) to correct
any that are not.

Even that will leave some difficulties, if a
relative path is retrieved from [glob] and
stored, and then a later mount of a filesystem
converts that stored path from relative to
absolute. I think that level of difficulty cannot
be avoided given the dynamic view of the
filesystem Tcl exposes. Any paths meant
to be kept around for any length of time at
all need to be in absolute form.

[Don Porter]

At least this example of the problem
seems to filter down to a flaw in the
TclFSMakePathRelative() which is
failing to, um, make the path relative. :)

[Don Porter]

The command [tk::dialog::file::JoinFile] is
an example of needing to work around this
issue.

[Don Porter]

Comments in the routine
TclFSMakePathRelative() indicate
an awareness of this issue, and
an intentional discarding of
cached intreps so that the ~foo
name can be re-interpreted later
without contradiction. The
recorded justification is that
"this is how glob is currently
defined".

This is looking like RFE/TIP
territory.

[Don Porter]

On the other hand, test filename-11.42,
which was put in place for the 8.4.0 release
explicitly tests that the elements returned
by [glob *] match what would be produced
by [file tail], which fails precisely on the sort
of filenames examined here.

[Don Porter]

tests filesystem-9.7,8,9 are present
to guarantee that the behavior reported
here continues unchanged.

It may be annoying and stupid, but it is
intentional. Not a bug.

Shifting this report over the Feature Request
tracker.