#101 Long file names / can't cd to / at PERL2EXE_STORAGE/File/Find.pm line 940

[Anonymous]

I am getting the error and no XML output for a CLOC run on the OSS project mojarra. The error is:

Can't cd to p:\analysis\downloads\481 [JavaServer Faces]\repositories/481727/MOJARRA_2_1X_ROLLING/test/agnostic/vdl/facelets/faces-config-extension/html5/src/test/java/com/sun/faces/test/agnostic/vdl/facelets/faces_config_extension/html5../../../../../../../../../../../../../.. at PERL2EXE_STORAGE/File/Find.pm line 940.

This problem in similar form occurs for other projects as well.

I am using cloc-1.60.exe on Windows 7 64bit, NTFS file system, with the following command line:

p:\analysis\binaries\cloc\cloc-1.60.exe --xml --by-file-by-lang --report-file p:\analysis\downloads\481 [JavaServer Faces]\loc_project_xml_481.xml p:\analysis\downloads\481 [JavaServer Faces]\repositories

TO REPRODUCE:

• Download the project MOJARRA via Subversion from https://svn.java.net/svn/mojarra~svn/branches/MOJARRA_2_1X_ROLLING/, using version from date '2014-03-07 3:21h'.
• Put this code into disk/directory p:/analysis/downloads/481 [JavaServer Faces]/repositories/481727/MOJARRA_2_1X_ROLLING/ on a Windows 7 64bit machine.
• Run the above command line

If you need any additional info, let me know.

[Phil Mayer]

Submitter is me, forgot to log in.

[Al Danial]

I did the following on a Linux machine using the current development snapshot of cloc:

svn export -r {'2014-03-07 03:21:00'} https://svn.java.net/svn/mojarra~svn/branches/MOJARRA_2_1X_ROLLING/
cloc --xml --by-file-by-lang --report-file loc_project_xml_481.xml MOJARRA_2_1X_ROLLING

There were no issues; report file is attached. Tomorrow I'll try on a Windows machine to see if I can duplicate the problem.

[Phil Mayer]

Thanks for investigating; I am interested to see whether this is a Windows issue!

[Phil Mayer]

Update: I just tried installing Strawberry Perl and running "perl cloc-1.60.pl" instead of cloc-1.60.exe. The error message differs slightly but points to the same problem:

P:\analysis\binaries\cloc>perl cloc-1.60.pl --xml --by-file-by-lang --report-file "p:\analysis\downloads\481 [JavaServer Faces]\loc_project_xml_481.xml" "p:\analysis\downloads\481 [JavaServer Faces]\repositories"
Can't cd to ../../../../../../../../../../../../../.. from p:\analysis\downloads\481 [JavaServer Faces]\repositories/481727/MOJARRA_2_1X_ROLLING/test/agnostic/vdl/facelets/faces-config-extension/html5/src/test/java/com/sun/faces/test/agnostic/vdl/facelets/faces_config_extension/html5 at C:/code/strawberry/perl/lib/File/Find.pm line 812.

:(

[Al Danial]

I just finished a successful run on Windows. Here's my theory on why your run fails: the target directory path name, "p:\analysis\downloads\481 [JavaServer Faces]\repositories/481727" has open and close brackets that confuse the File::Find module where the error occurs. (As you may know, in Perl, open/close brackets define a character class for regular expressions.)

To test that theory, copy the Mojarra code to another directory name that is as vanilla as they come, like c:\tmp, then run cloc against the source at that location.

[Anonymous]

Hi Al,

Some more experiments from my side have revealed the following issues. First, shortening the path works:

p:\analysis\binaries\cloc\cloc-1.60.exe --xml --by-file-by-lang --report-file "p:\X\loc_project_xml_481.xml" "p:\X"
   11098 text files.
    6574 unique files.
   19636 files ignored.
Wrote p:\X\loc_project_xml_481.xml

However, adding [] to the path shows no difference, so this does not seem to be the issue:

p:\analysis\binaries\cloc\cloc-1.60.exe --xml --by-file-by-lang --report-file "p:\X [Foo]\loc_project_xml_481.xml" "p:\X [Foo]"
   11098 text files.
    6574 unique files.
   19636 files ignored.
Wrote p:\X [Foo]\loc_project_xml_481.xml

Then again, LENGHTENING the path leads to the issue:

C:\Users\mayer>p:\analysis\binaries\cloc\cloc-1.60.exe --xml --by-file-by-lang --report-file "p:\hjsvadsvadzujsvadgfjvasdjhzvasdfjvasjdfvasjfdvasjdfvasjdfvasjdfvasdjfvajsdfvasjfvsdkjbfhvaksjbdf\loc_project_xml_481.xml" "p:\hjsvadsvadzujsvadgfjvasdjhzvasdfjvasjdfvasjfdvasjdfvasjdfvasjdfvasdjfvajsdfvasjfvsdkjbfhvaksjbdf"
Can't cd to p:\hjsvadsvadzujsvadgfjvasdjhzvasdfjvasjdfvasjfdvasjdfvasjdfvasjdfvasdjfvajsdfvasjfvsdkjbfhvaksjbdf/480563/trunk/jsf-ri/systest-per-webapp/myfaces-uidata-component-state-test/src/java/com/sun/faces/systest/myfaces_uidata_component_state_test../../../../../../.. at PERL2EXE_STORAGE/File/Find.pm line 940.

So, I conclude, long file names are the issue on Windows. Can you reproduce?

[Phil Mayer]

...previous anonymous post was me again, forgetting to log in. Grrr. Apparently needs moderator approval this time ;-)

[Al Danial]

Yes, it is a file path length issue on Windows. (The same long path I tried on Linux on an NFS file system worked, but I'm sure there's a point where it, too, breaks.)

[Anonymous]

Update with a possibly related error:

Can't opendir(p:\analysis\downloads\10680 [GoGoEgo]\repositories/20301/trunk/modules/com.solertium.gogoego.legacy.server.extensions.formforwarder/trunk/com.solertium.gogoego.legacy.server.extensions.formforwarder/src/com/solertium/gogoego/server/extensions/formforwarder): No such file or directory at p:\analysis\binaries\cloc\cloc-1.60.exe line 2896

[Mac]

I've got the same issue due to the way npm handles dependencies https://github.com/npm/npm/issues/3697
More annoyingly the trouble occurs on a folder that has been specified as ignored

"D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\packages\Isogeo.Build.1.0.3.10\tools\build\cloc.exe" D:\ccnet\projects\Isogeo.App\ContinuousBuild\work --exclude-dir=bin,bower_components,build,dist,doc,lib,node_modules,obj,packages,tmp,vendor,GeneratedCode --exclude-ext=csproj,dbmdl,proj,sln,sqlproj,suo,user --not-match-f="(^jquery.|.?min.js$|^_)" --3 --quiet --progress-rate=0 --xml --report_file="D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\tmp\cloc-results.xml"
Can't opendir(D:\ccnet\projects\Isogeo.App\ContinuousBuild\work/node_modules/grunt-contrib-imagemin/node_modules/gifsicle/node_modules/bin-wrapper/node_modules/download/node_modules/request/node_modules/form-data/node_modules/combined-stream/node_modules/delayed-stream/lib): No such file or directory
at D:\ccnet\projects\Isogeo.App\ContinuousBuild\work\packages\Isogeo.Build.1.0.3.10\tools\build\cloc.exe line 2896
Can't cd to D:\ccnet\projects\Isogeo.App\ContinuousBuild\work/node_modules/grunt-contrib-imagemin/node_modules/gifsicle/node_modules/bin-wrapper/node_modules/download/node_modules/request/node_modules/form-data/node_modules/combined-stream/node_modules/delayed-stream/lib../../.. at PERL2EXE_STORAGE/File/Find.pm line 940.
Can't stat initial working directory at PERL2EXE_STORAGE/File/Temp.pm line 898

[Al Danial]

The path "D:\ccnet\projects\Isogeo.App\ContinuousBuild\work/node_modules/grunt-contrib-imagemin/node_modules/gifsicle/node_modules/bin-wrapper/node_modules/download/node_modules/request/node_modules/form-data/node_modules/combined-stream/node_modules/delayed-stream/lib" has 259 characters, close to the maximum of 260 supported by Windows (ref http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx#maxpath). If the lib directory at the end has anything in it, the 260 characters will be exceeded.

I haven't proven it but I think there's a problem even sooner, after 255 just characters, with Perl's File::Find module.

My recommendation is to cd into the "D:\ccnet\projects\Isogeo.App\ContinuousBuild\work" then pass cloc the current directory, ".", and see if that shortens the path sufficiently.

Finally, regarding the ignored directory, the ignore step happens after the directory name is determined. In other words, the code can't ignore a directory until it knows its full name, and the failure happens then.

[Mac]

I understand the path is way too long for Windows to handle, but in this case CLOC was instructed to exclude all paths containing node_modules so there is no logical need for it to go that deep in the folder hierarchy (the first occurence happens way before the 260 characters limit).

• how about allowing CLOC to use the "\?\" prefix (cf. http://msdn.microsoft.com/en-us/library/Aa365247)? It would require using "\" instead of "/" as a path delimiters though.

• maybe the folder analysis algorithm could be improved so that it would not have to analyse subfolders of an excluded one? This alone would solve my particular problem.

[Al Danial]

OK I understand your point. cloc actually already does (that is, should do) what you describe in your second bullet point: if it encounters the name of an excluded directory, it stops its traversal and continues to the next directory returned by File::Find.

The code in question (source at http://sourceforge.net/p/cloc/code/HEAD/tree/trunk/cloc) begins at line 3047:

3037 sub files {                                  # {{{1
3038     # invoked by File::Find's find()   Populates global variables,
3039     # @file_list, %Ignored
3040     if ($opt_exclude_dir or $opt_exclude_list_file) {
3041         my $return = 0;
3042         foreach my $skip_dir (keys %Exclude_Dir) {
3043             # File::Find::dir used to always start with / but
3044             # newer versions (1.13) no longer do; have to correct for this
3045             my $dir = $File::Find::dir;
3046                $dir = "./$dir" unless $dir =~ m{^/};
3047             if ($dir =~ m{/\Q$skip_dir\E(/|$)} ) {
3048                 $Ignored{$File::Find::name} = "--exclude-dir=$skip_dir";
3049                 $return = 1;
3050                 last;
3051             }
3052         }
3053         return if $return;
3054     }

One problem has to do how File::Find presents path separators. The regex at line 3047 will only skip the directory if it is enclosed by /'s. If on Windows File::Find returns the path with \ separators then I'll need to tweak the regex. (My dev machine is Linux but I'll have access to Windows later in the week to see what's going on there.)

Incidentally, if you run cloc with the verbose level at 6 or greater (cloc -v 6 ...), you can follow exactly which directory is being investigated; look for lines that begin with "files(". Excluded directories shouldn't appear. Also helpful in this case is running with "--ignored=ignored.txt". The resulting ignored.txt file should contain file names which were in directories that were excluded.

If you can tell me whether or not excluded directories appear with a -v 6 run that would be helpful. If they do appear, the issue is with the path separators. If they don't appear, the issue is with File::Find and I'll need to investigate ways to prune the search tree (there is an option File::Find::prune) to terminate unwanted descents.

[Mac]

The topmost folder that should be excluded appears in the output (files(D:\ccnet\projects\Isogeo.App\ContinuousBuild\work/node_modules) zero size). This is the last line that appears in the output before the error above occurs on a subfolder.

The ignored files does not seem to be generated. Must be because of the exception?

[Al Danial]

Right, the ignored file isn't written until the (successful) end of the run.

In any case, I now know how to fix the problem once and for all. Got a good tip from http://stackoverflow.com/questions/12334037/perl-how-to-stop-filefind-entering-directory-recursively on calling a preprocessor with File::Find to prune directories before descending into them. I'll implement the fix this weekend, in the mean time will reopen this as a bug.

[Al Danial]

fixed with svn commit 380

[Al Danial]

v1.62 released