Menu
▾
▴
#1021 SysFileTree usage Access Violation
Under Windows XP Professional, SP1 32bit w/ ooRexx 4.1.0, when I try to use the results of SysFileTree of my C: Drive (64k+entries, avg entry length 102 bytes) an access violation occurs or I get "date" function argument errors randomly. Sometimes the .rex program even works as expected. When the errors occur is unpredictable as well as when it will run successflly. Frustrating! Tried to put in stops like Trace ?r or exit after certain points. Never have been able to catch what is causing the problem.
The function is rc = SysFileTree("C:*","f","BLS","*")
After SysFileTree is executed, the RC is 0 or the .rex program will exit displaying a message and the return code.
The .rex program goes on to examine the results of SysFileTree and select entries based on user command line selected search criteria. At random points in the seach loop I get either of the above errors. Have included a zip file containing two Dr Watson logs and a sample of the random "date" function error.
Regressed to ooRexx to 4.0.0 and still had the above problems.
Regressed to ooRexx to 3.2.0 and I no longer had any problems with ooRexx. Finished writing the code under 3.2.0 and it works as expected. After I finished writing the code I restored my Windows XP System back to the ooRexx 4.1.0 version and the problems are reoccuring again.
I need to move to a new computer and operating system (Windows 7 Home Premium 64 bit) and wanted to use the latest ooRexx 64 bit programs. I am writing .rex programs to replace 16 bit programs that no longer function on Windows 7. I've developed replacement programs on Windows XP and hope to port them to Windows 7. But, now I am concerned about ooRexx reliability on Windows 7.
Committed revision 8175. 4.1 fixes branch.
Well Mark, I believe I may have run into the major cause of the SysFileTree problems. The fix you gave me to test in July covers some random occurences of SysFileTree failures and needs to be included in ooRexx. But I didn't understand the real problem until recently when I changed ownership of everything on my C: drive as part of a test for another program I am writing. Using an application I found on the web, my directory count went from 23000+ to 54863. Windows 7 was hiding over 30000 directories as part of it's NTFS Authorization and Securityscheme. This was impemented beginning in Windows Vista (according to MSDN.com).
I reinstalled the test fix and I got some interesting information:
[ooRexx]test4b
Collecting SysFileTreeB information
Program: C:\ooRexx\test4B.rex, line 7
Error RC(98) Execution error
in recursiveFindFileB() at line: 2805 buffer too small; _snprintf rc:
ret data size: 260 buffer size: 261
[ooRexx]
Running another instance of test4b:
[ooRexx]test4b
Collecting SysFileTreeB information
Program: C:\ooRexx\test4B.rex, line 7
Error RC(98) Execution error
in recursiveFindFileB() at line: 2805 buffer too small; _snprintf rc:
ret data size: -1 buffer size: 265
[ooRexx]
The first occurence data size is "_MAX_PATH" size. The second occurrence indicates to me that the variable for the data size may not be defined properly (-1)?.
If oorexx is using "_MAX_PATH" as the determining factor of how big a buffer to get and the data to go into that buffer exceeds 259 bytes + any prepended data such as provided by "L" option of SysFileTree, ooRexx may overlay other items in the variable pool or system. I have 1561 directories that exceed _MAX_PATH, the longest being 326 bytes. My longest filename including .ext 167 bytes. Adding the 2 together you get 493 bytes. There are several articles at MSDN.com that discuss problems associated with _MAX_PATH and buffer overrun. The net is that a buffer to contain full(extended) path information should be 32767 bytes with a check that insures that the data won't exceed that buffer. This size is the maximum size for extended paths.
If you are able to come up with a fix I would be happy to test it because I can easily generate the problem at will. I would not recommend the program I used to unhide my entire directory structure to any production level system. It does expose everything to "everyone". But, since I keep full disk backups of my system I can easily restore my system after the test.
Hope this helps.
With all the other problems I've been having with SysFileTree the following escaped me. Look at the listings for my root directory on Windows 7 SP1 64bit, oorexx 4.1.0. Look at the size information for pagefile.sys and hiberfil.sys. Have attached the console log. This comment editor is even worse than the previous one.
The numbers produced by SysFileTree aren't even close. The root.rex program is a simple SysFileTree followed by a do loop to say the stem variables.
Hi Jerry,
SysFileTree() is limited to an unsigned 32-bit number for file size. So this:
2012-11-24 18:28:43 4276559872 A-H-S C:\pagefile.sys
11/24/2012 06:28 PM 12,866,494,464 pagefile.sys
is a permanent restriction, not a bug.
As far as MAX_PATh goes:
In the Windows API, the maximum length for a path is MAX_PATH, which is defined as 260 characters.
You wrote:
"The net is that a buffer to contain full(extended) path information should be 32767 bytes with a check that insures that the data won't exceed that buffer. This size is the maximum size for extended paths."
This is not completely correct. You can only have those longer path or file names when using the Unicode versions of the file management functions. ooRexx is compiled as an ANSI program and therefore the Unicode versions of the functions are not available. It is possible to create path and file names by using one application, that are not available to other applications.
SysFileTree is restricted to a maximum length for a path as the maximum for the ANSI versions of the file management functions.
You have this:
"Collecting SysFileTreeB information
Program: C:\ooRexx\test4B.rex, line 7
Error RC(98) Execution error
in recursiveFindFileB() at line: 2805 buffer too small; _snprintf rc:
ret data size: -1 buffer size: 265"
In the ooRexx version I gave you, that was the first approach I took, raising a condition if the buffer was going to be too small. Wiser heads than myself convinced me to not raise a condition but rather to deal internally with the too small buffer.
That is what the final version of the fix does. If a too small buffer is encountered a dynamic buffer is allocated that is big enough. I'll get you a build with the final version so you can test it.
The -1 you see is the return from _snpritf which merely indicates that not all of the characters could be stored in the buffer. In no case can there be a buffer overrun. What can happen is that the string will not be null-terminated. The raised condition prevented the non-terminated string from being used by the interpreter.
Before the changes made to SysFileTree, buffer overruns were possible and incidents like you reported in your recent post were what caused your original access violations and other oddities. I'll get you the latest build which should not raise a condition like you have seen recently.
But, again, from MSDN:
"Windows Vista NTFS file system supports paths up to 32768 characters in length, but only when using the Unicode APIs. When using long path names, prefix the path with the characters \?\ and use the Unicode versions of the C Runtime functions."
This is not possible in ooRexx because it does not use the Unicode versions of the C runtime functions.
This bug is fixed in the ooRexx 4.1.3 release.