My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit
Attached script 'test.rex' grows a file (in 1MB steps).
It then uses 2 methods of getting the file size:
The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.
When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder
Bottom lines from my log look similar to this:
4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)
Andre
Anonymous
Doing a scan of more than 400,000 files on my system with attached TstBugSize.rex, and querying the file size of each file with ...
file~length
.Stream~new(file)~query('SIZE')
SysFileTree file~absolutePath,Size.
... I'm seeing a few files, where those three size values disagree. E. g.
\pagefile.sys sizeFile=0 sizeStream=8253640704 sizeTree=3958673408
\PGPWDE01 sizeFile=0 sizeStream=1048576 sizeTree=1048576
Surprisingly I'm also seeing a few files (from a commercial product install) with path lengths larger than 260 bytes (didn't know that this was possible). For these (non-empty) files
file~length
reports 0.Stream~new(file)~query('SIZE')
reports the empty stringSysFileTree file~absolutePath,Size.
will fail asfile~absolutePath
reports the empty stringooRexx 4.2.0 & 5.0.0
Win 7 64-Bit
I suspect the the sizeFile=0 is an indication that a file is open, and therefore there is no exact file size until the file is closed. It does look like sizeTree has a 32bit variable, where a 64bit one should be used. It is also possible that sizeTree uses an outdated windows API. Note that for \pagefile.sys the sizeStream and sizeTree differ by exactly 4GB (102410241024*4 or 4,294,967,296 bytes).
FYI, this is the fix that works for me:
In \extensions\rexxutil\platform\windows\rexxutil.cpp in function 'formatFile'
In lines [1932], [1944] and [1956] change
into
In lines [1925], [1938] and [1948] change formatting parameter
%10lu
into
%10llu
I notice that some of the numbers are longer than the default Numeric Digits.
Les,
Note that the first statement in the attached test.rex that produced this output is:
Numeric Digits 15
It is no longer a problem for me because I now build my own ooRexx interpreter that includes the fix documented above.
Andre
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
Andre,
how about a solution that could become part of an official ooRexx distribution?
Something like a new switch?
Discussions are probably best on the developer list.
Cheers
---rony
Rony G. Flatscher (mobil/e)
Related
Bugs:
#1318Rony: By a 'new switch' do you mean a new option for SysFileTree? Perhaps 'G' for Giant files?
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
Les, yes, something like that. It is clear that SysFileTree is broken and needs a fix one way or another. ---rony
Rony G. Flatscher (mobil/e)
Related
Bugs:
#1318Rony,
I too think this fix should go into the official ooRexx distribution, but:
- this can only be done by someone with proper authorization (not me)
- two of our best ooRexx experts stated fixing this bug is not a good idea
(see https://sourceforge.net/p/oorexx/discussion/408478/thread/062cb660/#482a)
With all due respect to Rick and Jon, adding a new switch/flag/option will NOT break existing programs and will allow users who need to process very large files to use SysFileTree. My opinion, YMMV.
I never said this should not be fixed, I was just against that particular
fix, where the field would just get expanded unconditionally. My real
preference would be for people to not use SysFileTree at all for that sort
of stuff and use the .File class for retrieving file information. A new
option flag on SysFileTree would work also, although this should be
addressed for both Windows and *ix systems both.
Rick
On Sun, Aug 16, 2015 at 10:02 AM, Gil Barmwater orange-e@users.sf.net
wrote:
I propose the following resolution for this issue. 1) withdraw/close this bug ticket, 2) open a documentation bug ticket to add a warning that results for file size are incorrect if >= 4096MB, 3) open an RFE for a new option on SysFileTree, perhaps G for Gigabyte size files.
I don't see that any of that is necessary. The fix for this bug report can
be done by adding the new option.
On Sun, Aug 16, 2015 at 11:07 AM, Gil Barmwater orange-e@users.sf.net
wrote:
Sorry I wrote so briefly.
I have no objection to adding a new switch, I notice that Regina has a new switch 'H' (for huge) in v3.9.1 which seems to be for this circumstance, so would suggest we maintain compatibility.
Mark Hessling wrote (of Regina)
It would be good if our output with the 'H' switch then mirrored Regina.
The ooRexx change would need to be done for Unix and Windows at the same time
Jon
Regardless how you want to format SysFileTree output, we must first make sure that our (Windows) ooRexx code uses the correct file size internally.
Now it ignores the higher double-word (nFileSizeHigh) that the Windows API reports back to ooRexx, so we don't even know the correct size for files > 4096MB.
Look back in this thread for the line of code that corrects this.
I agree that next we must decide how to format filesizes bigger than 10 positions (9999999999 ~ 9536MB).
FYI, right now, the C formatting is '%10lu' both in our Unix and Windows code.
The new version in 5.0.0 trunk supports a new option "H"uge which will use a longer field size for the length and support sizes up to the full 64-bit size.
This has fixed only part of what I reported.
SysFileTree (when not using the new 'H' option) has 10 positions to display the filesize.
So theoretically all filesizes up to 9 999 999 999 bytes (a little over 9536,74 MB) have enough positions to show correctly.
But SysFileTree in r11795 (when used without the 'H' option) continues to show a wrong number for all filesizes between 4 294 967 296 (=4096MB) and 9 999 999 999 bytes.
Any plans to fix that?
Andre
And it will continue to do so for compatibility reasons. If you wish the
full size, you need to use the H option.
Rick
On Sun, Feb 24, 2019 at 10:29 AM AvdP avanderputten@users.sourceforge.net
wrote:
Related
Bugs:
#1318I fail to see why fixing the wrong filesizes between 4294967296 and 9999999999 bytes would introduce incompatibility.
I do understand for larger sizes that need more than the available 10 positions, but not for this specific range of numbers.
Ah, ok, I misunderstood what you were saying. Yes, this probably should be
fixed.
Rick
On Sun, Feb 24, 2019 at 10:41 AM AvdP avanderputten@users.sourceforge.net
wrote:
Related
Bugs:
#1318Additional fix committed [r1180]
Related
Commit: [r1180]
There's a column shift issue with large file sizes.
Committed revision [r11806] to fix.
Related
Commit: [r11806]
4.0 and 5.0 column layout is slightly different
Documented new SysFileTree H option with revision [r11808]