#1318 SysFileTree wrong for files > 4096 MB

[AvdP]

My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

• stream(bigfile, 'c', 'query size')
• sysfiletree()

The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)


Andre

[Erich]

Doing a scan of more than 400,000 files on my system with attached TstBugSize.rex, and querying the file size of each file with ...

1. file~length
2. .Stream~new(file)~query('SIZE')
3. SysFileTree file~absolutePath,Size.

... I'm seeing a few files, where those three size values disagree. E. g.

\pagefile.sys sizeFile=0 sizeStream=8253640704 sizeTree=3958673408
\PGPWDE01 sizeFile=0 sizeStream=1048576 sizeTree=1048576

Surprisingly I'm also seeing a few files (from a commercial product install) with path lengths larger than 260 bytes (didn't know that this was possible). For these (non-empty) files

• file~length reports 0
• .Stream~new(file)~query('SIZE') reports the empty string
• SysFileTree file~absolutePath,Size. will fail as file~absolutePath reports the empty string

---

ooRexx 4.2.0 & 5.0.0
Win 7 64-Bit

[Bruce]

I suspect the the sizeFile=0 is an indication that a file is open, and therefore there is no exact file size until the file is closed. It does look like sizeTree has a 32bit variable, where a 64bit one should be used. It is also possible that sizeTree uses an outdated windows API. Note that for \pagefile.sys the sizeStream and sizeTree differ by exactly 4GB (102410241024*4 or 4,294,967,296 bytes).

[AvdP]

FYI, this is the fix that works for me:

In \extensions\rexxutil\platform\windows\rexxutil.cpp in function 'formatFile'

In lines [1932], [1944] and [1956] change

wfd->nFileSizeLow);

into

(((int64_t)wfd->nFileSizeHigh) << 32) + (int64_t)wfd->nFileSizeLow);

In lines [1925], [1938] and [1948] change formatting parameter

%10lu

into

%10llu

[LesK]

I notice that some of the numbers are longer than the default Numeric Digits.

[AvdP]

Les,
Note that the first statement in the attached test.rex that produced this output is:
Numeric Digits 15

It is no longer a problem for me because I now build my own ooRexx interpreter that includes the fix documented above.

Andre

[Anonymous]

Andre,

how about a solution that could become part of an official ooRexx distribution?

Something like a new switch?

Discussions are probably best on the developer list.

Cheers

---rony

Rony G. Flatscher (mobil/e)

Am 16.08.2015 um 00:41 schrieb AvdP avanderputten@users.sf.net:

Les,
Note that the first statement in the attached test.rex that produced this output is:
Numeric Digits 15

It is no longer a problem for me because I now build my own ooRexx interpreter that includes the fix documented above.

Andre

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: open
Group: None
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sat Aug 15, 2015 10:03 PM UTC
Owner: nobody
Attachments:

test.rex (1.5 kB; application/octet-stream)
My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

stream(bigfile, 'c', 'query size')
sysfiletree()
The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)

Andre

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/oorexx/bugs/1318/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[LesK]

Rony: By a 'new switch' do you mean a new option for SysFileTree? Perhaps 'G' for Giant files?

[Anonymous]

Les, yes, something like that. It is clear that SysFileTree is broken and needs a fix one way or another. ---rony

Rony G. Flatscher (mobil/e)

Am 16.08.2015 um 06:22 schrieb LesK vmrexx@users.sf.net:

Rony: By a 'new switch' do you mean a new option for SysFileTree? Perhaps 'G' for Giant files?

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: open
Group: None
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sat Aug 15, 2015 10:41 PM UTC
Owner: nobody
Attachments:

test.rex (1.5 kB; application/octet-stream)
My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

stream(bigfile, 'c', 'query size')
sysfiletree()
The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)

Andre

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/oorexx/bugs/1318/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[AvdP]

Rony,
I too think this fix should go into the official ooRexx distribution, but:
- this can only be done by someone with proper authorization (not me)
- two of our best ooRexx experts stated fixing this bug is not a good idea
(see https://sourceforge.net/p/oorexx/discussion/408478/thread/062cb660/#482a)

[Gil Barmwater]

With all due respect to Rick and Jon, adding a new switch/flag/option will NOT break existing programs and will allow users who need to process very large files to use SysFileTree. My opinion, YMMV.

[Rick McGuire]

I never said this should not be fixed, I was just against that particular
fix, where the field would just get expanded unconditionally. My real
preference would be for people to not use SysFileTree at all for that sort
of stuff and use the .File class for retrieving file information. A new
option flag on SysFileTree would work also, although this should be
addressed for both Windows and *ix systems both.

Rick

On Sun, Aug 16, 2015 at 10:02 AM, Gil Barmwater orange-e@users.sf.net
wrote:

With all due respect to Rick and Jon, adding a new switch/flag/option will
NOT break existing programs and will allow users who need to process very
large files to use SysFileTree. My opinion, YMMV.

---

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: open
Group: None
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sun Aug 16, 2015 12:30 PM UTC
Owner: nobody
Attachments:

• test.rex (1.5 kB;
  application/octet-stream)

My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

• stream(bigfile, 'c', 'query size')
• sysfiletree()

The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)


Andre

---

Sent from sourceforge.net because you indicated interest in <
https://sourceforge.net/p/oorexx/bugs/1318/>

To unsubscribe from further messages, please visit <
https://sourceforge.net/auth/subscriptions/>

[Gil Barmwater]

I propose the following resolution for this issue. 1) withdraw/close this bug ticket, 2) open a documentation bug ticket to add a warning that results for file size are incorrect if >= 4096MB, 3) open an RFE for a new option on SysFileTree, perhaps G for Gigabyte size files.

[Rick McGuire]

I don't see that any of that is necessary. The fix for this bug report can
be done by adding the new option.

On Sun, Aug 16, 2015 at 11:07 AM, Gil Barmwater orange-e@users.sf.net
wrote:

I propose the following resolution for this issue. 1) withdraw/close this
bug ticket, 2) open a documentation bug ticket to add a warning that
results for file size are incorrect if >= 4096MB, 3) open an RFE for a new
option on SysFileTree, perhaps G for Gigabyte size files.

---

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: open
Group: None
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sun Aug 16, 2015 02:02 PM UTC
Owner: nobody
Attachments:

• test.rex (1.5 kB;
  application/octet-stream)

My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

• stream(bigfile, 'c', 'query size')
• sysfiletree()

The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)


Andre

---

Sent from sourceforge.net because you indicated interest in <
https://sourceforge.net/p/oorexx/bugs/1318/>

To unsubscribe from further messages, please visit <
https://sourceforge.net/auth/subscriptions/>

[Jon Wolfers]

Sorry I wrote so briefly.

I have no objection to adding a new switch, I notice that Regina has a new switch 'H' (for huge) in v3.9.1 which seems to be for this circumstance, so would suggest we maintain compatibility.

Mark Hessling wrote (of Regina)

... added a new option: H for specifying reporting of 'huge' files. If you need to parse the output of a directory and have files with a file size greater than 10 significant digits, the layout for all files will now be consistent (to 16 significant digits).

It would be good if our output with the 'H' switch then mirrored Regina.
The ooRexx change would need to be done for Unix and Windows at the same time

Jon

[AvdP]

Regardless how you want to format SysFileTree output, we must first make sure that our (Windows) ooRexx code uses the correct file size internally.

Now it ignores the higher double-word (nFileSizeHigh) that the Windows API reports back to ooRexx, so we don't even know the correct size for files > 4096MB.

Look back in this thread for the line of code that corrects this.

I agree that next we must decide how to format filesizes bigger than 10 positions (9999999999 ~ 9536MB).

FYI, right now, the C formatting is '%10lu' both in our Unix and Windows code.

[Rick McGuire]

The new version in 5.0.0 trunk supports a new option "H"uge which will use a longer field size for the length and support sizes up to the full 64-bit size.

[AvdP]

This has fixed only part of what I reported.

SysFileTree (when not using the new 'H' option) has 10 positions to display the filesize.
So theoretically all filesizes up to 9 999 999 999 bytes (a little over 9536,74 MB) have enough positions to show correctly.

But SysFileTree in r11795 (when used without the 'H' option) continues to show a wrong number for all filesizes between 4 294 967 296 (=4096MB) and 9 999 999 999 bytes.

Any plans to fix that?

Andre

[Rick McGuire]

And it will continue to do so for compatibility reasons. If you wish the
full size, you need to use the H option.

Rick

On Sun, Feb 24, 2019 at 10:29 AM AvdP avanderputten@users.sourceforge.net
wrote:

This has fixed only part of what I reported.

SysFileTree (when not using the new 'H' option) has 10 positions to
display the filesize.
So theoretically all filesizes up to 9 999 999 999 bytes (a little over
9536,74 MB) have enough positions to show correctly.

But SysFileTree in r11795 (when used without the 'H' option) continues to
show a wrong number for all filesizes between 4 294 967 296 (=4096MB) and 9
999 999 999 bytes.

Any plans to fix that?

Andre

---

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: accepted
Group: 5.0.0
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sat Feb 16, 2019 09:42 PM UTC
Owner: nobody
Attachments:

• test.rex (1.5 kB;
  application/octet-stream)

My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

• stream(bigfile, 'c', 'query size')
• sysfiletree()

The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)


Andre

---

Sent from sourceforge.net because you indicated interest in <
https://sourceforge.net/p/oorexx/bugs/1318/>

To unsubscribe from further messages, please visit <
https://sourceforge.net/auth/subscriptions/>

[AvdP]

I fail to see why fixing the wrong filesizes between 4294967296 and 9999999999 bytes would introduce incompatibility.
I do understand for larger sizes that need more than the available 10 positions, but not for this specific range of numbers.

[Rick McGuire]

Ah, ok, I misunderstood what you were saying. Yes, this probably should be
fixed.

Rick

On Sun, Feb 24, 2019 at 10:41 AM AvdP avanderputten@users.sourceforge.net
wrote:

I fail to see why fixing the wrong filesizes between 4294967296 and
9999999999 bytes would introduce incompatibility.
I do understand for larger sizes that need more than the available 10
positions, but not for this specific range of numbers.

---

[bugs:#1318] SysFileTree wrong for files > 4096 MB

Status: accepted
Group: 5.0.0
Created: Tue Dec 02, 2014 03:30 PM UTC by AvdP
Last Updated: Sun Feb 24, 2019 03:29 PM UTC
Owner: nobody
Attachments:

• test.rex (1.5 kB;
  application/octet-stream)

My Rexx: ooRexx.5.0.0.10860-x86_32.exe
My OS: Win7 32bit

Attached script 'test.rex' grows a file (in 1MB steps).

It then uses 2 methods of getting the file size:

• stream(bigfile, 'c', 'query size')
• sysfiletree()

The two size methods give identical results.
Until the file size reaches 4096MB when sysfiletree() restarts at '0'.

When you run this code:
- make sure the filesystem can handle files > 4GB
- consider disabling your anti-virus for better performance
- a logfile is created in the same folder

Bottom lines from my log look similar to this:

4092 MB = 4290772992 (by query size) / 4290772992 (by sysfiletree)
4093 MB = 4291821568 (by query size) / 4291821568 (by sysfiletree)
4094 MB = 4292870144 (by query size) / 4292870144 (by sysfiletree)
4095 MB = 4293918720 (by query size) / 4293918720 (by sysfiletree)
4096 MB = 4294967296 (by query size) / 0 (by sysfiletree)
4097 MB = 4296015872 (by query size) / 1048576 (by sysfiletree)
4098 MB = 4297064448 (by query size) / 2097152 (by sysfiletree)
4099 MB = 4298113024 (by query size) / 3145728 (by sysfiletree)
4100 MB = 4299161600 (by query size) / 4194304 (by sysfiletree)


Andre

---

Sent from sourceforge.net because you indicated interest in <
https://sourceforge.net/p/oorexx/bugs/1318/>

To unsubscribe from further messages, please visit <
https://sourceforge.net/auth/subscriptions/>

[Rick McGuire]

Additional fix committed [r1180]

[Erich]

There's a column shift issue with large file sizes.

2/24/19   7:49p      100000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p     1000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p    10000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p   100000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p  1000000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p  10000000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p  100000000000 -rw-rw-r--  /home/erichst/large
 2/24/19   7:49p  1000000000000 -rw-rw-r--  /home/erichst/large

Committed revision [r11806] to fix.

 2/24/19   8:03p      100000 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p     1000000 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p    10000000 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p   100000000 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p  1000000000 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p  9999999999 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p  9999999999 -rw-rw-r--  /home/erichst/large
 2/24/19   8:03p  9999999999 -rw-rw-r--  /home/erichst/large

[Erich]

4.0 and 5.0 column layout is slightly different

4.0
11/26/15   1:01p        1025  -rwxrwxr-x  /home/erichst/Tst.rex
5.0
12/05/18   8:07p        1884 -rwxrwxr-x  /home/erichst/Tst.rex

[Erich]

Documented new SysFileTree H option with revision [r11808]