#827 Improve stream reading speed on Windows

[Per Olov Jonsson]

This request relate to the slow I/O on Windows in 5.0.0 (and 5.1.0) compared to 4.2.0

I hesitate to call this a bug hence I file this as a feature request. It would be desirable to have a faster handling of stream methods arrayIn and lineIn on Windows. As can be seen in the attached overview reading files on Windows is 2-3 times slower on 5.0.0 than in 4.2. Also they are 5-10 times slower on Windows than on macOS and Linux.

I have enclosed 5 testcases and the results from reading a huge file (7.4 million lines), I have attached a small part of it, duplicate until a reasonable size for testing, the original file is to large for including.

I have enclosed also all the individual testruns in a zipfile.

[Anonymous]

From Jochem Peelen:
The first sentence of the description in my view could create the impression that I/O of 5.0.0. is generally slower than 4.2.0 under Windows 10. This is not my experience. The "old fashioned" program (similar to your readtest3.rex):

 infile = 'normal.xy'
 do i=1 while lines(infile)
      parse value linein(infile) with xkoo.i ykoo.i
 end

under 64 bit 4.2.0 rev 9940 ran on average for 72 seconds to process 3 million lines.
After installing 64 bit 5.0.0 rev 12584, runtime shrank to 34 seconds. (Lenovo E595, 2.1 GHz AMD processor, 1TB SCSI disk; Win 10 Pro 22H2).
In other words, the speed of a "legacy" reading loop is doubled just by going from 4.2.0 to 5.0.0 under Windows 10. It was speeded up further to just 6 (six) seconds runtime by rewriting based on Jon Wolfers' proposal:

 infile = .stream~new('normal.xy')
 inarray = infile~charIn(1,infile~chars)~makearray
 do i=1 for inarray~items
      parse value inarray[i] with xkoo.i ykoo.i
end

From my perspective, 5.0.0 I/O is quite impressive. But doubtless, your systematic testing turned up some unexpected ooRexx behaviour that should be investigated.