Hi,
by mistake firstly I reported the issue to GCC:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97215
Regarding a C source loading an entire 3.3GB file and checksumming it.
First, I use Intel v15.0 and GCC v7.3.0, on Windows 64bit.
For my dismay I encountered that Intel's binary loads and reports the correct checksum, whereas GCC's binary fails, after comparing the loaded content I saw that GCC loads all the file into a malloc-ed pool but without the last ~860 bytes?!
If you need to reproduce the issue - the two binaries (GCC and Intel) and the C source as well are here:
www.sanmayce.com/Nakamichi/Satanichi_aka_Nakamichi_2020-Jun-09_BUG_ZEROED-END.zip
The file being loaded is the Human Genome:
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.28_GRCh38.p13/GCA_000001405.28_GRCh38.p13_genomic.fna.gz
This bug never appeared with files 1GB or less in size, my guess, this is a clue.
These are the files:
06/11/2020 09:16 AM 1,316,439 Nakamichi_Ryuugan-ditto-1TB_btree.c 06/15/2019 02:37 AM 3,313,087,324 NCBI_FTP_Homo_sapiens_(human)_GCA_000001405.28_GRCh38.p13_genomic.fna 06/15/2019 02:37 AM 3,313,087,324 q 01/07/2018 05:26 PM 191,644 Satanichi_GCC730_64bit.exe 06/11/2020 09:16 AM 198,144 Satanichi_ICL150_64bit.exe
As you can see below, the same file is loaded differently into malloc-ed pool:
D:\Satanichi_aka_Nakamichi_2020-Jun-09>Satanichi_GCC730_64bit.exe q w 20 888 i ... Allocating Source-Buffer 3,159 MB ... Allocating Target-Buffer 3,191 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xc1d4,3f7f ... D:\Satanichi_aka_Nakamichi_2020-Jun-09> D:\Satanichi_aka_Nakamichi_2020-Jun-09>Satanichi_ICL150_64bit.exe q w 20 888 i Allocating Source-Buffer 3,159 MB ... Allocating Target-Buffer 3,191 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x81bd,fe4b ... D:\Satanichi_aka_Nakamichi_2020-Jun-09>
If you need more info, will add it...
Very much I would like to know what causes this anomaly/bug.
Georgi
Hello,
I am pretty sure that our fread/fwrite routines work pretty well. The
underlying issue here seems to be the missing use of "binary"-mode on
the fopen-call. Not sure what Intel's runtime assumes as default mode
on open for fopen, but MS' variant assumes text-mode, if no "b" is
within the mode-option.
Cheers,
Kai
Am So., 27. Sept. 2020 um 04:59 Uhr schrieb Sanmayce
sanmayce@users.sourceforge.net:
Related
Bugs: #855
The same said the GCC guy in the link above:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97215#c4
(In reply to Andrew Pinski from comment #2)
At 11,945th line I use:
1] As far as I investigated, the problem is that fread() reads less (around 860 bytes) than specified.
2] Also, as I wrote:
These two facts suggest the problem is not a mere binary mode issue, something weird happens...
Last edit: Sanmayce 2020-09-28
For what is worth the OS is Windows 7 Pro, also the quick checking of a textual file shows the GCC binary works "PROPERLY":
Kai, I'm using MinGW packages that have certainly that problem, are you 100% sure yours is working with e.g. 2+GB in one go?!
Still cannot figure out where the problem is, added the return value:
According to https://man7.org/linux/man-pages/man3/fread.3.html the return value is zero if error occurred.
In fact, it reads into the pool 3,313,087,324 - (~860) bytes?!
Tried the latest MinGW package from here: https://nuwen.net/mingw.html
It behaves the same way as the GCC730 one:
The files:
Again, working with 2-GB file (with lots of \r\n):
Don't know who else to ask, anyone?!
Hello,
fread actually returns a size_t. So maximal size of read per attempt
is on 32-bit 2^32-1, and on 64-bit 2^64-1 bytes.
If you want to process bigger data, I would recommend to operate here
in junks. The maximal read bytes by the streaming API depends on the
used open() variant for the file-descriptor. That is indeed a bit
annoying. So we provide here an alternative for fopen(). You could
try to replace fopen-call by fopen64()-call instead. See also there is
the fsetpos64, fgetpos64, _fseeki64, _ftelli64. ftello64, fseeko64,
ftello64, ... API, which can be pretty helpful to get proper support
for files >=2GB on Windows.
Hope this helps
Kai
Am Fr., 2. Okt. 2020 um 06:47 Uhr schrieb Sanmayce
sanmayce@users.sourceforge.net:
Related
Bugs: #855
Thanks for the fopen64 workaround, but the issue remains unaddressed!
Obviously, not in practice, so in order to avoid problems documentations should state this issue.