Menu

#230 overlapStoreBuild memory estimate wrong with big gzip files

overlapper
open
5
2012-12-20
2012-12-20
No

gzip returns an incorrect uncompressed file size for files larger than 2gb.

> ls -l
-rwxrwxrwx 1 xxx yyy 14478956391 Dec 19 14:51 000001.ovb.gz

> gzip -l 000001.ovb.gz
compressed uncompressed ratio uncompressed_name
14478956391 272803424 -5207.5% 000001.ovb

overlapStoreBuild uses this information to estimate the number of overlaps it can fit in a bucket. In these cases it will overestimate. In this particular case, it tried to put 60gb of uncompressed overlaps into 1gb memory.

Discussion


Log in to post a comment.