Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#146 Problems with PKFile larger than 2GB

open
nobody
5
2008-03-10
2008-03-10
No

It seems that the next implementation limit in the cache daemon code that we'll need to address is PKFile size. We recently had one cache at Intel repeatedly die with this message:

Exception while re-writing MPKFile /var/lib/vesta/cache/sCache/gran-16/60/26

************************************************************
FS::Failure:
opName = FS::Seek(std::istream) [istream::seekg]
arg(s) = offset = -2147475723, orig = 0
errno = 22 (Invalid argument)
************************************************************
VCache: CacheS.C:1745: void CacheS::VToSCache(const PKPrefix::T&, const BitVector*): Assertion `false' failed.

We haven't yet reproduced the problem in isolation so we don't know exactly where this incorrect seek is originating. However we do know that the MPKFile in question had multiple PKFiles, one of which was over 2GB. Specifically, here's a snipper from PrintMPKFile showing the sizes and file offsets of the PKFiles:

// <HeaderEntry>
pk = 602601f7fb887f32 746ed74a305bab19
offset = 118
offsetLoc = 34
pkLen = 2149914209

// <HeaderEntry>
pk = 6026934e3886ff17 154ae25770bd902b
offset = 2149914327
offsetLoc = 54
pkLen = 14443

// <HeaderEntry>
pk = 6026bf26ddacb8d8 34371565d62b66cc
offset = 2149928770
offsetLoc = 74
pkLen = 68952

// <HeaderEntry>
pk = 602680f1b8b01629 3f70fad85645ba9c
offset = 2149997722
offsetLoc = 94
pkLen = 10638

// <HeaderEntry>
pk = 6026bb4cf78a139f b081d4f716c1f120
offset = 2150008360
offsetLoc = 114
pkLen = 234

I think this particular problem is pretty clearly a signed/unsigned problem, as all of these offsets and lengths fit in a 32-bit unsigned integer. However, we should really perform a careful review of the cache code to make sure we're using 64-bit integers for all file offsets and lengths. Even a cursory look reveals the use of 32-bit integers in SMultiPKFileRep::HeaderEntry.

Discussion