I'm not a fan of adding a TFileOffsetPtr type and the same goes for the FSEEK define. All versions of genpat should support large files and to accomplish that we would have to stick with int32 seeks.
I've had a quick look, and one thing I've noticed is that GenPat needs the zlib library for the adler32() function.
I've had a go at making an absolute version of adler32(), see attachment.
The function prototype in the header has to be changed to size_t, other than that the function is self contained (I copy-pasted most of the code from zlib).
I'll also look at making a size_t version of md5_append() as well.
adler32/md5_append are not the main problems, they can be worked around like I did with adler32 in my patch and I prefer that to adding another implementation that we have to maintain.
The big problem is the fileoffset/seek and we really need to investigate what (if anything) is needed to get large file support working when you can only seek by int32. And is large file supported by the plugin?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I had a quick look, and looks like not right now. But it looks like SetFilePointer() will have to be swapped for SetFilePointerEx(). It appears that most of the reading and writing is based on the current position, so theoretically it can support large files.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks to me like there is one 64 bit seek that needs fixing but you don't need the Ex function to perform a 64 bit seek, just write a SetFilePointer64 helper function for the non-Ex version. This of course assumes that the patch file format itself supports large files and from the looks of it, it does not (The offset in the destination file is always 32 bits from the start of the file?).
Last edit: Anders 2014-10-14
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm considering somehow using an unused flag (if there is one) to indicate a large file, and putting a magic byte before or after each patch to make seeking easier using 32 bit seeks (lets say every 2GB and the magic number is the number of times to seek 2GB, make 0 invalid so 512GB - 4GB max).
So I take it backwards compatibility is a must?
The actual patch is split into blocks (16KB >=) and they are read and written in a loop, so a single patch can be bigger than 4GB. Highly unlikely, but possible.
The patches are automatically skipped (using the current position) if the patch doesn't match, so the actual position in the patch file isn't tracked by the plugin (which is a good thing).
Last edit: Jason 2014-10-15
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The number of patches in a patchfile is limited to "patches = patches & 0x00FFFFFF;"
The number of blocks in a patch is limited to UINT32.
bodySize (per patch?) is limited UINT32. This could probably be worked around in the generator by adding another patch when you get close to this limit.
The only real problem in the plugin is "SetFilePointer(hSource, temp, 0, FILE_BEGIN);" (writeDword(patch,current->sourceOffset);?) and it might be possible to use 0xffffffff as a magic value to signal that the offset is 64 bit and then store whatever is needed to go from 0xffffffff to the real offset in the next bytes?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That actually sounds like a better way of doing it. We could make it a fixed width (8 bytes split into high/low is logical), or a more dynamic way using a UTF-8 style approach, although almost everything else in the patch file is fixed width so it may not fit well for the intended purpose.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So I've done a little work (using your patch) and coded up a draft using 0xffffffff as a magic value.
One thing I noticed in GenPat is that TFileOffset appears to be 32 bit on 32 platforms and 64 bit on 64 platforms, which would mean 64 bit platforms would enjoy large file support where 32 bit platforms would suffer from the 2 GB limit (Linux x64 would be the first to benefit from this). The plugin .dll doesn't care, it's setup to handle both sizes.
Doesn't it defeat the purpose of adding large file support if it's only on x64?
I'm not a fan of adding a TFileOffsetPtr type and the same goes for the FSEEK define. All versions of genpat should support large files and to accomplish that we would have to stick with int32 seeks.
I've had a quick look, and one thing I've noticed is that GenPat needs the zlib library for the adler32() function.
I've had a go at making an absolute version of adler32(), see attachment.
The function prototype in the header has to be changed to size_t, other than that the function is self contained (I copy-pasted most of the code from zlib).
I'll also look at making a size_t version of md5_append() as well.
Last edit: Jason 2014-10-13
adler32/md5_append are not the main problems, they can be worked around like I did with adler32 in my patch and I prefer that to adding another implementation that we have to maintain.
The big problem is the fileoffset/seek and we really need to investigate what (if anything) is needed to get large file support working when you can only seek by int32. And is large file supported by the plugin?
I had a quick look, and looks like not right now. But it looks like SetFilePointer() will have to be swapped for SetFilePointerEx(). It appears that most of the reading and writing is based on the current position, so theoretically it can support large files.
It looks to me like there is one 64 bit seek that needs fixing but you don't need the Ex function to perform a 64 bit seek, just write a SetFilePointer64 helper function for the non-Ex version. This of course assumes that the patch file format itself supports large files and from the looks of it, it does not (The offset in the destination file is always 32 bits from the start of the file?).
Last edit: Anders 2014-10-14
Yep, that's what the offset looks like to me too.
I'm considering somehow using an unused flag (if there is one) to indicate a large file, and putting a magic byte before or after each patch to make seeking easier using 32 bit seeks (lets say every 2GB and the magic number is the number of times to seek 2GB, make 0 invalid so 512GB - 4GB max).
So I take it backwards compatibility is a must?
The actual patch is split into blocks (16KB >=) and they are read and written in a loop, so a single patch can be bigger than 4GB. Highly unlikely, but possible.
The patches are automatically skipped (using the current position) if the patch doesn't match, so the actual position in the patch file isn't tracked by the plugin (which is a good thing).
Last edit: Jason 2014-10-15
The number of patches in a patchfile is limited to "patches = patches & 0x00FFFFFF;"
The number of blocks in a patch is limited to UINT32.
bodySize (per patch?) is limited UINT32. This could probably be worked around in the generator by adding another patch when you get close to this limit.
The only real problem in the plugin is "SetFilePointer(hSource, temp, 0, FILE_BEGIN);" (writeDword(patch,current->sourceOffset);?) and it might be possible to use 0xffffffff as a magic value to signal that the offset is 64 bit and then store whatever is needed to go from 0xffffffff to the real offset in the next bytes?
That actually sounds like a better way of doing it. We could make it a fixed width (8 bytes split into high/low is logical), or a more dynamic way using a UTF-8 style approach, although almost everything else in the patch file is fixed width so it may not fit well for the intended purpose.
So I've done a little work (using your patch) and coded up a draft using 0xffffffff as a magic value.
One thing I noticed in GenPat is that TFileOffset appears to be 32 bit on 32 platforms and 64 bit on 64 platforms, which would mean 64 bit platforms would enjoy large file support where 32 bit platforms would suffer from the 2 GB limit (Linux x64 would be the first to benefit from this). The plugin .dll doesn't care, it's setup to handle both sizes.
Doesn't it defeat the purpose of adding large file support if it's only on x64?