Here are some changes I made to WillowSaveGame.cs. I am not one of the developers of
WillowTree# so these are purely unofficial modifications. This is a patched version of
WillowSaveGame.cs from WillowTree# beta 11. Feel free to use these changes if they are
useful to you. Just replace the WillowSaveGame.cs with this one and compile. The full
source code for WillowTree# is available at sourceforge.net. The changes.txt file is
purely informational and may be discarded.
The changes were confined to the reading (ReadWSG) and writing (SaveWSG) code starting at
Unknown9 and up through the end of echo list 2 to reflect the fact that there are not
always two echo lists in a save file. Attempting to read a second echo list when it was
not there would put the file pointer out of sync when it read the DLC data length and the
DLC bank size.
In my saves this would result in the bank size being corrupted any time a save with only
one playthrough was loaded then saved. It didn't appear to cause any other problems for
me but I can't say if this loss of sync may cause any other important data corruptions
in the DLC data or not.
There was a data element called SometimesNullsRevenge that was used in this section of code
that is no longer used but was not removed in case some external file references it. I
wanted to make sure the new WillowSaveGame.cs is completely plug-in compatible with the old
one in case someone uses it in conjunction with some software other than WillowTree#. I
also gleaned some information analyzing savegame files that could warrant renaming of a few
data items, but I didn't make any changes to the names at this time for the same reason.
The information:
unknownSaveValue – This is the total gameplay time elapsed, in seconds I believe.
unknown2 - this is the color of vehicle 1
unknown3 - this is the color of vehicle 2
unknown4 - this is the type of vehicle 1 (0 = rocket, 1 = machine gun)
unknown5 - this is the type of vehicle 2
unknown9 - this is the number of echo lists in the save file
I removed some ugly sanity checks on reading the echo list that appear unnecessary now
that the code knows how to tell that there is only one echo list.
I can be reached by email at matt911@users.sf.net if you have any direct questions or
comments about these code changes.
matt911 of sourceforge.net
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I feel kind of silly now, since I put up a newer WillowSaveGame.cs that already had some of this fixed. The thing is that I put it in a private forum. Either way, thanks for clearing up most of those unknowns! I've gone ahead and merged this info into my updated class, which you can find here: http://xanderchaos.com/WillowTree/WillowSaveGame.cs
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the link. I like the new design of WillowSaveGame.cs. It is much cleaner than the old code since you rewrote ReadString and WriteString instead of trying to individually account for the nulls in each spot that may encounter an empty string. I did the same thing in my own utility program to analyze the save files, but did not do it in the WillowSaveGame.cs modification since I wanted to make sure that someone that is already using WillowTree# beta 11 or a homemade utility
program that uses WillowSaveGame.cs can plug and play with no other modification. I made the minimum amount of changes to fix the problem with reading the DLC bank size.
I see a fault in your new version of the code where you removed the voodoo code that checked for an extra Int32 value that appears sometimes in the data stream before unknown 6, 7, 8, or 9. Failing to detect it will result in the echo list count at unknown9 being misread as zero so only one echo list will be read properly and who knows what will happen to the DLC data afterwards.
The extra int value seems very rare so it may be a hard problem to spot. I collected all my own PC save files and as many other save files for pc, ps3, and xbox360 as I could easily locate around the net when I analyzed the data and out of about 25 savegames I have only one that encounters this extra Int32. That save has the values {0, 1, 1, 0, 2} starting at unknown6 and ending at unknown9, which must be 1 or 2 in all cases so it is certainly not the fourth value and the echo list placement relative to it afterwards suggests that it must be the fifth value. Every other savegame I have has only four values in this range: {0, 0, 0, 1} for playthrough one saves or {0, 0, 0, 2} for playthrough two saves. I think the most likely scenario is that unknown7 is a flag or count for the optional exra value which would be written in the stream right after it appearing as the second 1 in {0, 1, 1, 0, 2}, but there are other possibilities.
In the original beta 11 code you accounted for the extra int32 value with SometimesNullsRevenge and wasNull when you realized the length of the echo list 1 is sometimes not where you expected because it was also shifted by that extra value inserted in the unknown6-unknown9 region. I basically did the same thing but moved the extra value check to Unknown9 since that must be read correctly to get the right number of echo lists so the file pointer is in the right spot to read the DLC data without corrupting it. As I said I suspect that Unknown7 is actually the cause of the extra value, but if it turns out that it is just a normal unrelated value variable then checking for the zero will be more reliable, so I stuck with that as in the original beta 11 code and if I detected zero at uknown9 I used wasNull to flag it and make sure I wrote an extra zero after unknown8 in SaveWSG to save it back the same way I read it. wasNull could be renamed to something more descriptive, but I think you will either need the null check or some kind of analysis of unknown7 or 8 to determine whether there's an extra Int32 in the stream to make sure the number of echos is read properly.
Here's a link for the only savegame I have with the extra value, but it is a PS3 save so you won't be able to test it yet with the new version since PS3 is big endian:
I also spotted a very minor technical detail that probably doesn't really matter, but I noticed that you set EchoPlaythroughs = 2 if FinishedPlaythrough1 = 1 in WriteWSG. You can finish playthrough 1 without starting playthrough 2 and you will have FinishedPlaythrough1 = 1 but only one echo list in your savegame. FinishedPlaythrough1 is set when you kill the creature at the final vault, but the echo list won't be added until you start in playthrough 2. This will create a second echo list even when one does not exist in the original file in some instances. That probably has zero practical effect on anything except that if you load then save the resulting savegame file wont be identical to the one you started with.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Very nice work da_fileserver! I can tell you are a big fan of ,NET, C#,, XML and object oriented technology in general. Your code is very thorough at enumerating every value in your classes as Properties, making extensive use of things like interfaces, class inheritance, derived classes, exception handling, and so on. I'm very much an old school C programmer who got stuck using modern day object oriented technologies unwillingly so I don't even fully understand how half this stuff works.
Your code contains so much object oriented stuff in fact that I haven't really figured out where the meat is in the code yet, but from what I did read through I think the extra int that occurs at Unknown7 or 8 in the original WillowSaveGame.cs code is a result of a collection of promo code activations. I look forward to examining your code more to figure out how these are handled.. Its lack of proper understanding of the file format in this area has made WillowSaveGame.cs unable to properly read all save files as evidenced by the loads of sanity checks needed to stop the program from crashing after unknown6. The DLC reading code is particularly ugly. I think it does mostly work but I can't even test it completely because it doesn't attempt to write files back the same way it read them. It moves items out of the DLC backpack into the regular character inventory area. This results in it being impossible to simply load a DLC savegame then save it back and use a file compare utility to make sure WillowTree# hasn't corrupted any data.
I don't personally understand why the game decides to use the DLC backpack area instead of the regular backpack area. It may be that it is due to a limitation on the number of items the regular backpack area can hold because of some fixed size arrays that are not large enough if a person has all the possible backpack upgrades available in the expansions. If that is the case then moving too many items from the DLC backpack to the regular inventory could cause crashes or unloadable savegame files. It could be that some DLC items have extra data that cannot be accounted for in the regular backpack, so again if you are moving items from it to the regular backpack you might achieve unwanted results. The whole policy of moving data this data routinely in WillowTree# is suspect.
Like you, I would like to see a savegame tool that can accurately save and edit without any corruptions. WillowTree# is quite close to this, but needs a few more fixes to WillowSaveGame.cs to achieve that and probably some changes to other WIllowTree# source files as well. XanderChaos recognizes this but I think he has tackled the problem at too high of a level by trying to completely rewrite WillowSaveGame in way that makes it incompatible with what he has elsewhere, causing him to have to make too many changes to handle easily. It likely would have been better to do this in smaller revisions than try to do it all at once. Making everyone wait for this to happen is leaving the whole project stalled in the meantime.
Beyond the basic functioning there are some ways that WillowTree#'s usability could be improved somewhat through an improved interface with context menus, keyboard support, drag-and-drop inventory management support, and other interface ease-of-use improvements. Those kind of improvements are not likely to be made unless the loading and saving engine is accurate and complete though. People won't want to spend the time to beautify and improve an engine whose core is not functioning properly. That is where I find myself at the moment, so I have spent some time looking at ways to improve WillowSaveGame.cs. Your code will offer some helpful insight here and possibly with the meanings of the individual ChallengeData values, which I can tell from his new WillowSaveGame.cs version that XanderChaos is planning to parse and allow editing of instead of simply copying them as a block like it does curently. It would require quite an extensive rewrite of WillowTree# to use the code directly, so I don't suspect that will happen. It looks like a very solid foundation for a brand new character editor project though.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
da_fileserver, I had some time to adapt your code to my savegame validation tool and test it. First let me go over the steps I had to use to make it work, in case anyone else wants to try to use this code, then in the next reply I will go over the results.
First off when I tried to compile, it complained that my assembly was not CLSCompliant. I'm not sure what the purpose of CLS compliance is and have no idea how to turn it on in the IDE if there is a way, but I found that adding the following before the namespace line in any of my cs files would do it:
[assembly:CLSCompliant(true)]{/code]Next there were some complaints in ArchiveStreamWriter,cs and ArchiveStreamReader.cs that my assembly was not marked as unsafe. I don't know how to turn unsafe on and off and I didn't see any reason to use unsafe code here since this is not a time critical application, so in each of those source files I went to the top and commented out #define USE_UNSAFE_CODE to fix those errors.Finally there was two calls to int.ByteSwap which doesn't appear to be a normal method in .NET 4.0 and wasn't implemented anywhere that I could find. I had to implement an int class extension to solve this issue. To do this I made a new source file which I called intExtender.cs:[code]using System;using System.Collections.Generic;using System.Linq;using System.Text;namespace Borderlands{public static class Int32Extension{public static Int32 ByteSwap(this Int32 value){byte[] buffer=BitConverter.GetBytes(value);Array.Reverse(buffer);return BitConverter.ToInt32(buffer, 0);}}}[/code]With these changes I was able to compile and use your source code in my VisualStudio 2010 C# Express application.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Eww, I messed up the BBCode in the last reply so the formatting came out messed up. I don't see how a forum can exist without a message preview option. Anyway, on to the results of my testing:
The basic savegame reading and writing worked flawlessly. Without trying to parse and rebuild the expansion data, it was able to read and write all my PC (about 25 of these) and PS3 (5 of these) savegames without receiving exceptions, without modifying or rearranging the file format, and without introducing any data corruption so the output was byte-identical to the input.
When I tried to parse and rebuild the expansion data with PlayerExpansionData.Deserialize and PlayerExpansionData.Serialize, it was a mixed result: PS3 was a failure and PC was a success. The only PS3 file I have with expansion data failed the byte verification. The performance on PC files was flawless on all my legitimate savegame files, but I encountered an exception on one that is not legitimate. That PC savegame that had an extended dlc block with a bunch of zeroes after the normal DLC item/weapon data. That game did load properly in Borderlands and save the DLC area back the same way it read it with the zero padding after the dlc weapon/item data. The player in that savefile has completed playthrough 2 according to the mission completion log but he can only teleport to 5 locations in PT2, so I this is not an authentic save file and probably shouldn't be considered a real failure.
I tracked down the problem with the PS3 file in the code. There was a failure to conserve the byte order of the writer at one point when a temporary reader is used to build some data. That resulted in at least one data item being output in the wrong byte order. I looked at the serialize method and the input reader type is taken as ArchiveReader which doesn't even support big endian. After some minor changes to PlayerExpansionData.cs to fix this problem, I was able to succeed byte verification of the expansion data on my PS3 savegame file also. Your code needs the following modifications:or it cannot be used to deserialize into a big endian byte order properly:
In PlayerExpansionData.cs:
Change From:
public void Serialize(Unreal.ArchiveWriter writer)
Change to:
public void Serialize(Unreal.ArchiveStreamWriter writer)
About the CLS compliance, it doesn't really matter that much. It's was something that I did to make FxCop (and Visual Studio analysis) work without any warnings. Basically CLS compliant code means that it only publicly exposes CLS-compliant code (and does not use Microsoft-specific extensions like SByte, UInt16, UInt32, UInt64, etc.) But like I said in the readme, I was doing it in haste, so there was some stuff that I removed from the archive reading/writing source files which required setting the CLS compliance the way I did. But the way you added the compliance is correct. Though I usually put it into the Properties/AssemblyInfo.cs file.
Unsafe code can be enabled by going into the project settings (under Build) and checking the "Allow unsafe code" option. But you are right in that the performance isn't really critical.
I forgot that I was using the byte swapping extensions. I thought about including that file, but I didn't think I was using it. The version you're using is fine, though the one that I built is optimized a little better. (In mine I avoid allocating byte arrays by swapping it manually using shifts, ANDs, and ORs instead of the BitConverter and Array.Reverse.)
A lot of the code in the Unreal namespace is really ugly and hard to understand (ReadString in particular comes to mind). But I just wanted something that worked fairly well and quickly. The reason for a lot of the performance stuff is that the other project I was working on involved actually reading the Unreal packages, which are huge (and time becomes a little bit more critical at that point).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've updated my code a bit. From all appearances, it works with both PC and PS3 save games (and theoretically Xbox360 too). See changes.txt in the archive for the "complete" list of changes.
Also, it should be easier to read/write the DLC expansion data now, since I added a IsBigEndian property to the WillowSaveGame (and you can pass that to the constructor for SimpleArchiveReader/SimpleArchiveWriter). Thus your change, matt911, is not necessary. But thanks for the updates anyways.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You made it easier to set the byte order of the write stream by adding the constructor to SimpleArchiveWriter. That makes the code external to your savegame reader easier to do since I can now open the SimpleArchiveWriter and set the byte order in one line instead of two. However, you did not fix the underlying problem with PlayerExpansionData.Serialize() creating a secondary writer (tempWriter) to process data and not using the same byte order as the writer passed to the method (writer) was using. You still require almost the same changes. The new writer defaults to little endian and you never tell it any differently, so the particular section of the save file written by that temporary reader will only be correct on little endian machines. To set the new writer to the correct byte order you need to read the byte order from the original reader, but the ArchiveWriter type doesn't support or expose IsBigEndian. To get access to IsBigEndian you need to use one of the more specific writer types that guarantees that functionality for the input parameter, either SimpleArchiveWriter or ArchiveStreamWriter. That was the purpose for me saying that you needed this change:
Change From:
public void Serialize(Unreal.ArchiveWriter writer)
Change to:
public void Serialize(Unreal.ArchiveStreamWriter writer)
Once you have an input writer that you can figure out the byte order of, then you need to pass that byte order on when you pass the duties off to the temporary writer. That is the purpose of this change:.
In case you do not have a PS3 Borderlands file with expansion data to attempt to write to confirm that the byte order is still incorrect starting at the variable that tells the number of bank slots, here is a link to one available at GameFAQs:
You are totally correct about that. I completely spaced. I fixed that and re-uploaded revision 2. No need to download it again if you applied that fix already (other than a small change in changes.txt and readme.txt).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Link: http://www.mediafire.com/?17osgp9a6jg8cc6
Here are some changes I made to WillowSaveGame.cs. I am not one of the developers of
WillowTree# so these are purely unofficial modifications. This is a patched version of
WillowSaveGame.cs from WillowTree# beta 11. Feel free to use these changes if they are
useful to you. Just replace the WillowSaveGame.cs with this one and compile. The full
source code for WillowTree# is available at sourceforge.net. The changes.txt file is
purely informational and may be discarded.
The changes were confined to the reading (ReadWSG) and writing (SaveWSG) code starting at
Unknown9 and up through the end of echo list 2 to reflect the fact that there are not
always two echo lists in a save file. Attempting to read a second echo list when it was
not there would put the file pointer out of sync when it read the DLC data length and the
DLC bank size.
In my saves this would result in the bank size being corrupted any time a save with only
one playthrough was loaded then saved. It didn't appear to cause any other problems for
me but I can't say if this loss of sync may cause any other important data corruptions
in the DLC data or not.
There was a data element called SometimesNullsRevenge that was used in this section of code
that is no longer used but was not removed in case some external file references it. I
wanted to make sure the new WillowSaveGame.cs is completely plug-in compatible with the old
one in case someone uses it in conjunction with some software other than WillowTree#. I
also gleaned some information analyzing savegame files that could warrant renaming of a few
data items, but I didn't make any changes to the names at this time for the same reason.
The information:
unknownSaveValue – This is the total gameplay time elapsed, in seconds I believe.
unknown2 - this is the color of vehicle 1
unknown3 - this is the color of vehicle 2
unknown4 - this is the type of vehicle 1 (0 = rocket, 1 = machine gun)
unknown5 - this is the type of vehicle 2
unknown9 - this is the number of echo lists in the save file
I removed some ugly sanity checks on reading the echo list that appear unnecessary now
that the code knows how to tell that there is only one echo list.
I can be reached by email at matt911@users.sf.net if you have any direct questions or
comments about these code changes.
I feel kind of silly now, since I put up a newer WillowSaveGame.cs that already had some of this fixed. The thing is that I put it in a private forum. Either way, thanks for clearing up most of those unknowns! I've gone ahead and merged this info into my updated class, which you can find here:
http://xanderchaos.com/WillowTree/WillowSaveGame.cs
Thank you for the link. I like the new design of WillowSaveGame.cs. It is much cleaner than the old code since you rewrote ReadString and WriteString instead of trying to individually account for the nulls in each spot that may encounter an empty string. I did the same thing in my own utility program to analyze the save files, but did not do it in the WillowSaveGame.cs modification since I wanted to make sure that someone that is already using WillowTree# beta 11 or a homemade utility
program that uses WillowSaveGame.cs can plug and play with no other modification. I made the minimum amount of changes to fix the problem with reading the DLC bank size.
I see a fault in your new version of the code where you removed the voodoo code that checked for an extra Int32 value that appears sometimes in the data stream before unknown 6, 7, 8, or 9. Failing to detect it will result in the echo list count at unknown9 being misread as zero so only one echo list will be read properly and who knows what will happen to the DLC data afterwards.
The extra int value seems very rare so it may be a hard problem to spot. I collected all my own PC save files and as many other save files for pc, ps3, and xbox360 as I could easily locate around the net when I analyzed the data and out of about 25 savegames I have only one that encounters this extra Int32. That save has the values {0, 1, 1, 0, 2} starting at unknown6 and ending at unknown9, which must be 1 or 2 in all cases so it is certainly not the fourth value and the echo list placement relative to it afterwards suggests that it must be the fifth value. Every other savegame I have has only four values in this range: {0, 0, 0, 1} for playthrough one saves or {0, 0, 0, 2} for playthrough two saves. I think the most likely scenario is that unknown7 is a flag or count for the optional exra value which would be written in the stream right after it appearing as the second 1 in {0, 1, 1, 0, 2}, but there are other possibilities.
In the original beta 11 code you accounted for the extra int32 value with SometimesNullsRevenge and wasNull when you realized the length of the echo list 1 is sometimes not where you expected because it was also shifted by that extra value inserted in the unknown6-unknown9 region. I basically did the same thing but moved the extra value check to Unknown9 since that must be read correctly to get the right number of echo lists so the file pointer is in the right spot to read the DLC data without corrupting it. As I said I suspect that Unknown7 is actually the cause of the extra value, but if it turns out that it is just a normal unrelated value variable then checking for the zero will be more reliable, so I stuck with that as in the original beta 11 code and if I detected zero at uknown9 I used wasNull to flag it and make sure I wrote an extra zero after unknown8 in SaveWSG to save it back the same way I read it. wasNull could be renamed to something more descriptive, but I think you will either need the null check or some kind of analysis of unknown7 or 8 to determine whether there's an extra Int32 in the stream to make sure the number of echos is read properly.
Here's a link for the only savegame I have with the extra value, but it is a PS3 save so you won't be able to test it yet with the new version since PS3 is big endian:
http://www.mediafire.com/?8scphsjp8cja5nd
I also spotted a very minor technical detail that probably doesn't really matter, but I noticed that you set EchoPlaythroughs = 2 if FinishedPlaythrough1 = 1 in WriteWSG. You can finish playthrough 1 without starting playthrough 2 and you will have FinishedPlaythrough1 = 1 but only one echo list in your savegame. FinishedPlaythrough1 is set when you kill the creature at the final vault, but the echo list won't be added until you start in playthrough 2. This will create a second echo list even when one does not exist in the original file in some instances. That probably has zero practical effect on anything except that if you load then save the resulting savegame file wont be identical to the one you started with.
Here's some code I've been working on for a while for Borderlands. It should be fairly complete (except for some of the expansion data fields).
http://tommunism.net/downloads/Borderlands%20Save%20Format%20r1.rar
Note that the three unknowns in SaveProfile are not used at all, and the latter two are almost certainly derived from uninitialized memory.
Very nice work da_fileserver! I can tell you are a big fan of ,NET, C#,, XML and object oriented technology in general. Your code is very thorough at enumerating every value in your classes as Properties, making extensive use of things like interfaces, class inheritance, derived classes, exception handling, and so on. I'm very much an old school C programmer who got stuck using modern day object oriented technologies unwillingly so I don't even fully understand how half this stuff works.
Your code contains so much object oriented stuff in fact that I haven't really figured out where the meat is in the code yet, but from what I did read through I think the extra int that occurs at Unknown7 or 8 in the original WillowSaveGame.cs code is a result of a collection of promo code activations. I look forward to examining your code more to figure out how these are handled.. Its lack of proper understanding of the file format in this area has made WillowSaveGame.cs unable to properly read all save files as evidenced by the loads of sanity checks needed to stop the program from crashing after unknown6. The DLC reading code is particularly ugly. I think it does mostly work but I can't even test it completely because it doesn't attempt to write files back the same way it read them. It moves items out of the DLC backpack into the regular character inventory area. This results in it being impossible to simply load a DLC savegame then save it back and use a file compare utility to make sure WillowTree# hasn't corrupted any data.
I don't personally understand why the game decides to use the DLC backpack area instead of the regular backpack area. It may be that it is due to a limitation on the number of items the regular backpack area can hold because of some fixed size arrays that are not large enough if a person has all the possible backpack upgrades available in the expansions. If that is the case then moving too many items from the DLC backpack to the regular inventory could cause crashes or unloadable savegame files. It could be that some DLC items have extra data that cannot be accounted for in the regular backpack, so again if you are moving items from it to the regular backpack you might achieve unwanted results. The whole policy of moving data this data routinely in WillowTree# is suspect.
Like you, I would like to see a savegame tool that can accurately save and edit without any corruptions. WillowTree# is quite close to this, but needs a few more fixes to WillowSaveGame.cs to achieve that and probably some changes to other WIllowTree# source files as well. XanderChaos recognizes this but I think he has tackled the problem at too high of a level by trying to completely rewrite WillowSaveGame in way that makes it incompatible with what he has elsewhere, causing him to have to make too many changes to handle easily. It likely would have been better to do this in smaller revisions than try to do it all at once. Making everyone wait for this to happen is leaving the whole project stalled in the meantime.
Beyond the basic functioning there are some ways that WillowTree#'s usability could be improved somewhat through an improved interface with context menus, keyboard support, drag-and-drop inventory management support, and other interface ease-of-use improvements. Those kind of improvements are not likely to be made unless the loading and saving engine is accurate and complete though. People won't want to spend the time to beautify and improve an engine whose core is not functioning properly. That is where I find myself at the moment, so I have spent some time looking at ways to improve WillowSaveGame.cs. Your code will offer some helpful insight here and possibly with the meanings of the individual ChallengeData values, which I can tell from his new WillowSaveGame.cs version that XanderChaos is planning to parse and allow editing of instead of simply copying them as a block like it does curently. It would require quite an extensive rewrite of WillowTree# to use the code directly, so I don't suspect that will happen. It looks like a very solid foundation for a brand new character editor project though.
da_fileserver, I had some time to adapt your code to my savegame validation tool and test it. First let me go over the steps I had to use to make it work, in case anyone else wants to try to use this code, then in the next reply I will go over the results.
First off when I tried to compile, it complained that my assembly was not CLSCompliant. I'm not sure what the purpose of CLS compliance is and have no idea how to turn it on in the IDE if there is a way, but I found that adding the following before the namespace line in any of my cs files would do it:
Eww, I messed up the BBCode in the last reply so the formatting came out messed up. I don't see how a forum can exist without a message preview option. Anyway, on to the results of my testing:
The basic savegame reading and writing worked flawlessly. Without trying to parse and rebuild the expansion data, it was able to read and write all my PC (about 25 of these) and PS3 (5 of these) savegames without receiving exceptions, without modifying or rearranging the file format, and without introducing any data corruption so the output was byte-identical to the input.
When I tried to parse and rebuild the expansion data with PlayerExpansionData.Deserialize and PlayerExpansionData.Serialize, it was a mixed result: PS3 was a failure and PC was a success. The only PS3 file I have with expansion data failed the byte verification. The performance on PC files was flawless on all my legitimate savegame files, but I encountered an exception on one that is not legitimate. That PC savegame that had an extended dlc block with a bunch of zeroes after the normal DLC item/weapon data. That game did load properly in Borderlands and save the DLC area back the same way it read it with the zero padding after the dlc weapon/item data. The player in that savefile has completed playthrough 2 according to the mission completion log but he can only teleport to 5 locations in PT2, so I this is not an authentic save file and probably shouldn't be considered a real failure.
I tracked down the problem with the PS3 file in the code. There was a failure to conserve the byte order of the writer at one point when a temporary reader is used to build some data. That resulted in at least one data item being output in the wrong byte order. I looked at the serialize method and the input reader type is taken as ArchiveReader which doesn't even support big endian. After some minor changes to PlayerExpansionData.cs to fix this problem, I was able to succeed byte verification of the expansion data on my PS3 savegame file also. Your code needs the following modifications:or it cannot be used to deserialize into a big endian byte order properly:
In PlayerExpansionData.cs:
Change From:
public void Serialize(Unreal.ArchiveWriter writer)
Change to:
public void Serialize(Unreal.ArchiveStreamWriter writer)
Before the line:
entry.Serialize(tempWriter);
Add this:
tempWriter.IsBigEndian = writer.IsBigEndian;
About the CLS compliance, it doesn't really matter that much. It's was something that I did to make FxCop (and Visual Studio analysis) work without any warnings. Basically CLS compliant code means that it only publicly exposes CLS-compliant code (and does not use Microsoft-specific extensions like SByte, UInt16, UInt32, UInt64, etc.) But like I said in the readme, I was doing it in haste, so there was some stuff that I removed from the archive reading/writing source files which required setting the CLS compliance the way I did. But the way you added the compliance is correct. Though I usually put it into the Properties/AssemblyInfo.cs file.
Unsafe code can be enabled by going into the project settings (under Build) and checking the "Allow unsafe code" option. But you are right in that the performance isn't really critical.
I forgot that I was using the byte swapping extensions. I thought about including that file, but I didn't think I was using it. The version you're using is fine, though the one that I built is optimized a little better. (In mine I avoid allocating byte arrays by swapping it manually using shifts, ANDs, and ORs instead of the BitConverter and Array.Reverse.)
A lot of the code in the Unreal namespace is really ugly and hard to understand (ReadString in particular comes to mind). But I just wanted something that worked fairly well and quickly. The reason for a lot of the performance stuff is that the other project I was working on involved actually reading the Unreal packages, which are huge (and time becomes a little bit more critical at that point).
I've updated my code a bit. From all appearances, it works with both PC and PS3 save games (and theoretically Xbox360 too). See changes.txt in the archive for the "complete" list of changes.
http://tommunism.net/downloads/Borderlands%20Save%20Format%20r2.rar
Also, it should be easier to read/write the DLC expansion data now, since I added a IsBigEndian property to the WillowSaveGame (and you can pass that to the constructor for SimpleArchiveReader/SimpleArchiveWriter). Thus your change, matt911, is not necessary. But thanks for the updates anyways.
You made it easier to set the byte order of the write stream by adding the constructor to SimpleArchiveWriter. That makes the code external to your savegame reader easier to do since I can now open the SimpleArchiveWriter and set the byte order in one line instead of two. However, you did not fix the underlying problem with PlayerExpansionData.Serialize() creating a secondary writer (tempWriter) to process data and not using the same byte order as the writer passed to the method (writer) was using. You still require almost the same changes. The new writer defaults to little endian and you never tell it any differently, so the particular section of the save file written by that temporary reader will only be correct on little endian machines. To set the new writer to the correct byte order you need to read the byte order from the original reader, but the ArchiveWriter type doesn't support or expose IsBigEndian. To get access to IsBigEndian you need to use one of the more specific writer types that guarantees that functionality for the input parameter, either SimpleArchiveWriter or ArchiveStreamWriter. That was the purpose for me saying that you needed this change:
Change From:
public void Serialize(Unreal.ArchiveWriter writer)
Change to:
public void Serialize(Unreal.ArchiveStreamWriter writer)
Once you have an input writer that you can figure out the byte order of, then you need to pass that byte order on when you pass the duties off to the temporary writer. That is the purpose of this change:.
Before the line:
entry.Serialize(tempWriter);
Add this:
tempWriter.IsBigEndian = writer.IsBigEndian;
With your new functionality to SimpleArchiveReader you could instead do this for that part:
Change this:
using (Unreal.SimpleArchiveWriter tempWriter = new Unreal.SimpleArchiveWriter(tempStream, true))
To this:
using (Unreal.SimpleArchiveWriter tempWriter = new Unreal.SimpleArchiveWriter(tempStream, true, writer.IsBigEndian))
In case you do not have a PS3 Borderlands file with expansion data to attempt to write to confirm that the byte order is still incorrect starting at the variable that tells the number of bank slots, here is a link to one available at GameFAQs:
http://db.gamefaqs.com/console/ps3/save/borderlands_eu_a.zip
You are totally correct about that. I completely spaced. I fixed that and re-uploaded revision 2. No need to download it again if you applied that fix already (other than a small change in changes.txt and readme.txt).