#26 Twinkle File Won't Open

[Anonymous]

I created a database file using TwinkEdit. I saved it, studied it on my palm, and then sent it back to my computer to update. The columns and everything load, but none of the cards. The console prints out: "Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space".

[Nobody/Anonymous]

Works fine on palm, but TwinkEdit won't open.

[flight16]

Logged In: YES
user_id=667415
Originator: NO

Sorry, that was me.

[flight16]

Logged In: YES
user_id=667415
Originator: NO

It seems populate() isn't working when the file is opened. Around line 167 of TwinkRecord.java inside populate() there is a line: " byte[] data = new byte[len];". The value for len, which is gotten from a "raf.readInt()" is some random number (nothing reasonable... it's huge). Also, populate() is getting called around 24 times before the "while (dataLen > 0) " loop is even getting entered.

Perhaps this is due to my buggy XML import code. It's just odd it still works on the palm...

[flight16]

Logged In: YES
user_id=667415
Originator: NO

I added a static variable to TwinkRecord, set to 0, and incremented it when dataLen > 0 in parse(). This causes the while() loop to be skipped for the first occasion that dataLen > 0 (ie. the time when it found the huge len) and load everything afterwards. Surprisingly, this works. The data loads fine. All of the cards are there. None are missing.

I'm going to have to look closer at the format (specifically the header) of Twinke's binary files...

[flight16]

Logged In: YES
user_id=667415
Originator: NO

One more peculiarity: With the code in place that skips the first record with dataLen > 0 (TwinkRecord's populate() ), I imported XML data, and saved. I can open the file just fine now in TwinkEdit.

The only think I can think of that I did on the palm that is at all non-routine is add a field to the back of the card (unhide it), change font size, and then enable "write answer". I'm not sure if I did all of that directly before the file got corrupted, but somewhere thereabout.

[Michael Duersch]

Logged In: YES
user_id=1336289
Originator: NO

Sorry it's taken me so long to respond. I've been super busy with looking for a house and starting a new job. I'll download the file and see what I can find.

[flight16]

Logged In: YES
user_id=667415
Originator: NO

Don't worry if you aren't able to get around to it for a while. It sounds like you have a lot more important things to be doing. Good luck with the new job and house hunting.

The bug hasn't happened since when I opened this ticket, and I've been importing several files per week.

[Jason Reaves]

Logged In: YES
user_id=1523185
Originator: NO

I encountered a very similar problem (same console error) after deleting a lot of cards (including all of the rows within one category) in TwinkEdit, HotSynch'ing, and then re-opening the database in TwinkEdit. The database seemed okay on the Palm, but none of my cards showed up in TwinkEdit, and I wasn't able to view Categories or other information about the database in TwinkEdit.

I did find a workaround, however: by creating a new database on the Palm and then merging the original with the new one and synching back to the computer, I was able to open the new datbase in TwinkEdit without any problems.

[flight16]

Logged In: YES
user_id=667415
Originator: NO

I have tracked down the problem to be reproducible for a new file created in TwinkEdit and then edited!

1. Make a new file.
2. Make some categories.
3. Make some cards (with about 8 - 20 characters of junk)
4. Assign cards to categories.
5. Delete 4 characters off the front of any card.
6. Save.
7. Reopen... you can't.
NOTE: if you only delete 3 characters in step 4, everything works fine.

So files become unopenable if you delete a bunch of categories, or delete 4 chars of text from a card side.

I first noticed this with SJIS encoding. But then it was reproduced with default PALM encoding. So it doesn't seem to be character set related.

I ran further test cases. I made 5 copies of the file. The original, then one for each increment of characters I deleted. Then I looked at the hex file and the results are what I expected (these are the trailing contents of the file):

0 chars deleted: 66792061646679206100
1 chars deleted: 79206164667920610000
2 chars deleted: 20616466792061006100
3 chars deleted: 61646679206100206100
4 chars deleted: 64667920610079206100

You can see the last two bytes, the "00", move two bytes over every time a character gets deleted.

At first I thought that the problem could lie elsewhere (ie. pointer offsets when writing fields). But now I wonder if the problem really does lie in these ending bytes. My reasoning is that when I deleted the trailing "79206100" off the last case's file, to reveal the "real" trailing 00 bytes at the end (truncating the file), it opened again in TwinkEdit. Furthermore, you can copy the "corrupt" part of the last test case to a good file, and it will cause the good file to become bad.

I know this isn't very precise, but it is a reproducible bug, with a reproducible working work-around.

[Michael Duersch]

Logged In: YES
user_id=1336289
Originator: NO

The more I think about it, this probably is a random access bug. This could be a beast to track done (which I'm still going to do), but for now, I think a work-around is probably the best solution...

[flight16]

Logged In: YES
user_id=667415
Originator: NO

Good news!

I did a grep for .length() and found one instance of RandomAccessFile using it. On about line 386, "record[i].size = raf.length() - record[i].offset;" This is very curious. Looking at source forge's CVS, this line first turned up in TwinkEdit revision 1.15: the first with image support.

Points:
1. This seems to be the ONLY place in all of TwinkEdit that calls .length() on a random access file. And it is for setting the record.size of the very last record. This happened in all cases with the bug.
2. As proven earlier, doing so much as modifying the last dozen bits of a random access file causes it to become corrupt (which also rules out problems with all other explicitly written offsets and lengths in the file).
3. RandomAccessFile is not truncating. When you truncate the file it fixes the bug.

I think this is the single line of code that is both causing the bug and affected by the work around I described. So simply calling RandomAccessFile.setLength(0) -should- be all that is needed to permanently fix this bug. I have class right now, so I will check tonight...