DataCrow Backup Files greater than 4.28 GB are corrupt!

Volkmar
2013-01-27
2013-02-08
  • Volkmar

    Volkmar - 2013-01-27

    Hello,

    there's definitely a major problem with the file size of backups.
    All backups larger than 4.28 GB (file size > 2^32) are corrupt and cannot be restored!
    (see as well my comment on restore problems with 3.9.19).

    In fact this is a problem of all versions.

    I've got the approval by testing as follows.

    • installed 3.9.19
    • restored my last non-corrupt backup file with size 4.227.760.128 Byte (size on hard disk)

    • added three movies in the movie module by adding the same existing item as new for three times

    • generated a new backup (file size = 4,245,307,392 byte on hard disk => < 2^32)
    • deleted one of the items added
    • tried to restore this backup with version 3.9.19 (originating version of backup was recognized, restore was completely successful)

    • added six movies in the movie module by adding the same existing item again as new for six times

    • generated a new backup (file size = 4,282,331,136 byte on hard disk => < 2^32)
    • deleted one of the items added
    • tried to restore this backup with version 3.9.19 (originating version of backup was recognized, restore was completely successful)

    • again added six movies in the movie module by adding the same existing item as new for six times

    • generated a new backup (file size = 4,319,354,880 byte on hard disk => > 2^32 )
      (There was no program message that the backup failed!)
    • deleted one of the items added
    • tried to restore this backup with version 3.9.19 (originating version of backup was NOT recognized, restore failed, backup file is corrupt)

    I can send you the backup files if you like to check this!

    I'm looking forward to get an answer for the following questions:

    • Is there a chance that this problem can be fixed so that I can go on using DataCrow to administrate my large movie collection?
    • If so: how can I move about 150 to 200 new movies and masses of edits from my actual version 3.9.14 to a fixed version?

    Best Regards
    Volkmar

     
  • Jeffrey Thayer

    Jeffrey Thayer - 2013-01-30

    I may have the same problem, but there is a work around I found by accident, seems crazy, but you have to leave 3.9.18 in place or the version your running now, I was running 3.9.18,, went to update to the new version (3.9.19) and everything seemed to go great until you restore your backup from 3.9.18 to 3.9.19 it would hang up and totally become unresponsive so when you stop the restore process by Ctrl - Alt - Del and do the same thing to stop the program itself, and when you opened it back up it would revert to 3.9.18. So in all my testing on 3 machines to get log files for Robert, I found it will work great as long as you leave the old version in place, it seems to piggy back the old version as long it is place,,,,, hope this work around works for you until Robert fixes, this bug,,, I am still waiting to hear from him to about this,, I am working with just a little over 600 movies. Again I hope this works for you

    Good luck
    Jeff

     
  • Volkmar

    Volkmar - 2013-01-30

    Thanks for your advice, Jeff! But actually in my case I don't see any sense in updating or adding entries to Data Crow as long as this is not solved yet.
    My backup file has a size of nearly 5 GB (2640 movies).
    But I'm quite hopeful that Robert will find a solution to the problem in general, because the backup file is a zip archive of some directories and files. The only problem is the size of the zip file. So: if the backup process would provide just a simple directory structure, this would be the minimal solution to the problem in general.

    In fact this would not solve my problem. I think, only a certain patch of version 3.9.14 could help me to get out of this dilemma without repeating the work of last month. Nevertheless I'm lucky that I still got a backup that is smaller than the critical size. And would I have updated to 3.9.15 instantly when it came out, the mess would be less... :-)

    Best Regards
    Volkmar

     
  • Jeffrey Thayer

    Jeffrey Thayer - 2013-01-31

    Your Welcome Volkmar. I emailed Robert the problem when it happened, about 4 or 5 days ago, gave him detail by detail of the 2 bugs that were going on,,, I have not heard a thing back except, that he wanted to know what platform I was using,, and that he had no problems and it was working great on his machines, I just hope he fixes it soon, I am sure he will,, like I tell everyone Data-Crow is hands down the best data cataloger out there

     
  • Robert Jan van der Waals

    Fixed with version 3.9.20. java normally does not support zip files larger then 4.28GB. But I have found another software that allowed me to do that.

    Jeffrey, for you it would be good and try using the -userdir:<your user="" folder=""> parameter:
    java -Xmx512m -jar datacrow.jar -userdir:./data

    In the above example the data folder will always be relative to your installation folder. I think you have set the user folder to point to your 3.9.18 installation folder. With the above option your user folder will always point to the current installation folder. Either change the shortcut and append the -userdir: option or change the .bat or .sh file.

    Best regards, Robert

     
  • Volkmar

    Volkmar - 2013-02-03

    My trials to transfer my data from 3.9.14 to 3.9.20 seemed to look hopefully. I made a copy of my 3.9.14 installation and started 3.9.20 with parameter -userdir pointing at the copied directory of 3.9.14. After I copied the files from directory ./data to ./database I could see all my movies. After I copied all files from ./webapp to ./webroot I also found my images at the right places. But all my changes to edit windows are gone. But OK that's something I can rearrange in a few hours. But when I tried to make my first backup of the saved data it failed showing the following error message:

     
    Last edit: Volkmar 2013-02-03
  • Volkmar

    Volkmar - 2013-02-03

    Sorry, maybe I'm too hysterical... ;-)
    After clicking OK it went on. Let's see what it will bring....

    Thanks for your anyway, Robert!

    Volkmar

     
  • Robert Jan van der Waals

    Where did you indicate your Data Crow user folder to be? Please don't tell you picked the installation folder as the user folder :-) Note that the backup process has been chaged but this only is for new backups. The corrupt backups due to the filesize limit cannot be restored (!).

    Second is that if the drive you are making the backup to has a FAT partition, the backup can still not be larger then 4 GB due to the 4GB limit on these kind of drives.

     
  • Volkmar

    Volkmar - 2013-02-03

    FAT partition
    :-)) It's one of my 3TB Caviar disks. So it's quite fat but it needs NTFS

    I'm well aware that I cannot use any corrupt files. (boy!) That's why I tried to suggest a patch of 3.9.14. But I think I won't need it.

    Here's the command line I used:
    "C:\Program Files (x86)\Java\jre6\bin\java.exe" -Xmx512m -jar "C:\Program Files (x86)\Data Crow\datacrow.jar" -userdir:"C:\Program Files (x86)\Data Crow_3.9.14 - Kopie"

    After I copied some .properties files from the old ./data directory to the new ./setting/modules directory I managed to get all my look & feel changes show up, too. So at first glance it looks quite well. And much, much better, than I ever expected. The other scenario would have been to delete all images from about 200 movies in the actual 3.9.14 installation and add them to the new version after restore of the "reliefed" backup.

    Next I will make another backup and install 3.9.20 again to try reading the new backup files...

     
  • Volkmar

    Volkmar - 2013-02-03

    Yippie :-)

    Now everything works fine! The error message was indeed because I had to use the 3.9.14 installation directory. After deleting the copy of "datacrow.exe" it didn't show up again.

    After reinstalling 3.9.20 I could restore the intermediate backup without any problems to the specified new user directory. I think that I didn't have to repeat a month's work is owed to Data Crow's robust and clever architecture.

    Thanks again for your support, Robert, and Best Regards

    Volkmar

     
  • Jeremy

    Jeremy - 2013-02-03

    I am completely confused. I don't understand how to pick the user data location. I never see an options for that when I am installing Data Crow. Could you please explain how this can be done.

     
  • Volkmar

    Volkmar - 2013-02-04

    Hi Jeremy,

    (I presume you're working with a Windows system as well)

    if you install the actual version of Data Crow it asks you where you want to have the user directory. If there's already an existing installation it could be necessary to rename its installation directory.

    Another way is, to start Data Crow from the command line and use the parameter -userdir:

    (see my command line above). If you try to take over a former installation always make a complete copy of the installation directory and use this one for experiments. Delete datacrow.exe before running a backup. Keep in mind that some directory names and some file locations have been changed since 3.9.14. Robert can explain this much better for sure.

    Good luck,
    Volkmar

     
  • Robert Jan van der Waals

    I have added checks now for the next version to prevent users from selecting the installation folder as the user folder. The backup and restore process can;t cope with this . You will be allowed to select a sub directory of your installation folder. So the following is allowed (given that C:\program files\Data Crow is the installation folder):

    C:\Program Files\Data Crow\Data
    but not:
    C:\program files\Data Crow

    Cheers, Robert

     
    • Jeremy

      Jeremy - 2013-02-08

      Thanks for working on this. When you said the backup restore can't cope with it, does that mean that backing up or restoring isn't working? Just checking because it is locking up when I try to restore. There is an option on startup to pick the user folder.

      Edit: Just a note. It didn't lock up the second time I tried. The .bck file changed to a .zip when I tried to restore the first time and when I restored from that .zip file it ran for a couple of seconds and said the restore was complete, but did not restore.

       
      Last edit: Jeremy 2013-02-08
  • Jeremy

    Jeremy - 2013-02-08

    Thanks Volkmar for the reply, I am using Linux, does this change the procedure. I think I misunderstood what the installation was asking for. I thought the directory I chose was the installation path, I didn't realize it was asking for the user directory path. Am I thinking about this correctly now?

    Edit: The linux version doesn't give an option to pick the user directory during installation, only the installation directory.

     
    Last edit: Jeremy 2013-02-08

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks