#157 dump fails to restore incremental if directory was removed

open
nobody
restore (48)
7
2014-08-01
2012-06-21
Frank Bicknell
No

dump fails to restore incremental if directory was removed

I know I saw this in archives, but can't find it in the bug list.
The upshot of the archives was that someone found it not to be a problem
for dump v 0.4b42

Re: [Dump-users] removing directories in incremental backups
From: Christoph Holz <ch@ra...> - 2012-03-14 17:39

I found out that this does not happen with dump-0.4b42.

Regards,
Christoph

I'm using dump 0.4b44 on Ubuntu 12.04.

Here's a test rundown:

For the test backups, I started with a clean filesystem and created
testing/foo directory in the filesystem. Then performed the level
0 backup. I removed the directory 'foo' and performed a level 1 backup.

The backups went something like this:

bick-ubtu3:~$ sudo mkfs -t ext4 /dev/mapper/loop0p1
mke2fs 1.42 (29-Nov-2011)
...
Writing superblocks and filesystem accounting information: done

bick-ubtu3:~$ sudo mount /dev/mapper/loop0p1 /mnt/t
bick-ubtu3:~$ cd /mnt/t
bick-ubtu3:/mnt/t$ ls -l
total 16
drwx------ 2 root root 16384 Jun 21 13:02 lost+found
bick-ubtu3:/mnt/t$ sudo mkdir -p testing/foo
bick-ubtu3:/mnt/t$ sudo dump -u -f /tmp/level0.dump /mnt/t
DUMP: Date of this level 0 dump: Thu Jun 21 13:03:22 2012
DUMP: Dumping /dev/mapper/loop0p1 (/mnt/t) to /tmp/level0.dump
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 299 blocks.
DUMP: Volume 1 started with block 1 at: Thu Jun 21 13:03:24 2012
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: Closing /tmp/level0.dump
DUMP: Volume 1 completed at: Thu Jun 21 13:03:24 2012
DUMP: Volume 1 280 blocks (0.27MB)
DUMP: 280 blocks (0.27MB) on 1 volume(s)
DUMP: finished in less than a second
DUMP: Date of this level 0 dump: Thu Jun 21 13:03:22 2012
DUMP: Date this dump completed: Thu Jun 21 13:03:24 2012
DUMP: Average transfer rate: 0 kB/s
DUMP: DUMP IS DONE
bick-ubtu3:/mnt/t$ sudo rmdir testing/foo/
bick-ubtu3:/mnt/t$ sudo dump -1 -u -f /tmp/level1.dump /mnt/t
DUMP: Date of this level 1 dump: Thu Jun 21 13:04:12 2012
DUMP: Date of last level 0 dump: Thu Jun 21 13:03:22 2012
DUMP: Dumping /dev/mapper/loop0p1 (/mnt/t) to /tmp/level1.dump
DUMP: Label: none
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 284 blocks.
DUMP: Volume 1 started with block 1 at: Thu Jun 21 13:04:13 2012
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: Closing /tmp/level1.dump
DUMP: Volume 1 completed at: Thu Jun 21 13:04:13 2012
DUMP: Volume 1 280 blocks (0.27MB)
DUMP: 280 blocks (0.27MB) on 1 volume(s)
DUMP: finished in less than a second
DUMP: Date of this level 1 dump: Thu Jun 21 13:04:12 2012
DUMP: Date this dump completed: Thu Jun 21 13:04:13 2012
DUMP: Average transfer rate: 0 kB/s
DUMP: DUMP IS DONE
bick-ubtu3:/mnt/t$ cd
bick-ubtu3:~$ sudo umount /mnt/t

Here's how the restore goes:

bick-ubtu3:~$ sudo mkfs -t ext4 /dev/mapper/loop0p1
mke2fs 1.42 (29-Nov-2011)
...
Writing superblocks and filesystem accounting information: done

bick-ubtu3:~$ sudo mount /dev/mapper/loop0p1 /mnt/t
bick-ubtu3:~$ cd /mnt/t
bick-ubtu3:/mnt/t$ sudo restore -rf /tmp/level0.dump
restore: ./lost+found: File exists
bick-ubtu3:/mnt/t$ ls -l
total 1736
drwx------ 2 root root 16384 Jun 21 13:02 lost+found
-rw------- 1 root root 1756736 Jun 21 13:04 restoresymtable
drwxr-xr-x 3 root root 4096 Jun 21 13:02 testing
bick-ubtu3:/mnt/t$ sudo restore -rvf /tmp/level1.dump
Verify tape and initialize maps
Input is from a local file/pipe
Input block size is 32
Dump date: Thu Jun 21 13:04:12 2012
Dumped from: Thu Jun 21 13:03:22 2012
Level 1 dump of /mnt/t on bick-ubtu3:/dev/mapper/loop0p1
Label: none
Begin incremental restore
Initialize symbol table.
Extract directories from tape
Mark entries to be removed.
rename ./testing/foo to ./testing/RSTTMP0655362
Calculate node updates.
Find unreferenced names.
deleteino: out of range 0
abort? [yn]

So either the bug is back or it never left (sorry Christoph).

This makes it impossible to restore incrementals unless you're very
very lucky. I first encountered the problem while attempting to do a
test restore of my system ... glad I checked!

Discussion

1 2 > >> (Page 1 of 2)
  • Frank Bicknell
    Frank Bicknell
    2012-06-21

    seems important as this is a restore failure

     
  • Frank Bicknell
    Frank Bicknell
    2012-06-21

    • priority: 5 --> 7
     
  • Frank Bicknell
    Frank Bicknell
    2012-06-21

    Tried with an ext2 filesystem; same result.

    However, I did answer "n" twice:

    ----------------------------------
    rename ./testing/foo to ./testing/RSTTMP0155649
    Calculate node updates.
    Find unreferenced names.
    deleteino: out of range 0
    abort? [yn] n
    deleteino: 0 not found
    abort? [yn] n
    ^Crestore interrupted, continue? [yn] n
    ----------------------------------

    And after the second 'n', it launched into CPU 100% land. I left it
    there until I was pretty sure it wasn't coming out and as you see above
    interrupted it.

    A subsequent umount and fsck found the filesystem ok. Just that funnly
    "leftover" where foo was before.

    So I've decided to try this tactic on my own system test I was running
    earlier and see if it does the same for a live system.

    More to come.

     
  • Frank Bicknell
    Frank Bicknell
    2012-06-22

    I went back and looked up Christoph's e-mail here in sourceforge and it finally dawned on me that he was trying to say that back-revving to 4b42 was what produced a successful result.

    So I tracked down this version in binary form and he's right: it works! So not sure whether it was 43 or 44 that introduced the problem, but from the email thread it was broken in 43.

    My test result (anti climactic now)

    ------------------------------------------
    ...
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done

    bick-ubtu3:~$ sudo mount /dev/mapper/loop0p1 /mnt/t
    bick-ubtu3:~$ cd /mnt/t
    bick-ubtu3:/mnt/t$ sudo restore -rf /tmp/level0.dump
    restore: ./lost+found: File exists
    bick-ubtu3:/mnt/t$ sudo restore -rvf /tmp/level1.dump
    Verify tape and initialize maps
    Input is from a local file/pipe
    Input block size is 32
    Dump date: Fri Jun 22 09:06:36 2012
    Dumped from: Fri Jun 22 09:05:59 2012
    Level 1 dump of /mnt/t on bick-ubtu3:/dev/mapper/loop0p1
    Label: none
    Begin incremental restore
    Initialize symbol table.
    Extract directories from tape
    Mark entries to be removed.
    rename ./testing/foo to ./testing/RSTTMP0131074
    Calculate node updates.
    Find unreferenced names.
    Remove old nodes (directories).
    Remove node ./testing/RSTTMP0131074
    Extract new leaves.
    Check pointing the restore
    Add links
    Set directory mode, owner, and times.
    Check the symbol table.
    Check pointing the restore
    bick-ubtu3:/mnt/t$ ls -lR
    .:
    total 1736
    drwx------ 2 root root 16384 Jun 22 09:04 lost+found
    -rw------- 1 root root 1756656 Jun 22 09:07 restoresymtable
    drwxr-xr-x 2 root root 4096 Jun 22 09:06 testing
    ls: cannot open directory ./lost+found: Permission denied

    ./testing:
    total 0
    bick-ubtu3:/mnt/t$
    ------------------------------------------

    Perfect!

    So I guess just consider this a long-winded submission of a bug report for 4b43 as tested in 4b44...

     
  • Frank Bicknell
    Frank Bicknell
    2012-06-22

    I was able to successfully restore dumps (including incremental) created with 0.4b44 using b42. So clearly the problem is in restore, not dump.

     
  • sangamesh
    sangamesh
    2012-07-13

    I had the same problem and decided to dig in a little bit. I found the problem to be that the deleted directory's entry structure is getting added twice to the remove list. In "findunreflinks" function in restore.c the deleted directory's entry structure is added again to the remove list, this makes the removelist a circular linked list with the entrys next poniting to itself and then it goes into an infinite loop.

     
  • John Cairns
    John Cairns
    2014-01-20

    I experienced this bug as well. I was able to work around by using restore in interactive mode (-i).

     
  • Andy Valencia
    Andy Valencia
    2014-04-11

    Just bit by this in Debian Wheezy, fully up-to-date.
    So, the maintainer realizes that the headline is basically "dump project no longer supports incremental backups"? I had to go dig up a version 42 and build it for myself to get us back online here. Sobering.

     
  • same here. just happened. versio 44 cannot restore, downgrade to 42 allowed me to restore.

     
  • Ben Harris
    Ben Harris
    2014-08-01

    My testing indicates that this bug was introduced in revision 1.39 of restore/restore.c. If I build 0.4b44 with restore.c rolled back to 1.38, the bug doesn't manifest. If I roll restore.c forward to 1.39, the bug appears.

    I've attached a tarball of a minimal test case to save others else the trouble of creating one. It contains level0.dump and level1.dump, and running the following as root should demonstrate the problem:

    mkdir restoretest
    cd restoretest
    restore -rf ../level0.dump
    restore -rf ../level1.dump
    
     
    Attachments
1 2 > >> (Page 1 of 2)