#26 Remove_file program corrupts archives.

[J. Scott Edwards]

Sometimes running the remove_file program will corrupt something in the archive (most likely a reference_list) and the archive is corrupted. After that doing almost any file operation asserts:

list_files: objectify.c:1743: nwos_read_object_from_disk_and_decrypt: Assertion `block_contains_object(scrambled, ref)' failed.

[J. Scott Edwards]

I was wrong about it corrupting the reference list. The actual problem is that if there are files in the archive that are duplicates it does not detect that and removes all of the data, checksums, etc., thereby trashing the duplicate file. For example if you do the following:

$ export OBJECTIFY_PRIVATE_PATH=/tmp/objectify_test/private.obj
$ export OBJECTIFY_LOG_FILE_PATH=/tmp/objectify_test/objectify_log
$ export OBJECTIFY_BACKUP_PATH=
$ prep_disk 32M
$ mkdir xxx
$ cat >> xxx/junk.txt
Hello, it's me!
$ mkdir yyy
$ cat >> yyy/dup.txt
Hello, it's me!
$ import_file --very-low-security -r xxx yyy
$ remove_file --all-traces xxx/junk.txt
$ export_file yyy/dup.txt

You will get the following error:

num_refs: 1
yyy/dup.txt: file size: 16
export_file: reference_list.c:253: read_ref_list_into_cache: Assertion `nwos_read_object_from_disk_and_decrypt(&cache->ref, &cache->first_block->list, 256, ivec, nwos_random_sequence[0])' failed.
Aborted (core dumped)

[J. Scott Edwards]

I have decided not to fix this problem right now because the remove_file program causes other problems and the correct way to fix this is to have it detect if it is going to break anything by removing all traces of the file and not allow that option. That of course means other options need to be available, like --remove-path.

Also if creating backup (.dif) files is enabled running remove_file will create a broken .dif file because currently .dif files cannot deal with blocks being deleted from the archive (they can only add or change blocks).

My conclusion is that remove_file should not be used, until all of these problems can be fixed. I am going to remove it from the Makefile until I have time to properly fix it.