[Lessfs-users] LessFS crash: key pair not found

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I've found LessFS to be somewhat unreliable in the latest release
(1.5.8). I've not tried any other versions. I compiled it
--with-berkeleydb.

I've been experimenting using linux containers (lxc), and
deduplicating them using lessfs. The described behavior has occurred
within containers running on lessfs. I've not encountered the behavior
prior to using lessfs.

The behavior that I see has been transient failures such as files not
found one minute, then found the next. For example, when installing
debian packages in a lessfs mount, I regularly encountered this:

Preparing to replace libcomerr2 1.41.3-1 (using
.../libcomerr2_1.41.12-4stable1_i386.deb) ...
Unpacking replacement libcomerr2 ...
dpkg: error processing
/var/cache/apt/archives/libcomerr2_1.41.12-4stable1_i386.deb
(--unpack):
 failed to unlink `//lib/libcom_err.so.2.1.dpkg-tmp': No such file or directory
Errors were encountered while processing:
 /var/cache/apt/archives/libcomerr2_1.41.12-4stable1_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

But attempt to run apt-get again, and it works perfectly.

Also encountered when installing packages (this generates a fair
amount of IO, so it seems like a good test), after some amount of
time, I get the following error, which completely breaks lessfs, and I
have to unmount/remount:

Unpacking replacement tzdata ...
dpkg: error processing
/var/cache/apt/archives/tzdata_2011k-0squeeze1_all.deb (--unpack):
 unable to install new version of
`./usr/share/zoneinfo/right/Greenwich': Software caused connection
abort
E: Sub-process /usr/bin/dpkg exited unexpectedly

>From syslog:

Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:57 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: btbin_write_dup : write failed
DB_KEYEXIST: Key/data pair already exists 5980855433845166676 database
6

>From /var/log/lessfs--bdb_err.txt:

root@zapazoid:~# cat /var/log/lessfs-bdb_err.txt
Duplicate data items are not supported with sorted data

OS: Debian 6.0.3 (Squeeze), linux kernel 2.6.32
Hardware: a Intel P4 / 2G RAM, and a standard hard drive (no ssd involved).
Software:
    * LessFS 1.5.8
    * Berkely DB v4.8

LessFS config:

root@zapazoid:~# grep -v '^#' /etc/lessfs-lxc.cfg
DEBUG = 5
HASHNAME=MHASH_TIGER192
HASHLEN = 24
BLOCKDATA_IO_TYPE=file_io
BLOCKDATA_PATH=/var/lib/lessfs-lxc/dta/blockdata.dta
META_PATH=/var/lib/lessfs-lxc/mta
META_BS=1048576
CACHESIZE=512
COMMIT_INTERVAL=5
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
MAX_THREADS=2
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=2560000000
SYNC_RELAX=0
BACKGROUND_DELETE=on
ENCRYPT_DATA=on
ENCRYPT_META=off
ENABLE_TRANSACTIONS=on
BLKSIZE=131072
COMPRESSION=qlz

MIN_SPACE_CLEAN=10
MIN_SPACE_FREE=1

With DB_CONFIG:

root@zapazoid:~# grep -v '^#' /var/lib/lessfs-lxc/mta/DB_CONFIG
set_cachesize   1       0    2
set_flags       DB_LOG_AUTOREMOVE
set_lg_regionmax        52428800
set_lg_max              10485760
set_lk_max_locks   150000
set_lk_max_lockers 150000
set_lk_max_objects 150000
set_lg_bsize    2097152
set_lg_max      8388608

Any ideas as to what I should tune or investigate?

-davidc