lessfs-users Mailing List for lessfs - data deduplication for less
Brought to you by:
mruijter
Menu
▾
▴
lessfs-users — Users mailinglist
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(2) |
Jul
(5) |
Aug
|
Sep
(2) |
Oct
(2) |
Nov
|
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
(7) |
Jun
(2) |
Jul
|
Aug
|
Sep
(1) |
Oct
(1) |
Nov
|
Dec
(2) |
| 2012 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2013 |
Jan
|
Feb
|
Mar
|
Apr
(5) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
|
From: David N. <do...@ma...> - 2017-11-04 02:58:36
|
I am running lessfs for the first time.
Attached is the cfg file the log file and a gdb analysis of the events
leading up to the SIGABORT (in gdb a SIGUSER1 is generated instead, I'm
not sure why. Maybe the SIGABOUT comes afterwards).
# equery l -p lessfs
* Searching for lessfs ...
[-P-] [ ] sys-fs/lessfs-1.5.13:0
[IP-] [ ] sys-fs/lessfs-1.7.0:0
# lessfs --version
-------------------------------------------------------
lessfs 1.7.0
Usage: lessfs [/path_to_config.cfg] [mount_point] <FUSE OPTIONS>
Example :
mklessfs /etc/lessfs.cfg
lessfs /etc/lessfs.cfg /mnt
A high performance example with big_writes.
(Requires kernel 2.6.26 or higher and a recent version of fuse.)
lessfs /etc/lessfs.cfg /fuse -o use_ino,readdir_ino,default_permissions,\
allow_other,big_writes,max_read=131072,max_write=131072
-------------------------------------------------------
(gdb) bt
#0 0x00007ffff73fba87 in kill () at ../sysdeps/unix/syscall-template.S:84
#1 0x00000000004157ff in exitFunc () at lib_safe.c:230
#2 0x00000000004164d4 in s_open2 (pathname=pathname@entry=0x6441f0
"/mnt/md1/lessfs/dta/blockdata.dta/replog.dta", flags=flags@entry=262210,
mode=mode@entry=448) at lib_safe.c:493 #3 0x000000000041fdc9 in bdb_open
() at lib_bdb.c:331 #4 0x000000000040f06f in parseconfig
(mklessfs=mklessfs@entry=0, force_optimize=force_optimize@entry=false) at
lib_common.c:3728 #5 0x00000000004041bb in main (argc=<optimized out>,
argv=0x7fffffffb338) at lessfs.c:2147
I realize that I'm using a larger hash value then is the max according to
the manual (I tried changing this to the maximum value without result).
The sha512sum uses 512 bits and there is not way around that so I
am assuming that the docs are out of date with respect to that, if not
please tell me.
Thanks,
David
|
|
From: David H. R. C. <da...@ze...> - 2013-04-27 01:29:01
|
On Tue, Apr 16, 2013 at 01:22:37AM +0000, Gruher, Joseph R wrote: > > >>>> However, when I try to configure lessfs with the --with-hamsterdb flag, I get this error: > >>>> > >>>> configure: error: "Hamsterdb is not found" > > >> -----Original Message----- > >> From: David H. Rhodes Clymer [mailto:da...@ze...] > >> I would guess that it is not finding the development files for hamsterdb. > >> Where was hamsterdb installed? Perhaps you will have to mess with > >> CPPFLAGS, LIBS, or LDFLAGS. > > Thanks for responding. On the hamsterdb install I just ran the "make install", so wherever installs to by default. > > I don't see any obvious clues in lessfs config.*. I have attached for reference. > > >> You didn't mention what version of lessfs you were trying to compile. > > This is with lessfs 1.5.13. It does seem to compile OK if I run make but since the configure throws an error I assume it is not set up correctly. I can even run the binaries but nothing seems to happen when I move data into the mounted lessfs (for example database never grows at all no matter how much random data I copy into the lessfs mount). I think these problems go back to hamsterdb and the configure failure. I am going to try with BDB and see if I get a better result. > > >> Does your config.log give you any hints? > > I don't see anything obvious. I have attached for reference. > It does appear to find header files for hamsterdb. However, looking at your config.log (starting line 1184), it appears that a test for a particular hamsterdb function fails. That suggests to me that the version of hamsterdb that lessfs requires is different from the version it is finding. -davidc |
|
From: David H. R. C. <da...@ze...> - 2013-04-16 01:41:36
|
On Tue, Apr 09, 2013 at 05:20:20PM +0000, Gruher, Joseph R wrote: > Hello- > > I am trying to set up lessfs with hamsterdb. I have hamsterdb installed, basically I copied down the 2.1.0 tree, extracted, configured, then ran make and make install. Everything ran successfully as far as I can tell. > > However, when I try to configure lessfs with the --with-hamsterdb flag, I get this error: > > configure: error: "Hamsterdb is not found" > > Does anyone have any suggestions as to the cause? I am running on Ubuntu 12.10. I can provide a more complete log of how the system has been set up if that would be of interest. > I would guess that it is not finding the development files for hamsterdb. Where was hamsterdb installed? Perhaps you will have to mess with CPPFLAGS, LIBS, or LDFLAGS. You didn't mention what version of lessfs you were trying to compile. Does your config.log give you any hints? -davidc |
|
From: Gruher, J. R <jos...@in...> - 2013-04-16 01:24:38
|
>-----Original Message----- >From: Gruher, Joseph R >Sent: Monday, April 15, 2013 6:23 PM >To: 'David H. Rhodes Clymer' >Cc: les...@li... >Subject: RE: [Lessfs-users] hamsterdb is not found > > >>>>> However, when I try to configure lessfs with the --with-hamsterdb flag, I >get this error: >>>>> >>>>> configure: error: "Hamsterdb is not found" > >>> -----Original Message----- >>> From: David H. Rhodes Clymer [mailto:da...@ze...] >>> I would guess that it is not finding the development files for hamsterdb. >>> Where was hamsterdb installed? Perhaps you will have to mess with >>> CPPFLAGS, LIBS, or LDFLAGS. > >Thanks for responding. On the hamsterdb install I just ran the "make install", >so wherever installs to by default. > >I don't see any obvious clues in lessfs config.*. I have attached for reference. > >>> You didn't mention what version of lessfs you were trying to compile. > >This is with lessfs 1.5.13. It does seem to compile OK if I run make but since >the configure throws an error I assume it is not set up correctly. I can even >run the binaries but nothing seems to happen when I move data into the >mounted lessfs (for example database never grows at all no matter how much >random data I copy into the lessfs mount). I think these problems go back to >hamsterdb and the configure failure. I am going to try with BDB and see if I >get a better result. > >>> Does your config.log give you any hints? > >I don't see anything obvious. I have attached for reference. Whoops, now with attachment included. |
|
From: Gruher, J. R <jos...@in...> - 2013-04-16 01:22:45
|
>>>> However, when I try to configure lessfs with the --with-hamsterdb flag, I get this error: >>>> >>>> configure: error: "Hamsterdb is not found" >> -----Original Message----- >> From: David H. Rhodes Clymer [mailto:da...@ze...] >> I would guess that it is not finding the development files for hamsterdb. >> Where was hamsterdb installed? Perhaps you will have to mess with >> CPPFLAGS, LIBS, or LDFLAGS. Thanks for responding. On the hamsterdb install I just ran the "make install", so wherever installs to by default. I don't see any obvious clues in lessfs config.*. I have attached for reference. >> You didn't mention what version of lessfs you were trying to compile. This is with lessfs 1.5.13. It does seem to compile OK if I run make but since the configure throws an error I assume it is not set up correctly. I can even run the binaries but nothing seems to happen when I move data into the mounted lessfs (for example database never grows at all no matter how much random data I copy into the lessfs mount). I think these problems go back to hamsterdb and the configure failure. I am going to try with BDB and see if I get a better result. >> Does your config.log give you any hints? I don't see anything obvious. I have attached for reference. |
|
From: Gruher, J. R <jos...@in...> - 2013-04-09 17:20:28
|
Hello- I am trying to set up lessfs with hamsterdb. I have hamsterdb installed, basically I copied down the 2.1.0 tree, extracted, configured, then ran make and make install. Everything ran successfully as far as I can tell. However, when I try to configure lessfs with the --with-hamsterdb flag, I get this error: configure: error: "Hamsterdb is not found" Does anyone have any suggestions as to the cause? I am running on Ubuntu 12.10. I can provide a more complete log of how the system has been set up if that would be of interest. Thanks!! -Joe Gruher |
|
From: Dominic R. <do...@ti...> - 2012-07-26 10:19:31
|
I am experimenting with lessfs on Ubuntu 12.04 using the configuration suggested by Mark at http://www.lessfs.com/wordpress/?p=707, but with lessfs 1.5.12 and hamsterdb 2.0.4 (and snappy 1.0.5). I have used Mark's lessfs.cfg file (http://www.lessfs.com/wordpress/wp-content/lessfs.cfg) but as I am testing with a single-processor VM I set MAX_THREADS=1. The machine has 1GB RAM. I set about copying (rsync) some data (a lot of files, mostly quite small Word or pdf files) into the lessfs mount. Things go quickly for a while (and there are pauses every so often which I guess are when lessfs flushes data to disk) but then become very slow indeed (say 1 x 30K file per second), and never speed up again. This seems similar to the problems in a previous posting (https://groups.google.com/forum/?fromgroups#!topic/lessfs/cA2M6Czc3Us) but that was where tokyocabinet was being used, and the suggestions relate to _BS configuration options which don't seem apply when using hamsterdb: man lessfs states: 'The hamsterdb database does not require the _BS lines since hamsterdb does not use hash databases'. Does this apply to all *_BS configuration options, I presume so because none are set in Mark's lessfs.cfg? Size of underlying datastore (sudo du -h --summarize /data): 589MB Size of stored files (sudo du -h --summarize /mnt/less): 869M (62 min to calculate this!!!) Number of stored files (sudo find /mnt/less -type f | wc -l): 22675 Can anyone suggest why lessfs is so very slow on my system and how I can improve it? I have tried starting lessfs with '-o big_writes,max_read=131072,max_write=131072', or not: doesn't seem to make any difference to performance, maybe the setting BLKSIZE=131072 in lessfs.cfg means these settings are used anyway? Thanks for any help Dominic -- *TimeDicer* <http://www.timedicer.co.uk>: Free File Recovery from Whenever |
|
From: Robby P. <rpe...@xs...> - 2012-03-10 06:45:31
|
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font size="-1"><font face="DejaVu Sans">Hi,<br>
<br>
I had a power failure while data was being copied to a lessfs
mount.<br>
<br>
<b>lessfs.cfg</b><br>
<br>
BLOCKDATA_IO_TYPE=file_io<br>
BLOCKDATA_PATH=/mnt/backup/dta/blockdata.dta<br>
META_PATH=/mnt/backup/mta <br>
META_BS=1048576<br>
CACHESIZE=512<br>
COMMIT_INTERVAL=10<br>
MAX_THREADS=16<br>
DYNAMIC_DEFRAGMENTATION=on<br>
COREDUMPSIZE=2560000000<br>
SYNC_RELAX=0<br>
BACKGROUND_DELETE=on<br>
ENCRYPT_DATA=off<br>
ENCRYPT_META=off<br>
ENABLE_TRANSACTIONS=on<br>
BLKSIZE=131072<br>
COMPRESSION=qlz15<br>
<br>
On init startup I get:<br>
<br>
<b>/var/log/messages</b><br>
<br>
Mar 10 08:12:44 mail lessfs[32039]: Blocksize = 131072 bytes<br>
Mar 10 08:12:44 mail lessfs[32039]: MIN_SPACE_CLEAN is not set,
lessfs runs -ENOSPC when reaching MIN_SPACE_FREE<br>
Mar 10 08:12:44 mail lessfs[32039]: The selected data store is
file_io.<br>
Mar 10 08:12:44 mail lessfs[32039]: Lessfs transaction support
is enabled.<br>
Mar 10 08:12:44 mail lessfs[32039]: config->blockdata =
/mnt/backup/dta/blockdata.dta<br>
Mar 10 08:12:44 mail lessfs[32039]: compression = qlz15<br>
Mar 10 08:12:44 mail lessfs[32039]: Threaded background delete
is enabled<br>
Mar 10 08:12:44 mail lessfs[32039]: Hash MHASH_TIGER192 has been
selected<br>
Mar 10 08:12:44 mail lessfs[32039]: Lessfs uses a 24 bytes long
hash.<br>
Mar 10 08:12:44 mail lessfs[32039]: Automatic defragmentation is
enabled.<br>
Mar 10 08:12:44 mail lessfs[32039]: cache 4096 data blocks<br>
<br>
<b>/var/log/syslog</b><br>
<br>
Mar 10 08:12:44 mail lessfs[32039]: Open DB handle failed:
Invalid argument<br>
<br>
<b>/var/log/lessfs-bdb_err.txt</b><br>
<br>
file unknown has LSN 223/6046595, past end of log at 223/6021599<br>
Commonly caused by moving a database from one database
environment<br>
to another without clearing the database LSNs, or by removing
all of<br>
the log files from a database environment<br>
file unknown has LSN 223/6027912, past end of log at 223/6021599<br>
Commonly caused by moving a database from one database
environment<br>
to another without clearing the database LSNs, or by removing
all of<br>
the log files from a database environment<br>
file metadata.db has LSN 223/6027912, past end of log at
223/6021691<br>
Commonly caused by moving a database from one database
environment<br>
to another without clearing the database LSNs, or by removing
all of<br>
the log files from a database environment<br>
__db_meta_setup: /mnt/backup/mta/metadata.db: unexpected file
type or format<br>
<br>
lessfs is version 1.5.9 64-bit<br>
mount is /mnt/backup/lessfs<br>
<br>
Any assistance would be appreciated.<br>
<br>
Regards<br>
<br>
Robby<br>
<br>
</font></font>
</body>
</html>
|
|
From: Nils K. <ni...@ib...> - 2011-12-29 09:09:16
|
Hey there, I had some time to investigate my problems a bit deeper, it seems to me that the problem is with the database - either I use Berkely DB (4.8 from Ubuntu repository) or hamsterdb 1.1.15 (self compiled) I get the same issues as described below. Mark - any hints on which database version you use resp. compile options? Cheers, Nils > Hi! > > I am using lessfs-1.6.0-alpha0 and I just wonder if it is only me suffering > instability issues copy-/deleting files? > I use berkelydb and qlz compression and of course I'd like to test the new > feature multifile_io. > Am I doing something incredible stupid or is it just not yet working > properly? > Please see my configuration below. > Basesystem is Ubuntu Oneric Ocelot. > > When I launch lessfs /etc/lessfs.cfg mnt/ -o debug > FUSE library version: 2.8.4 > nullpath_ok: 0 > unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 > INIT: 7.16 > flags=0x0000007b > max_readahead=0x00020000 > INIT: 7.12 > flags=0x00000031 > max_readahead=0x00000080 > max_write=0x00020000 > unique: 1, success, outsize: 40 > unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56 > getattr / > unique: 2, success, outsize: 120 > unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 47 > LOOKUP /.Trash > getattr /.Trash > unique: 3, error: -2 (No such file or directory), outsize: 16 > unique: 4, opcode: LOOKUP (1), nodeid: 1, insize: 52 > LOOKUP /.Trash-1000 > getattr /.Trash-1000 > unique: 4, error: -2 (No such file or directory), outsize: 16 > <... snipp ...> > > > nothing exiting happends, even when the thing crashes, it silenty > dies, leaving > no trace :-/ > The operation causing the crash mubles something like rm: fts_read > failed: Transport > endpoint is not connected. > > Any suggestions? Thanks for your attention, > Nils > > > ############################################################################### > # Enable informational messages about compression. > DEBUG = 5 > HASHNAME=MHASH_TIGER192 > # The (chopped) hashlen in bytes, minimum is 20. > HASHLEN = 24 > BLOCKDATA_IO_TYPE=multifile_io > BLOCKDATA_PATH=/home/nils/data/dta/blockdata.dta > # > META_PATH=/home/nils/data/mta > META_BS=1048576 > # > CACHESIZE=512 > # Flush data to disk after X seconds. > COMMIT_INTERVAL=10 > # > LISTEN_IP=127.0.0.1 > LISTEN_PORT=100 > # Not more then 2 on most machines. > MAX_THREADS=16 > DYNAMIC_DEFRAGMENTATION=on > COREDUMPSIZE=2560000000 > # Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS. > SYNC_RELAX=0 > # When BACKGROUND_DELETE=on lessfs will spawn a thread to delete > # a file as a background task. This is a recently added feature > # and is therefore disabled by default. > BACKGROUND_DELETE=on > # Requires openssl and lessfs has to be configured with --with-crypto > ENCRYPT_DATA=off > # ENCRYPT_META on or off, default is off > # Requires ENCRYPT_DATA=on and is otherwise ignored. > ENCRYPT_META=off > # You don't like fsck? > ENABLE_TRANSACTIONS=on > # Select a blocksize to fit your needs. > BLKSIZE=131072 > COMPRESSION=qlz > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Lessfs-users mailing list > Les...@li... > https://lists.sourceforge.net/lists/listinfo/lessfs-users > |
|
From: Nils K. <ni...@ib...> - 2011-12-01 17:37:17
|
Hi!
I am using lessfs-1.6.0-alpha0 and I just wonder if it is only me suffering
instability issues copy-/deleting files?
I use berkelydb and qlz compression and of course I'd like to test the new
feature multifile_io.
Am I doing something incredible stupid or is it just not yet working properly?
Please see my configuration below.
Basesystem is Ubuntu Oneric Ocelot.
When I launch lessfs /etc/lessfs.cfg mnt/ -o debug
FUSE library version: 2.8.4
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.16
flags=0x0000007b
max_readahead=0x00020000
INIT: 7.12
flags=0x00000031
max_readahead=0x00000080
max_write=0x00020000
unique: 1, success, outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
unique: 2, success, outsize: 120
unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 47
LOOKUP /.Trash
getattr /.Trash
unique: 3, error: -2 (No such file or directory), outsize: 16
unique: 4, opcode: LOOKUP (1), nodeid: 1, insize: 52
LOOKUP /.Trash-1000
getattr /.Trash-1000
unique: 4, error: -2 (No such file or directory), outsize: 16
<... snipp ...>
nothing exiting happends, even when the thing crashes, it silenty
dies, leaving
no trace :-/
The operation causing the crash mubles something like rm: fts_read
failed: Transport
endpoint is not connected.
Any suggestions? Thanks for your attention,
Nils
###############################################################################
# Enable informational messages about compression.
DEBUG = 5
HASHNAME=MHASH_TIGER192
# The (chopped) hashlen in bytes, minimum is 20.
HASHLEN = 24
BLOCKDATA_IO_TYPE=multifile_io
BLOCKDATA_PATH=/home/nils/data/dta/blockdata.dta
#
META_PATH=/home/nils/data/mta
META_BS=1048576
#
CACHESIZE=512
# Flush data to disk after X seconds.
COMMIT_INTERVAL=10
#
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
# Not more then 2 on most machines.
MAX_THREADS=16
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=2560000000
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
# When BACKGROUND_DELETE=on lessfs will spawn a thread to delete
# a file as a background task. This is a recently added feature
# and is therefore disabled by default.
BACKGROUND_DELETE=on
# Requires openssl and lessfs has to be configured with --with-crypto
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off
# You don't like fsck?
ENABLE_TRANSACTIONS=on
# Select a blocksize to fit your needs.
BLKSIZE=131072
COMPRESSION=qlz
|
|
From: David R. C. <da...@ze...> - 2011-10-13 19:52:53
|
I've found LessFS to be somewhat unreliable in the latest release
(1.5.8). I've not tried any other versions. I compiled it
--with-berkeleydb.
I've been experimenting using linux containers (lxc), and
deduplicating them using lessfs. The described behavior has occurred
within containers running on lessfs. I've not encountered the behavior
prior to using lessfs.
The behavior that I see has been transient failures such as files not
found one minute, then found the next. For example, when installing
debian packages in a lessfs mount, I regularly encountered this:
Preparing to replace libcomerr2 1.41.3-1 (using
.../libcomerr2_1.41.12-4stable1_i386.deb) ...
Unpacking replacement libcomerr2 ...
dpkg: error processing
/var/cache/apt/archives/libcomerr2_1.41.12-4stable1_i386.deb
(--unpack):
failed to unlink `//lib/libcom_err.so.2.1.dpkg-tmp': No such file or directory
Errors were encountered while processing:
/var/cache/apt/archives/libcomerr2_1.41.12-4stable1_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
But attempt to run apt-get again, and it works perfectly.
Also encountered when installing packages (this generates a fair
amount of IO, so it seems like a good test), after some amount of
time, I get the following error, which completely breaks lessfs, and I
have to unmount/remount:
Unpacking replacement tzdata ...
dpkg: error processing
/var/cache/apt/archives/tzdata_2011k-0squeeze1_all.deb (--unpack):
unable to install new version of
`./usr/share/zoneinfo/right/Greenwich': Software caused connection
abort
E: Sub-process /usr/bin/dpkg exited unexpectedly
>From syslog:
Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:52 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:57 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:58 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: count_dirlinks : DB_NOTFOUND:
No matching key/data pair found
Oct 13 14:19:59 zapazoid lessfs[26263]: btbin_write_dup : write failed
DB_KEYEXIST: Key/data pair already exists 5980855433845166676 database
6
>From /var/log/lessfs--bdb_err.txt:
root@zapazoid:~# cat /var/log/lessfs-bdb_err.txt
Duplicate data items are not supported with sorted data
OS: Debian 6.0.3 (Squeeze), linux kernel 2.6.32
Hardware: a Intel P4 / 2G RAM, and a standard hard drive (no ssd involved).
Software:
* LessFS 1.5.8
* Berkely DB v4.8
LessFS config:
root@zapazoid:~# grep -v '^#' /etc/lessfs-lxc.cfg
DEBUG = 5
HASHNAME=MHASH_TIGER192
HASHLEN = 24
BLOCKDATA_IO_TYPE=file_io
BLOCKDATA_PATH=/var/lib/lessfs-lxc/dta/blockdata.dta
META_PATH=/var/lib/lessfs-lxc/mta
META_BS=1048576
CACHESIZE=512
COMMIT_INTERVAL=5
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
MAX_THREADS=2
DYNAMIC_DEFRAGMENTATION=on
COREDUMPSIZE=2560000000
SYNC_RELAX=0
BACKGROUND_DELETE=on
ENCRYPT_DATA=on
ENCRYPT_META=off
ENABLE_TRANSACTIONS=on
BLKSIZE=131072
COMPRESSION=qlz
MIN_SPACE_CLEAN=10
MIN_SPACE_FREE=1
With DB_CONFIG:
root@zapazoid:~# grep -v '^#' /var/lib/lessfs-lxc/mta/DB_CONFIG
set_cachesize 1 0 2
set_flags DB_LOG_AUTOREMOVE
set_lg_regionmax 52428800
set_lg_max 10485760
set_lk_max_locks 150000
set_lk_max_lockers 150000
set_lk_max_objects 150000
set_lg_bsize 2097152
set_lg_max 8388608
Any ideas as to what I should tune or investigate?
-davidc
|
|
From: Eduardo T. <etr...@gm...> - 2011-09-16 15:09:12
|
Hi, I've packaged lessfs for Debian. Because of its policies and package warnings I have had to: * write manual pages for binaries that didn't have one * fix existing manual pages (escape the "-" and stuff like that) and correct spelling Please download the debian specific source code from here[0]. You will find the manuals under the debian folder and the patches under the debian/patch folder. It'd be great if you could add (some of) them to your code. * build without openssl support I'd like to ask you to add a LICENSE file with the GPLvx text and the additional text here[1], the one that mentions the exception to be able to link with OpenSSL. If you'd like to know more about the problem between OpenSSL license and the GPL this is a good source of information[2]. * I had to tweak the init.d script. This is a wishlist kind of request: could you create a PIDFILE? I tried using the --make-pidfile with start-stop-daemon but it doesn't work because you fork from the main program. So I wrote a bit of a hack to make it work (debian/lessfs.init in the source[0]) but *the* solution would be to have your daemon create the appropriate file. Thanks, it's a great piece of software, I'm testing it for a backup drive (2T) and I'd love to use it for the imap folders at work. I hope a Debian package will make more people test (more easily) lessfs. Eduardo. [0] http://programoj.esperanto.org.uy/angle/debian/lessfs/lessfs_1.5.4-1.debian.tar.gz [1] http://lists.debian.org/debian-legal/2004/05/msg00595.html [2] http://people.gnome.org/~markmc/openssl-and-the-gpl.html |
|
From: Kuniyasu S. <k.s...@ai...> - 2011-06-16 11:32:05
|
Dear, I presented my paper entitled "Analyze Disk Access Pattern of File Systems for Content Addressable Storage" at Linux Symposium 2011, yesterday. The slide is downloable. http://www.slideshare.net/suzaki/linux-symposium-2011-analysis-of-disk-access-patterns-on-file-systems-for-content-addressable-storage # The paper will be issued soon. It evaluate affinity between file systems and CAS (fixed size deduplication). I evaluate the difference the effect of deduplication when same contents are saved on different file systems (ext3, ext4, JFS, XFS, ReiserFS, nilfs, btrfs, FAT32 and NTFS). I think the situation will be same virtual disk (VMDK) on lessfs. Even if file system level deduplication, same problem will occur when the file system accept virtual block device file. Please give me comments, if you have. ------ suzaki |
|
From: Adam T. W. <awi...@wh...> - 2011-06-06 13:13:39
|
Quoting Paul Dickson <ass...@gm...>: > Is anyone using lessfs in a production environment? For what purpose? How > long have you been doing so? How much data are you storing on it? Are you > replicating? What issues have you run into? ?? I'm also curious if anyone is using this in production and any feedback, blog posts, etc... |
|
From: Märt B. <spa...@gm...> - 2011-05-24 18:14:57
|
Hi!
I can't find the bugzilla and sourceforge won't let me send a message to
mruijter. So I'll file my bug report here.
lessfs-1.4.2 crashes on my gentoo box with segfault. My use flags are:
USE="crypt -debug -lzo -memtrace". The segfault is created in
lib_common.c line 1276:
if (0 != strcmp(config->blockdata_io_type, "file_io")) {
In my case, config->blockdata_io_type is null and the program crashes.
blockdata_io_type is usually set in lib_common.c:3116 (parseconfig),
except #ifndef HAMSTERDB. Possible fix could be adding the following
lines (maybe? is "tokyocabinet" correct?):
@3153
#ifndef HAMSTERDB
+ config->blockdata_io_type = "tokyocabinet";
config->blockdatabs = read_val("BLOCKDATA_BS");
LINFO("The selected data store is tokyocabinet.");
#else
Regards,
Märt Bakhoff
|
|
From: Kuniyasu S. <k.s...@ai...> - 2011-05-24 08:34:31
|
Dear, I'm looking for a method to evaluate deduplication. Do you know any good methods? Recently I tried files which repeats 256 bytes random data or 257 bytes random data. Variable-length deduplication will deduplicate both files well. Fixex-length deduplication will deduplicate the file with 256 bytes random data, but will not deduplicate the file with 257 bytes random data. ----- Kuniyasu Suzaki, National Institute of Advanced Industrial Science and Technology, http://staff.aist.go.jp/k.suzaki |
|
From: Nick W. <ni...@re...> - 2011-05-24 00:19:50
|
On Sun, May 22, 2011 at 11:26 PM, Haug, Christoph <chr...@we...> wrote: > Hi Nick, > > to compile lessfs you need also tokyocabinet. Tokyocabinet is not an optional library. Install the tokiokabinet sources an then it's done well. > > Christoph Thanks Christoph - That was it. Now if I can just get CentOS 5.6 to use blocks >4K :). Let me know if there is anyway to do this, other than upgrading the kernel. Too bad CentOS 6 isn't out yet. Nick |
|
From: Nick W. <ni...@re...> - 2011-05-23 02:48:57
|
Hi All, I'm trying to build 1.4.2 with berkelydb, I've got Berkeley DB 4.8 installed from sources, as well as openssl and fuse, but when I do: ./configure --with-berkeleydb --with-crypto I get: . . . checking for DEPS... configure: error: Package requirements (tokyocabinet >= 1.4.21 fuse >= 2.8.0 openssl > 0.9.5) were not met: No package 'tokyocabinet' found . . . Why is it still looking for tokyocabinet? Can I somehow exclude it? Nick -- Nick Webb Principal Data Protection Specialist http://www.redwireservices.com Red Wire Services, LLC PO Box 46454 Seattle, WA 98146 |
|
From: Haug, C. <chr...@we...> - 2011-05-14 12:31:26
|
Hi, if found a bug in lessfs. If lessfs remove data from the filesystem and the system is reboot the lessfs process is killing by the system. A remount after rebooting doesn't work. The System is blocked by lessfs. I can't browse the folder's in the hierarchy who i try to mount lessfs. Sorry for my English. |
|
From: Paul D. <ass...@gm...> - 2011-05-09 18:59:18
|
Beauller? On Thu, May 5, 2011 at 12:49 PM, Paul Dickson <ass...@gm...> wrote: > Is anyone using lessfs in a production environment? For what purpose? How > long have you been doing so? How much data are you storing on it? Are you > replicating? What issues have you run into? > |
|
From: Paul D. <ass...@gm...> - 2011-05-05 16:49:31
|
Is anyone using lessfs in a production environment? For what purpose? How long have you been doing so? How much data are you storing on it? Are you replicating? What issues have you run into? |
|
From: Marcin M. <ma...@me...> - 2011-02-08 13:27:05
|
W dniu 07.02.2011 13:08, David Touzeau pisze: > Dear i think i have a libmash issue > > i try to compile lessfs-1.3.3 > > i receive this compilation error : > lib_common.c:2975: error: ‘MHASH_SNEFRU256’ undeclared (first use in > this function) > make[1]: *** [lessfs-lib_common.o] Error 1 > make[1]: Leaving directory `/root/APP_LESSFS-2011-02-07-13' > make: *** [all] Error > > > How can i fix it ? Hello, i suppose devs doesn't read this mailing list:( Those errors refers to mhash, maybe you don't have mhash installed? (or mhash-devel , i don't know centos, i'm guessing). Regards |
|
From: David T. <da...@to...> - 2011-02-07 16:42:49
|
Le lundi 07 février 2011 à 13:08 +0100, David Touzeau a écrit : > Dear i think i have a libmash issue > > i try to compile lessfs-1.3.3 > > i receive this compilation error : > > make[1]: Entering directory `/root/APP_LESSFS-2011-02-07-13' > if gcc -DHAVE_CONFIG_H -I. -I. -I. -D_REENTRANT -D__USE_LARGEFILE64 > -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG > -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g > -Wall -Wextra -O2 -MT lessfs-lib_hamster.o -MD -MP -MF > ".deps/lessfs-lib_hamster.Tpo" -c -o lessfs-lib_hamster.o `test -f > 'lib_hamster.c' || echo './'`lib_hamster.c; \ > then mv -f ".deps/lessfs-lib_hamster.Tpo" > ".deps/lessfs-lib_hamster.Po"; else rm -f > ".deps/lessfs-lib_hamster.Tpo"; exit 1; fi > if gcc -DHAVE_CONFIG_H -I. -I. -I. -D_REENTRANT -D__USE_LARGEFILE64 > -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG > -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g > -Wall -Wextra -O2 -MT lessfs-lib_common.o -MD -MP -MF > ".deps/lessfs-lib_common.Tpo" -c -o lessfs-lib_common.o `test -f > 'lib_common.c' || echo './'`lib_common.c; \ > then mv -f ".deps/lessfs-lib_common.Tpo" ".deps/lessfs-lib_common.Po"; > else rm -f ".deps/lessfs-lib_common.Tpo"; exit 1; fi > lib_common.c: In function ‘check_datafile_sanity’: > lib_common.c:142: warning: comparison between signed and unsigned > lib_common.c: In function ‘logiv’: > lib_common.c:231: warning: unused parameter ‘msg’ > lib_common.c:231: warning: unused parameter ‘bhash’ > lib_common.c: In function ‘loghash’: > lib_common.c:257: warning: unused parameter ‘msg’ > lib_common.c:257: warning: unused parameter ‘bhash’ > lib_common.c: In function ‘readBlock’: > lib_common.c:1288: warning: comparison between signed and unsigned > lib_common.c:1296: warning: cast from pointer to integer of different > size > lib_common.c:1312: warning: cast to pointer from integer of different > size > lib_common.c:1242: warning: unused parameter ‘filename’ > lib_common.c: In function ‘update_filesize’: > lib_common.c:1513: warning: comparison between signed and unsigned > lib_common.c:1517: warning: comparison between signed and unsigned > lib_common.c:1527: warning: comparison between signed and unsigned > lib_common.c:1531: warning: comparison between signed and unsigned > lib_common.c: In function ‘db_unlink_file’: > lib_common.c:1624: warning: comparison between signed and unsigned > lib_common.c: In function ‘tc_compress’: > lib_common.c:1753: warning: comparison between signed and unsigned > lib_common.c:1759: warning: comparison between signed and unsigned > lib_common.c:1765: warning: comparison between signed and unsigned > lib_common.c: In function ‘partial_truncate_block’: > lib_common.c:1857: warning: unused variable ‘encrypted’ > lib_common.c: In function ‘flush_wait’: > lib_common.c:2138: warning: cast to pointer from integer of different > size > lib_common.c: In function ‘cook_cache’: > lib_common.c:2156: warning: unused parameter ‘ksize’ > lib_common.c: In function ‘purge_read_cache’: > lib_common.c:2206: warning: cast to pointer from integer of different > size > lib_common.c:2187: warning: unused parameter ‘caller’ > lib_common.c: In function ‘db_fs_truncate’: > lib_common.c:2324: warning: unused parameter ‘bname’ > lib_common.c: In function ‘fil_fuse_info’: > lib_common.c:2435: warning: unused parameter ‘fi’ > lib_common.c: In function ‘locks_to_dir’: > lib_common.c:2451: warning: unused parameter ‘fi’ > lib_common.c: In function ‘parseconfig’: > lib_common.c:2919: error: ‘MHASH_TIGER192’ undeclared (first use in this > function) > lib_common.c:2919: error: (Each undeclared identifier is reported only > once > lib_common.c:2919: error: for each function it appears in.) > lib_common.c:2967: error: ‘MHASH_WHIRLPOOL’ undeclared (first use in > this function) > lib_common.c:2975: error: ‘MHASH_SNEFRU256’ undeclared (first use in > this function) > make[1]: *** [lessfs-lib_common.o] Error 1 > make[1]: Leaving directory `/root/APP_LESSFS-2011-02-07-13' > make: *** [all] Error > > > How can i fix it ? > > Best regards > > > ------------------------------------------------------------------------------ > The modern datacenter depends on network connectivity to access resources > and provide services. The best practices for maximizing a physical server's > connectivity to a physical network are well understood - see how these > rules translate into the virtual world? > http://p.sf.net/sfu/oracle-sfdevnlfb > _______________________________________________ > Lessfs-users mailing list > Les...@li... > https://lists.sourceforge.net/lists/listinfo/lessfs-users I have tried with 2.0 Alpha, but same issue make all-am make[1]: Entering directory `/root/lessfs-2.0.0-palpha2' gcc -DHAVE_CONFIG_H -I. -D_REENTRANT -D__USE_LARGEFILE64 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g -O2 -MT lessfs-lib_commercial.o -MD -MP -MF .deps/lessfs-lib_commercial.Tpo -c -o lessfs-lib_commercial.o `test -f 'lib_commercial.c' || echo './'`lib_commercial.c mv -f .deps/lessfs-lib_commercial.Tpo .deps/lessfs-lib_commercial.Po gcc -DHAVE_CONFIG_H -I. -D_REENTRANT -D__USE_LARGEFILE64 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g -O2 -MT lessfs-lib_common.o -MD -MP -MF .deps/lessfs-lib_common.Tpo -c -o lessfs-lib_common.o `test -f 'lib_common.c' || echo './'`lib_common.c lib_common.c: In function ‘parseconfig’: lib_common.c:2492: error: ‘MHASH_TIGER192’ undeclared (first use in this function) lib_common.c:2492: error: (Each undeclared identifier is reported only once lib_common.c:2492: error: for each function it appears in.) lib_common.c:2540: error: ‘MHASH_WHIRLPOOL’ undeclared (first use in this function) lib_common.c:2548: error: ‘MHASH_SNEFRU256’ undeclared (first use in this function) make[1]: *** [lessfs-lib_common.o] Error 1 make[1]: Leaving directory `/root/lessfs-2.0.0-palpha2' make: *** [all] Error 2 |
|
From: David T. <da...@to...> - 2011-02-07 12:50:32
|
Dear i think i have a libmash issue i try to compile lessfs-1.3.3 i receive this compilation error : make[1]: Entering directory `/root/APP_LESSFS-2011-02-07-13' if gcc -DHAVE_CONFIG_H -I. -I. -I. -D_REENTRANT -D__USE_LARGEFILE64 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g -Wall -Wextra -O2 -MT lessfs-lib_hamster.o -MD -MP -MF ".deps/lessfs-lib_hamster.Tpo" -c -o lessfs-lib_hamster.o `test -f 'lib_hamster.c' || echo './'`lib_hamster.c; \ then mv -f ".deps/lessfs-lib_hamster.Tpo" ".deps/lessfs-lib_hamster.Po"; else rm -f ".deps/lessfs-lib_hamster.Tpo"; exit 1; fi if gcc -DHAVE_CONFIG_H -I. -I. -I. -D_REENTRANT -D__USE_LARGEFILE64 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Di686 -D SYSLOG -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/kerberos/include -g -Wall -Wextra -O2 -MT lessfs-lib_common.o -MD -MP -MF ".deps/lessfs-lib_common.Tpo" -c -o lessfs-lib_common.o `test -f 'lib_common.c' || echo './'`lib_common.c; \ then mv -f ".deps/lessfs-lib_common.Tpo" ".deps/lessfs-lib_common.Po"; else rm -f ".deps/lessfs-lib_common.Tpo"; exit 1; fi lib_common.c: In function ‘check_datafile_sanity’: lib_common.c:142: warning: comparison between signed and unsigned lib_common.c: In function ‘logiv’: lib_common.c:231: warning: unused parameter ‘msg’ lib_common.c:231: warning: unused parameter ‘bhash’ lib_common.c: In function ‘loghash’: lib_common.c:257: warning: unused parameter ‘msg’ lib_common.c:257: warning: unused parameter ‘bhash’ lib_common.c: In function ‘readBlock’: lib_common.c:1288: warning: comparison between signed and unsigned lib_common.c:1296: warning: cast from pointer to integer of different size lib_common.c:1312: warning: cast to pointer from integer of different size lib_common.c:1242: warning: unused parameter ‘filename’ lib_common.c: In function ‘update_filesize’: lib_common.c:1513: warning: comparison between signed and unsigned lib_common.c:1517: warning: comparison between signed and unsigned lib_common.c:1527: warning: comparison between signed and unsigned lib_common.c:1531: warning: comparison between signed and unsigned lib_common.c: In function ‘db_unlink_file’: lib_common.c:1624: warning: comparison between signed and unsigned lib_common.c: In function ‘tc_compress’: lib_common.c:1753: warning: comparison between signed and unsigned lib_common.c:1759: warning: comparison between signed and unsigned lib_common.c:1765: warning: comparison between signed and unsigned lib_common.c: In function ‘partial_truncate_block’: lib_common.c:1857: warning: unused variable ‘encrypted’ lib_common.c: In function ‘flush_wait’: lib_common.c:2138: warning: cast to pointer from integer of different size lib_common.c: In function ‘cook_cache’: lib_common.c:2156: warning: unused parameter ‘ksize’ lib_common.c: In function ‘purge_read_cache’: lib_common.c:2206: warning: cast to pointer from integer of different size lib_common.c:2187: warning: unused parameter ‘caller’ lib_common.c: In function ‘db_fs_truncate’: lib_common.c:2324: warning: unused parameter ‘bname’ lib_common.c: In function ‘fil_fuse_info’: lib_common.c:2435: warning: unused parameter ‘fi’ lib_common.c: In function ‘locks_to_dir’: lib_common.c:2451: warning: unused parameter ‘fi’ lib_common.c: In function ‘parseconfig’: lib_common.c:2919: error: ‘MHASH_TIGER192’ undeclared (first use in this function) lib_common.c:2919: error: (Each undeclared identifier is reported only once lib_common.c:2919: error: for each function it appears in.) lib_common.c:2967: error: ‘MHASH_WHIRLPOOL’ undeclared (first use in this function) lib_common.c:2975: error: ‘MHASH_SNEFRU256’ undeclared (first use in this function) make[1]: *** [lessfs-lib_common.o] Error 1 make[1]: Leaving directory `/root/APP_LESSFS-2011-02-07-13' make: *** [all] Error How can i fix it ? Best regards |
|
From: Emmanuel F. <ef...@in...> - 2010-10-06 11:47:08
|
I've built a small munin monitor for lessfs. It simply parses .lessfs/lessfs_stats to build a digest of used space, saved space and duplicate files. Find it here : http://update.intellique.com/pub/munin-lessfs-0.1.tgz Debian version : http://update.intellique.com/pub/munin-lessfs_0.1_all.deb Enjoy. -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <ef...@in...> | +33 1 78 94 84 02 ------------------------------------------------------------------------ |
