[Squashfs-devel] New Parallel Mksquashfs available

[Phillip L. <phi...@gm...>]

Hi

I'm pleased to announce a new parallel version of Mksquashfs.  This is
currently available in CVS on Sourceforge in the Squashfs-tools directory
(in the directory par_mksquashfs).  The new parallel Mksquashfs builds as
par_mksquashfs, it will replace the current Mksquashfs in CVS if no
problems are reported.

Parallel Mksquashfs has the following improvements:

1. Parallel compression.  By default as many compression and fragment
compression threads are created as there are available processors.
This significantly speeds up performance on SMP systems.

2. File input and filesystem output is peformed in parallel on separate
threads to maximise I/O performance.  Even on single processor systems
this speeds up performance by at least 10%.

3. Appending has been significantly improved, and files within the
filesystem being appended to are no longer scanned and checksummed.  This
significantly improves append time for large filesystems.

4. File duplicate checking has been optimised, and split into two separate
phases.  Only files which are considered possible duplicates after the
first phase are checksummed and cached in memory.  As the use of swap
memory was found to significantly impact performance, the amount of
memory used to cache the file is now a command line option,  by default
this is 512 Mbytes.

The new par_mksquashfs program has a couple of extra command line options
which can be used to control the new features:

-processors <processors>

This specifies the number of processors used by par_mksquashfs.
By default this is the number of available processors.

-read_queue <size in Mbytes>

This specifies the size of the file input queue used by the reader thread.
This defaults to 64 Mbytes.

-write_queue <size in Mbytes>

This specifies the size of the filesystem output queue used by the
writer thread.  It also specifies the maximum cache used in file
duplicate detection (the output queue is shared between these tasks).
This defaults to 512 Mbytes.

Performance results:

The following results give an indication of the speed improvements.  Two
example filesystems were tested, a liveCD filesystem (about 1.8 Gbytes
uncompressed), and my home directory consisting largely of text files
(about 1.3 Gbytes uncompressed).  Tests were run on a single core
and a dual core system.

Dual Core (AMDx2 3800+) system:
Source directories on ext3.

LiveCD, old mksquashfs:

real    11m48.401s
user    9m27.056s
sys     0m15.281s

LiveCD, new par_mksquashfs:

real	4m8.736s
user	7m11.771s
sys	0m27.749s

"Home", old mksquashfs:

real	4m34.360s
user	3m54.007s
sys	0m32.155s

"Home", new par_mksquashfs:

real	1m27.381s
user	2m7.304s
sys	0m17.234s

Single Core PowerBook (PowerPC G4 1.5 GHz Ubuntu Linux)
Source directories on ext3.

LiveCD, old mksquashs:

real	11m38.472s
user	9m6.137s
sys	0m23.799s

LiveCD,  par_mksquashfs:

real	10m5.572s
user	8m59.921s
sys	0m16.145s

"Home", old mksquashfs:

real	3m42.298s
user	2m49.478s
sys	0m13.675s

"Home", new par_mksquashfs:

real	3m9.178s
user	2m50.699s
sys	0m9.069s


Please give the new par_mksquashfs a go!  I'll be interested in any
performance results obtained, especially from SMP machines larger than my
dual-core AMD box, as this will give an indication of the scalability of
the code.  Obviously, I'm also interested in any problems, deadlocks, low
performance etc.

Phillip Lougher