Recent changes to Home

Home modified by Josef Ezra

Josef Ezra — Tue, 07 Jan 2020 22:06:48 -0000

--- v8
+++ v9
@@ -11,7 +11,6 @@
 Usage
 -----
 
-
 % slimfastq *file.fastq* *new-file.sfq*   : compress *file.fastq* to *new-file.sfq*
 % slimfastq -1 *file.fastq* *new-file.sfq*: compress *file.fastq* to *new-file.sfq*, using little CPU/memory resources
                                            (-1 to -4 are levels of compression/resources trade-offs, -3 is default)
@@ -24,7 +23,6 @@
 pipe usage:
 % gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
 % slimfastq *file.sfq* | md5sum -                    : get the checksum of the decompressed file without creating a file
-
 

 The multi threads FAQ
@@ -52,12 +50,16 @@
 "make test" will compress, decompress and compare all the fastq files in the ./samples dir.
 Some testing tips to check your own fastq files:
 1) slimfastq is a lossless compression and posix pipes friendly, therefore it's easy to check the integrity of a large file with checksums:
+
 % md5sum large-file.fq
 % ./slimfastq largefile.fq -O /tmp/tst.sfq
 % ./slimfastq /tmp/tst.sfq | md5sum -
+
 (if md5sum don't match, one can use ./tools/mydiff.pl to the bad line. And I'll be grateful for a bug report)
 2) use time. Example:
+
 % /usr/bin/time -f  " IO : io=%I faults=%F\n MEM: max=%M kb Average=%K kb\n CPU: Percentage=%P real=%e sys=%S user=%U"  slimfastq large-file.fq /tmp/a.tst -O
+
 3) Performance wise, a single file compression/decompression is not very interesting. The script tools/slimfastq.multi can be used to evaluate performance of
 concurrent files compression/decompression. This script can be edited to use with other compression softwares - general or fastq specific.
 4) Using slimfastq.multi, try to increase/decrease thread count to find the optimal number for a specific system.

Home modified by Josef Ezra

Josef Ezra — Tue, 07 Jan 2020 22:05:12 -0000

--- v7
+++ v8
@@ -10,7 +10,7 @@

 Usage
 -----
-+

 % slimfastq *file.fastq* *new-file.sfq*   : compress *file.fastq* to *new-file.sfq*
 % slimfastq -1 *file.fastq* *new-file.sfq*: compress *file.fastq* to *new-file.sfq*, using little CPU/memory resources
@@ -25,7 +25,7 @@
 % gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
 % slimfastq *file.sfq* | md5sum -                    : get the checksum of the decompressed file without creating a file

-

+

 The multi threads FAQ
 ---------------------

Home modified by Josef Ezra

Josef Ezra — Tue, 07 Jan 2020 22:03:15 -0000

--- v6
+++ v7
@@ -31,8 +31,9 @@
 ---------------------
 The main reason slimfastq is a single thread application is to avoid the overhead of semaphores, L2 flushes, and context
 switches. The goal is to focus on the speed of N files compression/decompression instead of a single file.
-...
-
+Use slimfastq.multi script (located under tools/) to compress/decompress multiple files in parallel. The '-h' argument, as
+expected, will provide help. This script can be easily edited for a sepcial setup. Please do not hesitate to email me if
+any help is needed.

 Compile
 -------
@@ -46,7 +47,7 @@
 * Recompile with optimization flags for the profiler generated data (use 'make slimfastq.opt').
 (For whatever it's worth, the author of this page could not notice any significant performance change yield by the optimized compilation.)

-Testing
+Test
 -------
 "make test" will compress, decompress and compare all the fastq files in the ./samples dir.
 Some testing tips to check your own fastq files:

Home modified by Josef Ezra

Josef Ezra — Tue, 07 Jan 2020 22:00:44 -0000

--- v5
+++ v6
@@ -23,7 +23,7 @@

 pipe usage:
 % gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
-% slimfastq *file.sfq* | md5sum -                    : get the checksum of the decompressed file without create a file
+% slimfastq *file.sfq* | md5sum -                    : get the checksum of the decompressed file without creating a file

Home modified by Josef Ezra

Josef Ezra — Tue, 07 Jan 2020 21:59:58 -0000

--- v4
+++ v5
@@ -1,48 +1,71 @@
-
 slimfastq
 =========

-Welcome to slimfastq wiki page!
-----
-
-slimfastq is a cli application that compresses/decompresses fastq files (a raw product of genome sequencing). 
-
-If you wish to download only the core essential to build slimfastq please download the latest release.
-
-If you wish to read/alter the sources it is recommended to checkout the master repository via git. This repository is kept stable and contain a variety of fastq samples for testing. 
-
-
-Slimfastq features
------
+slimfastq would efficiently compresses/decompresses fastq files. It features:

 * High compression ratio
-* High compression Speed
-* Relatively low cpu and memory usage
+* Relatively low CPU, memory usage
 * Truly lossless compression/decompression
-
+* Posix piping friendly (i.e. fastq input/output steam is serialzed during compression/decompression)

 Usage
 -----
++% slimfastq *file.fastq* *new-file.sfq*   : compress *file.fastq* to *new-file.sfq*
+% slimfastq -1 *file.fastq* *new-file.sfq*: compress *file.fastq* to *new-file.sfq*, using little CPU/memory resources
+                                           (-1 to -4 are levels of compression/resources trade-offs, -3 is default)

-    % slimfastq *file.fastq* *new-file.sfq*   : compress *file.fastq* to *new-file.sfq* 
-    % slimfastq -1 *file.fastq* *new-file.sfq*: compress *file.fastq* to *new-file.sfq*, using little cpu/memory resources
-                                           (-1 to -4 are levels of compression/resources tradeoffs, -3 is default)
-    
-    % slimfastq *file.sfq*                    : decompress *file.sfq* to stdout (format is determined by stamp, not name)
-    % slimfastq *file.sfq* *file.fastq        : decompress *file.sfq* to *file.fastq*
-    
-    % slimfastq -h                            : get help
-    
-    % gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
+% slimfastq *file.sfq*                    : decompress *file.sfq* to stdout (format is determined by stamp, not name)
+% slimfastq *file.sfq* *file.fastq        : decompress *file.sfq* to *file.fastq*
+
+% slimfastq -h                            : get help
+
+pipe usage:
+% gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
+% slimfastq *file.sfq* | md5sum -                    : get the checksum of the decompressed file without create a file

 

+The multi threads FAQ

+---------------------

+The main reason slimfastq is a single thread application is to avoid the overhead of semaphores, L2 flushes, and context

+switches. The goal is to focus on the speed of N files compression/decompression instead of a single file.

+...

+

+

+Compile

+-------

+Simple compilation:

+ run "make" or "make test"

+(If the compilation fails, please let me know).

+

+Profile optimized compilation:

+ Compile slimfastq in a gcc profile generator mode (use 'make slimfastq.prof').

+ Compress/decompress some of your fastq files - this will generate some .gcda files.

+* Recompile with optimization flags for the profiler generated data (use 'make slimfastq.opt').

+(For whatever it's worth, the author of this page could not notice any significant performance change yield by the optimized compilation.)

+

+Testing

+-------

+"make test" will compress, decompress and compare all the fastq files in the ./samples dir.

+Some testing tips to check your own fastq files:

+1) slimfastq is a lossless compression and posix pipes friendly, therefore it's easy to check the integrity of a large file with checksums:

+% md5sum large-file.fq

+% ./slimfastq largefile.fq -O /tmp/tst.sfq

+% ./slimfastq /tmp/tst.sfq | md5sum -

+(if md5sum don't match, one can use ./tools/mydiff.pl to the bad line. And I'll be grateful for a bug report)

+2) use time. Example:

+% /usr/bin/time -f  " IO : io=%I faults=%F\n MEM: max=%M kb Average=%K kb\n CPU: Percentage=%P real=%e sys=%S user=%U"  slimfastq large-file.fq /tmp/a.tst -O

+3) Performance wise, a single file compression/decompression is not very interesting. The script tools/slimfastq.multi can be used to evaluate performance of

+concurrent files compression/decompression. This script can be edited to use with other compression softwares - general or fastq specific.

+4) Using slimfastq.multi, try to increase/decrease thread count to find the optimal number for a specific system.
Install

-Compile via 'make', copy the executable "slimfastq" to any location.
+After compile

+ run "sudo make install"

+ Alternatively to "make install", copy the "slimfastq" executable and the "tools/slimfastq.multi" script to any location.
License

@@ -50,20 +73,10 @@
Platform

-slimfastq was developed and optimized for x86_64 GNU/Linux. For other system's support requests, please contact us. 

+slimfastq was developed and optimized for x86_64 GNU/Linux and Darwin OS. For other system's support requests, please contact Josef Ezra.
Contact

Josef Ezra  (jezra at infinidat.com), (jezra at cpan.org)

-
Project Members:
	
		James Bonfield
Josef Ezra (admin)
		
	
-
------
-Notes
-Version 1.03: Fixed a bug when compressed data exceed 4GB on certain systems. (Special thanks to Christopher Black!)

Home modified by Josef Ezra

Josef Ezra — Thu, 30 Oct 2014 12:56:31 -0000

--- v3
+++ v4
@@ -1,16 +1,60 @@
+
+slimfastq
+=========
+
 Welcome to slimfastq wiki page!
+----

-This cli application can be used to compress fastq files (a raw product of genome sequencing). 
+slimfastq is a cli application that compresses/decompresses fastq files (a raw product of genome sequencing). 

 If you wish to download only the core essential to build slimfastq please download the latest release.

 If you wish to read/alter the sources it is recommended to checkout the master repository via git. This repository is kept stable and contain a variety of fastq samples for testing. 

+
+Slimfastq features
 -----

-*Notes*
+* High compression ratio
+* High compression Speed
+* Relatively low cpu and memory usage
+* Truly lossless compression/decompression

-**Version 1.03**: Fixed a bug when compressed data exceed 4GB on certain systems. (Special thanks to Christopher Black!) 
+
+Usage
+-----
+
+
+    % slimfastq *file.fastq* *new-file.sfq*   : compress *file.fastq* to *new-file.sfq* 
+    % slimfastq -1 *file.fastq* *new-file.sfq*: compress *file.fastq* to *new-file.sfq*, using little cpu/memory resources
+                                           (-1 to -4 are levels of compression/resources tradeoffs, -3 is default)
+    
+    % slimfastq *file.sfq*                    : decompress *file.sfq* to stdout (format is determined by stamp, not name)
+    % slimfastq *file.sfq* *file.fastq        : decompress *file.sfq* to *file.fastq*
+    
+    % slimfastq -h                            : get help
+    
+    % gzip -dc *file.fastq.gz* | slimfastq -f *file.sfq* : convert from gzip to sfq format (and save a lot of disk space)
+
+

+
+
+Install
+-------
+Compile via 'make', copy the executable "slimfastq" to any location.
+
+
+License
+-------
+The BSD 3-Clause
+
+Platform
+--------
+slimfastq was developed and optimized for x86_64 GNU/Linux. For other system's support requests, please contact us.
+
+Contact
+-------
+Josef Ezra (jezra at infinidat.com), (jezra at cpan.org)

@@ -18,3 +62,8 @@

+-----
+
+Notes
+
+Version 1.03: Fixed a bug when compressed data exceed 4GB on certain systems. (Special thanks to Christopher Black!)

Home modified by Josef Ezra

Josef Ezra — Sun, 29 Jun 2014 22:56:25 -0000

--- v2
+++ v3
@@ -1,22 +1,20 @@
-Welcome to your slimfastq wiki page!
+Welcome to slimfastq wiki page!

 This cli application can be used to compress fastq files (a raw product of genome sequencing). 

-If you wish to download only the core essential to build slimfastq please use:
-[[download_button]]
+If you wish to download only the core essential to build slimfastq please download the latest release.

 If you wish to read/alter the sources it is recommended to checkout the master repository via git. This repository is kept stable and contain a variety of fastq samples for testing. 

 -----

-News
-----
+*Notes*

-**Version 1.03**: Fixed a bug when compressed data exceed 4GB on certain systems. (Thank you Chris Black!) 
+**Version 1.03**: Fixed a bug when compressed data exceed 4GB on certain systems. (Special thanks to Christopher Black!) 

 -----

 [[members limit=20]]

+[[download_button]]

-The wiki uses [Markdown](/p/slimfastq/wiki/markdown_syntax/) syntax.

Home modified by Josef Ezra

Josef Ezra — Sun, 29 Jun 2014 22:51:50 -0000

--- v1
+++ v2
@@ -1,8 +1,22 @@
-Welcome to your wiki!
+Welcome to your slimfastq wiki page!

-This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].
+This cli application can be used to compress fastq files (a raw product of genome sequencing). 
+
+If you wish to download only the core essential to build slimfastq please use:
+[[download_button]]
+
+If you wish to read/alter the sources it is recommended to checkout the master repository via git. This repository is kept stable and contain a variety of fastq samples for testing. 
+
+-----
+
+News
+----
+
+**Version 1.03**: Fixed a bug when compressed data exceed 4GB on certain systems. (Thank you Chris Black!) 
+
+-----
+
+[[members limit=20]]
+

 The wiki uses [Markdown](/p/slimfastq/wiki/markdown_syntax/) syntax.
-
-[[members limit=20]]
-[[download_button]]

Home modified by Josef Ezra

Josef Ezra — Thu, 12 Dec 2013 13:03:45 -0000

Welcome to your wiki!

This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].

The wiki uses Markdown syntax.

Project Members:

Josef Ezra (admin)