Portable multithreading for LZMA sdk

  • cJ

    cJ - 2007-07-27

    Hi devs,

    I'd like to know if some day we'll be able to enjoy multithreaded compression in linux/... which might be possible by switching from the current windows implementation of multithreading to a more portable threading library.

    Is such a work on the roadmap ?


    • Igor Pavlov

      Igor Pavlov - 2007-07-27

      Check p7zip.

      • cJ

        cJ - 2007-07-27

        p7zip can't do stream compression AFAIK

        • my space

          my space - 2007-07-27

          > p7zip can't do stream compression AFAIK

          in the p7zip package you can find contrib/gzip-like_CLI_wrapper_for_7z/p7zip.

          it's a wrapper around 7za that produce .7z archive but behaves like gzip/bzip2

          So, you can do :

          > echo "foobar" | p7zip | p7zip -d

          Is it what you wanted ?

          • cJ

            cJ - 2007-07-27

            This wrapper seems to create a temporary file, it contains :
            7z a ${tmp} -si >/dev/null
            cat ${tmp}

            And 7zip / p7zip can't use both -si and -so commands at the same time, since archive creation needs to seek in the created archive.

            At the contrary lzma compressor can.

            The lzma executable, provided by the lzma sdk, isn't multithreaded, but you can use it as a stream compressor without having temp files, which is a good point (behaves like bzip2 or gzip or compress...) :
            cat mybigfile | lzma e -si -so > mybigfile.lzma

            If lzma could be multithreaded it would be perfect.
            That's the point of my post : the lzma sdk seems to have multithreading code (correct me if I'm wrong) but it is windows-only code, so I'm thinking about using more portable thread libraries...

            • Igor Pavlov

              Igor Pavlov - 2007-07-28

              To myspace:
              Can lzma (LZMA_Alone) from p7zip use two threads as lzma for windows?

              • my space

                my space - 2007-07-30

                > Can lzma (LZMA_Alone) from p7zip use two threads as lzma for windows?

                I made a mistake in the makefile, so lzma in p7zip does not use 2 threads ...

                You can wait for the next release,
                or you can follow this procedure :

                1) cd CPP\7zip\Compress\LZMA_Alone

                2) edit makefile

                3) replace "-DBENCH_MT" with "-DCOMPRESS_MF_MT -DBENCH_MT"

                4) replace
                MatchFinder.o \ with
                MatchFinder.o \ MatchFinderMt.o \

                5) add
                MatchFinderMt.o : ../../../../C/Compress/Lz/MatchFinderMt.c
                    $(CC) $(CFLAGS) ../../../../C/Compress/Lz/MatchFinderMt.c
                Remark : as this forum does not keep the format, please add "tab" before $(CC)

                6) make clean

                7) make

                Then benchmark :

                ./lzma b -mt1

                ./lzma b -mt2

                • cJ

                  cJ - 2007-07-31

                  At first I thought you were talking about LZMA sdk files, but I've mistaken.
                  I didn't know that p7zip contained completely different files, at least I thought the lzma library was identical...

                  Well, building the program is OK, thank you !

                  If I don't reply, it means that everything is OK ;)

                  I think that it's great to have such a powerful and multithreaded stream compressor... lzma is very very good.
                  Thank you Igor for inventing it, and myspace, for the port !


Log in to post a comment.