E_NOTIMPL for stdin / stdout pipe

Help
2011-04-19
2013-05-28
1 2 > >> (Page 1 of 2)
  • Hi everyone,

    I'm new to 7zip, and I'm trying to use it for database backups.  So far it's a perfect solution with the threading, but I'm having problems using it as a pipe.  My goal is to cpio a database snapshot into 7zip, then directly to curl for an FTP upload.

    Here's an example:

    find /myfiles | cpio -ov | 7za a dummy -m0=lzma2 -mx=4 -mmt=8 -md32m -si -so | curl -T -  ftp://my.ftpserver/path/file.cpio.7z

    When I do this, I get

    System error:
    E_NOTIMPL

    Is this really not implemented due to 7zip needing to seek in the archive file?  Is it possible to keep with the lzma2 compression standard and still create a compression pipe like this?

    Thanks for any help!

     
  • Igor Pavlov
    Igor Pavlov
    2011-04-19

    7za a -an -txz -si -so

     
  • Great, that appears to work!  What exactly do the parameters -an and -txz do?  I can't find those in the documentation.

    Also, does this change the compression format at all?  Running 'file' on an archive created without -so gives me:

    $ file test2.cpio.7z
    test2.cpio.7z: 7-zip archive data, version 0.3

    But using this command line:

    find /files  | /bin/cpio -ov  | 7za a -an -txz -m0=lzma2 -mx=4 -mmt=8 -md32m-si -so | curl -T - ftp://myserver/test.cpio.7z

    appears to create a different format:

    $ file test.cpio.7z
    test.cpio.7z: data

    Uncompressing it still works fine, so I'm just curious what the difference is.

    Thanks for the help!

     
  • Igor Pavlov
    Igor Pavlov
    2011-04-19

    it's not .7z format, it's xz:
    http://en.wikipedia.org/wiki/Xz

    -an - to disable archive name in command.
    -txz - sets .xz format

     
  • I see…  so xz if uses LZMA2 as well, is it just that the container format is a little different between xz and 7zip, even though the compression algorithm is the same?  I noticed that both output files end up being almost the same size.

     
  • Igor Pavlov
    Igor Pavlov
    2011-04-19

    .xz - is container for one file.
    .7z - is container for many files. and .7z creating requires seek operation. So you can't use it with -so.

     
  • Dmitry Katsubo
    Dmitry Katsubo
    2012-04-22

    It's strange that without -si it does not work…

     
  • Dmitry Katsubo
    Dmitry Katsubo
    2012-04-23

    Ah, I've missed that you want to get data from STDIN… -si blocks until some input happens. However without it 7z does not work:

    # 7za a -an -txz -so / > /dev/null
    Scanning

    Creating archive stdout

    System error:
    E_INVALIDARG

    # 7za a dummy -txz -so / > /dev/null
    Scanning

    Creating archive stdout

    System error:
    E_INVALIDARG

     
  • Igor Pavlov
    Igor Pavlov
    2012-04-24

    -txz format supports only one file per archive.
    "/" folder contains many files.
    Create one archive from that folder, for example .tar. Then use .xz.

     
  • frapas
    frapas
    2013-02-15

    Hi
    using the following syntax I get "System error: unspecified error" after a while:

    7z.exe -an -txz -so a  large_file.bak | curl.exe -k -upload-file - https://upload.mycompany.com/upload/large_file.xz

    large_file.bak is a 2GB that compress to 150MB

    whith smaller (for example 80 MB) file the same command works without problems!

    version: 7-Zip (A) 9.20  32 bit
    OS: windows 7 64 bit

    thanks

     
  • frapas
    frapas
    2013-02-15

    with:
    7z.exe 64 bit version on Windows 7 64 bit: same error
    in ubuntu linux: no problem !

    thanks

     
  • Igor Pavlov
    Igor Pavlov
    2013-02-15

    It's difficult to find the reason now.
    Maybe there is BUG in stdout mode writing code in 7-Zip.
    Do you have problems with
    7z.exe -an -txz -so a large_file.bak  > a.xz
    command?

     
  • frapas
    frapas
    2013-02-15

    Hi Igor,
    thank for the support.

    No there is no error, "Everything is Ok" with:

    7z.exe -an -txz -so a large_file.bak  > a.xz

     
  • frapas
    frapas
    2013-02-16

    I forgot to say that the same command, with the same file, works without any problem for  other archive types (gzip2 e gz tested).

    thanks

     
  • Igor Pavlov
    Igor Pavlov
    2013-02-16

    Please try more commands with curl:
    a -txz -mmt1
    a -tbzip2 -mmt1
    a -tgzip

    try also extracting commands with curl:
    e a.xz
    e a.gz
    e a.bz2

    Files for tests must be big.

     
  • frapas
    frapas
    2013-02-16

    Hi,
    with -mmt1 seems to works !

    7z -an -txz -mmt1  -so a  large_file.bak | curl -k -upload-file - https://upload.mycompany.com/upload/large_file.xz

    The upload is still running but I think it will finish without errors.
    Without -mmt1 it stop very soon.

    For my needs 1 thread is good, i have to use this command in servers and I don't want interfere causing high cpu load.

    thanks a lot !

     
  • Igor Pavlov
    Igor Pavlov
    2013-02-16

    I still want to fix the problem for any case.
    So try more commands:
    -txz -mmt2
    -txz -mmt4 -mx1
    -txz -mmt4 -mx3
    -txz -mmt4 -mx5
    -txz -mmt4 -mx7

     
  • frapas
    frapas
    2013-02-16

    OK… I immediatley test this cases.

    Meanwhile the upload (-mmt1) is completed .

    upload  with  -txz -mmt2 is in progress (seems to work)

     
  • frapas
    frapas
    2013-02-16

    update:

    -txz -mmt1             OK
    -txz -mmt2             upload in progress
    -txz -mmt4 -mx1   upload in progress
    -txz -mmt4 -mx3   upload in progress
    -txz -mmt4 -mx5   System error
    -txz -mmt4 -mx7   System error

     
  • Igor Pavlov
    Igor Pavlov
    2013-02-16

    more tests:
    -txz -mmt4 -mx5  -md16
    -txz -mmt4 -mx5  -md17
    -txz -mmt4 -mx5  -md18
    -txz -mmt4 -mx5  -md19
    -txz -mmt4 -mx5  -md20
    -txz -mmt4 -mx5  -md21
    -txz -mmt4 -mx5  -md22
    -txz -mmt4 -mx5  -md23
    -txz -mmt4 -mx5  -md24

     
  • frapas
    frapas
    2013-02-17

    Exuseme for the delay… the tests are a bit slow … I can post the results tomorow

    thanks

     
  • frapas
    frapas
    2013-02-17

    I'm repeating the tests, but I'm having random errors.
    At this point I suspect  that the problem is not  in 7zip but peraphs in web server timeout.
    I notice that if use -mmt1 switch, 7zip produce bytes in very fluid mode (the bytes arrive continuosly).
    Instead without the -mmt1 I notice 7zip doing pauses of several seconds in the output stream between a write and another

    With -mmtoff the output stream is fluid.
    With -mmt1 the output stream is fluid.
    With -mmt2 the output stream is fluid.
    With -mmt3 the output stream is fluid.
    With -mmt4 the output stream is NOT fluid.
    With -mmt5 the output stream is NOT fluid.
    With -mmt6 the output stream is NOT fluid.
    With -mmt7 the output stream is NOT fluid.
    With -mmt8 the output stream is NOT fluid.
    With -mmton the output stream is NOT fluid.

    I'm testing on a  Intel i7-2720QM  ( Quad-Core )

    Is this regular ?

    What type of switch do you suggest for piping the 7zip output to an upload process ?

    Thanks a lot !

     
  • Igor Pavlov
    Igor Pavlov
    2013-02-18

    In the following tests I try to find exact block size with errors:
    -txz -mmt4 -mx5  -md16
    -txz -mmt4 -mx5  -md17
    -txz -mmt4 -mx5  -md18
    -txz -mmt4 -mx5  -md19
    -txz -mmt4 -mx5  -md20
    -txz -mmt4 -mx5  -md21
    -txz -mmt4 -mx5  -md22
    -txz -mmt4 -mx5  -md23
    -txz -mmt4 -mx5  -md24

     
  • frapas
    frapas
    2013-02-18

    Hi,
    here are the results:

    test_.txt no switch..... System error 
    test_mt2................ OK
    test_mt4_x1............. OK 
    test_mt4_x3............. OK
    test_mt4_x5............. System error
    test_mt4_x5_md16........ OK
    test_mt4_x5_md17........ OK
    test_mt4_x5_md18........ OK
    test_mt4_x5_md19........ OK
    test_mt4_x5_md20........ OK
    test_mt4_x5_md21........ OK
    test_mt4_x5_md22........ OK
    test_mt4_x5_md23........ OK
    test_mt4_x5_md24........ System error
    test_mt4_x7............. Error: Can't allocate required memory!
    
     
  • frapas
    frapas
    2013-02-18

    nb:
    I discovered that the two errors are caused by web server timeout:

    test_mt4_x5…………. System error
    test_mt4_x5_md24…….. System error

    Apache error 408: Request time-out

     
1 2 > >> (Page 1 of 2)