I'm new to 7zip, and I'm trying to use it for database backups. So far it's a perfect solution with the threading, but I'm having problems using it as a pipe. My goal is to cpio a database snapshot into 7zip, then directly to curl for an FTP upload.
Is this really not implemented due to 7zip needing to seek in the archive file? Is it possible to keep with the lzma2 compression standard and still create a compression pipe like this?
Thanks for any help!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I see… so xz if uses LZMA2 as well, is it just that the container format is a little different between xz and 7zip, even though the compression algorithm is the same? I noticed that both output files end up being almost the same size.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It's difficult to find the reason now.
Maybe there is BUG in stdout mode writing code in 7-Zip.
Do you have problems with
7z.exe -an -txz -so a large_file.bak > a.xz
command?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
-txz -mmt1 OK
-txz -mmt2 upload in progress
-txz -mmt4 -mx1 upload in progress
-txz -mmt4 -mx3 upload in progress
-txz -mmt4 -mx5 System error
-txz -mmt4 -mx7 System error
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm repeating the tests, but I'm having random errors.
At this point I suspect that the problem is not in 7zip but peraphs in web server timeout.
I notice that if use -mmt1 switch, 7zip produce bytes in very fluid mode (the bytes arrive continuosly).
Instead without the -mmt1 I notice 7zip doing pauses of several seconds in the output stream between a write and another
With -mmtoff the output stream is fluid.
With -mmt1 the output stream is fluid.
With -mmt2 the output stream is fluid.
With -mmt3 the output stream is fluid.
With -mmt4 the output stream is NOT fluid.
With -mmt5 the output stream is NOT fluid.
With -mmt6 the output stream is NOT fluid.
With -mmt7 the output stream is NOT fluid.
With -mmt8 the output stream is NOT fluid.
With -mmton the output stream is NOT fluid.
I'm testing on a Intel i7-2720QM ( Quad-Core )
Is this regular ?
What type of switch do you suggest for piping the 7zip output to an upload process ?
Thanks a lot !
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
test_.txt no switch..... System error
test_mt2................ OK
test_mt4_x1............. OK
test_mt4_x3............. OK
test_mt4_x5............. System error
test_mt4_x5_md16........ OK
test_mt4_x5_md17........ OK
test_mt4_x5_md18........ OK
test_mt4_x5_md19........ OK
test_mt4_x5_md20........ OK
test_mt4_x5_md21........ OK
test_mt4_x5_md22........ OK
test_mt4_x5_md23........ OK
test_mt4_x5_md24........ System error
test_mt4_x7............. Error: Can't allocate required memory!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi everyone,
I'm new to 7zip, and I'm trying to use it for database backups. So far it's a perfect solution with the threading, but I'm having problems using it as a pipe. My goal is to cpio a database snapshot into 7zip, then directly to curl for an FTP upload.
Here's an example:
find /myfiles | cpio -ov | 7za a dummy -m0=lzma2 -mx=4 -mmt=8 -md32m -si -so | curl -T - ftp://my.ftpserver/path/file.cpio.7z
When I do this, I get
System error:
E_NOTIMPL
Is this really not implemented due to 7zip needing to seek in the archive file? Is it possible to keep with the lzma2 compression standard and still create a compression pipe like this?
Thanks for any help!
7za a -an -txz -si -so
Great, that appears to work! What exactly do the parameters -an and -txz do? I can't find those in the documentation.
Also, does this change the compression format at all? Running 'file' on an archive created without -so gives me:
$ file test2.cpio.7z
test2.cpio.7z: 7-zip archive data, version 0.3
But using this command line:
find /files | /bin/cpio -ov | 7za a -an -txz -m0=lzma2 -mx=4 -mmt=8 -md32m-si -so | curl -T - ftp://myserver/test.cpio.7z
appears to create a different format:
$ file test.cpio.7z
test.cpio.7z: data
Uncompressing it still works fine, so I'm just curious what the difference is.
Thanks for the help!
it's not .7z format, it's xz:
http://en.wikipedia.org/wiki/Xz
-an - to disable archive name in command.
-txz - sets .xz format
I see… so xz if uses LZMA2 as well, is it just that the container format is a little different between xz and 7zip, even though the compression algorithm is the same? I noticed that both output files end up being almost the same size.
.xz - is container for one file.
.7z - is container for many files. and .7z creating requires seek operation. So you can't use it with -so.
It's strange that without -si it does not work…
Ah, I've missed that you want to get data from STDIN… -si blocks until some input happens. However without it 7z does not work:
Creating archive stdout
System error:
E_INVALIDARG
# 7za a dummy -txz -so / > /dev/null
Scanning
Creating archive stdout
System error:
E_INVALIDARG
-txz format supports only one file per archive.
"/" folder contains many files.
Create one archive from that folder, for example .tar. Then use .xz.
Hi
using the following syntax I get "System error: unspecified error" after a while:
7z.exe -an -txz -so a large_file.bak | curl.exe -k -upload-file - https://upload.mycompany.com/upload/large_file.xz
large_file.bak is a 2GB that compress to 150MB
whith smaller (for example 80 MB) file the same command works without problems!
version: 7-Zip (A) 9.20 32 bit
OS: windows 7 64 bit
thanks
with:
7z.exe 64 bit version on Windows 7 64 bit: same error
in ubuntu linux: no problem !
thanks
It's difficult to find the reason now.
Maybe there is BUG in stdout mode writing code in 7-Zip.
Do you have problems with
7z.exe -an -txz -so a large_file.bak > a.xz
command?
Hi Igor,
thank for the support.
No there is no error, "Everything is Ok" with:
7z.exe -an -txz -so a large_file.bak > a.xz
I forgot to say that the same command, with the same file, works without any problem for other archive types (gzip2 e gz tested).
thanks
Please try more commands with curl:
a -txz -mmt1
a -tbzip2 -mmt1
a -tgzip
try also extracting commands with curl:
e a.xz
e a.gz
e a.bz2
Files for tests must be big.
Hi,
with -mmt1 seems to works !
7z -an -txz -mmt1 -so a large_file.bak | curl -k -upload-file - https://upload.mycompany.com/upload/large_file.xz
The upload is still running but I think it will finish without errors.
Without -mmt1 it stop very soon.
For my needs 1 thread is good, i have to use this command in servers and I don't want interfere causing high cpu load.
thanks a lot !
I still want to fix the problem for any case.
So try more commands:
-txz -mmt2
-txz -mmt4 -mx1
-txz -mmt4 -mx3
-txz -mmt4 -mx5
-txz -mmt4 -mx7
OK… I immediatley test this cases.
Meanwhile the upload (-mmt1) is completed .
upload with -txz -mmt2 is in progress (seems to work)
update:
-txz -mmt1 OK
-txz -mmt2 upload in progress
-txz -mmt4 -mx1 upload in progress
-txz -mmt4 -mx3 upload in progress
-txz -mmt4 -mx5 System error
-txz -mmt4 -mx7 System error
more tests:
-txz -mmt4 -mx5 -md16
-txz -mmt4 -mx5 -md17
-txz -mmt4 -mx5 -md18
-txz -mmt4 -mx5 -md19
-txz -mmt4 -mx5 -md20
-txz -mmt4 -mx5 -md21
-txz -mmt4 -mx5 -md22
-txz -mmt4 -mx5 -md23
-txz -mmt4 -mx5 -md24
Exuseme for the delay… the tests are a bit slow … I can post the results tomorow
thanks
I'm repeating the tests, but I'm having random errors.
At this point I suspect that the problem is not in 7zip but peraphs in web server timeout.
I notice that if use -mmt1 switch, 7zip produce bytes in very fluid mode (the bytes arrive continuosly).
Instead without the -mmt1 I notice 7zip doing pauses of several seconds in the output stream between a write and another
With -mmtoff the output stream is fluid.
With -mmt1 the output stream is fluid.
With -mmt2 the output stream is fluid.
With -mmt3 the output stream is fluid.
With -mmt4 the output stream is NOT fluid.
With -mmt5 the output stream is NOT fluid.
With -mmt6 the output stream is NOT fluid.
With -mmt7 the output stream is NOT fluid.
With -mmt8 the output stream is NOT fluid.
With -mmton the output stream is NOT fluid.
I'm testing on a Intel i7-2720QM ( Quad-Core )
Is this regular ?
What type of switch do you suggest for piping the 7zip output to an upload process ?
Thanks a lot !
In the following tests I try to find exact block size with errors:
-txz -mmt4 -mx5 -md16
-txz -mmt4 -mx5 -md17
-txz -mmt4 -mx5 -md18
-txz -mmt4 -mx5 -md19
-txz -mmt4 -mx5 -md20
-txz -mmt4 -mx5 -md21
-txz -mmt4 -mx5 -md22
-txz -mmt4 -mx5 -md23
-txz -mmt4 -mx5 -md24
Hi,
here are the results:
nb:
I discovered that the two errors are caused by web server timeout:
test_mt4_x5…………. System error
test_mt4_x5_md24…….. System error
Apache error 408: Request time-out