7-zip (or at least 7z) seems to buffer it's output. When run normally this isn't a problem, but if piping the data through tee it can cause issues. For example, when I run the following command:
7z x testfile.7z | tee logfile.txt
tee will display the output to stdout in addition to writing it to the log file. However, since 7z buffers its output, nothing gets written until either the process is complete or sufficient data has been output to fill the buffer. While this may be just an annoyance in most circumstances, it's particularly troublesome with the user is prompted for input: if I run the above command a second time, it'll prompt me to overwrite the existing files, however I'll never see that prompt because the prompt is in the output buffer. As a result, I'll see no output at all and it'll appear as if 7z hung.
Is there any way to disable this output buffer? or perhaps force 7z to flush the buffer before a user prompt? I've written a script that uses 7-Zip on the backend to handle file extraction, and I need the output from tee for debugging - it mostly works great, but being unable to prompt the user for overwrites, etc. is causing several problems.
I've seen this behavior with multiple implementations of tee under both Windows and Linux, so I'm reasonably sure 7z is doing the buffering; I do not see the same behavior with other CLI utilities. Any advice is most definitely appreciated.
Thanks!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It seems in the latest version (p7zip 9.20.1) only the case of interactive queries has been fixed, like if files exist already.
I still get other messages (like "Extracting …") not until 7z is ready.
Is this planned to be fixed?
Also, which source files should I take a look for to implement the flushes myself?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think I got this fixed. Patch file: http://pastebin.com/ZW87afGE
There is similar code in Client7z.cpp, but that seems to not be used. Not sure if there's a patch needed as well.
I think it would be reasonable to include this in the next release, as I think flushing after every line should be the expected behavior of 7z.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Igor, it's required if you run 7z in external environment, for example PeaZip, but needs to inform user about progress and/or redirect requests to user
i would vote for adding special option for reducing amount of output for the cases when it's required to improve perfromance, and disable buffering of stdout. it may be even better to use stderr for interaction with the user, allowing to extract data to stdout, but it will be larger change
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
bulat:
1) 7-Zip can extract data to stdout.
2) Initial post was about output buffering for case with prompts for input. I suppose I've fixed it. So if there is BUG for these cases, I want to know details.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1. Igor, i mean that extracted data may interfere with service output
2. yes, but then another user requests for disabling buffering at all, and i said you why it may be required
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1. When 7-zip extracts data to stdout, it prints messages to stderr.
2. But that user doesn't mention any motivations for that request.
I just want to know motivations from user that requests some feature.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ipavlov I have a batch file that calls a second batch file and it's output is either redirected to a file or copied using tee.exe
At the same time I want to be able to see the progress of 7zip and this can not be done due to buffering.
I used a great number of command line utilities and they do not buffer the output. What's the purpose? The performance overhead is minimal.
The best thing would be to add a switch to be able to disable the buffering.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2012-10-12
+1
I use 7-zip in a GUI application and the progress (=stdout/stderr output) is shown in the GUI.
I needed to patch 7-zip before I was able to do this…
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It seems that the latest version of 7z.exe still does not give output when redirected. So during compress operation if there are 5 files that are 100mb and you are redirecting the output of 7z.exe to text file it will not output anything until its finished. Is there any chance that a switch or something may be made available in future to force 7z.exe to flush output after ever line so its possible to tell more accurately its progress when its outputs redirected?
Many thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
7-zip (or at least 7z) seems to buffer it's output. When run normally this isn't a problem, but if piping the data through tee it can cause issues. For example, when I run the following command:
7z x testfile.7z | tee logfile.txt
tee will display the output to stdout in addition to writing it to the log file. However, since 7z buffers its output, nothing gets written until either the process is complete or sufficient data has been output to fill the buffer. While this may be just an annoyance in most circumstances, it's particularly troublesome with the user is prompted for input: if I run the above command a second time, it'll prompt me to overwrite the existing files, however I'll never see that prompt because the prompt is in the output buffer. As a result, I'll see no output at all and it'll appear as if 7z hung.
Is there any way to disable this output buffer? or perhaps force 7z to flush the buffer before a user prompt? I've written a script that uses 7-Zip on the backend to handle file extraction, and I need the output from tee for debugging - it mostly works great, but being unable to prompt the user for overwrites, etc. is causing several problems.
I've seen this behavior with multiple implementations of tee under both Windows and Linux, so I'm reasonably sure 7z is doing the buffering; I do not see the same behavior with other CLI utilities. Any advice is most definitely appreciated.
Thanks!
I'll fix that code in next version. Thanks!
Awesome, thanks Igor. I'll keep an eye out for it.
Hi Igor, I just tried 7za version 9.20 and I believe it still seems to be buffering any idea when you can get around to a fix. Thank you very much.
I suppose I've fixed it. Check again.
Check also case when it asks password.
It seems in the latest version (p7zip 9.20.1) only the case of interactive queries has been fixed, like if files exist already.
I still get other messages (like "Extracting …") not until 7z is ready.
Is this planned to be fixed?
Also, which source files should I take a look for to implement the flushes myself?
I think I got this fixed. Patch file: http://pastebin.com/ZW87afGE
There is similar code in Client7z.cpp, but that seems to not be used. Not sure if there's a patch needed as well.
I think it would be reasonable to include this in the next release, as I think flushing after every line should be the expected behavior of 7z.
Any news on this issue. I use 7zip 9.20 and it's not fixed.
eugenec76:
What cases do you need to disable buffering and why?
Igor, it's required if you run 7z in external environment, for example PeaZip, but needs to inform user about progress and/or redirect requests to user
i would vote for adding special option for reducing amount of output for the cases when it's required to improve perfromance, and disable buffering of stdout. it may be even better to use stderr for interaction with the user, allowing to extract data to stdout, but it will be larger change
bulat:
1) 7-Zip can extract data to stdout.
2) Initial post was about output buffering for case with prompts for input. I suppose I've fixed it. So if there is BUG for these cases, I want to know details.
1. Igor, i mean that extracted data may interfere with service output
2. yes, but then another user requests for disabling buffering at all, and i said you why it may be required
1. When 7-zip extracts data to stdout, it prints messages to stderr.
2. But that user doesn't mention any motivations for that request.
I just want to know motivations from user that requests some feature.
2. i've translated motivation from one SREP user. finally, i've modified SREP to call setbuf(stderr,NULL)
Ipavlov I have a batch file that calls a second batch file and it's output is either redirected to a file or copied using tee.exe
At the same time I want to be able to see the progress of 7zip and this can not be done due to buffering.
I used a great number of command line utilities and they do not buffer the output. What's the purpose? The performance overhead is minimal.
The best thing would be to add a switch to be able to disable the buffering.
+1
I use 7-zip in a GUI application and the progress (=stdout/stderr output) is shown in the GUI.
I needed to patch 7-zip before I was able to do this…
Hi Igor,
It seems that the latest version of 7z.exe still does not give output when redirected. So during compress operation if there are 5 files that are 100mb and you are redirecting the output of 7z.exe to text file it will not output anything until its finished. Is there any chance that a switch or something may be made available in future to force 7z.exe to flush output after ever line so its possible to tell more accurately its progress when its outputs redirected?
Many thanks.
Maybe later.
Great, thank you.