7-zip output buffering

Jared B.
2010-05-10
2013-07-02
  • Jared B.
    Jared B.
    2010-05-10

    7-zip (or at least 7z) seems to buffer it's output.  When run normally this isn't a problem, but if piping the data through tee it can cause issues.  For example, when I run the following command:

    7z x testfile.7z | tee logfile.txt

    tee will display the output to stdout in addition to writing it to the log file.  However, since 7z buffers its output, nothing gets written until either the process is complete or sufficient data has been output to fill the buffer.  While this may be just an annoyance in most circumstances, it's particularly troublesome with the user is prompted for input: if I run the above command a second time, it'll prompt me to overwrite the existing files, however I'll never see that prompt because the prompt is in the output buffer.  As a result, I'll see no output at all and it'll appear as if 7z hung.

    Is there any way to disable this output buffer?  or perhaps force 7z to flush the buffer before a user prompt?  I've written a script that uses 7-Zip on the backend to handle file extraction, and I need the output from tee for debugging - it mostly works great, but being unable to prompt the user for overwrites, etc. is causing several problems.

    I've seen this behavior with multiple implementations of tee under both Windows and Linux, so I'm reasonably sure 7z is doing the buffering; I do not see the same behavior with other CLI utilities.  Any advice is most definitely appreciated.

    Thanks!

     
  • Igor Pavlov
    Igor Pavlov
    2010-05-11

    I'll fix that code in next version. Thanks!

     
  • Jared B.
    Jared B.
    2010-05-11

    Awesome, thanks Igor.  I'll keep an eye out for it.

     
  • Andrew
    Andrew
    2010-11-30

    Hi Igor, I just tried 7za version 9.20 and I believe it still seems to be buffering any idea when you can get around to a fix. Thank you very much.

     
  • Igor Pavlov
    Igor Pavlov
    2010-12-01

    I suppose I've fixed it. Check again.
    Check also case when it asks password.

     
  • pcworld
    pcworld
    2011-10-31

    It seems in the latest version (p7zip 9.20.1) only the case of interactive queries has been fixed, like if files exist already.
    I still get other messages (like "Extracting …") not until 7z is ready.
    Is this planned to be fixed?

    Also, which source files should I take a look for to implement the flushes myself?

     
  • pcworld
    pcworld
    2011-10-31

    I think I got this fixed. Patch file: http://pastebin.com/ZW87afGE
    There is similar code in Client7z.cpp, but that seems to not be used. Not sure if there's a patch needed as well.

    I think it would be reasonable to include this in the next release, as I think flushing after every line should be the expected behavior of 7z.

     
  • Any news on this issue. I use 7zip 9.20 and it's not fixed.

     
  • Igor Pavlov
    Igor Pavlov
    2012-10-08

    eugenec76:
    What cases do you need to disable buffering and why?

     
  • Igor, it's required if you run 7z in external environment, for example PeaZip, but needs to inform user about progress and/or redirect requests to user

    i would vote for adding special option for reducing amount of output for the cases when it's required to improve perfromance, and disable buffering of stdout. it may be even better to use stderr for interaction with the user, allowing to extract data to stdout, but it will be larger change

     
  • Igor Pavlov
    Igor Pavlov
    2012-10-08

    bulat:
    1) 7-Zip can extract data to stdout.
    2) Initial post was about output buffering for case with prompts for input. I suppose I've fixed it.  So if there is BUG for these cases, I want to know details.

     
  • 1. Igor, i mean that extracted data may interfere with service output
    2. yes, but then another user requests for disabling buffering at all, and i said you why it may be required

     
  • Igor Pavlov
    Igor Pavlov
    2012-10-08

    1. When 7-zip extracts data to stdout, it prints messages to stderr.
    2. But that user doesn't mention any motivations for that request.
    I just want to know motivations from user that requests some feature.

     
  • 2. i've translated motivation from one SREP user. finally, i've modified SREP to call setbuf(stderr,NULL)

     
  • Ipavlov I have a batch file that calls a second batch file and it's output is either redirected to a file or copied using tee.exe

    At the same time I want to be able to see the progress of 7zip and this can not be done due to buffering.

    I used a great number of command line utilities and they do not buffer the output. What's the purpose? The performance overhead is minimal.

    The best thing would be to add a switch to be able to disable the buffering.

     

  • Anonymous
    2012-10-12

    +1
    I use 7-zip in a GUI application and the progress (=stdout/stderr output) is shown in the GUI.
    I needed to patch 7-zip before I was able to do this…

     
  • Buzz
    Buzz
    2013-06-28

    Hi Igor,

    It seems that the latest version of 7z.exe still does not give output when redirected. So during compress operation if there are 5 files that are 100mb and you are redirecting the output of 7z.exe to text file it will not output anything until its finished. Is there any chance that a switch or something may be made available in future to force 7z.exe to flush output after ever line so its possible to tell more accurately its progress when its outputs redirected?

    Many thanks.

     
  • Igor Pavlov
    Igor Pavlov
    2013-06-29

    Maybe later.

     
  • Buzz
    Buzz
    2013-07-02

    Great, thank you.