Re: [Kaldi-users] -u flag in dnn python scripts

[Karel V. <ive...@fi...>]

Hi,
I committed updated version of python scripts which is using buffering 
(no -u flag).
Perhaps I put it there when I was developing the scripts while debugging 
and forgot
to remove it later on. Many thanks for pointing it out.
Karel


Dne 2. 12. 2013 18:30, Daniel Povey napsal(a):
> Thanks a lot for pointing this out.
> Cc'ing Karel who maintains this part.
> Dan
>
>
> On Mon, Dec 2, 2013 at 3:39 AM, Josef Novak
> <jos...@gm...> wrote:
>> Hi,
>>
>>    I have been digging around a bit in the dnn-related python scripts and had
>> a question.
>>
>>    All of the the python scripts are called with the -u flag.  This means
>> 'use unbuffered IO', which in turn means that every single byte is flushed
>> to disk.
>>
>>    I noticed this because utils/gen_rbm_init.py seemed to be taking an
>> inordinate amount of time (1+ hours on the WSJ for each call to the script).
>>
>>    In the case where all the data is on the local drive, and particularly if
>> the local drive is an SSD, this would barely be noticeable - it is about 4x
>> slower.  But any kind of network latency massively magnifies the penalty.
>> In my case, with a locally networked drive, the difference was:
>> ----------------------------------------------------
>> #!/usr/bin/python -u:  1+ hours to write each N.rbm.init file
>> #!/usr/bin/python:      10 seconds to write each N.rbm.init file
>> ----------------------------------------------------
>>    my understanding is that the -u flag also suppresses automatic newline
>> conversion in 2.x - ensuring that Windows/Linux/OSX results would be
>> equivalent/interoperable.  This was the only reason I could come up with for
>> why one might want to do this.  However the potential time penalty for this
>> is pretty massive, so I thought it worth pointing out.
>>
>> http://docs.python.org/2/using/cmdline.html#cmdoption-u
>>
>> http://docs.python.org/2/glossary.html#term-universal-newlines
>>
>>    If there is some other motivation for using this flag I would also be
>> curious to know what it might be.
>>
>>    Best regards,
>>      Joe
>>
>> ------------------------------------------------------------------------------
>> Rapidly troubleshoot problems before they affect your business. Most IT
>> organizations don't have a clear picture of how application performance
>> affects their revenue. With AppDynamics, you get 100% visibility into your
>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
>> Pro!
>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Kaldi-users mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>