Re: [Kaldi-users] -u flag in dnn python scripts

[Daniel P. <dp...@gm...>]

Thanks a lot for pointing this out.
Cc'ing Karel who maintains this part.
Dan


On Mon, Dec 2, 2013 at 3:39 AM, Josef Novak
<jos...@gm...> wrote:
> Hi,
>
>   I have been digging around a bit in the dnn-related python scripts and had
> a question.
>
>   All of the the python scripts are called with the -u flag.  This means
> 'use unbuffered IO', which in turn means that every single byte is flushed
> to disk.
>
>   I noticed this because utils/gen_rbm_init.py seemed to be taking an
> inordinate amount of time (1+ hours on the WSJ for each call to the script).
>
>   In the case where all the data is on the local drive, and particularly if
> the local drive is an SSD, this would barely be noticeable - it is about 4x
> slower.  But any kind of network latency massively magnifies the penalty.
> In my case, with a locally networked drive, the difference was:
> ----------------------------------------------------
> #!/usr/bin/python -u:  1+ hours to write each N.rbm.init file
> #!/usr/bin/python:      10 seconds to write each N.rbm.init file
> ----------------------------------------------------
>   my understanding is that the -u flag also suppresses automatic newline
> conversion in 2.x - ensuring that Windows/Linux/OSX results would be
> equivalent/interoperable.  This was the only reason I could come up with for
> why one might want to do this.  However the potential time penalty for this
> is pretty massive, so I thought it worth pointing out.
>
> http://docs.python.org/2/using/cmdline.html#cmdoption-u
>
> http://docs.python.org/2/glossary.html#term-universal-newlines
>
>   If there is some other motivation for using this flag I would also be
> curious to know what it might be.
>
>   Best regards,
>     Joe
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>