Re: [proteowizard-support] Error running MSConvert

[Chambers, M. <mat...@gm...>]

In fact the program is supposed to terminate after an exception while converting, but it's loading spectra on multiple threads (maybe this 
in itself should be configurable and disabled on Wine), so when one thread dies, but another thread is waiting indefinitely, the program 
termination is delayed indefinitely. I guess such an option could cause the error handler to just abort() rather than trying to let threads 
terminate gracefully. I'll think about it.

Thanks,
-Matt


On 3/15/2019 2:37 PM, Manuel Belmadani wrote:
> Not that I can think of right now. The problem with that it appears msconvert.exe was still running at times (what I could tell from top, 
> or maybe it was just the docker image) so the cpu/memory usage was never at 0.0, just unusually low (5-10% or something).  It might be 
> also hard to tell whether a process is just on an Ssl state for a valid reason or if it's actually stuck.
>
> This isn't super elegant but the one way I would think of working around this is to add an option (say `--no-exception-tolerance`) that 
> would capture the output of stderr and add a watch to check for `[C:\pwiz\msconvert.exe] Caught unknown exception.`, and in this case kill 
> the process if it's still running. It's sort of hacky but if the user specifies `--no-exception-tolerance` then it would make sense in my 
> opinion to explicitly kill the process when an exception is detected.
>
> Let me know if you need more information on this issue, thanks for looking into it!
>
> On Fri, Mar 15, 2019 at 11:23 AM Chambers, Matthew <mat...@gm... <mailto:mat...@gm...>> wrote:
>
>     Indefinite hangs are certainly extremely undesirable, but if a vendor DLL makes a blocking I/O call and it doesn't time out for some
>     OS reason, there's not much we can do about it. It would need an out-of-process watcher to look when the CPU usage had gone to 0 for a
>     certain amount of time. Kind of like the "timeout" command, but conditioned on activity rather than absolute time. Do you know of
>     something like that?
>
>     On 3/15/2019 2:08 PM, Manuel Belmadani wrote:
>>     Hi,
>>
>>     It doesn't do it consistently actually, I reran it after I noticed and it terminated pretty quickly.
>>
>>     This was run on a Google Cloud Compute VM running Ubuntu 18.04 using data stored on a local disk (local to the VM). I don't think
>>     it's revelant in this case but I was running 7 concurrent calls (suing `xargs -P7`) with the same command this over 24 files and this
>>     is the only one that failed.
>>
>>     I noticed a similar issue when using gscfuse (a network mounted filesystem using the google cloud buckets) on multiple concurrent
>>     processes (using different files in each process). gscfuse is known to have issues when multiple process try to read from a mount at
>>     the same time, so the performance got quite bad (as expected) but it also seemed to leave the processes in a zombie state (Ssl state
>>     Interruptible sleep, multithreaded).
>>
>>     Sorry I don't have more information on how to reproduce this consistently. It would be nice if the docker image would exit/terminate
>>     (or have an option to specify that) when that failure occurs.
>>
>>     Hope this is useful! And thank you for your development/support of this software.
>>
>>     On Fri, Mar 15, 2019 at 10:58 AM Chambers, Matthew <mat...@gm... <mailto:mat...@gm...>> wrote:
>>
>>         Hi Manuel,
>>
>>         Does this error repeat at the same scan every time? If so, does it also happen on native Windows? If not, does it always error,
>>         or only sometimes?
>>
>>         Thanks for the report,
>>         -Matt
>>
>>
>>         On 3/15/2019 1:12 PM, Manuel Belmadani wrote:
>>>         Hi,
>>>
>>>         Encountered a crash using the msconvert docker image on Linux.
>>>
>>>         Process was hanging a few days after it stopped working. Noticed the STDERR had this info. It seems like the output mzML file is
>>>         not empty either. It's about 1.5Gb (gzipped) while the other
>>>         .RAW files are 2.5Gb and completed correctly and have nothing in the STDERR logs.
>>>         *
>>>         *
>>>         *COMMAND:*
>>>         docker run --rm -e WINEDEBUG=-all -v /home/z/local/RAW:/data -v /home/z/development/TMT/RawData:/RawData
>>>         chambm/pwiz-skyline-i-agree-to-the-vendor-licenses wine msconvert /data/tmt_b04_07.raw --gzip -o /RawData
>>>         *
>>>         *
>>>         *STDERR:*
>>>         $ cat RawData/rushtmt_b04_07.raw.err
>>>         [SpectrumWorkerThreads::work] error in thread: [SpectrumList_Thermo::spectrum()] Unknown exception retrieving spectrum
>>>         "controllerType=0 controllerNumber=1 scan=43319"
>>>         [C:\pwiz\msconvert.exe] Caught unknown exception.
>>>         Please report this error to su...@pr... <mailto:su...@pr...>.
>>>         Attach the command output and this version information in your report:
>>>
>>>         ProteoWizard release: 3.0.19056 (6b6b0a2b4)
>>>         Build date: Feb 25 2019 17:17:16
>>>
>>>         *STDOUT:*
>>>         RawData/rushtmt_b04_07.raw.log
>>>         format: mzML
>>>             m/z: Compression-None, 64-bit
>>>             intensity: Compression-None, 32-bit
>>>             rt: Compression-None, 64-bit
>>>         ByteOrder_LittleEndian
>>>          indexed="true"
>>>         outputPath: /RawData
>>>         extension: .mzML.gz
>>>         contactFilename:
>>>         runIndexSet:
>>>
>>>         spectrum list filters:
>>>
>>>         chromatogram list filters:
>>>
>>>         filenames:
>>>           /data\rushtmt_b04_07.raw
>>>
>>>         processing file: /data\rushtmt_b04_07.raw
>>>         calculating source file checksums
>>>         writing output file: /RawData\rushtmt_b04_07.mzML.gz
>>>
>>>
>>>
>>>         _______________________________________________
>>>         proteowizard-support mailing list
>>>         pro...@li...  <mailto:pro...@li...>
>>>         https://lists.sourceforge.net/lists/listinfo/proteowizard-support
>>
>