Menu
▾
▴
Re: [scipion-users] Bayesian polishing
|
From: Dmitry A. S. <sem...@gm...> - 2021-06-10 19:52:44
|
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} </style><![endif]--><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style></head><body lang=EN-US link=blue vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-GB>NO, not yet.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>That is the next thing I plan to do.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Thank you!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>I will check the RAM while running the polishing protocol.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Sincerely,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Dmitry</span><o:p></o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='border:none;padding:0in'><b>From: </b><a href="mailto:sha...@gm...">Grigory Sharov</a><br><b>Sent: </b>Thursday, June 10, 2021 9:36 PM<br><b>To: </b><a href="mailto:sci...@li...">Mailing list for Scipion users</a><br><b>Subject: </b>Re: [scipion-users] Bayesian polishing</p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>I see, thanks for the clarification. Have you tried to run with all frames (default) but with a smaller particle subset, e.g. 3000-5000 particles?</p><div><p class=MsoNormal>You can monitor RAM usage when the protocol starts to see if that is an issue.</p></div><div><p class=MsoNormal><br clear=all></p><div><div><div><div><div><div><div><div><div><div><div><div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>Best regards,<br>Grigory</span><span style='color:black'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>--------------------------------------------------------------------------------</span><span style='color:black'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;color:black'>Grigory Sharov, Ph.D.<br><br>MRC Laboratory of Molecular Biology,<br>Francis Crick Avenue,<br>Cambridge Biomedical Campus,<br>Cambridge CB2 0QH, UK.<br>tel. </span><span style='color:black'><a href="tel:+44%201223%20267228" target="_blank"><span style='font-size:9.5pt;color:#1155CC'>+44 (0) 1223 267228</span></a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:9.5pt;color:black'>e-mail: </span><span style='color:black'><a href="mailto:gs...@mr..." target="_blank"><span style='font-size:9.5pt;color:#1155CC'>gs...@mr...</span></a><o:p></o:p></span></p></div></div></div></div></div></div></div></div></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Thu, Jun 10, 2021 at 8:32 PM Dmitry A. Semchonok <<a href="mailto:sem...@gm...">sem...@gm...</a>> wrote:</p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>Sorry Grigory,</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>I would have explained that abbreviation. </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>This means a) 1 movie frame and b) 5 movie frames.</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>a)</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=369 height=255 style='width:3.8385in;height:2.651in' id="gmail-m_3230183901105949274Picture_x0020_3" src="cid:image005.png@01D75E42.E7190CB0"></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=234 height=233 style='width:2.4322in;height:2.4218in' id="gmail-m_3230183901105949274Picture_x0020_4" src="cid:image006.png@01D75E42.E7190CB0"></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>And even with Threads 1 MPI 1 the 5 movie frames fail</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=362 height=238 style='width:3.7656in;height:2.4739in' id="gmail-m_3230183901105949274Picture_x0020_5" src="cid:image007.png@01D75E42.E7190CB0"></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=1213 height=176 style='width:12.6302in;height:1.8333in' id="gmail-m_3230183901105949274Picture_x0020_6" src="cid:image008.png@01D75E42.E7190CB0"></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>Thanks again</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>Sincerely,</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB>Dmitry</span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sent from <a href="https://go.microsoft.com/fwlink/?LinkId=550986" target="_blank">Mail</a> for Windows 10</p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b>From: </b><a href="mailto:sha...@gm..." target="_blank">Grigory Sharov</a><br><b>Sent: </b>Thursday, June 10, 2021 8:27 PM<br><b>To: </b><a href="mailto:sci...@li..." target="_blank">Mailing list for Scipion users</a><br><b>Subject: </b>Re: [scipion-users] Bayesian polishing</p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have no idea what 1e or 5e means, I see you are using 14 threads..</p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all></p><div><div><div><div><div><div><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>Best regards,<br>Grigory</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>--------------------------------------------------------------------------------</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;color:black'>Grigory Sharov, Ph.D.<br><br>MRC Laboratory of Molecular Biology,<br>Francis Crick Avenue,<br>Cambridge Biomedical Campus,<br>Cambridge CB2 0QH, UK.<br>tel. </span><span style='color:black'><a href="tel:+44%201223%20267228" target="_blank"><span style='font-size:9.5pt;color:#1155CC'>+44 (0) 1223 267228</span></a></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;color:black'>e-mail: </span><span style='color:black'><a href="mailto:gs...@mr..." target="_blank"><span style='font-size:9.5pt;color:#1155CC'>gs...@mr...</span></a></span></p></div></div></div></div></div></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Thu, Jun 10, 2021 at 7:25 PM Dmitry A. Semchonok <<a href="mailto:sem...@gm..." target="_blank">sem...@gm...</a>> wrote:</p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thank you Grigory,</p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The interesting point is that if the one uses 1e - the process finishes without error.</p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>But 5e seems too be too much. </p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sincerely,</p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Dmitry</p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> </p></div><div id="gmail-m_3230183901105949274gmail-m_-6395269166704044080aqm-original"><div><div><p style='margin-bottom:8.0pt'><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>On June 10, 2021 5:20:06 PM Grigory Sharov <<a href="mailto:sha...@gm..." target="_blank">sha...@gm...</a>> wrote:</span></p><blockquote style='border:none;border-left:solid gray 1.0pt;padding:0in 0in 0in 5.0pt;margin-left:4.5pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Hi Dmitry,</span></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>I think you posted a similar error before. You are using too many particles for training and running out of RAM, hence your process is killed by Linux kernel.</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'><br clear=all></span></p><div><div><div><div><div><div><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>Best regards,<br>Grigory</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black'>--------------------------------------------------------------------------------</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;color:black'>Grigory Sharov, Ph.D.<br><br>MRC Laboratory of Molecular Biology,<br>Francis Crick Avenue,<br>Cambridge Biomedical Campus,<br>Cambridge CB2 0QH, UK.<br>tel. </span><span style='color:black'><a href="tel:+44%201223%20267228" target="_blank"><span style='font-size:9.5pt;color:#1155CC'>+44 (0) 1223 267228</span></a></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;color:black'>e-mail: </span><span style='color:black'><a href="mailto:gs...@mr..." target="_blank"><span style='font-size:9.5pt;color:#1155CC'>gs...@mr...</span></a></span></p></div></div></div></div></div></div></div></div></div></div></div></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>On Thu, Jun 10, 2021 at 4:02 PM Dmitry Semchonok <<a href="mailto:Sem...@gm..." target="_blank">Sem...@gm...</a>> wrote:</span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Dear colleagues,<br><br>After running Bayesian polishing script with 5e I got the following error<br><br><br><br>2232: GridSquare_26409500_Data_FoilHole_26633166_Data_26416121_26416123_20210330_110202_Fractions.mrc<br>02591: 1914: GridSquare_26409382_Data_FoilHole_26635345_Data_26416121_26416123_20210330_172633_Fractions.mrc<br>02592: 1985: GridSquare_26409382_Data_FoilHole_26635480_Data_26416121_26416123_20210330_201730_Fractions.mrc<br>02593: 2253: GridSquare_26409500_Data_FoilHole_26633222_Data_26416121_26416123_20210330_123213_Fractions.mrc<br>02594: 1404: GridSquare_26409268_Data_FoilHole_26621354_Data_26416121_26416123_20210329_092746_Fractions.mrc<br>02595: 361: GridSquare_26409080_Data_FoilHole_26614677_Data_26416121_26416123_20210328_095659_Fractions.mrc<br>02596: <br>02597: - Warning: this dataset does not contain 217187 particles (--min_p) in micrographs with at least 2 particles<br>02598: + preparing alignment data... <br>02599: Traceback (most recent call last):<br>02600: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 197, in run<br>02601: self._run()<br>02602: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 248, in _run<br>02603: resultFiles = self._runFunc()<br>02604: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 244, in _runFunc<br>02605: return self._func(*self._args)<br>02606: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/relion/protocols/protocol_bayesian_polishing.py", line 351, in trainOrPolishStep<br>02607: self.runJob(self._getProgram('relion_motion_refine'), args)<br>02608: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 1388, in runJob<br>02609: self._stepsExecutor.runJob(self._log, program, arguments, **kwargs)<br>02610: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 65, in runJob<br>02611: process.runJob(log, programName, params,<br>02612: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/utils/process.py", line 52, in runJob<br>02613: return runCommand(command, env, cwd)<br>02614: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/utils/process.py", line 67, in runCommand<br>02615: check_call(command, shell=True, stdout=sys.stdout, stderr=sys.stderr,<br>02616: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/subprocess.py", line 364, in check_call<br>02617: raise CalledProcessError(retcode, cmd)<br>02618: subprocess.CalledProcessError: Command ' relion_motion_refine --i Runs/004833_ProtRelionBayesianPolishing/input_particles.star --o Runs/004833_ProtRelionBayesianPolishing/extra --f Runs/004384_ProtRelionPostprocess/extra/postprocess.star --angpix_ref 0.59200 --corr_mic Runs/004833_ProtRelionBayesianPolishing/input_corrected_micrographs.star --first_frame 1 --last_frame 10 --min_p 217187 --eval_frac 0.500 --align_frac 0.500 --params3 --j 14 ' died with <Signals.SIGKILL: 9>.<br>02619: Protocol failed: Command ' relion_motion_refine --i Runs/004833_ProtRelionBayesianPolishing/input_particles.star --o Runs/004833_ProtRelionBayesianPolishing/extra --f Runs/004384_ProtRelionPostprocess/extra/postprocess.star --angpix_ref 0.59200 --corr_mic Runs/004833_ProtRelionBayesianPolishing/input_corrected_micrographs.star --first_frame 1 --last_frame 10 --min_p 217187 --eval_frac 0.500 --align_frac 0.500 --params3 --j 14 ' died with <Signals.SIGKILL: 9>.<br>02620: FAILED: trainOrPolishStep, step 2, time 2021-06-09 11:27:00.176491<br>02621: *** Last status is failed <br>02622: ------------------- PROTOCOL FAILED (DONE 2/2)<br><br>Do you know how to fix that?<br><br><br>Thank you<br><br>Sincerely,<br>Dmitry<br><br><br><br><br>_______________________________________________<br>scipion-users mailing list<br><a href="mailto:sci...@li..." target="_blank">sci...@li...</a><br><a href="https://lists.sourceforge.net/lists/listinfo/scipion-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/scipion-users</a></span></p></blockquote></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>_______________________________________________</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>scipion-users mailing list</span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'><a href="mailto:scipion-users%40lists.sourceforge.net" target="_blank">sci...@li...</a></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'><a href="https://lists.sourceforge.net/lists/listinfo/scipion-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/scipion-users</a></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span></p></div></blockquote></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:4.8pt'> </p></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:4.8pt'> </p><p class=MsoNormal><o:p> </o:p></p></div></body></html> |
