From: Grigory S. <sha...@gm...> - 2020-12-30 17:05:07
|
CC for the records: The problem was in xmipp center particles protocol that did not preserve micnames. I've reported this to the xmipp dev team which will be on it. Best regards, Grigory -------------------------------------------------------------------------------- Grigory Sharov, Ph.D. MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge CB2 0QH, UK. tel. +44 (0) 1223 267228 <+44%201223%20267228> e-mail: gs...@mr... On Wed, Dec 30, 2020 at 4:23 PM Dmitry A. Semchonok <sem...@gm...> wrote: > Thank you very much, Grigory, for your detailed investigation and > recommendations! > > > Sincerely, > Dmitry > > On December 30, 2020 6:16:12 PM Grigory Sharov <sha...@gm...> > wrote: > >> Hi, just confirmed, polishing fails with the same error as you reported. >> I'll inform the xmipp team to fix this. Thanks a lot for reporting this! >> >> In Relion motioncor you set the number of patches to 1x1 which means you >> only did global movie alignment. It's okay, but probably better results can >> be obtained with a number of patches 5x5 (see relion tutorial). >> >> Best regards, >> Grigory >> >> >> -------------------------------------------------------------------------------- >> Grigory Sharov, Ph.D. >> >> MRC Laboratory of Molecular Biology, >> Francis Crick Avenue, >> Cambridge Biomedical Campus, >> Cambridge CB2 0QH, UK. >> tel. +44 (0) 1223 267228 <+44%201223%20267228> >> e-mail: gs...@mr... >> >> >> On Wed, Dec 30, 2020 at 4:11 PM Dmitry A. Semchonok <sem...@gm...> >> wrote: >> >>> Dear Grigory, >>> >>> Thank you! >>> >>> So now it is clear that the xmipp centering doesnot preserve the >>> required info for Bayesian polishing - micName. >>> Maybe it can be corrected by Scipion/xmipp team sometime? >>> >>> I have a question - what do you mean by >>> >>> 1) you run only global mc without local patch-based alignment >>>> >>> >>> Usually I run relion - motioncor protocol for the motion alignment, and >>> continue further with the image processing. >>> >>> Am I doing something wrong? >>> >>> What could you recommend to run inorder to correct for the motion? >>> >>> Thank you! >>> >>> Sincerely, >>> Dmitry >>> >>> On December 30, 2020 5:47:22 PM Grigory Sharov <sha...@gm...> >>> wrote: >>> >>>> Hi again, >>>> >>>> I think I found the culprit which is xmipp center particles protocol. >>>> It does not preserve coordinate micName. I will run polishing to confirm. >>>> >>>> PS. Some things I've noticed: >>>> 1) you run only global mc without local patch-based alignment >>>> 2) for all cryosparc protocols numbers of mpi/threads are irrelevant >>>> since they are not available in cryosparc for modification. So you could >>>> use always 1 mpi, 1 thread >>>> >>>> >>>> Best regards, >>>> Grigory >>>> >>>> >>>> -------------------------------------------------------------------------------- >>>> Grigory Sharov, Ph.D. >>>> >>>> MRC Laboratory of Molecular Biology, >>>> Francis Crick Avenue, >>>> Cambridge Biomedical Campus, >>>> Cambridge CB2 0QH, UK. >>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>> e-mail: gs...@mr... >>>> >>>> >>>> On Wed, Dec 30, 2020 at 2:20 PM Grigory Sharov < >>>> sha...@gm...> wrote: >>>> >>>>> Thanks! That's quite a workflow. I did a simple test (below) already >>>>> and it worked. So must be some other combination of protocols. I'll try to >>>>> follow your project with my test data. >>>>> [image: image.png] >>>>> Best regards, >>>>> Grigory >>>>> >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> Grigory Sharov, Ph.D. >>>>> >>>>> MRC Laboratory of Molecular Biology, >>>>> Francis Crick Avenue, >>>>> Cambridge Biomedical Campus, >>>>> Cambridge CB2 0QH, UK. >>>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>>> e-mail: gs...@mr... >>>>> >>>>> >>>>> On Wed, Dec 30, 2020 at 1:59 PM Dmitry Semchonok <sem...@gm...> >>>>> wrote: >>>>> >>>>>> Dear Grigory, >>>>>> >>>>>> >>>>>> >>>>>> Please find the file in the attachment. >>>>>> >>>>>> >>>>>> >>>>>> I also progressed further, trying to re-extract the good particles >>>>>> and run the Bayesian polishing in relion protocols above only (not yet >>>>>> there). >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> *От: *Grigory Sharov <sha...@gm...> >>>>>> *Отправлено: *30 декабря 2020 г. в 14:51 >>>>>> *Кому: *Dmitry A. Semchonok <sem...@gm...> >>>>>> *Тема: *Re: [scipion-users] Bayesian polishing issue >>>>>> >>>>>> >>>>>> >>>>>> Sure, you click Ctrl+A to select all protocols, then press Export and >>>>>> choose a json file to save. The file will be very small so you can attach >>>>>> it to the email. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> >>>>>> Best regards, >>>>>> Grigory >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------------- >>>>>> >>>>>> Grigory Sharov, Ph.D. >>>>>> >>>>>> MRC Laboratory of Molecular Biology, >>>>>> Francis Crick Avenue, >>>>>> Cambridge Biomedical Campus, >>>>>> Cambridge CB2 0QH, UK. >>>>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>>>> >>>>>> e-mail: gs...@mr... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 30, 2020 at 12:50 PM Dmitry A. Semchonok < >>>>>> sem...@gm...> wrote: >>>>>> >>>>>> Dear Grigory, >>>>>> >>>>>> >>>>>> >>>>>> I already thought about that, but I forgot how to do that. >>>>>> >>>>>> >>>>>> >>>>>> Could you please remind me how can I copy my workflow? >>>>>> >>>>>> >>>>>> >>>>>> Thank you, >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> On December 30, 2020 2:10:53 PM Grigory Sharov < >>>>>> sha...@gm...> wrote: >>>>>> >>>>>> Hi Dmitry, >>>>>> >>>>>> >>>>>> >>>>>> Could you possibly send me your workflow for the project? Since there >>>>>> are so many cryosparc protocols and I'm not familiar with the code it will >>>>>> take time to figure it out. >>>>>> >>>>>> >>>>>> >>>>>> Grigory >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 29, 2020, 14:54 Dmitry Semchonok <sem...@gm...> >>>>>> wrote: >>>>>> >>>>>> I run the motion correction with relion – motioncor >>>>>> >>>>>> Is there a way to do the cryosparc- motion correction inside the >>>>>> SCIPON3 (I was not aware) ? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *От: *Grigory Sharov <sha...@gm...> >>>>>> *Отправлено: *29 декабря 2020 г. в 16:45 >>>>>> *Кому: *Mailing list for Scipion users >>>>>> <sci...@li...> >>>>>> *Тема: *Re: [scipion-users] Bayesian polishing issue >>>>>> >>>>>> >>>>>> >>>>>> Oh, I see. You are probably one of the first users trying to combine >>>>>> cryosparc and relion protocols in scipion. I will investigate whether >>>>>> cryosparc plugin does not keep the original micrographName and get back to >>>>>> you. >>>>>> >>>>>> >>>>>> >>>>>> Did you run motion correction also in cryosparc or in relion? >>>>>> >>>>>> >>>>>> Best regards, >>>>>> Grigory >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------------- >>>>>> >>>>>> Grigory Sharov, Ph.D. >>>>>> >>>>>> MRC Laboratory of Molecular Biology, >>>>>> Francis Crick Avenue, >>>>>> Cambridge Biomedical Campus, >>>>>> Cambridge CB2 0QH, UK. >>>>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>>>> >>>>>> e-mail: gs...@mr... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 29, 2020 at 2:36 PM Dmitry Semchonok <sem...@gm...> >>>>>> wrote: >>>>>> >>>>>> Dear Grigory, >>>>>> >>>>>> >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> >>>>>> That is quite interesting result. >>>>>> >>>>>> Could you please check the new Bayesian run (I copied the files and >>>>>> will send you with Wetransfer again) >>>>>> >>>>>> >>>>>> >>>>>> If the problem remains the same it means that the cryosparc 3D refine >>>>>> output can not be followed up with relion Bayesian polishing. >>>>>> >>>>>> >>>>>> >>>>>> Please see the project tree. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> What is your opinion? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Отправлено из приложения "Почта >>>>>> <https://go.microsoft.com/fwlink/?LinkId=550986>" для Windows 10 >>>>>> >>>>>> >>>>>> >>>>>> *От: *Grigory Sharov <sha...@gm...> >>>>>> *Отправлено: *29 декабря 2020 г. в 16:22 >>>>>> *Кому: *Mailing list for Scipion users >>>>>> <sci...@li...> >>>>>> *Тема: *Re: [scipion-users] Bayesian polishing issue >>>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> Thank you for sending over the files. rlnMicrographName in the >>>>>> input_particles.star and input_corrected_micrographs.star are completely >>>>>> different. I do not know how you have generated your particles, but you >>>>>> cannot run polishing with them. >>>>>> >>>>>> >>>>>> >>>>>> rlnMicrographName >>>>>> >>>>>> Best regards, >>>>>> Grigory >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------------- >>>>>> >>>>>> Grigory Sharov, Ph.D. >>>>>> >>>>>> MRC Laboratory of Molecular Biology, >>>>>> Francis Crick Avenue, >>>>>> Cambridge Biomedical Campus, >>>>>> Cambridge CB2 0QH, UK. >>>>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>>>> >>>>>> e-mail: gs...@mr... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 29, 2020 at 11:32 AM Grigory Sharov < >>>>>> sha...@gm...> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> You forgot to attach Runs/004138_ProtRelionBayesianPolishing/input_particles.star >>>>>> file >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 29, 2020, 11:26 Dmitry Semchonok <sem...@gm...> >>>>>> wrote: >>>>>> >>>>>> Dear Grigory, >>>>>> >>>>>> >>>>>> >>>>>> Please find the file attached >>>>>> >>>>>> >>>>>> >>>>>> Thank you! >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 29, 2020 at 12:28 AM Grigory Sharov < >>>>>> sha...@gm...> wrote: >>>>>> >>>>>> Hi Dmitry, >>>>>> >>>>>> >>>>>> >>>>>> this usually happens if you have moved your raw movie files from >>>>>> where Scipion expects them to be. >>>>>> >>>>>> >>>>>> >>>>>> Could you please send me >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_particles.star and >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_corrected_micrographs.star >>>>>> files? >>>>>> >>>>>> >>>>>> Best regards, >>>>>> Grigory >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------------- >>>>>> >>>>>> Grigory Sharov, Ph.D. >>>>>> >>>>>> MRC Laboratory of Molecular Biology, >>>>>> Francis Crick Avenue, >>>>>> Cambridge Biomedical Campus, >>>>>> Cambridge CB2 0QH, UK. >>>>>> tel. +44 (0) 1223 267228 <+44%201223%20267228> >>>>>> >>>>>> e-mail: gs...@mr... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Dec 28, 2020 at 10:16 PM Dmitry Semchonok < >>>>>> sem...@gm...> wrote: >>>>>> >>>>>> Dear colleagues, >>>>>> >>>>>> >>>>>> >>>>>> *Bayesian polishing issue –* >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 00008: workingDir: Runs/004138_ProtRelionBayesianPolishing >>>>>> >>>>>> 00009: runMode: Restart >>>>>> >>>>>> 00010: MPI: 1 >>>>>> >>>>>> 00011: threads: 14 >>>>>> >>>>>> 00012: Starting at step: 1 >>>>>> >>>>>> 00013: Running steps >>>>>> >>>>>> 00014: STARTED: convertInputStep, step 1, time 2020-12-28 >>>>>> 23:03:24.304754 >>>>>> >>>>>> 00015: Converting set from >>>>>> 'Runs/003937_ProtRelionRefine3D/particles.sqlite' into >>>>>> 'Runs/004138_ProtRelionBayesianPolishing/input_particles.star' >>>>>> >>>>>> 00016: convertBinaryFiles: creating soft links. >>>>>> >>>>>> 00017: Root: >>>>>> Runs/004138_ProtRelionBayesianPolishing/input/particles/input -> >>>>>> Runs/000427_ProtRelionExtractParticles/extra >>>>>> >>>>>> 00018: FINISHED: convertInputStep, step 1, time 2020-12-28 >>>>>> 23:04:33.416996 >>>>>> >>>>>> 00019: STARTED: trainOrPolishStep, step 2, time 2020-12-28 >>>>>> 23:04:33.422574 >>>>>> >>>>>> 00020: relion_motion_refine --i >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_particles.star --o >>>>>> Runs/004138_ProtRelionBayesianPolishing/extra --f >>>>>> Runs/004089_ProtRelionPostprocess/extra/postprocess.star --angpix_ref >>>>>> 0.59200 --corr_mic >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_corrected_micrographs.star >>>>>> --first_frame 3 --last_frame 0 --min_p 421029 --eval_frac 0.500 >>>>>> --align_frac 0.500 --params3 --j 14 >>>>>> >>>>>> 00021: + Reading >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_particles.star.. >>>>>> >>>>>> 00022: in: >>>>>> /home/user/Data/Software/Scipion3/software/em/relion-3.1.0/src/jaz/micrograph_handler.cpp, >>>>>> line 261 >>>>>> >>>>>> 00023: ERROR: >>>>>> >>>>>> 00024: There is no movie metadata STAR file for any micrographs! >>>>>> >>>>>> 00025: === Backtrace === >>>>>> >>>>>> 00026: relion_motion_refine(_ZN11RelionErrorC1ERKSsS1_l+0x41) >>>>>> [0x453571] >>>>>> >>>>>> 00027: >>>>>> relion_motion_refine(_ZN17MicrographHandler11loadInitialERKSt6vectorI13MetaDataTableSaIS1_EEbRiRdRSs+0xb95) >>>>>> [0x531275] >>>>>> >>>>>> 00028: >>>>>> relion_motion_refine(_ZN17MicrographHandler4initERKSt6vectorI13MetaDataTableSaIS1_EEbiRiRdRSs+0x78) >>>>>> [0x533a88] >>>>>> >>>>>> 00029: relion_motion_refine(_ZN13MotionRefiner4initEv+0x546) >>>>>> [0x47d1a6] >>>>>> >>>>>> 00030: relion_motion_refine(main+0x31) [0x428841] >>>>>> >>>>>> 00031: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff26cfb7555] >>>>>> >>>>>> 00032: relion_motion_refine() [0x42955f] >>>>>> >>>>>> 00033: ================== >>>>>> >>>>>> 00034: ERROR: >>>>>> >>>>>> 00035: There is no movie metadata STAR file for any micrographs! >>>>>> >>>>>> 00036: Traceback (most recent call last): >>>>>> >>>>>> 00037: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", >>>>>> line 189, in run >>>>>> >>>>>> 00038: self._run() >>>>>> >>>>>> 00039: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", >>>>>> line 240, in _run >>>>>> >>>>>> 00040: resultFiles = self._runFunc() >>>>>> >>>>>> 00041: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", >>>>>> line 236, in _runFunc >>>>>> >>>>>> 00042: return self._func(*self._args) >>>>>> >>>>>> 00043: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/relion/protocols/protocol_bayesian_polishing.py", >>>>>> line 309, in trainOrPolishStep >>>>>> >>>>>> 00044: self.runJob(prog, args) >>>>>> >>>>>> 00045: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", >>>>>> line 1357, in runJob >>>>>> >>>>>> 00046: self._stepsExecutor.runJob(self._log, program, >>>>>> arguments, **kwargs) >>>>>> >>>>>> 00047: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", >>>>>> line 65, in runJob >>>>>> >>>>>> 00048: process.runJob(log, programName, params, >>>>>> >>>>>> 00049: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/process.py", >>>>>> line 51, in runJob >>>>>> >>>>>> 00050: return runCommand(command, env, cwd) >>>>>> >>>>>> 00051: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/process.py", >>>>>> line 66, in runCommand >>>>>> >>>>>> 00052: check_call(command, shell=True, stdout=sys.stdout, >>>>>> stderr=sys.stderr, >>>>>> >>>>>> 00053: File >>>>>> "/home/user/.conda/envs/.scipion3env/lib/python3.8/subprocess.py", line >>>>>> 364, in check_call >>>>>> >>>>>> 00054: raise CalledProcessError(retcode, cmd) >>>>>> >>>>>> 00055: subprocess.CalledProcessError: Command 'relion_motion_refine >>>>>> --i Runs/004138_ProtRelionBayesianPolishing/input_particles.star --o >>>>>> Runs/004138_ProtRelionBayesianPolishing/extra --f >>>>>> Runs/004089_ProtRelionPostprocess/extra/postprocess.star --angpix_ref >>>>>> 0.59200 --corr_mic >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_corrected_micrographs.star >>>>>> --first_frame 3 --last_frame 0 --min_p 421029 --eval_frac 0.500 >>>>>> --align_frac 0.500 --params3 --j 14 ' returned non-zero exit status 1. >>>>>> >>>>>> 00056: Protocol failed: Command 'relion_motion_refine --i >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_particles.star --o >>>>>> Runs/004138_ProtRelionBayesianPolishing/extra --f >>>>>> Runs/004089_ProtRelionPostprocess/extra/postprocess.star --angpix_ref >>>>>> 0.59200 --corr_mic >>>>>> Runs/004138_ProtRelionBayesianPolishing/input_corrected_micrographs.star >>>>>> --first_frame 3 --last_frame 0 --min_p 421029 --eval_frac 0.500 >>>>>> --align_frac 0.500 --params3 --j 14 ' returned non-zero exit status 1. >>>>>> >>>>>> 00057: FAILED: trainOrPolishStep, step 2, time 2020-12-28 >>>>>> 23:04:45.953315 >>>>>> >>>>>> 00058: *** Last status is failed >>>>>> >>>>>> 00059: ------------------- PROTOCOL FAILED (DONE 2/2) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Any ideas? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Dmitry >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> > > > |