Menu
▾
▴
Re: [scipion-users] Export particles from cryosparc
|
From: Dmitry S. <Sem...@gm...> - 2021-05-18 12:25:12
|
Dear Pablo, Thank you. I heard about this option. For that I guess the https://pypi.org/project/cs2star/ <https://pypi.org/project/cs2star/> needs to be installed. In Cryosparc itself there is an option to export files. And then what we get is the mrc files with different number of particles in each. It appeared to be possible to rename mrc —> to —>mrcs. Then SCIPION can import those particles. Currently the problem is that not relion nor cryosparc can run these particles. Relion stops with error: 00001: RUNNING PROTOCOL ----------------- 00002: Hostname: cryoem01 00003: PID: 46455 00004: pyworkflow: 3.0.13 00005: plugin: relion 00006: plugin v: 3.1.2 00007: currentDir: /data1/ScipionUserData/projects/Caro__helix 00008: workingDir: Runs/001546_ProtRelionClassify2D 00009: runMode: Continue 00010: MPI: 3 00011: threads: 3 00012: Starting at step: 1 00013: Running steps 00014: STARTED: convertInputStep, step 1, time 2021-05-18 12:33:26.198123 00015: Converting set from 'Runs/001492_ProtUserSubSet/particles.sqlite' into 'Runs/001546_ProtRelionClassify2D/input_particles.star' 00016: convertBinaryFiles: creating soft links. 00017: Root: Runs/001546_ProtRelionClassify2D/extra/input -> Runs/001057_ProtImportParticles/extra 00018: FINISHED: convertInputStep, step 1, time 2021-05-18 12:33:27.117588 00019: STARTED: runRelionStep, step 2, time 2021-05-18 12:33:27.145974 00020: mpirun -np 3 `which relion_refine_mpi` --i Runs/001546_ProtRelionClassify2D/input_particles.star --particle_diameter 690 --zero_mask --K 64 --norm --scale --o Runs/001546_ProtRelionClassify2D/extra/relion --oversampling 1 --flatten_solvent --tau2_fudge 2.0 --iter 25 --offset_range 5.0 --offset_step 2.0 --psi_step 10.0 --dont_combine_weights_via_disc --scratch_dir /data1/new_scratch/ --pool 3 --gpu --j 3 00021: RELION version: 3.1.2 00022: Precision: BASE=double, CUDA-ACC=single 00023: 00024: === RELION MPI setup === 00025: + Number of MPI processes = 3 00026: + Number of threads per MPI process = 3 00027: + Total number of threads therefore = 9 00028: + Leader (0) runs on host = cryoem01 00029: + Follower 1 runs on host = cryoem01 00030: + Follower 2 runs on host = cryoem01 00031: ================= 00032: uniqueHost cryoem01 has 2 ranks. 00033: GPU-ids not specified for this rank, threads will automatically be mapped to available devices. 00034: Thread 0 on follower 1 mapped to device 0 00035: Thread 1 on follower 1 mapped to device 0 00036: Thread 2 on follower 1 mapped to device 0 00037: GPU-ids not specified for this rank, threads will automatically be mapped to available devices. 00038: Thread 0 on follower 2 mapped to device 1 00039: Thread 1 on follower 2 mapped to device 1 00040: Thread 2 on follower 2 mapped to device 1 00041: Running CPU instructions in double precision. 00042: + WARNING: Changing psi sampling rate (before oversampling) to 5.625 degrees, for more efficient GPU calculations 00043: + On host cryoem01: free scratch space = 448.252 Gb. 00044: Copying particles to scratch directory: /data1/new_scratch/relion_volatile/ 00045: 000/??? sec ~~(,_,"> [oo] 00046: 1/ 60 sec ~~(,_,">in: /opt/Scipion3/software/em/relion-3.1.2/src/rwMRC.h, line 192 00047: ERROR: 00048: readMRC: Image number 11 exceeds stack size 1 of image 000011@Runs/001546_ProtRelionClassify2D/extra/input/1024562735536827037_FoilHole_1618719_Data_1621438_1621440_20200703_085118_Fractions_patch_aligned_doseweighted_particles.mrcs 00049: === Backtrace === 00050: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x4786a1] 00051: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN5ImageIdE7readMRCElbRK8FileName+0x99f) [0x4b210f] 00052: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN5ImageIdE5_readERK8FileNameR13fImageHandlerblbb+0x17b) [0x4b407b] 00053: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN10Experiment22copyParticlesToScratchEibbd+0xda7) [0x5b8f87] 00054: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN14MlOptimiserMpi18initialiseWorkLoadEv+0x210) [0x498540] 00055: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(_ZN14MlOptimiserMpi10initialiseEv+0x9aa) [0x49ab2a] 00056: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi(main+0x55) [0x4322a5] 00057: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f657f51a555] 00058: /opt/Scipion3/software/em/relion-3.1.2/bin/relion_refine_mpi() [0x435fbf] 00059: ================== 00060: ERROR: 00061: readMRC: Image number 11 exceeds stack size 1 of image 000011@Runs/001546_ProtRelionClassify2D/extra/input/1024562735536827037_FoilHole_1618719_Data_1621438_1621440_20200703_085118_Fractions_patch_aligned_doseweighted_particles.mrcs 00062: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 00063: Traceback (most recent call last): 00064: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 197, in run 00065: self._run() 00066: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 248, in _run 00067: resultFiles = self._runFunc() 00068: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 244, in _runFunc 00069: return self._func(*self._args) 00070: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/relion/protocols/protocol_base.py", line 811, in runRelionStep 00071: self.runJob(self._getProgram(), params) 00072: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 1388, in runJob 00073: self._stepsExecutor.runJob(self._log, program, arguments, **kwargs) 00074: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 65, in runJob 00075: process.runJob(log, programName, params, 00076: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/utils/process.py", line 52, in runJob 00077: return runCommand(command, env, cwd) 00078: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/site-packages/pyworkflow/utils/process.py", line 67, in runCommand 00079: check_call(command, shell=True, stdout=sys.stdout, stderr=sys.stderr, 00080: File "/usr/local/miniconda/envs/scipion3/lib/python3.8/subprocess.py", line 364, in check_call 00081: raise CalledProcessError(retcode, cmd) 00082: subprocess.CalledProcessError: Command ' mpirun -np 3 `which relion_refine_mpi` --i Runs/001546_ProtRelionClassify2D/input_particles.star --particle_diameter 690 --zero_mask --K 64 --norm --scale --o Runs/001546_ProtRelionClassify2D/extra/relion --oversampling 1 --flatten_solvent --tau2_fudge 2.0 --iter 25 --offset_range 5.0 --offset_step 2.0 --psi_step 10.0 --dont_combine_weights_via_disc --scratch_dir /data1/new_scratch/ --pool 3 --gpu --j 3' returned non-zero exit status 1. 00083: Protocol failed: Command ' mpirun -np 3 `which relion_refine_mpi` --i Runs/001546_ProtRelionClassify2D/input_particles.star --particle_diameter 690 --zero_mask --K 64 --norm --scale --o Runs/001546_ProtRelionClassify2D/extra/relion --oversampling 1 --flatten_solvent --tau2_fudge 2.0 --iter 25 --offset_range 5.0 --offset_step 2.0 --psi_step 10.0 --dont_combine_weights_via_disc --scratch_dir /data1/new_scratch/ --pool 3 --gpu --j 3' returned non-zero exit status 1. 00084: FAILED: runRelionStep, step 2, time 2021-05-18 12:33:29.230213 00085: *** Last status is failed 00086: ------------------- PROTOCOL FAILED (DONE 2/3) Cryosparc (in SCIPION) requires CTF to run. Thais is where I am now. Perhaps there is a solution? Sincerely, Dmitry > On 18. May 2021, at 14:16, Pablo Conesa <pc...@cn...> wrote: > > Dear Dmitry, the import of CS metadata files (*.cs) is not supported in Scipion. Does CS has an option to export to star files. It rings a bell. > > On 18/5/21 9:53, Dmitry Semchonok wrote: >> <>Dear Grigory, >> >> >> The files are in mrc format. >> >> >> Please, let me try to explain more plan: >> >> I have a project in cryosparc. There I have cryosparc selected 2D classes. I want to export the particles of those classes into SCIPION. >> >> So I I pressed Export (fig 1) and the program(cryosparc) created the folder with mrc + other files (fig 2;3). I looked into J48 and found many *.mrc files of the particles. But it is not 1 mrc = 1 particle. It seems to be a mrc stuck - so I have several files inside 1 *.mrc (fig 4) (you can also notice that they all have different sizes) >> >> So I need to export them somehow in SCIPION >> >> For that, I used the SCIPION export - images protocol where for the files to add I put *.mrc. But the protocol seems to be added only 1 mrc as 1 picture and instead of having 46392 particles I have ~600 particles. >> >> (Also the geometry seems not preserved). >> >> >> So my question how to export the particles from cryosparc into SCIPION correctly? >> >> >> Thank you! >> >> >> >> >> >> https://disk.yandex.com/d/Fv3Q1lpwEzSisg <https://disk.yandex.com/d/Fv3Q1lpwEzSisg> >> >> Sincerely, >> >> Dmitry >> >> >> >> >>> On 17. May 2021, at 18:12, Grigory Sharov <sha...@gm... <mailto:sha...@gm...>> wrote: >>> >>> Hi Dmitry, >>> >>> mrc stacks should have "mrcs" extension. Is this the problem you are getting? >>> >>> Best regards, >>> Grigory >>> >>> -------------------------------------------------------------------------------- >>> Grigory Sharov, Ph.D. >>> >>> MRC Laboratory of Molecular Biology, >>> Francis Crick Avenue, >>> Cambridge Biomedical Campus, >>> Cambridge CB2 0QH, UK. >>> tel. +44 (0) 1223 267228 <tel:+44%201223%20267228> >>> e-mail: gs...@mr... <mailto:gs...@mr...> >>> >>> >>> On Mon, May 17, 2021 at 4:49 PM Dmitry Semchonok <Sem...@gm... <mailto:Sem...@gm...>> wrote: >>> <>Dear colleagues, >>> >>> I would like to export the particles from cryosparc to SCIPION. >>> >>> How to do that? >>> >>> >>> >>> >>> >>> What I tried: >>> >>> >>> 1. In cryosparc I pressed Export – to export the particles I am interested in. >>> >>> 2. In the folder Export – I found many mrc stacks with particles in each. >>> >>> 3. I tried to export them to SCIPION using Export particles but instead of reading each stack and combine them in the 1 dataset I received 1 particle / per each mrc stack. >>> >>> >>> Any ideas? >>> >>> >>> Thank you >>> >>> Sincerely, >>> >>> Dmitry >>> >>> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... <mailto:sci...@li...> >>> https://lists.sourceforge.net/lists/listinfo/scipion-users <https://lists.sourceforge.net/lists/listinfo/scipion-users> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... <mailto:sci...@li...> >>> https://lists.sourceforge.net/lists/listinfo/scipion-users <https://lists.sourceforge.net/lists/listinfo/scipion-users> >> >> >> >> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... <mailto:sci...@li...> >> https://lists.sourceforge.net/lists/listinfo/scipion-users <https://lists.sourceforge.net/lists/listinfo/scipion-users> > -- > Pablo Conesa - Madrid Scipion <http://scipion.i2pc.es/> team > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users |
