Menu
▾
▴
scipion-users — Mailing list for Scipion users
You can subscribe to this list here.
| 2016 |
Jan
(2) |
Feb
(13) |
Mar
(9) |
Apr
(4) |
May
(5) |
Jun
(2) |
Jul
(8) |
Aug
(3) |
Sep
(25) |
Oct
(7) |
Nov
(49) |
Dec
(15) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2017 |
Jan
(24) |
Feb
(36) |
Mar
(53) |
Apr
(44) |
May
(37) |
Jun
(34) |
Jul
(12) |
Aug
(15) |
Sep
(14) |
Oct
(9) |
Nov
(9) |
Dec
(7) |
| 2018 |
Jan
(16) |
Feb
(9) |
Mar
(27) |
Apr
(39) |
May
(8) |
Jun
(24) |
Jul
(22) |
Aug
(11) |
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2019 |
Jan
(4) |
Feb
(5) |
Mar
|
Apr
(1) |
May
(21) |
Jun
(13) |
Jul
(31) |
Aug
(22) |
Sep
(9) |
Oct
(19) |
Nov
(24) |
Dec
(12) |
| 2020 |
Jan
(30) |
Feb
(12) |
Mar
(16) |
Apr
(4) |
May
(37) |
Jun
(17) |
Jul
(19) |
Aug
(15) |
Sep
(26) |
Oct
(84) |
Nov
(64) |
Dec
(55) |
| 2021 |
Jan
(18) |
Feb
(58) |
Mar
(26) |
Apr
(88) |
May
(51) |
Jun
(36) |
Jul
(31) |
Aug
(37) |
Sep
(79) |
Oct
(15) |
Nov
(29) |
Dec
(8) |
| 2022 |
Jan
(5) |
Feb
(8) |
Mar
(29) |
Apr
(21) |
May
(11) |
Jun
(11) |
Jul
(18) |
Aug
(16) |
Sep
(6) |
Oct
(10) |
Nov
(23) |
Dec
(1) |
| 2023 |
Jan
(18) |
Feb
|
Mar
(4) |
Apr
|
May
(3) |
Jun
(10) |
Jul
(1) |
Aug
|
Sep
|
Oct
(1) |
Nov
(3) |
Dec
(5) |
| 2024 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2025 |
Jan
(1) |
Feb
|
Mar
|
Apr
(5) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Ian L. R. <ian...@ln...> - 2019-12-05 10:54:14
|
Pablo, I would love to join the slack community! ________________________________ From: Pablo Conesa <pc...@cn...> Sent: Thursday, December 5, 2019 4:41 To: sci...@li... <sci...@li...> Subject: Re: [scipion-users] Greetings and a question We are also planning to organize a course for developers this summer. Keep tunned. On 4/12/19 19:52, Pablo Conesa wrote: Dear Ian, I guess you've already found the developers page<https://scipion-em.github.io/docs/docs/developer/developers.html>, there are lots of info but also work to be done. But recently we updated it. More precisely answering your questions: Scipion has it's own model where Em "concepts" are reflected. Micrograph is here -->https://github.com/I2PC/scipion/blob/master/pyworkflow/em/data.py#L642 They are pointers, all the binary files remain in the file system and only the paths to them are kept in the sqlites files. Usually the protocol you implement needs to take that into consideration, see an example here: https://github.com/scipion-em/scipion-em-sphire/blob/master/sphire/protocols/protocol_cryolo_picking.py#L188 But many of the protocols have somehow a "possible" conversion step. We run a slack workspace for developers and facility managers to have more direct conversations. I can add you if you want. Keep asking your doubts, we are here to help you. All the best, Pablo On 4/12/19 16:26, Ian Liu Rodrigues wrote: Dear Scipion community, I'm Ian Liu Rodrigues, working with software development at the National Laboratory of Nanotechnology, Brazil. I'm trying to understand how the plugin architecture works so I can make my own plugins. Here are my questions: * How is a micrograph represented inside Scipion? * Is the image imported into the database or just pointers to external files? * How does Scipion knows when to convert an input to a program? I'm very new to this, so sorry if I'm asking anything basic 😉 Kind regards, 👨💻 Ian L. Rodrigues 🏢 CNPEM, LNNano, CME 📞 +55 19 3512 1144 Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente. Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately. _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - Madrid Scipion<http://scipion.i2pc.es> team _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - Madrid Scipion<http://scipion.i2pc.es> team |
|
From: Pablo C. <pc...@cn...> - 2019-12-05 07:42:01
|
We are also planning to organize a course for developers this summer. Keep tunned. On 4/12/19 19:52, Pablo Conesa wrote: > > Dear Ian, > > I guess you've already found the developers page > <https://scipion-em.github.io/docs/docs/developer/developers.html>, > there are lots of info but also work to be done. But recently we > updated it. > > > More precisely answering your questions: > > > Scipion has it's own model where Em "concepts" are reflected. > Micrograph is here > -->https://github.com/I2PC/scipion/blob/master/pyworkflow/em/data.py#L642 > > They are pointers, all the binary files remain in the file system > and only the paths to them are kept in the sqlites files. > > Usually the protocol you implement needs to take that into > consideration, see an example here: > https://github.com/scipion-em/scipion-em-sphire/blob/master/sphire/protocols/protocol_cryolo_picking.py#L188 > > > But many of the protocols have somehow a "possible" conversion step. > > > We run a slack workspace for developers and facility managers to have > more direct conversations. I can add you if you want. > > > Keep asking your doubts, we are here to help you. > > > All the best, Pablo > > > > On 4/12/19 16:26, Ian Liu Rodrigues wrote: >> Dear Scipion community, >> >> I'm Ian Liu Rodrigues, working with software development at the >> National Laboratory of Nanotechnology, Brazil. I'm trying to >> understand how the plugin architecture works so I can make my own >> plugins. Here are my questions: >> >> * How is a micrograph represented inside Scipion? >> * Is the image *imported* into the database or just pointers to >> external files? >> * How does Scipion knows when to convert an input to a program? >> >> I'm very new to this, so sorry if I'm asking anything basic 😉 >> >> Kind regards, >> >> 👨💻 Ian L. Rodrigues >> 🏢 CNPEM, LNNano, CME >> 📞 +55 19 3512 1144 >> >> >> Aviso Legal: Esta mensagem e seus anexos podem conter informações >> confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo >> e considere eventual consulta ao remetente antes de copiá-la, >> divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, >> por favor avise o remetente e apague-a imediatamente. >> >> Disclaimer: This email and its attachments may contain confidential >> and/or privileged information. Observe its content carefully and >> consider possible querying to the sender before copying, disclosing >> or distributing it. If you have received this email by mistake, >> please notify the sender and delete it immediately. >> >> >> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users > -- > Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* |
|
From: Pablo C. <pc...@cn...> - 2019-12-04 18:53:07
|
Dear Ian, I guess you've already found the developers page <https://scipion-em.github.io/docs/docs/developer/developers.html>, there are lots of info but also work to be done. But recently we updated it. More precisely answering your questions: Scipion has it's own model where Em "concepts" are reflected. Micrograph is here -->https://github.com/I2PC/scipion/blob/master/pyworkflow/em/data.py#L642 They are pointers, all the binary files remain in the file system and only the paths to them are kept in the sqlites files. Usually the protocol you implement needs to take that into consideration, see an example here: https://github.com/scipion-em/scipion-em-sphire/blob/master/sphire/protocols/protocol_cryolo_picking.py#L188 But many of the protocols have somehow a "possible" conversion step. We run a slack workspace for developers and facility managers to have more direct conversations. I can add you if you want. Keep asking your doubts, we are here to help you. All the best, Pablo On 4/12/19 16:26, Ian Liu Rodrigues wrote: > Dear Scipion community, > > I'm Ian Liu Rodrigues, working with software development at the > National Laboratory of Nanotechnology, Brazil. I'm trying to > understand how the plugin architecture works so I can make my own > plugins. Here are my questions: > > * How is a micrograph represented inside Scipion? > * Is the image *imported* into the database or just pointers to > external files? > * How does Scipion knows when to convert an input to a program? > > I'm very new to this, so sorry if I'm asking anything basic 😉 > > Kind regards, > > 👨💻 Ian L. Rodrigues > 🏢 CNPEM, LNNano, CME > 📞 +55 19 3512 1144 > > > Aviso Legal: Esta mensagem e seus anexos podem conter informações > confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e > considere eventual consulta ao remetente antes de copiá-la, divulgá-la > ou distribuí-la. Se você recebeu esta mensagem por engano, por favor > avise o remetente e apague-a imediatamente. > > Disclaimer: This email and its attachments may contain confidential > and/or privileged information. Observe its content carefully and > consider possible querying to the sender before copying, disclosing or > distributing it. If you have received this email by mistake, please > notify the sender and delete it immediately. > > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* |
|
From: Ian L. R. <ian...@ln...> - 2019-12-04 17:00:55
|
Dear Scipion community, I'm Ian Liu Rodrigues, working with software development at the National Laboratory of Nanotechnology, Brazil. I'm trying to understand how the plugin architecture works so I can make my own plugins. Here are my questions: * How is a micrograph represented inside Scipion? * Is the image imported into the database or just pointers to external files? * How does Scipion knows when to convert an input to a program? I'm very new to this, so sorry if I'm asking anything basic 😉 Kind regards, 👨💻 Ian L. Rodrigues 🏢 CNPEM, LNNano, CME 📞 +55 19 3512 1144 Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente. Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately. |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-27 14:11:09
|
Hi again, You could in principle have both and alternate...but in your case I think it is not worth. Even if you have 2GPU, you can use more than 1 MPI per GPU and also threads. So the 16 cores will be busy if you want to get the GPU busy as well. Yes, any consideration about the combination of MPI/threads/cores and GPUs that are valid for Relion, also apply when using Relion within Scipion. Best, Jose Miguel On Wed, Nov 27, 2019 at 3:04 PM Derek Logan <der...@bi...> wrote: > Hi Jose Miguel, > > Thanks for your rapid reply! I'll try what you suggest. It wasn't clear to > me from a first read of the Relion 3.0 paper in eLife whether GPU and CPU > acceleration were mutually exclusiveonly at run time or at the time of > compilation. I assumed it was at run time only. Basically we are looking > for a way to use our 14 idle CPU cores when 2 of them are running processes > using the GPUs, so the GPU/CPU optimisation conflict would not matter. > > No, we don't have the icc compiler and anyway our CPUs are AMD, so I don't > know if icc would offer any benefits? I haven't had to think so much about > hardware and compilation for the past 10-15 years as a protein > crystallographer ;-) > > Maybe these questions are better asked on a Relion forum anyway... > > /Derek > > On 27 Nov 2019, at 14:53, Jose Miguel de la Rosa Trevin < > del...@gm...> wrote: > > Hi Derek, > > If you want to have more control about Relion installation (and > compilation flags) > probably is a better idea to do it outside Scipion and then create the > proper link > in scipion/software/em/relion-3.0 -> /path/to/relion-3.0 > > Regarding CPU optimization, remember that this is exclusive with GPU > acceleration, > which I assume you want. Moreover, I'm not completely sure if for the > complete CPU > optimization, you need the Intel compiler (not GCC) which is not free. > > Kind regards, > Jose Miguel > > > > On Wed, Nov 27, 2019 at 2:21 PM Derek Logan < > der...@bi...> wrote: > >> Hi, >> >> I recently installed scipion 2.0 from source on a machine with an AMD >> 2950X 16-core CPU and two RTX 2080 Ti GPUs. I then installed Relion 3.0 >> using the plugin interface and gcc 7.4.0. How can I be sure that Relion has >> been compiled with the CPU acceleration that was introduced in Relion 3.0? >> I was not attentive to this aspect during the installation and now I don't >> know where to find the log files that might give me a clue. If Relion was >> not compiled with CPU acceleration, how can I redo things so that it is? >> Finally, when running e.g. 3D refinement, do we have to explicitly write >> "--cpu" in the "Additional arguments" box? >> >> Thanks in advance >> Derek >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Derek L. <der...@bi...> - 2019-11-27 14:03:44
|
Hi Jose Miguel, Thanks for your rapid reply! I'll try what you suggest. It wasn't clear to me from a first read of the Relion 3.0 paper in eLife whether GPU and CPU acceleration were mutually exclusiveonly at run time or at the time of compilation. I assumed it was at run time only. Basically we are looking for a way to use our 14 idle CPU cores when 2 of them are running processes using the GPUs, so the GPU/CPU optimisation conflict would not matter. No, we don't have the icc compiler and anyway our CPUs are AMD, so I don't know if icc would offer any benefits? I haven't had to think so much about hardware and compilation for the past 10-15 years as a protein crystallographer ;-) Maybe these questions are better asked on a Relion forum anyway... /Derek On 27 Nov 2019, at 14:53, Jose Miguel de la Rosa Trevin <del...@gm...<mailto:del...@gm...>> wrote: Hi Derek, If you want to have more control about Relion installation (and compilation flags) probably is a better idea to do it outside Scipion and then create the proper link in scipion/software/em/relion-3.0 -> /path/to/relion-3.0 Regarding CPU optimization, remember that this is exclusive with GPU acceleration, which I assume you want. Moreover, I'm not completely sure if for the complete CPU optimization, you need the Intel compiler (not GCC) which is not free. Kind regards, Jose Miguel On Wed, Nov 27, 2019 at 2:21 PM Derek Logan <der...@bi...<mailto:der...@bi...>> wrote: Hi, I recently installed scipion 2.0 from source on a machine with an AMD 2950X 16-core CPU and two RTX 2080 Ti GPUs. I then installed Relion 3.0 using the plugin interface and gcc 7.4.0. How can I be sure that Relion has been compiled with the CPU acceleration that was introduced in Relion 3.0? I was not attentive to this aspect during the installation and now I don't know where to find the log files that might give me a clue. If Relion was not compiled with CPU acceleration, how can I redo things so that it is? Finally, when running e.g. 3D refinement, do we have to explicitly write "--cpu" in the "Additional arguments" box? Thanks in advance Derek _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-27 13:53:31
|
Hi Derek, If you want to have more control about Relion installation (and compilation flags) probably is a better idea to do it outside Scipion and then create the proper link in scipion/software/em/relion-3.0 -> /path/to/relion-3.0 Regarding CPU optimization, remember that this is exclusive with GPU acceleration, which I assume you want. Moreover, I'm not completely sure if for the complete CPU optimization, you need the Intel compiler (not GCC) which is not free. Kind regards, Jose Miguel On Wed, Nov 27, 2019 at 2:21 PM Derek Logan <der...@bi...> wrote: > Hi, > > I recently installed scipion 2.0 from source on a machine with an AMD > 2950X 16-core CPU and two RTX 2080 Ti GPUs. I then installed Relion 3.0 > using the plugin interface and gcc 7.4.0. How can I be sure that Relion has > been compiled with the CPU acceleration that was introduced in Relion 3.0? > I was not attentive to this aspect during the installation and now I don't > know where to find the log files that might give me a clue. If Relion was > not compiled with CPU acceleration, how can I redo things so that it is? > Finally, when running e.g. 3D refinement, do we have to explicitly write > "--cpu" in the "Additional arguments" box? > > Thanks in advance > Derek > > > > > > > > > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Derek L. <der...@bi...> - 2019-11-27 13:21:01
|
Hi, I recently installed scipion 2.0 from source on a machine with an AMD 2950X 16-core CPU and two RTX 2080 Ti GPUs. I then installed Relion 3.0 using the plugin interface and gcc 7.4.0. How can I be sure that Relion has been compiled with the CPU acceleration that was introduced in Relion 3.0? I was not attentive to this aspect during the installation and now I don't know where to find the log files that might give me a clue. If Relion was not compiled with CPU acceleration, how can I redo things so that it is? Finally, when running e.g. 3D refinement, do we have to explicitly write "--cpu" in the "Additional arguments" box? Thanks in advance Derek |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-12 18:11:56
|
No problem. Thanks for letting me know. On Tue, Nov 12, 2019, 6:01 PM Karim Rafie <kar...@um...> wrote: > Hi Jose and all, > > > I found it after searching a bit more. Sorry for the spam. > > > Best wishes, > > Karim > > ############################################## > > Karim Rafie; PhD, AMRSC > > Postdoctoral Research Fellow > > Carlson Lab > > Department of Medical Biochemistry and Biophysics > > Umeå University > > Umeå / Sweden > ------------------------------ > *From:* Karim Rafie <kar...@um...> > *Sent:* 12 November 2019 17:02:50 > *To:* Mailing list for Scipion users > *Subject:* Re: [scipion-users] Error during particle extraction after > Localized reconstruction job > > Hi Jose, > > Thanks a lot for the help. It does make sense. I did look for the > localised extraction protocol but couldn't find it. Is there a particular > plug in that I need to have for it? When choosing the localised > reconstruction tool set from the drop down menu I only get the compute sub > particle job. > > Best, > Karim > > Get Outlook for Android <https://aka.ms/ghei36> > > > > > On Tue, Nov 12, 2019 at 3:17 PM +0100, "Jose Miguel de la Rosa Trevin" < > del...@gm...> wrote: > > Dear Karim, >> >> The computed sub-particles are kind of particles, but not exactly, since >> they come from other particles, not micrographs. >> That's is the error here..expecting micrograph in these protocols you >> tried. Still, I understand that might be confusing >> and we should put some validation to prevent it. >> >> You should use instead of the localrec - localized extraction protocol as >> shown below: >> >> [image: image.png] >> >> Hope that helps, >> Best, >> Jose Miguel >> >> >> >> On Tue, Nov 12, 2019 at 3:09 PM Karim Rafie <kar...@um...> wrote: >> >>> Dear all, >>> >>> >>> I'm trying to perform a localized reconstruction on a sub-particle of my >>> structure. I used the "compute sub-particles" job in scipion-v2.0 to >>> generate the coordinates of the sub-particles, however when I want to >>> extract them with relion-extract I get the following error message: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *" Traceback (most recent call last): 00011: File >>> "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 186, in run >>> 00012: self._run() 00013: File >>> "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 1283, in >>> _run 00014: self._insertAllSteps() # Define steps for execute later >>> 00015: File >>> "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line >>> 171, in _insertAllSteps 00016: micDict = self._loadInputList() >>> 00017: File >>> "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line >>> 359, in _loadInputList 00018: micDict, self.micsClosed = >>> _loadMics(coordMics) 00019: File >>> "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line >>> 350, in _loadMics 00020: lambda mic: mic.getMicName()) 00021: >>> File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", >>> line 340, in _loadSet 00022: micKey = getKeyFunc(item) 00023: >>> File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", >>> line 350, in <lambda> 00024: lambda mic: mic.getMicName()) 00025: >>> AttributeError: 'Particle' object has no attribute 'getMicName' 00026: >>> ------------------- PROTOCOL FAILED (DONE 0/0)"* >>> >>> >>> I get the same error, when wanting to extract the particles with xmipp. >>> Is this a bug or am I doing something wrong? >>> >>> >>> Any help would be greatly appreciated. >>> >>> >>> Best wishes, >>> >>> Karim >>> >>> ############################################## >>> >>> Karim Rafie; PhD, AMRSC >>> >>> Postdoctoral Research Fellow >>> >>> Carlson Lab >>> >>> Department of Medical Biochemistry and Biophysics >>> >>> Umeå University >>> >>> Umeå / Sweden >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Karim R. <kar...@um...> - 2019-11-12 17:00:47
|
Hi Jose and all, I found it after searching a bit more. Sorry for the spam. Best wishes, Karim ############################################## Karim Rafie; PhD, AMRSC Postdoctoral Research Fellow Carlson Lab Department of Medical Biochemistry and Biophysics Umeå University Umeå / Sweden ________________________________ From: Karim Rafie <kar...@um...> Sent: 12 November 2019 17:02:50 To: Mailing list for Scipion users Subject: Re: [scipion-users] Error during particle extraction after Localized reconstruction job Hi Jose, Thanks a lot for the help. It does make sense. I did look for the localised extraction protocol but couldn't find it. Is there a particular plug in that I need to have for it? When choosing the localised reconstruction tool set from the drop down menu I only get the compute sub particle job. Best, Karim Get Outlook for Android<https://aka.ms/ghei36> On Tue, Nov 12, 2019 at 3:17 PM +0100, "Jose Miguel de la Rosa Trevin" <del...@gm...<mailto:del...@gm...>> wrote: Dear Karim, The computed sub-particles are kind of particles, but not exactly, since they come from other particles, not micrographs. That's is the error here..expecting micrograph in these protocols you tried. Still, I understand that might be confusing and we should put some validation to prevent it. You should use instead of the localrec - localized extraction protocol as shown below: [image.png] Hope that helps, Best, Jose Miguel On Tue, Nov 12, 2019 at 3:09 PM Karim Rafie <kar...@um...<mailto:kar...@um...>> wrote: Dear all, I'm trying to perform a localized reconstruction on a sub-particle of my structure. I used the "compute sub-particles" job in scipion-v2.0 to generate the coordinates of the sub-particles, however when I want to extract them with relion-extract I get the following error message: " Traceback (most recent call last): 00011: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 186, in run 00012: self._run() 00013: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 1283, in _run 00014: self._insertAllSteps() # Define steps for execute later 00015: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 171, in _insertAllSteps 00016: micDict = self._loadInputList() 00017: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 359, in _loadInputList 00018: micDict, self.micsClosed = _loadMics(coordMics) 00019: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in _loadMics 00020: lambda mic: mic.getMicName()) 00021: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 340, in _loadSet 00022: micKey = getKeyFunc(item) 00023: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in <lambda> 00024: lambda mic: mic.getMicName()) 00025: AttributeError: 'Particle' object has no attribute 'getMicName' 00026: ------------------- PROTOCOL FAILED (DONE 0/0)" I get the same error, when wanting to extract the particles with xmipp. Is this a bug or am I doing something wrong? Any help would be greatly appreciated. Best wishes, Karim ############################################## Karim Rafie; PhD, AMRSC Postdoctoral Research Fellow Carlson Lab Department of Medical Biochemistry and Biophysics Umeå University Umeå / Sweden _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users |
|
From: Karim R. <kar...@um...> - 2019-11-12 16:03:23
|
Hi Jose, Thanks a lot for the help. It does make sense. I did look for the localised extraction protocol but couldn't find it. Is there a particular plug in that I need to have for it? When choosing the localised reconstruction tool set from the drop down menu I only get the compute sub particle job. Best, Karim Get Outlook for Android<https://aka.ms/ghei36> On Tue, Nov 12, 2019 at 3:17 PM +0100, "Jose Miguel de la Rosa Trevin" <del...@gm...<mailto:del...@gm...>> wrote: Dear Karim, The computed sub-particles are kind of particles, but not exactly, since they come from other particles, not micrographs. That's is the error here..expecting micrograph in these protocols you tried. Still, I understand that might be confusing and we should put some validation to prevent it. You should use instead of the localrec - localized extraction protocol as shown below: [image.png] Hope that helps, Best, Jose Miguel On Tue, Nov 12, 2019 at 3:09 PM Karim Rafie <kar...@um...<mailto:kar...@um...>> wrote: Dear all, I'm trying to perform a localized reconstruction on a sub-particle of my structure. I used the "compute sub-particles" job in scipion-v2.0 to generate the coordinates of the sub-particles, however when I want to extract them with relion-extract I get the following error message: " Traceback (most recent call last): 00011: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 186, in run 00012: self._run() 00013: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 1283, in _run 00014: self._insertAllSteps() # Define steps for execute later 00015: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 171, in _insertAllSteps 00016: micDict = self._loadInputList() 00017: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 359, in _loadInputList 00018: micDict, self.micsClosed = _loadMics(coordMics) 00019: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in _loadMics 00020: lambda mic: mic.getMicName()) 00021: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 340, in _loadSet 00022: micKey = getKeyFunc(item) 00023: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in <lambda> 00024: lambda mic: mic.getMicName()) 00025: AttributeError: 'Particle' object has no attribute 'getMicName' 00026: ------------------- PROTOCOL FAILED (DONE 0/0)" I get the same error, when wanting to extract the particles with xmipp. Is this a bug or am I doing something wrong? Any help would be greatly appreciated. Best wishes, Karim ############################################## Karim Rafie; PhD, AMRSC Postdoctoral Research Fellow Carlson Lab Department of Medical Biochemistry and Biophysics Umeå University Umeå / Sweden _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-12 14:16:00
|
Dear Karim, The computed sub-particles are kind of particles, but not exactly, since they come from other particles, not micrographs. That's is the error here..expecting micrograph in these protocols you tried. Still, I understand that might be confusing and we should put some validation to prevent it. You should use instead of the localrec - localized extraction protocol as shown below: [image: image.png] Hope that helps, Best, Jose Miguel On Tue, Nov 12, 2019 at 3:09 PM Karim Rafie <kar...@um...> wrote: > Dear all, > > > I'm trying to perform a localized reconstruction on a sub-particle of my > structure. I used the "compute sub-particles" job in scipion-v2.0 to > generate the coordinates of the sub-particles, however when I want to > extract them with relion-extract I get the following error message: > > > > > > > > > > > > > > > > > > > *" Traceback (most recent call last): 00011: File > "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 186, in run > 00012: self._run() 00013: File > "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 1283, in > _run 00014: self._insertAllSteps() # Define steps for execute later > 00015: File > "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line > 171, in _insertAllSteps 00016: micDict = self._loadInputList() > 00017: File > "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line > 359, in _loadInputList 00018: micDict, self.micsClosed = > _loadMics(coordMics) 00019: File > "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line > 350, in _loadMics 00020: lambda mic: mic.getMicName()) 00021: > File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", > line 340, in _loadSet 00022: micKey = getKeyFunc(item) 00023: > File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", > line 350, in <lambda> 00024: lambda mic: mic.getMicName()) 00025: > AttributeError: 'Particle' object has no attribute 'getMicName' 00026: > ------------------- PROTOCOL FAILED (DONE 0/0)"* > > > I get the same error, when wanting to extract the particles with xmipp. Is > this a bug or am I doing something wrong? > > > Any help would be greatly appreciated. > > > Best wishes, > > Karim > > ############################################## > > Karim Rafie; PhD, AMRSC > > Postdoctoral Research Fellow > > Carlson Lab > > Department of Medical Biochemistry and Biophysics > > Umeå University > > Umeå / Sweden > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Karim R. <kar...@um...> - 2019-11-12 14:08:47
|
Dear all, I'm trying to perform a localized reconstruction on a sub-particle of my structure. I used the "compute sub-particles" job in scipion-v2.0 to generate the coordinates of the sub-particles, however when I want to extract them with relion-extract I get the following error message: " Traceback (most recent call last): 00011: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 186, in run 00012: self._run() 00013: File "/usr/local/bin/scipion/pyworkflow/protocol/protocol.py", line 1283, in _run 00014: self._insertAllSteps() # Define steps for execute later 00015: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 171, in _insertAllSteps 00016: micDict = self._loadInputList() 00017: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 359, in _loadInputList 00018: micDict, self.micsClosed = _loadMics(coordMics) 00019: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in _loadMics 00020: lambda mic: mic.getMicName()) 00021: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 340, in _loadSet 00022: micKey = getKeyFunc(item) 00023: File "/usr/local/bin/scipion/pyworkflow/em/protocol/protocol_particles.py", line 350, in <lambda> 00024: lambda mic: mic.getMicName()) 00025: AttributeError: 'Particle' object has no attribute 'getMicName' 00026: ------------------- PROTOCOL FAILED (DONE 0/0)" I get the same error, when wanting to extract the particles with xmipp. Is this a bug or am I doing something wrong? Any help would be greatly appreciated. Best wishes, Karim ############################################## Karim Rafie; PhD, AMRSC Postdoctoral Research Fellow Carlson Lab Department of Medical Biochemistry and Biophysics Umeå University Umeå / Sweden |
|
From: Pablo C. <pc...@cn...> - 2019-11-09 16:08:33
|
Please, if someone is interested in this follow this thread at github issues: https://github.com/I2PC/scipion/issues/2066 On 9/11/19 13:41, Juha Huiskonen wrote: > Dear All, > > I am having the same issue. I have data collected in two different > sessions and they have been exported from two different raw data > folders. When I combine two SetOfParticels using 'scipion - join sets' > and try to run 'relion - 3D auto-refine' I get the following error: > > 00029: File > "/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/site-packages/relion/convert/convert.py", > line 757, in convertBinaryFiles > 00030: pw.utils.createLink(rootDir, outputRoot) > 00031: File > "/projappl/project_2001566/apps/scipion/2.0/pyworkflow/utils/path.py", > line 250, in createLink > 00032: relsource = join(relpath(sourcedir, destdir), > basename(source)) > 00033: File > "/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/posixpath.py", > line 428, in relpath > 00034: raise ValueError("no path specified") > 00035: ValueError: no path specified > 00036: Protocol failed: no path specified > 00037: FAILED: convertInputStep, step 1 > > Best wishes, > Juha > > On Thu, Nov 7, 2019 at 6:07 PM Dmitry Semchonok <Sem...@gm... > <mailto:Sem...@gm...>> wrote: > > Dear colleagues, > > I have 2 sets of particles from different micrographs that have > the same pixel size. > > I want to join (combine) them and run all together. > > What can be the strategy to do that? > > > What I did > > a)Combined 2 sets of micrographs (with which I worked independently). > > Combined 2 sets of particles; > > Tried to extract the combo coordinates from the combined micrographs. > > Error --- part of them extracted; part of them has wrong coordinates. > > And even with wrong coordinates if I run the 3d auto refinement I > get the error Protocol failed: no path specified. > > b)Combined all the micrographs in 1 project. > > Run MotionCor + CTF estimation > > Somehow import the combined coordinates from a) to this new project. > > Re-extract the coordinates. > > So I have a questions: > > -Why a) doesn’t work? Is there a bug? > > -How to export/import the combined coordinates into the new project? > > Amy other suggestions? > > p.s. when/if I succeed with a) with the particle polishing and ctf > refinement work? The 2 datasets have different dose rate info. > > Thank you! > > Sincerely, > > Dmitry > > _______________________________________________ > scipion-users mailing list > sci...@li... > <mailto:sci...@li...> > https://lists.sourceforge.net/lists/listinfo/scipion-users > > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* |
|
From: Juha H. <juh...@he...> - 2019-11-09 12:42:08
|
Dear All,
I am having the same issue. I have data collected in two different
sessions and they have been exported from two different raw data folders.
When I combine two SetOfParticels using 'scipion - join sets' and try to
run 'relion - 3D auto-refine' I get the following error:
00029: File
"/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/site-packages/relion/convert/convert.py",
line 757, in convertBinaryFiles
00030: pw.utils.createLink(rootDir, outputRoot)
00031: File
"/projappl/project_2001566/apps/scipion/2.0/pyworkflow/utils/path.py", line
250, in createLink
00032: relsource = join(relpath(sourcedir, destdir), basename(source))
00033: File
"/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/posixpath.py",
line 428, in relpath
00034: raise ValueError("no path specified")
00035: ValueError: no path specified
00036: Protocol failed: no path specified
00037: FAILED: convertInputStep, step 1
Best wishes,
Juha
On Thu, Nov 7, 2019 at 6:07 PM Dmitry Semchonok <Sem...@gm...> wrote:
> Dear colleagues,
>
> I have 2 sets of particles from different micrographs that have the same
> pixel size.
>
>
>
> I want to join (combine) them and run all together.
>
> What can be the strategy to do that?
>
>
>
>
> What I did
>
> a) Combined 2 sets of micrographs (with which I worked
> independently).
>
> Combined 2 sets of particles;
>
> Tried to extract the combo coordinates from the combined micrographs.
>
> Error --- part of them extracted; part of them has wrong coordinates.
>
>
>
> And even with wrong coordinates if I run the 3d auto refinement I get the
> error Protocol failed: no path specified.
>
>
>
> b) Combined all the micrographs in 1 project.
>
> Run MotionCor + CTF estimation
>
> Somehow import the combined coordinates from a) to this new project.
>
> Re-extract the coordinates.
>
>
>
>
>
> So I have a questions:
>
> - Why a) doesn’t work? Is there a bug?
>
> - How to export/import the combined coordinates into the new
> project?
>
> Amy other suggestions?
>
>
> p.s. when/if I succeed with a) with the particle polishing and ctf
> refinement work? The 2 datasets have different dose rate info.
>
> Thank you!
>
>
>
> Sincerely,
>
> Dmitry
>
>
>
>
>
>
>
>
> _______________________________________________
> scipion-users mailing list
> sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scipion-users
>
|
|
From: Dmitry S. <Sem...@gm...> - 2019-11-07 16:06:06
|
Dear colleagues, I have 2 sets of particles from different micrographs that have the same pixel size. I want to join (combine) them and run all together. What can be the strategy to do that? What I did a) Combined 2 sets of micrographs (with which I worked independently). Combined 2 sets of particles; Tried to extract the combo coordinates from the combined micrographs. Error --- part of them extracted; part of them has wrong coordinates. And even with wrong coordinates if I run the 3d auto refinement I get the error Protocol failed: no path specified. b) Combined all the micrographs in 1 project. Run MotionCor + CTF estimation Somehow import the combined coordinates from a) to this new project. Re-extract the coordinates. So I have a questions: - Why a) doesn’t work? Is there a bug? - How to export/import the combined coordinates into the new project? Amy other suggestions? p.s. when/if I succeed with a) with the particle polishing and ctf refinement work? The 2 datasets have different dose rate info. Thank you! Sincerely, Dmitry |
|
From: Pablo C. <pc...@cn...> - 2019-11-05 09:58:48
|
Hi Daniel. CUDA variable is there just for Xmipp. It will not affect Relion compilation or others. I recommend you to leave CUDA to True, xmipp has a movie correlation method in GPU and they are porting other method to GPU. You can set OPENCV=False if you hit compilation errors. On 1/11/19 11:22, Daniel Larsson wrote: > > >> On 2019-11-01, at 11:14, Daniel Larsson <dan...@ic... >> <mailto:dan...@ic...>> wrote: >> >> >> >>> On 2019-10-31, at 11:32, Pablo Conesa <pc...@CN... >>> <mailto:pc...@CN...>> wrote: >>> >>> Hi Daniel, >>> >>> About XMIPP: >>> >>> You probably have the plugin (python code that wrapps xmipp) but not >>> the actual xmipp. >>> >>> Try scipion installb >>> >>> I'm getting: >>> >>> xmippBin_Centos 3.19.04 [ ] >>> xmippBin_Debian 3.19.04 [ ] >>> xmippSrc 3.19.04 [X] >>> >>> First ones are compiled xmipp for Centos and Debian. >>> >>> 3rd are sources and will require compiling xmipp, but probably safer >>> in terms of dependencies. >>> >>> >> Running "scipion installb" revealed that I indeed do not have xmipp >> installed. I did not realize that you could drill down for Xmipp in >> the Plugin Manager GUI. When I tried installing xmippSrc, it turns >> out that I have a too old g++ compiler (needs 4.8 or later). I guess >> that is why I did not get xmipp in the first place. Now I figured out >> how to use g++ version 7.3.1, but run into problems with missing >> opencv headers. I don't think I will need such features and wondering >> how I should customize my installation to run with "--no-opencv". >> Where should I edit? > > Ok, found the OPENCV = True/False flag in config/scipion.conf thanks > to some googling: > https://sourceforge.net/p/scipion/mailman/message/36738786/ > > Also while digging around in the scipion.conf file, I realized that I > have CUDA = False. I guess I should remove all plugins and recompile > them with the CUDA = True if I want Relion with CUDA, right? > >>> >>> About CONDA_ACTIVATION_CMD: >>> >>> Some plugins, (in your case sphire) require to have conda installed. >>> Conda installation ask you to "add itself" to the shell so you have >>> conda in the PATH. >>> >>> When doing so, python calls, in some cases are messed up due to >>> having conda and its python exposed in the terminal. >>> >>> We recommend not to have conda initialized in the terminal and tell >>> scipion how to activate conda using the variable "CONDA_ACTIVATION_CMD" >>> >>> >>> In my case I got this in the <Scipion-home>/config/scipion.conf >>> under BUILD section: >>> >>> CONDA_ACTIVATION_CMD = eval "$(/extra/miniconda3/bin/conda >>> shell.bash hook)" >>> >>> NOTE: Conda usually tells you that command. >>> seehttps://github.com/scipion-em/scipion-em-sphire#cryolo-integration >>> >> Ok. I don't think there is Conda available. I'll uninstall >> Sphire/cryolo for the time being. >> >>> >>> About Locascale: Locscale plugin has a dependency to Eman plugin, >>> try to install eman plugin: >>> >>> scipion installp -p scipion-em-eman2 >>> >> Ok. Fixed that! >> >>> >>> >>> >>> On 31/10/19 10:55, Daniel Larsson wrote: >>>> So I just installed Scipion 2.0 on a CentOS cluster and installed some plugins using the Plugin Manger and get some warnings. First, I installed Xmipp, but I stil get a warning about missing Xmipp binaries when I start Scipion. I ran the scipion installp --help --checkUpdates command (see below for the output). There are also warnings about locscale and eman2/PyQt4. And what is the CONDA warning about? Any suggestions? >>>> >>>> Regards, >>>> Daniel >>>> >>>> >>>> >>>> >>>> >>>>> scipion installp --help --checkUpdates >>>> Scipion v2.0 (2019-04-23) Diocletian >>>> >>>>>>>>> python /home/larsson/software/scipion_v2.0/pyworkflow/install/install-plugin.py /home/larsson/software/scipion_v2.0/scipion installp --help --checkUpdates >>>> >>> WARNING: Xmipp binaries not found. Ghost active.....BOOOOOO! >>>> > Please install Xmipp to get full functionality. >>>> (Configuration->Plugins->scipion-em-xmipp in Scipion manager window) >>>> >>>> > error when importing from eman2: No module named PyQt4 >>>> Called by /home/larsson/software/scipion_v2.0/software/lib/python2.7/site-packages/locscale/constants.py, line 34 >>>> Check the plugin manager (Configuration->Plugins in Scipion manager window) >>>> or use 'scipion installp --help --checkUpdates' in the command line to check for upgrades, >>>> it could be a versions compatibility issue. >>>> >>>> Warning: couldn't find Plugin for scipion-em-locscale >>>> Dirname: locscale >>>> WARNING!!: CONDA_ACTIVATION_CMD variable not defined. Relying on conda being in the PATH >>>> usage: /home/larsson/software/scipion_v2.0/scipion installp [-h] [--noBin] [-p pluginName [pipVersion ...]] >>>> >>>> optional arguments: >>>> -h, --help show help >>>> --noBin Optional flag to install plugins only as a python module, >>>> without installing the plugin binaries. This will affect >>>> all plugins specified in the command. >>>> --checkUpdates Optional flag to check which plugins have new releases. >>>> -p pluginName [pluginVersion ...], --plugin pluginName [pluginVersion ...] >>>> - pluginName: the name of the plugin to install from the list >>>> of available plugins shown below. >>>> - pluginVersion: (optional) pip version to install. If not specified, >>>> will install the latest compatible with current Scipion. >>>> --devel Optional flag to indicate that we will pass install sources instead >>>> of pip names. Sources might be local paths or git urls. With local >>>> paths, will do pip install -e (editable mode). It is expected to find >>>> the plugin name in the basename of the path or in the repo name. >>>> (i.e. it needs to match the one specified in setup.py). E.g: >>>> scipion installp -p path/to/pluginName --devel >>>> scipion installp -phttps://github.com/someOrg/pluginName.git --devel >>>> -j j Number of CPUs to use for compilation >>>> >>>> Example: /home/larsson/software/scipion_v2.0/scipion installp -p scipion-em-grigoriefflab 1.0.1 -p scipion-em-relion -p scipion-em-eman2 >>>> >>>> Available plugins: ([ ] not installed, [X] seems already installed) >>>> >>>> scipion-em-appion 1.0.2 [X] >>>> scipion-em-atomstructutils 1.0.7 [X] >>>> scipion-em-atsas 1.0.6 [X] >>>> scipion-em-bamfordlab [ ] >>>> scipion-em-bsoft 1.0.3 [X] >>>> scipion-em-ccp4 1.0.12 [X] >>>> scipion-em-chimera 1.0.2 [X] >>>> scipion-em-cryoef 1.0.5 [X] >>>> scipion-em-cryomethods [ ] >>>> scipion-em-cryosparc2 1.0.7 [X] >>>> scipion-em-eman2 [ ] >>>> scipion-em-empiar 1.0.3 [X] >>>> scipion-em-emxlib [ ] >>>> scipion-em-esrf [ ] >>>> scipion-em-gautomatch 1.0.7 [X] >>>> scipion-em-gctf 1.0.7 [X] >>>> scipion-em-grigoriefflab 1.0.6 [X] >>>> scipion-em-imagic [ ] >>>> scipion-em-ispyb [ ] >>>> scipion-em-localrec 2.0 [X] >>>> scipion-em-locscale 1.0.5 [X] >>>> scipion-em-motioncorr 1.0.4 [X] >>>> scipion-em-nysbc [ ] >>>> scipion-em-phenix [ ] >>>> scipion-em-powerfit [ ] >>>> scipion-em-relion 1.0.7 [X] >>>> scipion-em-resmap 1.0.5 [X] >>>> scipion-em-simple [ ] >>>> scipion-em-sphire 1.2.9 [X] >>>> scipion-em-spider [ ] >>>> scipion-em-topaz [ ] >>>> scipion-em-xmipp 19.4.3 [X] >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här:http://www.uu.se/om-uu/dataskydd-personuppgifter/ >>>> >>>> E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here:http://www.uu.se/en/about-uu/data-protection-policy >>>> >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> -- >>> Pablo Conesa -*MadridScipion <http://scipion.i2pc.es/>team* >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> <mailto:sci...@li...> >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> <mailto:sci...@li...> >> https://lists.sourceforge.net/lists/listinfo/scipion-users > > > > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users -- Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team* |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-02 10:45:18
|
Ok, thanks a lot for the feedback! Please don't hesitate to let us know. On Sat, Nov 2, 2019 at 11:42 AM Juha Huiskonen <juh...@he...> wrote: > It's possible that my database file was corrupted. When I copied run.db > (from the most recent successful run) over project.sqlite things started to > work. However at the same time I added SQLITE_TMPDIR environmental > variable pointing to a large scratch partition. So either of these could > have been the solution... Our other users who have seen the same error will > keep testing and I will report back. > > On Sat, Nov 2, 2019 at 12:32 PM Jose Miguel de la Rosa Trevin < > del...@gm...> wrote: > >> Hum...probably Sqlite library generates some of these temporary files >> under the hood >> while operating with databases if necessary. As far as I know, these >> temporary files >> are in the same places of the main sqlite file. We don't use some of the >> operations >> described there, so I guess not all are used in our case. Anyway, non of >> these are >> handled by our code...I'm curious what is causing your troubles. >> >> On Fri, Nov 1, 2019 at 1:12 PM Juha Huiskonen <juh...@he...> >> wrote: >> >>> So no temporary files whatsoever? Nothing like the ones explained here? >>> --> https://www.sqlite.org/tempfiles.html >>> >>> On Fri, Nov 1, 2019 at 1:50 PM Jose Miguel de la Rosa Trevin < >>> del...@gm...> wrote: >>> >>>> Hi, >>>> >>>> All sqlite databases are written inside the project folder. Some are >>>> run.db inside each run folder. >>>> Others are the .sqlite files for output sets. >>>> >>>> Best, >>>> Jose Miguel >>>> >>>> >>>> >>>> On Fri, Nov 1, 2019 at 12:33 PM Juha Huiskonen < >>>> juh...@he...> wrote: >>>> >>>>> Hi Jose Miquel, >>>>> >>>>> We need to investigate and it would be good to know where the file is >>>>> written? Is it only the one written in the project directory (we do have >>>>> plenty of space there)? Or are there temporary files written somewhere >>>>> else? (If yes where? /tmp? ~/?) >>>>> >>>>> Best wishes, >>>>> Juha >>>>> >>>>> >>>>> On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < >>>>> del...@gm...> wrote: >>>>> >>>>>> Hi Juha, >>>>>> >>>>>> Many temporary SQL database are kept in memory and final output >>>>>> written to disk. >>>>>> Have you check quota errors or if there is space left on disk? >>>>>> >>>>>> Best, >>>>>> Jose Miguel >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen < >>>>>> juh...@he...> wrote: >>>>>> >>>>>>> Dear All, >>>>>>> >>>>>>> We are seeing this error with large datasets (over 1M particles). >>>>>>> Where is Scipion writing temporary sql files? I suspect there might be a >>>>>>> limit in the file system, causing this error. Or can it be because of >>>>>>> something else? Have others experienced this error and solved it? >>>>>>> >>>>>>> Best wishes, >>>>>>> Juha >>>>>>> _______________________________________________ >>>>>>> scipion-users mailing list >>>>>>> sci...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>>> >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>> _______________________________________________ >>>>> scipion-users mailing list >>>>> sci...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>> >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Juha H. <juh...@he...> - 2019-11-02 10:42:28
|
It's possible that my database file was corrupted. When I copied run.db (from the most recent successful run) over project.sqlite things started to work. However at the same time I added SQLITE_TMPDIR environmental variable pointing to a large scratch partition. So either of these could have been the solution... Our other users who have seen the same error will keep testing and I will report back. On Sat, Nov 2, 2019 at 12:32 PM Jose Miguel de la Rosa Trevin < del...@gm...> wrote: > Hum...probably Sqlite library generates some of these temporary files > under the hood > while operating with databases if necessary. As far as I know, these > temporary files > are in the same places of the main sqlite file. We don't use some of the > operations > described there, so I guess not all are used in our case. Anyway, non of > these are > handled by our code...I'm curious what is causing your troubles. > > On Fri, Nov 1, 2019 at 1:12 PM Juha Huiskonen <juh...@he...> > wrote: > >> So no temporary files whatsoever? Nothing like the ones explained here? >> --> https://www.sqlite.org/tempfiles.html >> >> On Fri, Nov 1, 2019 at 1:50 PM Jose Miguel de la Rosa Trevin < >> del...@gm...> wrote: >> >>> Hi, >>> >>> All sqlite databases are written inside the project folder. Some are >>> run.db inside each run folder. >>> Others are the .sqlite files for output sets. >>> >>> Best, >>> Jose Miguel >>> >>> >>> >>> On Fri, Nov 1, 2019 at 12:33 PM Juha Huiskonen < >>> juh...@he...> wrote: >>> >>>> Hi Jose Miquel, >>>> >>>> We need to investigate and it would be good to know where the file is >>>> written? Is it only the one written in the project directory (we do have >>>> plenty of space there)? Or are there temporary files written somewhere >>>> else? (If yes where? /tmp? ~/?) >>>> >>>> Best wishes, >>>> Juha >>>> >>>> >>>> On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < >>>> del...@gm...> wrote: >>>> >>>>> Hi Juha, >>>>> >>>>> Many temporary SQL database are kept in memory and final output >>>>> written to disk. >>>>> Have you check quota errors or if there is space left on disk? >>>>> >>>>> Best, >>>>> Jose Miguel >>>>> >>>>> >>>>> >>>>> On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen < >>>>> juh...@he...> wrote: >>>>> >>>>>> Dear All, >>>>>> >>>>>> We are seeing this error with large datasets (over 1M particles). >>>>>> Where is Scipion writing temporary sql files? I suspect there might be a >>>>>> limit in the file system, causing this error. Or can it be because of >>>>>> something else? Have others experienced this error and solved it? >>>>>> >>>>>> Best wishes, >>>>>> Juha >>>>>> _______________________________________________ >>>>>> scipion-users mailing list >>>>>> sci...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>>> >>>>> _______________________________________________ >>>>> scipion-users mailing list >>>>> sci...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>> >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-02 10:30:38
|
Hum...probably Sqlite library generates some of these temporary files under the hood while operating with databases if necessary. As far as I know, these temporary files are in the same places of the main sqlite file. We don't use some of the operations described there, so I guess not all are used in our case. Anyway, non of these are handled by our code...I'm curious what is causing your troubles. On Fri, Nov 1, 2019 at 1:12 PM Juha Huiskonen <juh...@he...> wrote: > So no temporary files whatsoever? Nothing like the ones explained here? > --> https://www.sqlite.org/tempfiles.html > > On Fri, Nov 1, 2019 at 1:50 PM Jose Miguel de la Rosa Trevin < > del...@gm...> wrote: > >> Hi, >> >> All sqlite databases are written inside the project folder. Some are >> run.db inside each run folder. >> Others are the .sqlite files for output sets. >> >> Best, >> Jose Miguel >> >> >> >> On Fri, Nov 1, 2019 at 12:33 PM Juha Huiskonen < >> juh...@he...> wrote: >> >>> Hi Jose Miquel, >>> >>> We need to investigate and it would be good to know where the file is >>> written? Is it only the one written in the project directory (we do have >>> plenty of space there)? Or are there temporary files written somewhere >>> else? (If yes where? /tmp? ~/?) >>> >>> Best wishes, >>> Juha >>> >>> >>> On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < >>> del...@gm...> wrote: >>> >>>> Hi Juha, >>>> >>>> Many temporary SQL database are kept in memory and final output written >>>> to disk. >>>> Have you check quota errors or if there is space left on disk? >>>> >>>> Best, >>>> Jose Miguel >>>> >>>> >>>> >>>> On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen < >>>> juh...@he...> wrote: >>>> >>>>> Dear All, >>>>> >>>>> We are seeing this error with large datasets (over 1M particles). >>>>> Where is Scipion writing temporary sql files? I suspect there might be a >>>>> limit in the file system, causing this error. Or can it be because of >>>>> something else? Have others experienced this error and solved it? >>>>> >>>>> Best wishes, >>>>> Juha >>>>> _______________________________________________ >>>>> scipion-users mailing list >>>>> sci...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>>> >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Juha H. <juh...@he...> - 2019-11-01 12:12:35
|
So no temporary files whatsoever? Nothing like the ones explained here? --> https://www.sqlite.org/tempfiles.html On Fri, Nov 1, 2019 at 1:50 PM Jose Miguel de la Rosa Trevin < del...@gm...> wrote: > Hi, > > All sqlite databases are written inside the project folder. Some are > run.db inside each run folder. > Others are the .sqlite files for output sets. > > Best, > Jose Miguel > > > > On Fri, Nov 1, 2019 at 12:33 PM Juha Huiskonen <juh...@he...> > wrote: > >> Hi Jose Miquel, >> >> We need to investigate and it would be good to know where the file is >> written? Is it only the one written in the project directory (we do have >> plenty of space there)? Or are there temporary files written somewhere >> else? (If yes where? /tmp? ~/?) >> >> Best wishes, >> Juha >> >> >> On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < >> del...@gm...> wrote: >> >>> Hi Juha, >>> >>> Many temporary SQL database are kept in memory and final output written >>> to disk. >>> Have you check quota errors or if there is space left on disk? >>> >>> Best, >>> Jose Miguel >>> >>> >>> >>> On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen < >>> juh...@he...> wrote: >>> >>>> Dear All, >>>> >>>> We are seeing this error with large datasets (over 1M particles). >>>> Where is Scipion writing temporary sql files? I suspect there might be a >>>> limit in the file system, causing this error. Or can it be because of >>>> something else? Have others experienced this error and solved it? >>>> >>>> Best wishes, >>>> Juha >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-01 11:49:02
|
Hi, All sqlite databases are written inside the project folder. Some are run.db inside each run folder. Others are the .sqlite files for output sets. Best, Jose Miguel On Fri, Nov 1, 2019 at 12:33 PM Juha Huiskonen <juh...@he...> wrote: > Hi Jose Miquel, > > We need to investigate and it would be good to know where the file is > written? Is it only the one written in the project directory (we do have > plenty of space there)? Or are there temporary files written somewhere > else? (If yes where? /tmp? ~/?) > > Best wishes, > Juha > > > On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < > del...@gm...> wrote: > >> Hi Juha, >> >> Many temporary SQL database are kept in memory and final output written >> to disk. >> Have you check quota errors or if there is space left on disk? >> >> Best, >> Jose Miguel >> >> >> >> On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen < >> juh...@he...> wrote: >> >>> Dear All, >>> >>> We are seeing this error with large datasets (over 1M particles). Where >>> is Scipion writing temporary sql files? I suspect there might be a limit in >>> the file system, causing this error. Or can it be because of something >>> else? Have others experienced this error and solved it? >>> >>> Best wishes, >>> Juha >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Juha H. <juh...@he...> - 2019-11-01 11:32:53
|
Hi Jose Miquel, We need to investigate and it would be good to know where the file is written? Is it only the one written in the project directory (we do have plenty of space there)? Or are there temporary files written somewhere else? (If yes where? /tmp? ~/?) Best wishes, Juha On Fri, Nov 1, 2019 at 1:12 PM Jose Miguel de la Rosa Trevin < del...@gm...> wrote: > Hi Juha, > > Many temporary SQL database are kept in memory and final output written to > disk. > Have you check quota errors or if there is space left on disk? > > Best, > Jose Miguel > > > > On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen <juh...@he...> > wrote: > >> Dear All, >> >> We are seeing this error with large datasets (over 1M particles). Where >> is Scipion writing temporary sql files? I suspect there might be a limit in >> the file system, causing this error. Or can it be because of something >> else? Have others experienced this error and solved it? >> >> Best wishes, >> Juha >> _______________________________________________ >> scipion-users mailing list >> sci...@li... >> https://lists.sourceforge.net/lists/listinfo/scipion-users >> > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Jose M. de la R. T. <del...@gm...> - 2019-11-01 11:11:11
|
Hi Juha, Many temporary SQL database are kept in memory and final output written to disk. Have you check quota errors or if there is space left on disk? Best, Jose Miguel On Fri, Nov 1, 2019 at 11:47 AM Juha Huiskonen <juh...@he...> wrote: > Dear All, > > We are seeing this error with large datasets (over 1M particles). Where > is Scipion writing temporary sql files? I suspect there might be a limit in > the file system, causing this error. Or can it be because of something > else? Have others experienced this error and solved it? > > Best wishes, > Juha > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
|
From: Juha H. <juh...@he...> - 2019-11-01 10:47:36
|
Dear All, We are seeing this error with large datasets (over 1M particles). Where is Scipion writing temporary sql files? I suspect there might be a limit in the file system, causing this error. Or can it be because of something else? Have others experienced this error and solved it? Best wishes, Juha |
1 message has been excluded from this view by a project administrator.
