Menu
▾
▴
Re: [scipion-users] Scipion v1.1 difficulty continuing relion2d & relating joined particle set to micrographs via .xmd
|
From: Matthews-Palmer, T. R. S. <t.m...@im...> - 2018-03-21 13:53:46
|
Dear Jose Miguel, Again a delayed reply, sorry. The 1.2 candidate branch did fix the issue. :-) In my case the issue was only with loading lists of particle sets, and not other types of objects. All the best, Teige On 1 Mar 2018, at 12:29, Jose Miguel de la Rosa Trevin <del...@gm...<mailto:del...@gm...>> wrote: No worries! Can you try the version 1.2 release candidate (branch release-1.1.facilities-devel)?? We fixed an important issue that caused what you describe....a very long time when selecting any object as input. Was in big projects and with many 2D classifications. But I can't remember if this fix was introduced in 1.1 or not. Cheers, Jose Miguel On Thu, Mar 1, 2018 at 1:25 PM, Matthews-Palmer, Teige Rowan Seal <t.m...@im...<mailto:t.m...@im...>> wrote: Hi Jose Miguel, Sorry my reply is so late. I think that the problem might have been caused by me interrupting the process of generating the sqlite files, by closing the results viewer. In my case they were taking a very long time, like 10 minutes, (the file is >1GB) and I didn’t know that the delay was because an sqlite file needed to be generated. I am also experiencing that loading the list of particle sets for input particles in any protocol run window, takes a very long time. It is taking more than 10 minutes to open the sub-window list of particle sets. This is the same on both duplicates of a project on two (powerful) machines. It seems to be because the project is quite large. Does this sound normal? Is there anything I could do about it..? Scipion version is 1.1. Thanks for your help. Teige On 9 Feb 2018, at 09:16, Jose Miguel de la Rosa Trevin <del...@gm...<mailto:del...@gm...>> wrote: Hi Teige, Can you check on the run folder (under ProjectFolder/Runs/0000ID_ProtRelion2D or similar) in the extra folder, if there are zero-size sqlite files? You could try to delete the generated sqlite files there and try the Analyze Results button again...still it is weird why it has failed in your case. (and it seems to others as well as mentioned by Pablo). Best, Jose Miguel On Thu, Feb 8, 2018 at 10:06 PM, Matthews-Palmer, Teige Rowan Seal <t.m...@im...<mailto:t.m...@im...>> wrote: Dear Jose, Thanks for the quick & very helpful reply. I had a problem with generating the scipion output with the protocol viewer (see image). Only the last iteration generated _size, but the classavg images are not available. I tried “Show classification in Scipion” for many other iterations and they all show classavgs but as empty classes i.e. the other data is not generated properly. I just used the classID to pick the right subset and it seems to have worked, so I will continue like you suggest, thanks. Thanks a lot for explaining how to properly continue a relion2d run in scipion! Also thanks for pointing out that micrographName is what keeps track of the micrographs. I spent a bit of effort trying to trace it in the star files and understand why scipion doesn’t mind the non-unique micID, but I didn’t work it out. :-) That was a great help, thanks again. All the best, Teige On 8 Feb 2018, at 19:56, Jose Miguel de la Rosa Trevin <del...@gm...<mailto:del...@gm...>> wrote: Dear Teige, Thanks for providing feedback about your use of Scipion...find below some answers to your questions. On Thu, Feb 8, 2018 at 8:16 PM, Matthews-Palmer, Teige Rowan Seal <t.m...@im...<mailto:t.m...@im...>> wrote: Dear Scipion Users, When running a large relion 2D classification, which fails unavoidably (this class of error: https://github.com/3dem/relion/issues/155) at a late iteration but has not reached the specified 25 iterations, scipion has not processed the relion output in its sqlite database (I’m guessing) and so when analysing the completed iterations, information is missing e.g. size=0 for all classes. Therefore subsets of particles cannot be selected to remove bad classes, re-extract with recentering and keep processing. This is effectively a roadblock to continue processing inside scipion. This is the default behaviour of many protocols wrapping iterative programs, Scipion only generate the output at the last iteration. We didn't wanted to generate output for every iteration because it will create many more objects and probably most of them are not desired by the user. So, when you click in the Analyze Results, you have an option to visualize the results in Scipion and convert the result of a given iteration to Scipion format. In this example, you could visualize the last iterate (option Show classification in Scipion) and from there you can create the subset of particles to continue. Then for the steps that you want to do, you should use the 'extract - coordinates' protocol where you provide the subset of particles and you can apply the shifts to re-center the particles. In your case, since your particles come from merging from two datasets of micrographs, you will need to join the micrographs to obtain a full set of micrographs and then extract the particles base on that joined set and using the original pixel size. Is there a way to get the scipion DB / GUI to recover the information from any given iteration? (Tangentially, although running a relion command with —continue flag worked, in the scipion GUI the continue option failed, perhaps because ‘consider previous alignment’ was set to No? Scipion responded by starting at iteration 1, threatening to delete the successful iterations so far - which took ~2weeks! From there on scipion could not be persuaded to continue from iteration 18, and I do not know what would happen if I let it continue what looks like a restart.) Here the thing that might be confusing is the Relion continue option and the Scipion continue ones, that are different. In Scipion, when you use Continue mode, it will try to continue from the last completed step, but in the case of Relion, the whole execution of the program is a big step, so Scipion continue does not work here. What you can do, is make a copy of the protocol and select the option in the form to 'Continue from a previous Run' (at the top of the form) and then provide the other run and the desired iteration (by default the last). I hope that I could clarify this instead of confusing you more. I can take the optimiser.star file to relion and make a subset selection, fine. However, now I would like to re-extract particles with recentering, and un-bin the particles later. These both seem to be difficult now. Especially because I joined two sets of particles before binning, and the MicrographIDs were not adjusted to be unique values in the unioned set. We noticed this problem at the begining of merging set of micrographs...so, you don't need to worry about the unique integer ID, which is re-generated when using the merge. We introduced a micName, that is kind of unique string identifier base on the filename. So, even if you import to set of movies and take particles from then, you can later join the micrographs (or movies) and re-extract the particles and we match the micName instead of the micID. If I import the subset particle.star back in to scipion, it generates outputMicrographs & outputParticles - the Micrograph info is wrong & can’t find the micrographs, it thinks there are 5615 micrographs, but this is wrong since there are now 1114 non-unique micIDs. Does anyone know how the relationship between particles in the joined set and their micrographs can be re-established? I recommend you here going the previously mentioned procedure...creating the subset - extracting coordinates (applying shifts) - extracting particles from the joined set of micrograhs. Anyway, you are right that the created SetOfMicrographs is not properly set in this case. I think that we have been using the import particles when importing completely from Relion, where we could generate a new SetOfMicrographs based on the .star files that make sense. I was trying to import some particles recently from Relion and trying to match to existing micrographs in the project and I found a bug because the micName was not properly set from the star file. So we need to fix this soon. for xmipp particle extraction, the image.xmd files (/star files) lose the connection to the micrographs because the _micrograph field is /tmp/[…]noDust.xmp rather than the .mrc output from motioncorr. Other than that there is _micrographId ; does anyone know where _micrographId is related to the corresponding micrograph? At this point I hope that you have a better idea of the relation of micId and micName , which is used in CTFs, particles and coordinates to match the corresponding micrograph. Any help is greatly appreciated. Please let me know if I could clarify some concepts and you can continue with your processing. Please do not hesitate to contact us if you have any other issue or feedback as well. All the best, Teige Kind regards Jose Miguel ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org<http://slashdot.org/>! http://sdm.link/slashdot _______________________________________________ scipion-users mailing list sci...@li...<mailto:sci...@li...> https://lists.sourceforge.net/lists/listinfo/scipion-users |
