Menu
▾
▴
Re: [scipion-users] Scipion v1.1 difficulty continuing relion2d & relating joined particle set to micrographs via .xmd
|
From: Jose M. de la R. T. <del...@gm...> - 2018-03-01 12:29:17
|
No worries! Can you try the version 1.2 release candidate (branch release-1.1.facilities-devel)?? We fixed an important issue that caused what you describe....a very long time when selecting any object as input. Was in big projects and with many 2D classifications. But I can't remember if this fix was introduced in 1.1 or not. Cheers, Jose Miguel On Thu, Mar 1, 2018 at 1:25 PM, Matthews-Palmer, Teige Rowan Seal < t.m...@im...> wrote: > Hi Jose Miguel, > > Sorry my reply is so late. I think that the problem might have been caused > by me interrupting the process of generating the sqlite files, by closing > the results viewer. In my case they were taking a very long time, like 10 > minutes, (the file is >1GB) and I didn’t know that the delay was because an > sqlite file needed to be generated. > > I am also experiencing that loading the list of particle sets for input > particles in any protocol run window, takes a very long time. It is taking > more than 10 minutes to open the sub-window list of particle sets. This is > the same on both duplicates of a project on two (powerful) machines. It > seems to be because the project is quite large. Does this sound normal? Is > there anything I could do about it..? Scipion version is 1.1. > > Thanks for your help. > Teige > > > On 9 Feb 2018, at 09:16, Jose Miguel de la Rosa Trevin < > del...@gm...> wrote: > > Hi Teige, > > Can you check on the run folder (under ProjectFolder/Runs/0000ID_ProtRelion2D > or similar) > in the extra folder, if there are zero-size sqlite files? You could try to > delete the generated > sqlite files there and try the Analyze Results button again...still it is > weird why it has failed > in your case. (and it seems to others as well as mentioned by Pablo). > > Best, > Jose Miguel > > > On Thu, Feb 8, 2018 at 10:06 PM, Matthews-Palmer, Teige Rowan Seal < > t.m...@im...> wrote: > >> Dear Jose, >> >> Thanks for the quick & very helpful reply. >> >> I had a problem with generating the scipion output with the protocol >> viewer (see image). Only the last iteration generated _size, but the >> classavg images are not available. I tried “Show classification in Scipion” >> for many other iterations and they all show classavgs but as empty classes >> i.e. the other data is not generated properly. I just used the classID to >> pick the right subset and it seems to have worked, so I will continue like >> you suggest, thanks. >> >> Thanks a lot for explaining how to properly continue a relion2d run in >> scipion! >> Also thanks for pointing out that micrographName is what keeps track of >> the micrographs. I spent a bit of effort trying to trace it in the star >> files and understand why scipion doesn’t mind the non-unique micID, but I >> didn’t work it out. :-) >> >> That was a great help, thanks again. >> All the best, >> Teige >> >> >> >> On 8 Feb 2018, at 19:56, Jose Miguel de la Rosa Trevin < >> del...@gm...> wrote: >> >> Dear Teige, >> >> Thanks for providing feedback about your use of Scipion...find below some >> answers to your questions. >> >> On Thu, Feb 8, 2018 at 8:16 PM, Matthews-Palmer, Teige Rowan Seal < >> t.m...@im...> wrote: >> >>> Dear Scipion Users, >>> >>> When running a large relion 2D classification, which fails unavoidably >>> (this class of error: https://github.com/3dem/relion/issues/155) at a >>> late iteration but has not reached the specified 25 iterations, scipion has >>> not processed the relion output in its sqlite database (I’m guessing) and >>> so when analysing the completed iterations, information is missing e.g. >>> size=0 for all classes. Therefore subsets of particles cannot be selected >>> to remove bad classes, re-extract with recentering and keep processing. >>> This is effectively a roadblock to continue processing inside scipion. >>> >> This is the default behaviour of many protocols wrapping iterative >> programs, Scipion only generate the output at the last iteration. We didn't >> wanted to generate output for every iteration because it will create many >> more objects and probably most of them are not desired by the user. So, >> when you click in the Analyze Results, you have an option to visualize the >> results in Scipion and convert the result of a given iteration to Scipion >> format. In this example, you could visualize the last iterate (option Show >> classification in Scipion) and from there you can create the subset of >> particles to continue. >> Then for the steps that you want to do, you should use the 'extract - >> coordinates' protocol where you provide the subset of particles and you can >> apply the shifts to re-center the particles. In your case, since your >> particles come from merging from two datasets of micrographs, you will need >> to join the micrographs to obtain a full set of micrographs and then >> extract the particles base on that joined set and using the original pixel >> size. >> >>> Is there a way to get the scipion DB / GUI to recover the information >>> from any given iteration? >>> >>> *(Tangentially, although running a relion command with —continue flag >>> worked, in the scipion GUI the continue option failed, perhaps because >>> ‘consider previous alignment’ was set to No? Scipion responded by starting >>> at iteration 1, threatening to delete the successful iterations so far - >>> which took ~2weeks! From there on scipion could not be persuaded to >>> continue from iteration 18, and I do not know what would happen if I let it >>> continue what looks like a restart.) * >>> >> Here the thing that might be confusing is the Relion continue option and >> the Scipion continue ones, that are different. In Scipion, when you use >> Continue mode, it will try to continue from the last completed step, but in >> the case of Relion, the whole execution of the program is a big step, so >> Scipion continue does not work here. What you can do, is make a copy of the >> protocol and select the option in the form to 'Continue from a previous >> Run' (at the top of the form) and then provide the other run and the >> desired iteration (by default the last). I hope that I could clarify this >> instead of confusing you more. >> >>> >>> I can take the optimiser.star file to relion and make a subset >>> selection, fine. >>> However, now I would like to re-extract particles with recentering, and >>> un-bin the particles later. These both seem to be difficult now. Especially >>> because I joined two sets of particles before binning, and the >>> MicrographIDs were *not* adjusted to be unique values in the unioned >>> set. >>> >> We noticed this problem at the begining of merging set of >> micrographs...so, you don't need to worry about the unique integer ID, >> which is re-generated when using the merge. We introduced a micName, that >> is kind of unique string identifier base on the filename. So, even if you >> import to set of movies and take particles from then, you can later join >> the micrographs (or movies) and re-extract the particles and we match the >> micName instead of the micID. >> >>> If I import the subset particle.star back in to scipion, it generates >>> outputMicrographs & outputParticles - the Micrograph info is wrong & can’t >>> find the micrographs, it thinks there are 5615 micrographs, but this is >>> wrong since there are now 1114 non-unique micIDs. >>> Does anyone know how the relationship between particles in the joined >>> set and their micrographs can be re-established? >>> >> I recommend you here going the previously mentioned procedure...creating >> the subset - extracting coordinates (applying shifts) - extracting >> particles from the joined set of micrograhs. >> Anyway, you are right that the created SetOfMicrographs is not properly >> set in this case. I think that we have been using the import particles when >> importing completely from Relion, where we could generate a new >> SetOfMicrographs based on the .star files that make sense. I was trying to >> import some particles recently from Relion and trying to match to existing >> micrographs in the project and I found a bug because the micName was not >> properly set from the star file. So we need to fix this soon. >> >>> >>> for xmipp particle extraction, the image.xmd files (/star files) lose >>> the connection to the micrographs because the _micrograph field >>> is /tmp/[…]noDust.xmp rather than the .mrc output from motioncorr. Other >>> than that there is _micrographId ; does anyone know where _micrographId is >>> related to the corresponding micrograph? >>> >> At this point I hope that you have a better idea of the relation of micId >> and micName , which is used in CTFs, particles and coordinates to match the >> corresponding micrograph. >> >>> >>> Any help is greatly appreciated. >>> >> Please let me know if I could clarify some concepts and you can continue >> with your processing. Please do not hesitate to contact us if you have any >> other issue or feedback as well. >> >>> All the best, >>> Teige >>> >> Kind regards >> Jose Miguel >> >> >>> >>> ------------------------------------------------------------ >>> ------------------ >>> Check out the vibrant tech community on one of the world's most >>> engaging tech sites, Slashdot.org <http://slashdot.org/>! >>> http://sdm.link/slashdot >>> _______________________________________________ >>> scipion-users mailing list >>> sci...@li... >>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>> >>> >> >> > > |
