Menu
▾
▴
Re: [scipion-users] Fwd: Scipion v1.1 difficulty continuing relion2d & relating joined particle set to micrographs via .xmd
|
From: Carlos O. S. <co...@cn...> - 2018-03-23 05:32:20
|
Dear Teige and all, sorry for joining this thread so late. Are the duplicates coming from two independent pickings or coordinate extraction? If so, the consensus picking will be able to remove those coordinates that are closer than a given distance. If not, I agree with José Miguel that something new should be written/adapted. But in this latter case, I would like to understand how these duplicates appeared. Kind regards, Carlos Oscar > ---------- Forwarded message ---------- > From: *Jose Miguel de la Rosa Trevin* <del...@gm... > <mailto:del...@gm...>> > Date: Wed, Mar 21, 2018 at 2:57 PM > Subject: Re: [scipion-users] Scipion v1.1 difficulty continuing > relion2d & relating joined particle set to micrographs via .xmd > To: "Matthews-Palmer, Teige Rowan Seal" > <t.m...@im... > <mailto:t.m...@im...>> > Cc: "sci...@li... > <mailto:sci...@li...>" > <sci...@li... > <mailto:sci...@li...>> > > > Great Teige! > > By the way, regarding the box related discussion in ccpem...I think > the protocol picking consensus can not > be used to remove duplicates. I have added a protocol named 'picking > differences' that use a negative > set of coordinates to remove from another set....I think that I can > easily modify that one to also remove > very close coordinates within the single input set. > > Best, > Jose Miguel > > > On Wed, Mar 21, 2018 at 2:53 PM, Matthews-Palmer, Teige Rowan Seal > <t.m...@im... > <mailto:t.m...@im...>> wrote: > > Dear Jose Miguel, > > Again a delayed reply, sorry. > The 1.2 candidate branch did fix the issue. :-) > In my case the issue was only with loading lists of particle sets, > and not other types of objects. > > All the best, > Teige > >> On 1 Mar 2018, at 12:29, Jose Miguel de la Rosa Trevin >> <del...@gm... <mailto:del...@gm...>> wrote: >> >> No worries! >> >> Can you try the version 1.2 release candidate (branch >> release-1.1.facilities-devel)?? >> We fixed an important issue that caused what you describe....a >> very long time >> when selecting any object as input. Was in big projects and with >> many 2D classifications. >> But I can't remember if this fix was introduced in 1.1 or not. >> >> Cheers, >> Jose Miguel >> >> >> On Thu, Mar 1, 2018 at 1:25 PM, Matthews-Palmer, Teige Rowan Seal >> <t.m...@im... >> <mailto:t.m...@im...>> wrote: >> >> Hi Jose Miguel, >> >> Sorry my reply is so late. I think that the problem might >> have been caused by me interrupting the process of generating >> the sqlite files, by closing the results viewer. In my case >> they were taking a very long time, like 10 minutes, (the file >> is >1GB) and I didn’t know that the delay was because an >> sqlite file needed to be generated. >> >> I am also experiencing that loading the list of particle sets >> for input particles in any protocol run window, takes a very >> long time. It is taking more than 10 minutes to open the >> sub-window list of particle sets. This is the same on both >> duplicates of a project on two (powerful) machines. It seems >> to be because the project is quite large. Does this sound >> normal? Is there anything I could do about it..? Scipion >> version is 1.1. >> >> Thanks for your help. >> Teige >> >> >>> On 9 Feb 2018, at 09:16, Jose Miguel de la Rosa Trevin >>> <del...@gm... <mailto:del...@gm...>> >>> wrote: >>> >>> Hi Teige, >>> >>> Can you check on the run folder (under >>> ProjectFolder/Runs/0000ID_ProtRelion2D or similar) >>> in the extra folder, if there are zero-size sqlite files? >>> You could try to delete the generated >>> sqlite files there and try the Analyze Results button >>> again...still it is weird why it has failed >>> in your case. (and it seems to others as well as mentioned >>> by Pablo). >>> >>> Best, >>> Jose Miguel >>> >>> >>> On Thu, Feb 8, 2018 at 10:06 PM, Matthews-Palmer, Teige >>> Rowan Seal <t.m...@im... >>> <mailto:t.m...@im...>> wrote: >>> >>> Dear Jose, >>> >>> Thanks for the quick & very helpful reply. >>> >>> I had a problem with generating the scipion output with >>> the protocol viewer (see image). Only the last iteration >>> generated _size, but the classavg images are not >>> available. I tried “Show classification in Scipion” for >>> many other iterations and they all show classavgs but as >>> empty classes i.e. the other data is not generated >>> properly. I just used the classID to pick the right >>> subset and it seems to have worked, so I will continue >>> like you suggest, thanks. >>> >>> Thanks a lot for explaining how to properly continue a >>> relion2d run in scipion! >>> Also thanks for pointing out that micrographName is what >>> keeps track of the micrographs. I spent a bit of effort >>> trying to trace it in the star files and understand why >>> scipion doesn’t mind the non-unique micID, but I didn’t >>> work it out. :-) >>> >>> That was a great help, thanks again. >>> All the best, >>> Teige >>> >>> >>> >>>> On 8 Feb 2018, at 19:56, Jose Miguel de la Rosa Trevin >>>> <del...@gm... >>>> <mailto:del...@gm...>> wrote: >>>> >>>> Dear Teige, >>>> >>>> Thanks for providing feedback about your use of >>>> Scipion...find below some answers to your questions. >>>> >>>> On Thu, Feb 8, 2018 at 8:16 PM, Matthews-Palmer, Teige >>>> Rowan Seal <t.m...@im... >>>> <mailto:t.m...@im...>> wrote: >>>> >>>> Dear Scipion Users, >>>> >>>> When running a large relion 2D classification, >>>> which fails unavoidably (this class of error: >>>> https://github.com/3dem/relion/issues/155 >>>> <https://github.com/3dem/relion/issues/155>) at a >>>> late iteration but has not reached the specified 25 >>>> iterations, scipion has not processed the relion >>>> output in its sqlite database (I’m guessing) and so >>>> when analysing the completed iterations, >>>> information is missing e.g. size=0 for all classes. >>>> Therefore subsets of particles cannot be selected >>>> to remove bad classes, re-extract with recentering >>>> and keep processing. >>>> This is effectively a roadblock to continue >>>> processing inside scipion. >>>> >>>> This is the default behaviour of many protocols >>>> wrapping iterative programs, Scipion only generate the >>>> output at the last iteration. We didn't wanted to >>>> generate output for every iteration because it will >>>> create many more objects and probably most of them are >>>> not desired by the user. So, when you click in the >>>> Analyze Results, you have an option to visualize the >>>> results in Scipion and convert the result of a given >>>> iteration to Scipion format. In this example, you could >>>> visualize the last iterate (option Show classification >>>> in Scipion) and from there you can create the subset of >>>> particles to continue. >>>> Then for the steps that you want to do, you should use >>>> the 'extract - coordinates' protocol where you provide >>>> the subset of particles and you can apply the shifts to >>>> re-center the particles. In your case, since your >>>> particles come from merging from two datasets of >>>> micrographs, you will need to join the micrographs to >>>> obtain a full set of micrographs and then extract the >>>> particles base on that joined set and using the >>>> original pixel size. >>>> >>>> Is there a way to get the scipion DB / GUI to >>>> recover the information from any given iteration? >>>> /(Tangentially, although running a relion command >>>> with —continue flag worked, in the scipion GUI the >>>> continue option failed, perhaps because ‘consider >>>> previous alignment’ was set to No? Scipion >>>> responded by starting at iteration 1, threatening >>>> to delete the successful iterations so far - which >>>> took ~2weeks! From there on scipion could not be >>>> persuaded to continue from iteration 18, and I do >>>> not know what would happen if I let it continue >>>> what looks like a restart.) >>>> / >>>> >>>> Here the thing that might be confusing is the Relion >>>> continue option and the Scipion continue ones, that are >>>> different. In Scipion, when you use Continue mode, it >>>> will try to continue from the last completed step, but >>>> in the case of Relion, the whole execution of the >>>> program is a big step, so Scipion continue does not >>>> work here. What you can do, is make a copy of the >>>> protocol and select the option in the form to 'Continue >>>> from a previous Run' (at the top of the form) and then >>>> provide the other run and the desired iteration (by >>>> default the last). I hope that I could clarify this >>>> instead of confusing you more. >>>> >>>> >>>> I can take the optimiser.star file to relion and >>>> make a subset selection, fine. >>>> However, now I would like to re-extract particles >>>> with recentering, and un-bin the particles later. >>>> These both seem to be difficult now. Especially >>>> because I joined two sets of particles before >>>> binning, and the MicrographIDs were *not* adjusted >>>> to be unique values in the unioned set. >>>> >>>> We noticed this problem at the begining of merging set >>>> of micrographs...so, you don't need to worry about the >>>> unique integer ID, which is re-generated when using the >>>> merge. We introduced a micName, that is kind of unique >>>> string identifier base on the filename. So, even if you >>>> import to set of movies and take particles from then, >>>> you can later join the micrographs (or movies) and >>>> re-extract the particles and we match the micName >>>> instead of the micID. >>>> >>>> If I import the subset particle.star back in to >>>> scipion, it generates outputMicrographs & >>>> outputParticles - the Micrograph info is wrong & >>>> can’t find the micrographs, it thinks there are >>>> 5615 micrographs, but this is wrong since there are >>>> now 1114 non-unique micIDs. >>>> Does anyone know how the relationship between >>>> particles in the joined set and their micrographs >>>> can be re-established? >>>> >>>> I recommend you here going the previously mentioned >>>> procedure...creating the subset - extracting >>>> coordinates (applying shifts) - extracting particles >>>> from the joined set of micrograhs. >>>> Anyway, you are right that the created SetOfMicrographs >>>> is not properly set in this case. I think that we have >>>> been using the import particles when importing >>>> completely from Relion, where we could generate a new >>>> SetOfMicrographs based on the .star files that make >>>> sense. I was trying to import some particles recently >>>> from Relion and trying to match to existing micrographs >>>> in the project and I found a bug because the micName >>>> was not properly set from the star file. So we need to >>>> fix this soon. >>>> >>>> >>>> for xmipp particle extraction, the image.xmd files >>>> (/star files) lose the connection to the >>>> micrographs because the _micrograph field >>>> is /tmp/[…]noDust.xmp rather than the .mrc output >>>> from motioncorr. Other than that there >>>> is _micrographId ; does anyone know where >>>> _micrographId is related to the corresponding >>>> micrograph? >>>> >>>> At this point I hope that you have a better idea of the >>>> relation of micId and micName , which is used in CTFs, >>>> particles and coordinates to match the corresponding >>>> micrograph. >>>> >>>> >>>> Any help is greatly appreciated. >>>> >>>> Please let me know if I could clarify some concepts and >>>> you can continue with your processing. Please do not >>>> hesitate to contact us if you have any other issue or >>>> feedback as well. >>>> >>>> All the best, >>>> Teige >>>> >>>> Kind regards >>>> Jose Miguel >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Check out the vibrant tech community on one of the >>>> world's most >>>> engaging tech sites, Slashdot.org >>>> <http://slashdot.org/>! http://sdm.link/slashdot >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> <mailto:sci...@li...> >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> <https://lists.sourceforge.net/lists/listinfo/scipion-users> >>>> >>>> >>> >>> >> >> > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > <mailto:sci...@li...> > https://lists.sourceforge.net/lists/listinfo/scipion-users > <https://lists.sourceforge.net/lists/listinfo/scipion-users> > > > > > -- > Prof. Jose-Maria Carazo > Biocomputing Unit, Head, CNB-CSIC > Spanish National Center for Biotechnology > Darwin 3, Universidad Autonoma de Madrid > 28049 Madrid, Spain > > > Cell: +34639197980 -- ------------------------------------------------------------------------ Carlos Oscar Sánchez Sorzano e-mail: co...@cn... Biocomputing unit http://i2pc.es/coss National Center of Biotechnology (CSIC) c/Darwin, 3 Campus Universidad Autónoma (Cantoblanco) Tlf: 34-91-585 4510 28049 MADRID (SPAIN) Fax: 34-91-585 4506 ------------------------------------------------------------------------ |
