Menu
▾
▴
Re: [scipion-users] Scipion v1.1 difficulty continuing relion2d & relating joined particle set to micrographs via .xmd
|
From: Jose M. de la R. T. <del...@gm...> - 2018-03-01 12:44:54
|
...apart from this bug-fix, there are many others for this new release, together with more features. On Thu, Mar 1, 2018 at 1:29 PM, Jose Miguel de la Rosa Trevin < del...@gm...> wrote: > No worries! > > Can you try the version 1.2 release candidate (branch > release-1.1.facilities-devel)?? > We fixed an important issue that caused what you describe....a very long > time > when selecting any object as input. Was in big projects and with many 2D > classifications. > But I can't remember if this fix was introduced in 1.1 or not. > > Cheers, > Jose Miguel > > > On Thu, Mar 1, 2018 at 1:25 PM, Matthews-Palmer, Teige Rowan Seal < > t.m...@im...> wrote: > >> Hi Jose Miguel, >> >> Sorry my reply is so late. I think that the problem might have been >> caused by me interrupting the process of generating the sqlite files, by >> closing the results viewer. In my case they were taking a very long time, >> like 10 minutes, (the file is >1GB) and I didn’t know that the delay was >> because an sqlite file needed to be generated. >> >> I am also experiencing that loading the list of particle sets for input >> particles in any protocol run window, takes a very long time. It is taking >> more than 10 minutes to open the sub-window list of particle sets. This is >> the same on both duplicates of a project on two (powerful) machines. It >> seems to be because the project is quite large. Does this sound normal? Is >> there anything I could do about it..? Scipion version is 1.1. >> >> Thanks for your help. >> Teige >> >> >> On 9 Feb 2018, at 09:16, Jose Miguel de la Rosa Trevin < >> del...@gm...> wrote: >> >> Hi Teige, >> >> Can you check on the run folder (under ProjectFolder/Runs/0000ID_ProtRelion2D >> or similar) >> in the extra folder, if there are zero-size sqlite files? You could try >> to delete the generated >> sqlite files there and try the Analyze Results button again...still it is >> weird why it has failed >> in your case. (and it seems to others as well as mentioned by Pablo). >> >> Best, >> Jose Miguel >> >> >> On Thu, Feb 8, 2018 at 10:06 PM, Matthews-Palmer, Teige Rowan Seal < >> t.m...@im...> wrote: >> >>> Dear Jose, >>> >>> Thanks for the quick & very helpful reply. >>> >>> I had a problem with generating the scipion output with the protocol >>> viewer (see image). Only the last iteration generated _size, but the >>> classavg images are not available. I tried “Show classification in Scipion” >>> for many other iterations and they all show classavgs but as empty classes >>> i.e. the other data is not generated properly. I just used the classID to >>> pick the right subset and it seems to have worked, so I will continue like >>> you suggest, thanks. >>> >>> Thanks a lot for explaining how to properly continue a relion2d run in >>> scipion! >>> Also thanks for pointing out that micrographName is what keeps track of >>> the micrographs. I spent a bit of effort trying to trace it in the star >>> files and understand why scipion doesn’t mind the non-unique micID, but I >>> didn’t work it out. :-) >>> >>> That was a great help, thanks again. >>> All the best, >>> Teige >>> >>> >>> >>> On 8 Feb 2018, at 19:56, Jose Miguel de la Rosa Trevin < >>> del...@gm...> wrote: >>> >>> Dear Teige, >>> >>> Thanks for providing feedback about your use of Scipion...find below >>> some answers to your questions. >>> >>> On Thu, Feb 8, 2018 at 8:16 PM, Matthews-Palmer, Teige Rowan Seal < >>> t.m...@im...> wrote: >>> >>>> Dear Scipion Users, >>>> >>>> When running a large relion 2D classification, which fails unavoidably >>>> (this class of error: https://github.com/3dem/relion/issues/155) at a >>>> late iteration but has not reached the specified 25 iterations, scipion has >>>> not processed the relion output in its sqlite database (I’m guessing) and >>>> so when analysing the completed iterations, information is missing e.g. >>>> size=0 for all classes. Therefore subsets of particles cannot be selected >>>> to remove bad classes, re-extract with recentering and keep processing. >>>> This is effectively a roadblock to continue processing inside scipion. >>>> >>> This is the default behaviour of many protocols wrapping iterative >>> programs, Scipion only generate the output at the last iteration. We didn't >>> wanted to generate output for every iteration because it will create many >>> more objects and probably most of them are not desired by the user. So, >>> when you click in the Analyze Results, you have an option to visualize the >>> results in Scipion and convert the result of a given iteration to Scipion >>> format. In this example, you could visualize the last iterate (option Show >>> classification in Scipion) and from there you can create the subset of >>> particles to continue. >>> Then for the steps that you want to do, you should use the 'extract - >>> coordinates' protocol where you provide the subset of particles and you can >>> apply the shifts to re-center the particles. In your case, since your >>> particles come from merging from two datasets of micrographs, you will need >>> to join the micrographs to obtain a full set of micrographs and then >>> extract the particles base on that joined set and using the original pixel >>> size. >>> >>>> Is there a way to get the scipion DB / GUI to recover the information >>>> from any given iteration? >>>> >>>> *(Tangentially, although running a relion command with —continue flag >>>> worked, in the scipion GUI the continue option failed, perhaps because >>>> ‘consider previous alignment’ was set to No? Scipion responded by starting >>>> at iteration 1, threatening to delete the successful iterations so far - >>>> which took ~2weeks! From there on scipion could not be persuaded to >>>> continue from iteration 18, and I do not know what would happen if I let it >>>> continue what looks like a restart.) * >>>> >>> Here the thing that might be confusing is the Relion continue option and >>> the Scipion continue ones, that are different. In Scipion, when you use >>> Continue mode, it will try to continue from the last completed step, but in >>> the case of Relion, the whole execution of the program is a big step, so >>> Scipion continue does not work here. What you can do, is make a copy of the >>> protocol and select the option in the form to 'Continue from a previous >>> Run' (at the top of the form) and then provide the other run and the >>> desired iteration (by default the last). I hope that I could clarify this >>> instead of confusing you more. >>> >>>> >>>> I can take the optimiser.star file to relion and make a subset >>>> selection, fine. >>>> However, now I would like to re-extract particles with recentering, and >>>> un-bin the particles later. These both seem to be difficult now. Especially >>>> because I joined two sets of particles before binning, and the >>>> MicrographIDs were *not* adjusted to be unique values in the unioned >>>> set. >>>> >>> We noticed this problem at the begining of merging set of >>> micrographs...so, you don't need to worry about the unique integer ID, >>> which is re-generated when using the merge. We introduced a micName, that >>> is kind of unique string identifier base on the filename. So, even if you >>> import to set of movies and take particles from then, you can later join >>> the micrographs (or movies) and re-extract the particles and we match the >>> micName instead of the micID. >>> >>>> If I import the subset particle.star back in to scipion, it generates >>>> outputMicrographs & outputParticles - the Micrograph info is wrong & can’t >>>> find the micrographs, it thinks there are 5615 micrographs, but this is >>>> wrong since there are now 1114 non-unique micIDs. >>>> Does anyone know how the relationship between particles in the joined >>>> set and their micrographs can be re-established? >>>> >>> I recommend you here going the previously mentioned procedure...creating >>> the subset - extracting coordinates (applying shifts) - extracting >>> particles from the joined set of micrograhs. >>> Anyway, you are right that the created SetOfMicrographs is not properly >>> set in this case. I think that we have been using the import particles when >>> importing completely from Relion, where we could generate a new >>> SetOfMicrographs based on the .star files that make sense. I was trying to >>> import some particles recently from Relion and trying to match to existing >>> micrographs in the project and I found a bug because the micName was not >>> properly set from the star file. So we need to fix this soon. >>> >>>> >>>> for xmipp particle extraction, the image.xmd files (/star files) lose >>>> the connection to the micrographs because the _micrograph field >>>> is /tmp/[…]noDust.xmp rather than the .mrc output from motioncorr. Other >>>> than that there is _micrographId ; does anyone know where _micrographId is >>>> related to the corresponding micrograph? >>>> >>> At this point I hope that you have a better idea of the relation of >>> micId and micName , which is used in CTFs, particles and coordinates to >>> match the corresponding micrograph. >>> >>>> >>>> Any help is greatly appreciated. >>>> >>> Please let me know if I could clarify some concepts and you can continue >>> with your processing. Please do not hesitate to contact us if you have any >>> other issue or feedback as well. >>> >>>> All the best, >>>> Teige >>>> >>> Kind regards >>> Jose Miguel >>> >>> >>>> >>>> ------------------------------------------------------------ >>>> ------------------ >>>> Check out the vibrant tech community on one of the world's most >>>> engaging tech sites, Slashdot.org <http://slashdot.org/>! >>>> http://sdm.link/slashdot >>>> _______________________________________________ >>>> scipion-users mailing list >>>> sci...@li... >>>> https://lists.sourceforge.net/lists/listinfo/scipion-users >>>> >>>> >>> >>> >> >> > |
