Re: [scipion-users] Scipion v1.1 difficulty continuing relion2d & relating joined particle set to micrographs via .xmd

[Jose M. de la R. T. <del...@gm...>]

Dear Teige,

Thanks for providing feedback about your use of Scipion...find below some
answers to your questions.

On Thu, Feb 8, 2018 at 8:16 PM, Matthews-Palmer, Teige Rowan Seal <
t.m...@im...> wrote:

> Dear Scipion Users,
>
> When running a large relion 2D classification, which fails unavoidably
> (this class of error: https://github.com/3dem/relion/issues/155) at a
> late iteration but has not reached the specified 25 iterations, scipion has
> not processed the relion output in its sqlite database (I’m guessing) and
> so when analysing the completed iterations, information is missing e.g.
> size=0 for all classes. Therefore subsets of particles cannot be selected
> to remove bad classes, re-extract with recentering and keep processing.
> This is effectively a roadblock to continue processing inside scipion.
>
This is the default behaviour of many protocols wrapping iterative
programs, Scipion only generate the output at the last iteration. We didn't
wanted to generate output for every iteration because it will create many
more objects and probably most of them are not desired by the user. So,
when you click in the Analyze Results, you have an option to visualize the
results in Scipion and convert the result of a given iteration to Scipion
format. In this example, you could visualize the last iterate (option Show
classification in Scipion) and from there you can create the subset of
particles to continue.
Then for the steps that you want to do, you should use the 'extract -
coordinates' protocol where you provide the subset of particles and you can
apply the shifts to re-center the particles. In your case, since your
particles come from merging from two datasets of micrographs, you will need
to join the micrographs to obtain a full set of micrographs and then
extract the particles base on that joined set and using the original pixel
size.

> Is there a way to get the scipion DB / GUI to recover the information from
> any given iteration?
>
> *(Tangentially, although running a relion command with —continue flag
> worked, in the scipion GUI the continue option failed, perhaps because
> ‘consider previous alignment’ was set to No? Scipion responded by starting
> at iteration 1, threatening to delete the successful iterations so far -
> which took ~2weeks! From there on scipion could not be persuaded to
> continue from iteration 18, and I do not know what would happen if I let it
> continue what looks like a restart.)*
>
Here the thing that might be confusing is the Relion continue option and
the Scipion continue ones, that are different. In Scipion, when you use
Continue mode, it will try to continue from the last completed step, but in
the case of Relion, the whole execution of the program is a big step, so
Scipion continue does not work here. What you can do, is make a copy of the
protocol and select the option in the form to 'Continue from a previous
Run' (at the top of the form) and then provide the other run and the
desired iteration (by default the last). I hope that I could clarify this
instead of confusing you more.

>
> I can take the optimiser.star file to relion and make a subset selection,
> fine.
> However, now I would like to re-extract particles with recentering, and
> un-bin the particles later. These both seem to be difficult now. Especially
> because I joined two sets of particles before binning, and the
> MicrographIDs were *not* adjusted to be unique values in the unioned set.
>
We noticed this problem at the begining of merging set of micrographs...so,
you don't need to worry about the unique integer ID, which is re-generated
when using the merge. We introduced a micName, that is kind of unique
string identifier base on the filename. So, even if you import to set of
movies and take particles from then, you can later join the micrographs (or
movies) and re-extract the particles and we match the micName instead of
the micID.

> If I import the subset particle.star back in to scipion, it generates
> outputMicrographs & outputParticles - the Micrograph info is wrong & can’t
> find the micrographs, it thinks there are 5615 micrographs, but this is
> wrong since there are now 1114 non-unique micIDs.
> Does anyone know how the relationship between particles in the joined set
> and their micrographs can be re-established?
>
I recommend you here going the previously mentioned procedure...creating
the subset - extracting coordinates (applying shifts) - extracting
particles from the joined set of micrograhs.
Anyway, you are right that the created SetOfMicrographs is not properly set
in this case. I think that we have been using the import particles when
importing completely from Relion, where we could generate a new
SetOfMicrographs based on the .star files that make sense. I was trying to
import some particles recently from Relion and trying to match to existing
micrographs in the project and I found a bug because the micName was not
properly set from the star file. So we need to fix this soon.

>
> for xmipp particle extraction, the image.xmd files (/star files) lose the
> connection to the micrographs because the _micrograph field
> is /tmp/[…]noDust.xmp rather than the .mrc output from motioncorr. Other
> than that there is  _micrographId ; does anyone know where _micrographId is
> related to the corresponding micrograph?
>
At this point I hope that you have a better idea of the relation of micId
and micName , which is used in CTFs, particles and coordinates to match the
corresponding micrograph.

>
> Any help is greatly appreciated.
>
Please let me know if I could clarify some concepts and you can continue
with your processing. Please do not hesitate to contact us if you have any
other issue or feedback as well.

> All the best,
> Teige
>
Kind regards
Jose Miguel


>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> scipion-users mailing list
> sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scipion-users
>
>