Re: [scipion-users] How to export files?

[Jose M. de la R. T. <del...@gm...>]

Hi Matt Harrington (disambiguation needed now ;)

As Matt Hutchison pointed out, you can created a new Scipion project for
each session, or more than one per session.
Then you could manually import templates for the pre-processing steps. Some
of these task are ease/automate by
the use of "session wizards", that are basically Python scripts (or whole
apps) that use the Scipion API to setup
in some case folders structures, basic initial parameters and the creation
of the project template. So, between one user
and the next one, should not be any issue, unless you run out of disk space
or you want to prevent one uses to see
the data from the previous one...in which case I think it should make more
sense having different OS users.

We have created different session wizards to setups things
and create the project template...taking care of some data management. In
our case (CryoEM facility in Stockholm), the wizard also
fetch information from the Booking system and the Application portal, where
we have information
about the project that the current session belongs (and the PI that should
be invoiced). These wizards
are ad-hoc, so no general for every place... you can see the Stockholm one
here: https://github.com/delarosatrevin/scipion-session
(See also attached image)

And some instructions for the internal users about how to use it:
https://github.com/delarosatrevin/scipion-session/wiki/Instructions-to-use-the-Session-Wizard-at-SciLifeLab
Grigory Sharov have also created one for the LMB:
https://github.com/azazellochg/scipion-session-lmb
And the web one being used in Madrid:
https://github.com/I2PC/EMadmin/tree/master

Although Scipion has been designed with extensibility in mind (through
adding new plugins, protocols, viewers, etc),
creating a new plugin for data transfer seems a bit overkilling. My
colleagues in Madrid has developed a protocol (or
script) to transfer the project data to an external hdd while the
pre-processing is going on. I haven't found it, but
they might comment more on this. If it is no longer used there, could be a
better starting point for your use case.

You might want to take a look at our paper about using Scipion for CryoEM
facilities here:
https://www.sciencedirect.com/science/article/pii/S1047847718302740

Best,
Jose Miguel



On Fri, May 24, 2019 at 9:19 PM Hutchison, Matt (NIH/NCI) [C] via
scipion-users <sci...@li...> wrote:

> Matt,
>
>
>
> The overall layout sounds similar to what we deal with here.  In our case,
> we have a project template that is executed using the scripting interface,
> so we run the same tasks during each imaging session but they’re created as
> separate projects at execution time, with a few variables set in the
> template (project name, pixel size, paths, ctf parameters).  It’s also
> possible to do something similar in the GUI by creating a project that can
> be imported w/copy at the start of each session to create a new project
> each time with the same basic template.  We use v1.2.1 right now, but I
> think that both of these are still available in v2.0.
>
>
>
> Also, can you share what precisely you’re doing when you see files being
> deleted from the project?  That sounds odd to me.
>
>
>
> *Matt Hutchison (Contractor)*
>
> Software Developer
>
> National Cryo-EM Facility
>
> Cancer Research Technology Program
>
> Frederick National Laboratory for Cancer Research
>
> Leidos Biomedical Research Inc.
>
> PO Box B Frederick, MD 21702
>
> Phone: 301-360-3483
>
> Email: tho...@ni...
>
>
>
> *From: *Matt Harrington <ma...@ms...>
> *Reply-To: *Mailing list for Scipion users <
> sci...@li...>
> *Date: *Friday, May 24, 2019 at 2:06 PM
> *To: *Mailing list for Scipion users <sci...@li...>
> *Subject: *Re: [scipion-users] How to export files?
>
>
>
> On Fri, May 24, 2019 at 9:23 AM Jose Miguel de la Rosa Trevin <
> del...@gm...> wrote:
>
>
>
> If I understood correctly your question, you are talking more about
> transferring the data of a pre-processing project.
>
>
>
> Correct.  Thanks for your thoughts.  Here are a few more details.  We run
> under a shared user account, and we're expecting that each person will use
> the same Scipion pre-processing pipeline.  It sounds like users should
> manually copy drift-corrected files out of the project folder before the
> next user comes along.  Otherwise, they'll be deleted.
>
>
>
> Managing data transfer/policies is beyond the scope of Scipion...and it is
> almost impossible to do it in a general (and robust) way that serve all the
> different situations.
>
>
>
> The good thing is that we can create our own export plug-in.  I assume
> this contains all we need to know:
> https://scipion-em.github.io/docs/docs/developer/creating-a-plugin.
>
>
>
> Matt
>
>
> _______________________________________________
> scipion-users mailing list
> sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scipion-users
>