Re: [scipion-users] scipion3, own special scipion conda - cluster submission problems/question

[Pablo C. <pc...@cn...>]

Let me see if I'm getting it.

Are you trying to avoid the launching script (scipion3) and instead you 
want to run "manually" a protocol?

If so, you need to replicate what the launching script does:


Here is what I've got in the launching script hightlighting the key lines:


#!/usr/bin/env python
# Scipion launcher
import os
import sys
from os.path import dirname, abspath, join, basename

# Set SCIPION_HOME to the location of this file
scipionHome = dirname(abspath(__file__))
os.environ["SCIPION_TESTS_CMD"] = basename(__file__) + " tests"
os.environ["LD_LIBRARY_PATH"] = 
":".join([os.environ.get("LD_LIBRARY_PATH", ""), join(scipionHome, 
"software", "lib")])
os.environ["PYTHONPATH"] = ":".join([os.environ.get("PYTHONPATH", ""), 
join(scipionHome, "software", "bindings")])

cmd = ""
if len(sys.argv) > 1 and sys.argv[1] == 'git':
     for repo in ['scipion-app', 'scipion-pyworkflow', 'scipion-em']:
         cmd += ("(cd "+join(scipionHome, repo)+" ; echo ' > in 
"+repo+":' ;"
                 " git "+' '.join(sys.argv[2:])+" ; echo) ; ")
else:
     # Activate the environment
     cmd = '. /home/pablo/software/scipion/.scipion3env/bin/activate && '
     cmd += "python -m scipion %s" % " ".join(sys.argv[1:])

# Set SCIPION_HOME
os.environ["SCIPION_HOME"] = scipionHome
exit(os.WEXITSTATUS(os.system(cmd)))


A short explanation:

LD_LIBRARY_PATH and PYTHONPATH are set to allow scipion use xmipp 
fileformats convertions, metadata utilities and it's viewer.

SCIPION_HOME, is mandatory, since now the path of pyworkflow (pip 
package) cannot determine scipion home as it was for scipion2.


python -m scipion is to run entry point commands but in your case you 
can skip this.




On 11/9/20 16:44, Lugmayr, Wolfgang wrote:
> Hi,
>
> I have re-installed recently scipion3 with its own conda environment. Mainly for the reason that the plugins could overwrite my conda installation e.g. mine is cryolo-1.7 and the scipion plugin creates a cryolo-1.7.2
>
> My new scipion3 conda looks now like:
> base                  *  /beegfs/cssb/software/em/scipion/3.0/anaconda3
> .scipion3env             /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env
> cryolo-1.7.2             /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/cryolo-1.7.2
> cryoloCPU-1.7.2          /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/cryoloCPU-1.7.2
> xmipp_DLTK_v0.3          /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/xmipp_DLTK_v0.3
> xmipp_MicCleaner         /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/xmipp_MicCleaner
>
> If I run everything locally via scipion3 the task runs fine.
>
> The task script for the cluster contains:
> :
> source /beegfs/cssb/software/em/scipion/3.0/anaconda3/etc/profile.d/conda.sh
> conda activate .scipion3env
> which python3
> ################################
> python3 /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/apps/pw_protocol_run.py "/home/lugmayr/ScipionUserData/projects/Scipion3_TestSuite" "Runs/000146_XmippProtMovieCorr/logs/run.db" 146
>
> Now I get the following error:
> ### jobid : 5452483  ###
> /beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/bin/python3
> Traceback (most recent call last):
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/apps/pw_protocol_run.py", line 39, in <module>
>      runProtocolMain(projPath, dbPath, protId)
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 2167, in runProtocolMain
>      protocol = getProtocolFromDb(projectPath, protDbPath, protId, chdir=True)
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 2249, in getProtocolFromDb
>      project.load(dbPath=os.path.join(projectPath, protDbPath), chdir=chdir,
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/project/project.py", line 244, in load
>      self._loadDb(dbPath)
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/project/project.py", line 302, in _loadDb
>      self.mapper = self.createMapper(absDbPath)
>    File "/beegfs/cssb/software/em/scipion/3.0/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/project/project.py", line 211, in createMapper
>      classesDict.update(self._domain.getMapperDict())
> AttributeError: 'NoneType' object has no attribute 'getMapperDict'
>
> Before when my base conda included all other scipion conda envs the cluster submission was fine.
> I have set autoactivation for my base to off in my .condarc.
>
> I have to use the python3 from the .scipion3env otherwise there would not be the pyworkflow package.
> What else could I change for submission?
>
> Cheers,
> Wolfgang
>
-- 
Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team*