From: Patrick G. <pg...@ma...> - 2018-04-18 18:14:14
|
Hi - I'm trying to configure Scipion cloned from the git repo on an Ubuntu 18.04 machine and get the following error message: ----------------- Checking paths in /root/scipion/config/scipion.conf ... Path to MPI_LIBDIR (/usr/lib64/mpi/gcc/openmpi/lib) should exist but it doesn't. Path to MPI_INCLUDE (/usr/lib64/mpi/gcc/openmpi/include) should exist but it doesn't. Cannot find file: /usr/lib64/mpi/gcc/openmpi/include/mpi.h Errors found. ----------------- I think it's looking for the MPI header file in the wrong location for Ubuntu? It's here: /usr/lib/x86_64-linux-gnu/openmpi/include/mpi.h Also, minor point. The installation instructions have a list of dependencies that need to be installed which includes "libreadline6". There's no installation candidate for this, I think it needs to be lib64readline6-dev instead. |
From: Pablo C. <pc...@cn...> - 2018-04-19 05:50:07
|
Hi Patrick, We have default values for the paths of several dependencies, like MPI or JAVA. Our current default values do not match Ubuntu paths. At this point you can: 1.- run: ./scipion config --overwrite This tries to guess those paths, and in Ubuntu 14 --> 16 works fine. Sorry never tried on ubuntu 18! 2.- Manually, edit the config file: this file can be found at <SCIPION_HOME>/config/scipion.conf Regarding the libreadline6,....could this be a change related to ubuntu 18? To me, it seems that libreadline6 hasn't made it to bionic, yet?: <https://packages.ubuntu.com/trusty/libreadline6,%20yet> Whereas, the dev version has made it! <https://packages.ubuntu.com/trusty/libreadline6-dev> I have added a note in our wiki!! Thanks for the feedback! Patrick, probably you are the first one to install Scipion in Ubuntu 18. Please let us know how it goes, whether if it goes right and more importantly if something does wrong. All the best, Pablo. On 18/04/18 19:40, Patrick Goetz wrote: > Hi - > > I'm trying to configure Scipion cloned from the git repo on an Ubuntu > 18.04 machine and get the following error message: > > ----------------- > Checking paths in /root/scipion/config/scipion.conf ... > Path to MPI_LIBDIR (/usr/lib64/mpi/gcc/openmpi/lib) should exist but > it doesn't. > Path to MPI_INCLUDE (/usr/lib64/mpi/gcc/openmpi/include) should > exist but it doesn't. > Cannot find file: /usr/lib64/mpi/gcc/openmpi/include/mpi.h > Errors found. > ----------------- > > I think it's looking for the MPI header file in the wrong location for > Ubuntu? It's here: > > /usr/lib/x86_64-linux-gnu/openmpi/include/mpi.h > > > Also, minor point. The installation instructions have a list of > dependencies that need to be installed which includes "libreadline6". > There's no installation candidate for this, I think it needs to be > lib64readline6-dev instead. > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users |
From: Patrick G. <pg...@ma...> - 2018-04-19 14:34:07
|
Hi Pablo - Thanks very much for your help with this. I will try the config suggestions you provided; meanwhile, regarding libreadline6: On 04/19/2018 12:49 AM, Pablo Conesa wrote: > To me, it seems that libreadline6 hasn't made it to bionic, yet?: > My bad; I should have checked the Xenial (16.04) package list first. However, it doesn't look like libreadline6 is going to make it into Bionic (18.04). Here is the related package list obtained by searching the package directories on packages.ubuntu.com for "libreadline": Package libreadline-dev [this is version 7] Package libreadline5 Package libreadline5-dbg Package libreadline6-dev Package libreadline7 Package libreadline7-dbg Looks like they went to version 7, while retaining version 5, but not 6. Is this a compile time or a run time dependency? If compile time, one would usually need the -dev version anyway. |
From: Pablo C. <pc...@cn...> - 2018-04-20 12:47:25
|
Hi Patrick, to be honest , I don't know how libreadline6 is used,,, I would say that since it is not _dev we don't need it at compile time but at runtime But I might be wrong. Try dev_7 ? Will it work? On 19/04/18 15:59, Patrick Goetz wrote: > Hi Pablo - > > Thanks very much for your help with this. I will try the config > suggestions you provided; meanwhile, regarding libreadline6: > > > On 04/19/2018 12:49 AM, Pablo Conesa wrote: >> To me, it seems that libreadline6 hasn't made it to bionic, yet?: >> > > > My bad; I should have checked the Xenial (16.04) package list first. > However, it doesn't look like libreadline6 is going to make it into > Bionic (18.04). Here is the related package list obtained by > searching the package directories on packages.ubuntu.com for > "libreadline": > > Package libreadline-dev [this is version 7] > > Package libreadline5 > > Package libreadline5-dbg > > Package libreadline6-dev > > Package libreadline7 > > Package libreadline7-dbg > > > Looks like they went to version 7, while retaining version 5, but not > 6. Is this a compile time or a run time dependency? If compile time, > one would usually need the -dev version anyway. > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users |
From: Patrick G. <pg...@ma...> - 2018-04-24 21:56:20
|
Hi - I've managed to install Scipion (with --no-opencv) on Ubuntu 18.04. Missing from the instructions is the need for this package: libreadline-dev The readline.h header file is needed for the sqlite compilation. Also, the file ~/scipion/config/scipion.conf required several modifications to MPI, CUDA, and Matlab paths. I also installed EMAN2 using: ./scipion install --no-xmipp eman-2.12 Now unfortunately I have a number of questions. 1. I cloned the git repo into /root, and now everything is installed under /root/scipion How should I make the software available to users? It seems that source and binaries are heavily intertwined here. The scipion command is in /root/scipion, but everything else is installed in /root/scipion/software? Presumably this also means setting some environment variables (at least a path)? 2. I tried running one of the small tests (scipion tests em.workflows.test_workflow_spiderMDA), but this failed with a couple of python runtime errors. These look like internal errors, but could this have something to do with the fact that I'm trying to run the test from a (non-graphical) terminal as root (because of the install location)? Fetching files of dataset "hemoglobin_mda"... ################################################# ** Running command: 'python /root/scipion/scipion runprotocol pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderConvert" "Runs/000002_ProtImportParticles/logs/run.db" 2' Scipion v1.2 (2018-04-02) Caligula >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderConvert" "Runs/000002_ProtImportParticles/logs/run.db" "2" stackFn: /root/ScipionUserData/projects/TestSpiderConvert/stack.stk ** Running command: /root/scipion/software/em/spider-21.13/spider/bin/spider_linux_mp_intel64 stk @cp_endian /bin/sh: 1: /root/scipion/software/em/spider-21.13/spider/bin/spider_linux_mp_intel64: not found [ FAILED ] TestSpiderConvert.test_convert Traceback (most recent call last): File "/root/scipion/software/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "/root/scipion/pyworkflow/tests/em/workflows/test_workflow_spiderMDA.py", line 62, in test_convert writeSetOfImages(protImport.outputParticles, stackFn, selFn) File "/root/scipion/pyworkflow/em/packages/spider/convert.py", line 86, in writeSetOfImages convertEndian(stackFn, imgSet.getSize()) File "/root/scipion/pyworkflow/em/packages/spider/convert.py", line 103, in convertEndian }, cwd=fnDir) File "/root/scipion/pyworkflow/em/packages/spider/spider.py", line 177, in runTemplate runScript(outputScript, ext, log, cwd) File "/root/scipion/pyworkflow/em/packages/spider/spider.py", line 183, in runScript runJob(log, SPIDER, args, env=dict(environment), cwd=cwd) File "/root/scipion/pyworkflow/utils/process.py", line 51, in runJob return runCommand(command, env, cwd) File "/root/scipion/pyworkflow/utils/process.py", line 65, in runCommand check_call(command, shell=True, stdout=sys.stdout, stderr=sys.stderr, env=env, cwd=cwd) File "/root/scipion/software/lib/python2.7/subprocess.py", line 186, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '/root/scipion/software/em/spider-21.13/spider/bin/spider_linux_mp_intel64 stk @cp_endian' returned non-zero exit status 127 ('Creating project at: ', '/root/ScipionUserData/projects/TestSpiderWorkflow/project.sqlite') >>>> python /root/scipion/scipion testdata --download hemoglobin_mda Scipion v1.2 (2018-04-02) Caligula >>>>> python scripts/sync_data.py --download hemoglobin_mda Selected datasets: hemoglobin_mda Local copy of dataset hemoglobin_mda detected. Checking for updates... Regenerating local MANIFEST... Verifying MD5s... ...done. Updated files: 0 ** Running command: 'python /root/scipion/scipion runprotocol pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" "Runs/000002_ProtImportParticles/logs/run.db" 2' Scipion v1.2 (2018-04-02) Caligula >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" "Runs/000002_ProtImportParticles/logs/run.db" "2" ** Running command: 'python /root/scipion/scipion runprotocol pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" "Runs/000070_SpiderProtFilter/logs/run.db" 70' Scipion v1.2 (2018-04-02) Caligula >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" "Runs/000070_SpiderProtFilter/logs/run.db" "70" >>> ERROR running protocol scipion - filter particles [ FAILED ] TestSpiderWorkflow.test_mdaWorkflow Traceback (most recent call last): File "/root/scipion/software/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "/root/scipion/pyworkflow/tests/em/workflows/test_workflow_spiderMDA.py", line 104, in test_mdaWorkflow self.launchProtocol(protFilter) File "/root/scipion/pyworkflow/tests/tests.py", line 106, in launchProtocol raise Exception("ERROR: Protocol not finished") Exception: ERROR: Protocol not finished [==========] run 2 tests (46.593 secs) [ FAILED ] 2 tests [ PASSED ] 0 tests |
From: Kyle D. <kyl...@ep...> - 2018-04-25 06:32:07
|
Hi Patrick, (I'm not a Scipion dev but have some suggestions.) On 04/24/2018 11:21 PM, Patrick Goetz wrote: > > > 2. > I tried running one of the small tests (scipion tests > em.workflows.test_workflow_spiderMDA), but this failed with a couple > of python runtime errors. These look like internal errors, but could > this have something to do with the fact that I'm trying to run the > test from a (non-graphical) terminal as root (because of the install > location)? > > > File "/root/scipion/software/lib/python2.7/subprocess.py", line 186, > in check_call > raise CalledProcessError(retcode, cmd) > CalledProcessError: Command > '/root/scipion/software/em/spider-21.13/spider/bin/spider_linux_mp_intel64 > stk @cp_endian' returned non-zero exit status 127 > ('Creating project at: ', > '/root/ScipionUserData/projects/TestSpiderWorkflow/project.sqlite') > >>>> python /root/scipion/scipion testdata --download hemoglobin_mda > > Actually it looks like that error is coming from a call to a system command and is not directly caused by anything in Python. Exit status 127 is returned when either 1) the file does not exist or 2) the file does not have execute permission for your user or group. Could you verify that spider_linux_mp_intel64 exists and has execute permissions set? You can use the command ls -l /root/scipion/software/em/spider-21.13/spider/bin/ | grep spider_linux_mp_intel64 and check that an 'x' exists for the user/group? (I assume it's the root user.) I would also think that you should give everyone read/write access to the ScipionUserData folder since you want multiple users to have access to it; this may not be true in your case since it is buried inside a folder belonging to root. Finally, there are log files for stdout and stderr that are written to the ScipionUserData folder when you run a workflow or test. In my experience you can get more information about any errors from these files. Hope this helps, Kyle -- Kyle M. Douglass, PhD Post-doctoral researcher The Laboratory of Experimental Biophysics EPFL, Lausanne, Switzerland http://kmdouglass.github.io http://leb.epfl.ch |
From: Patrick G. <pg...@ma...> - 2018-04-25 14:20:12
|
Hi Kyle On 04/25/2018 01:31 AM, Kyle Douglass wrote: > verify that spider_linux_mp_intel64 exists and has execute permissions > set? You can use the command > > ls -l /root/scipion/software/em/spider-21.13/spider/bin/ | grep > spider_linux_mp_intel64 > This was the problem. I didn't notice that the spider package needs to be installed separately even though it's right there in the documentation. Now the test works as advertised. Thanks for your help with this. |
From: Jose M. de la R. T. <del...@gm...> - 2018-04-25 07:07:47
|
Hi Patrick, Find some clarification below, on top of the suggestions from Kyle. On Tue, Apr 24, 2018 at 11:21 PM, Patrick Goetz <pg...@ma...> wrote: > Hi - > > I've managed to install Scipion (with --no-opencv) on Ubuntu 18.04. > > Missing from the instructions is the need for this package: > libreadline-dev > > The readline.h header file is needed for the sqlite compilation. > > Also, the file ~/scipion/config/scipion.conf required several > modifications to MPI, CUDA, and Matlab paths. > > I also installed EMAN2 using: ./scipion install --no-xmipp eman-2.12 > > Now unfortunately I have a number of questions. > > 1. > I cloned the git repo into /root, and now everything is installed under > > /root/scipion > > How should I make the software available to users? It seems that source > and binaries are heavily intertwined here. The scipion command is in > /root/scipion, but everything else is installed in /root/scipion/software? > > Presumably this also means setting some environment variables (at least a > path)? > It is true that source and binaries are intertwined now, we want to improve this in the future. We have designed scipion to avoid modifying you environment...so, user only need to point to the scipion python script from the installation folder. Two possibilities are: a) Define an alias to that: alias scipion='/path/to/scipion/scipion' b) Add the /path/to/scipion to the user PATH environment variable All the environment settings for each process are done on the fly, this means that will not interfere between packages or with the system libraries. > > 2. > I tried running one of the small tests (scipion tests > em.workflows.test_workflow_spiderMDA), but this failed with a couple of > python runtime errors. These look like internal errors, but could this > have something to do with the fact that I'm trying to run the test from a > (non-graphical) terminal as root (because of the install location)? > > As Kyle pointed out, the error is a missing binary file. My guess would be that you haven't installed spider (scipion install spider --no-xmipp) and you are trying to run a test that use it. The test data sets are written in /path/to/scipion/data/tests, so, if this folder belong to root, you will need to run the tests with root or give user access. Additionally, the projects that are generated by each test go to the same data folder of each user (~/ScipionUserData by default) Hope this helps Best, Jose Miguel > > Fetching files of dataset "hemoglobin_mda"... > ################################################# > ** Running command: 'python /root/scipion/scipion runprotocol > pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderConvert" > "Runs/000002_ProtImportParticles/logs/run.db" 2' > > Scipion v1.2 (2018-04-02) Caligula > > >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py > "/root/ScipionUserData/projects/TestSpiderConvert" > "Runs/000002_ProtImportParticles/logs/run.db" "2" > stackFn: /root/ScipionUserData/projects/TestSpiderConvert/stack.stk > ** Running command: /root/scipion/software/em/spid > er-21.13/spider/bin/spider_linux_mp_intel64 stk @cp_endian > /bin/sh: 1: /root/scipion/software/em/spider-21.13/spider/bin/spider_linux_mp_intel64: > not found > [ FAILED ] TestSpiderConvert.test_convert > > Traceback (most recent call last): > File "/root/scipion/software/lib/python2.7/unittest/case.py", line 329, > in run > testMethod() > File "/root/scipion/pyworkflow/tests/em/workflows/test_workflow_spiderMDA.py", > line 62, in test_convert > writeSetOfImages(protImport.outputParticles, stackFn, selFn) > File "/root/scipion/pyworkflow/em/packages/spider/convert.py", line 86, > in writeSetOfImages > convertEndian(stackFn, imgSet.getSize()) > File "/root/scipion/pyworkflow/em/packages/spider/convert.py", line > 103, in convertEndian > }, cwd=fnDir) > File "/root/scipion/pyworkflow/em/packages/spider/spider.py", line 177, > in runTemplate > runScript(outputScript, ext, log, cwd) > File "/root/scipion/pyworkflow/em/packages/spider/spider.py", line 183, > in runScript > runJob(log, SPIDER, args, env=dict(environment), cwd=cwd) > File "/root/scipion/pyworkflow/utils/process.py", line 51, in runJob > return runCommand(command, env, cwd) > File "/root/scipion/pyworkflow/utils/process.py", line 65, in runCommand > check_call(command, shell=True, stdout=sys.stdout, stderr=sys.stderr, > env=env, cwd=cwd) > File "/root/scipion/software/lib/python2.7/subprocess.py", line 186, in > check_call > raise CalledProcessError(retcode, cmd) > CalledProcessError: Command '/root/scipion/software/em/spi > der-21.13/spider/bin/spider_linux_mp_intel64 stk @cp_endian' returned > non-zero exit status 127 > ('Creating project at: ', '/root/ScipionUserData/project > s/TestSpiderWorkflow/project.sqlite') > >>>> python /root/scipion/scipion testdata --download hemoglobin_mda > > Scipion v1.2 (2018-04-02) Caligula > > >>>>> python scripts/sync_data.py --download hemoglobin_mda > Selected datasets: hemoglobin_mda > Local copy of dataset hemoglobin_mda detected. > Checking for updates... > Regenerating local MANIFEST... > Verifying MD5s... > ...done. Updated files: 0 > ** Running command: 'python /root/scipion/scipion runprotocol > pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" > "Runs/000002_ProtImportParticles/logs/run.db" 2' > > Scipion v1.2 (2018-04-02) Caligula > > >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py > "/root/ScipionUserData/projects/TestSpiderWorkflow" > "Runs/000002_ProtImportParticles/logs/run.db" "2" > ** Running command: 'python /root/scipion/scipion runprotocol > pw_protocol_run.py "/root/ScipionUserData/projects/TestSpiderWorkflow" > "Runs/000070_SpiderProtFilter/logs/run.db" 70' > > Scipion v1.2 (2018-04-02) Caligula > > >>>>> python /root/scipion/pyworkflow/apps/pw_protocol_run.py > "/root/ScipionUserData/projects/TestSpiderWorkflow" > "Runs/000070_SpiderProtFilter/logs/run.db" "70" > > >>> ERROR running protocol scipion - filter particles > [ FAILED ] TestSpiderWorkflow.test_mdaWorkflow > > Traceback (most recent call last): > File "/root/scipion/software/lib/python2.7/unittest/case.py", line 329, > in run > testMethod() > File "/root/scipion/pyworkflow/tests/em/workflows/test_workflow_spiderMDA.py", > line 104, in test_mdaWorkflow > self.launchProtocol(protFilter) > File "/root/scipion/pyworkflow/tests/tests.py", line 106, in > launchProtocol > raise Exception("ERROR: Protocol not finished") > Exception: ERROR: Protocol not finished > > [==========] run 2 tests (46.593 secs) > [ FAILED ] 2 tests > [ PASSED ] 0 tests > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
From: Patrick G. <pg...@ma...> - 2018-04-25 13:48:30
|
On 04/25/2018 02:07 AM, Jose Miguel de la Rosa Trevin wrote: > We have designed scipion to avoid modifying you environment...so, user > only need to point to the scipion python script from the installation > folder. Two possibilities are: > a) Define an alias to that: alias scipion='/path/to/scipion/scipion' > b) Add the /path/to/scipion to the user PATH environment variable > All the environment settings for each process are done on the fly, this > means that will not interfere between packages or with the system > libraries. > That's very nice, actually. So all I need to do is move the Scipion folder to a publicly accessible location (e.g. /opt/scipion or /usr/local/scipion), make sure the users can find the scipion command, and make sure that the read/write/execute permissions are properly set? |
From: Jose M. de la R. T. <del...@gm...> - 2018-04-25 14:06:44
|
The users only need read/execution permissions and a way to find the scipion script inside the installation folder. On Wed, Apr 25, 2018 at 3:48 PM, Patrick Goetz <pg...@ma...> wrote: > > On 04/25/2018 02:07 AM, Jose Miguel de la Rosa Trevin wrote: > >> We have designed scipion to avoid modifying you environment...so, user >> only need to point to the scipion python script from the installation >> folder. Two possibilities are: >> a) Define an alias to that: alias scipion='/path/to/scipion/scipion' >> b) Add the /path/to/scipion to the user PATH environment variable >> All the environment settings for each process are done on the fly, this >> means that will not interfere between packages or with the system libraries. >> >> > That's very nice, actually. So all I need to do is move the Scipion > folder to a publicly accessible location (e.g. /opt/scipion or > /usr/local/scipion), make sure the users can find the scipion command, and > make sure that the read/write/execute permissions are properly set? > > > > > > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |
From: Patrick G. <pg...@ma...> - 2018-04-25 14:16:04
|
On 04/25/2018 09:06 AM, Jose Miguel de la Rosa Trevin wrote: > The users only need read/execution permissions and a way to find the > scipion script inside the installation folder. > What about the ScipionUserData folder -- does this just need to be located in parallel to scipion, or is there a way I can put scipion into /opt and ScipionUserData in /var or /home/per_user? And presumably users would need permission to write to this folder? |
From: Jose M. de la R. T. <del...@gm...> - 2018-04-25 14:18:33
|
ScipionUserData is a folder independent from the scipion installation folder and each user has its own ScipionUserData...by default it goes to ~/ScipionUserData. On Wed, Apr 25, 2018 at 4:15 PM, Patrick Goetz <pg...@ma...> wrote: > On 04/25/2018 09:06 AM, Jose Miguel de la Rosa Trevin wrote: > >> The users only need read/execution permissions and a way to find the >> scipion script inside the installation folder. >> >> > What about the ScipionUserData folder -- does this just need to be located > in parallel to scipion, or is there a way I can put scipion into /opt and > ScipionUserData in /var or /home/per_user? And presumably users would need > permission to write to this folder? > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > scipion-users mailing list > sci...@li... > https://lists.sourceforge.net/lists/listinfo/scipion-users > |