Re: [scipion-users] MPI pyworkflow error

[Pablo C. <pc...@cn...>]

I've created an issue, so is not lost and will have a look.

https://github.com/scipion-em/scipion-pyworkflow/issues/137

On 10/9/20 18:34, David Maluenda wrote:
> Dear all,
> I have been getting python errors related to MPI runs on different 
> xmipp3 protocols (alternatively, I tested on relion MPI protocols, 
> which runs correctly). The same protocols run without problems if 
> using threads instead of MPI processes.
>
> It seems like a python2-to-python3 encoding-related issue, as the 
> representative example run traceback below:
>
> 00001:   RUNNING PROTOCOL -----------------
> 00002:   Hostname: node04
> 00003:   PID: 57882
> 00004:   pyworkflow: 3.0.1
> 00005:   plugin: xmipp3
> 00006:   currentDir: /home/ScipionUserData/projects/Test
> 00007:   workingDir: Runs/001782_XmippProtExtractParticles
> 00008:   runMode: Restart
> 00009:             MPI: 2
> 00010:         threads: 1
> 00011:    Starting at step: 1
> 00012:    Running steps
> 00013:   STARTED: extractMicrographStep, step 1, time 2020-08-17 
> 13:51:05.684077
> 00014:   Extracting micrograph: 
> Runs/001557_ProtImportMicrographs/extra/4k_10000001.mrcs
> 00015:   Sending environment to 1
> 00016: 'xmipp_coordinates_noisy_zones_filter' have failed for 
> 4k_10000001.mrcs micrograph. We continue...
> 00017:   Sending environment to 1
> 00018:   Traceback (most recent call last):
> 00019:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", 
> line 149, in run
> 00020:       self.step._run()  # not self.step.run 
> <http://self.step.run>() , to avoid race conditions
> 00021:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", 
> line 239, in _run
> 00022:       resultFiles = self._runFunc()
> 00023:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", 
> line 235, in _runFunc
> 00024:       return self._func(*self._args)
> 00025:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pwem/protocols/protocol_particles.py", 
> line 233, in extractMicrographStep
> 00026: self._extractMicrograph(mic, *args)
> 00027:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/xmipp3/protocols/protocol_extract_particles.py", 
> line 212, in _extractMicrograph
> 00028: self.runJob <http://self.runJob>('xmipp_transform_filter',
> 00029:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", 
> line 1337, in runJob
> 00030:       self._stepsExecutor.runJob 
> <http://stepsExecutor.runJob>(self._log, program, arguments, **kwargs)
> 00031:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", 
> line 352, in runJob
> 00032:       runJobMPI(programName, params, self.comm 
> <http://self.comm>, node,
> 00033:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", 
> line 90, in runJobMPI
> 00034:       send("env=%s" % dumps(env), mpiComm, mpiDest, 
> TAG_RUN_JOB+mpiDest)
> 00035:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", 
> line 77, in send
> 00036:       raise Exception(str(result).encode(encoding='UTF-8'))
> 00037:   Exception: b"a bytes-like object is required, not 'str'"
> 00038:   Protocol failed: b"a bytes-like object is required, not 'str'"
> 00039:   FAILED: extractMicrographStep, step 1, time 2020-08-17 
> 13:51:08.570543
> 00040:   ------------------- PROTOCOL FAILED (DONE 1/10)
>
> The errors come from the:
> 00035:     File 
> "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", 
> line 77, in send
> 00036:       raise Exception(str(result).encode(encoding='UTF-8'))
>
> I tested adding ".encode(encoding='UTF-8')" part to the original code, 
> which clearly did not solve the problem. From the traceback, the error 
> seems to come from the string sent by the mpi communications, which I 
> am not acquainted enough to solve.
>
> On a related note, what is the recommended procedure for updating the 
> scipion packages (not plugins)? I updated from the latest available 
> from PyPI.
> Best regards,
>
> Gabriel R. Schleder
> /Universidade Federal do ABC (UFABC)/
> /Laboratório Nacional de Nanotecnologia (LNNano/CNPEM)/
-- 
Pablo Conesa - *Madrid Scipion <http://scipion.i2pc.es> team*