Re: [scipion-users] MPI pyworkflow error

[Gabriel R. S. <gab...@ln...>]

Thanks Pablo.

Gabriel Ravanhani Schleder
Nanociências e Materiais avançados - Universidade Federal do ABC
Laboratório Nacional de Nanotecnologia (LNNano)/CNPEM

gab...@ln...<mailto:gab...@ln...>

[https://ci4.googleusercontent.com/proxy/IopjzWOb6ylR_GtG5u-Zq0Y1mfY38261k9R9cRTt4mvcFHhBJiQ9jdJqyZz9bFRhh-8Afvxzo1WXoiG6kICRxixgPuWH7LhkjY7V1aEp_qzCoLTmslvMDpNFwnT1I0LsPrG-vJPz0GuZ6ezdapZ0ysF75gEHWqCWTiyAg81-qkj08_Ziypasdao553zOQn1TYkVWf3diMECDEkRYoQ=s0-d-e1-ft#https://docs.google.com/uc?export=download&id=1VYc_gBmfC5pszxYlE4VRvmsYfUaCft-9&revid=0BzRCD-uA8R9PKzEyQVhwcFp4eEk2cVRXdjh0ZnYzODRQY0lnPQ][https://ci3.googleusercontent.com/proxy/MJIDN1yBIjQxusVQnL3j90ZVVJvigW9KnZqAZTmUe7nf_TIZEBsARPVrz6F5nWlGTQcTYyVvxxHozrAcCleyGYlyUft2g8x8T-epesE1Lz6o_EMGLYHkoEtjqdWE7Uf1JxyS=s0-d-e1-ft#https://drive.google.com/uc?id=1fGLfbjGvalYjDusrpZViRV-yodfbPA83&export=download]

On Sep 11 2020, at 2:44 am, Pablo Conesa <pc...@cn...> wrote:

I've created an issue, so is not lost and will have a look.

https://github.com/scipion-em/scipion-pyworkflow/issues/137

On 10/9/20 18:34, David Maluenda wrote:
Dear all,

I have been getting python errors related to MPI runs on different xmipp3 protocols (alternatively, I tested on relion MPI protocols, which runs correctly). The same protocols run without problems if using threads instead of MPI processes.

It seems like a python2-to-python3 encoding-related issue, as the representative example run traceback below:

00001:   RUNNING PROTOCOL -----------------
00002:   Hostname: node04
00003:   PID: 57882
00004:   pyworkflow: 3.0.1
00005:   plugin: xmipp3
00006:   currentDir: /home/ScipionUserData/projects/Test
00007:   workingDir: Runs/001782_XmippProtExtractParticles
00008:   runMode: Restart
00009:             MPI: 2
00010:         threads: 1
00011:    Starting at step: 1
00012:    Running steps
00013:   STARTED: extractMicrographStep, step 1, time 2020-08-17 13:51:05.684077
00014:   Extracting micrograph: Runs/001557_ProtImportMicrographs/extra/4k_10000001.mrcs
00015:   Sending environment to 1
00016:   'xmipp_coordinates_noisy_zones_filter' have failed for 4k_10000001.mrcs micrograph. We continue...
00017:   Sending environment to 1
00018:   Traceback (most recent call last):
00019:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 149, in run
00020:       self.step._run()  # not self.step.run<http://self.step.run>() , to avoid race conditions
00021:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 239, in _run
00022:       resultFiles = self._runFunc()
00023:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 235, in _runFunc
00024:       return self._func(*self._args)
00025:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pwem/protocols/protocol_particles.py", line 233, in extractMicrographStep
00026:       self._extractMicrograph(mic, *args)
00027:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/xmipp3/protocols/protocol_extract_particles.py", line 212, in _extractMicrograph
00028:       self.runJob<http://self.runJob>('xmipp_transform_filter',
00029:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 1337, in runJob
00030:       self._stepsExecutor.runJob<http://stepsExecutor.runJob>(self._log, program, arguments, **kwargs)
00031:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 352, in runJob
00032:       runJobMPI(programName, params, self.comm<http://self.comm>, node,
00033:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 90, in runJobMPI
00034:       send("env=%s" % dumps(env), mpiComm, mpiDest, TAG_RUN_JOB+mpiDest)
00035:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 77, in send
00036:       raise Exception(str(result).encode(encoding='UTF-8'))
00037:   Exception: b"a bytes-like object is required, not 'str'"
00038:   Protocol failed: b"a bytes-like object is required, not 'str'"
00039:   FAILED: extractMicrographStep, step 1, time 2020-08-17 13:51:08.570543
00040:   ------------------- PROTOCOL FAILED (DONE 1/10)

The errors come from the:
00035:     File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 77, in send
00036:       raise Exception(str(result).encode(encoding='UTF-8'))

I tested adding ".encode(encoding='UTF-8')" part to the original code, which clearly did not solve the problem. From the traceback, the error seems to come from the string sent by the mpi communications, which I am not acquainted enough to solve.

On a related note, what is the recommended procedure for updating the scipion packages (not plugins)? I updated from the latest available from PyPI.

Best regards,

Gabriel R. Schleder
Universidade Federal do ABC (UFABC)
Laboratório Nacional de Nanotecnologia (LNNano/CNPEM)
--
Pablo Conesa - Madrid Scipion<http://scipion.i2pc.es> team
_______________________________________________
scipion-users mailing list
sci...@li...
https://lists.sourceforge.net/lists/listinfo/scipion-users


Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.