Re: [scipion-users] MPI pyworkflow error

[Gabriel R. S. <gab...@ln...>]

I think this message might not have been seen.
Any idea if this is a general error?

Gabriel Ravanhani Schleder
Nanociências e Materiais avançados - Universidade Federal do ABC
Laboratório Nacional de Nanotecnologia (LNNano)/CNPEM
gab...@ln... (mailto:gab...@ln...)

On Aug 17 2020, at 7:43 pm, Gabriel Ravanhani Schleder <gab...@ln...> wrote:
>
> Dear all,
>
> I have been getting python errors related to MPI runs on different xmipp3 protocols (alternatively, I tested on relion MPI protocols, which runs correctly). The same protocols run without problems if using threads instead of MPI processes.
>
> It seems like a python2-to-python3 encoding-related issue, as the representative example run traceback below:
> 00001: RUNNING PROTOCOL -----------------
> 00002: Hostname: node04
> 00003: PID: 57882
> 00004: pyworkflow: 3.0.1
> 00005: plugin: xmipp3
> 00006: currentDir: /home/ScipionUserData/projects/Test
> 00007: workingDir: Runs/001782_XmippProtExtractParticles
> 00008: runMode: Restart
> 00009: MPI: 2
> 00010: threads: 1
> 00011: Starting at step: 1
> 00012: Running steps
> 00013: STARTED: extractMicrographStep, step 1, time 2020-08-17 13:51:05.684077
> 00014: Extracting micrograph: Runs/001557_ProtImportMicrographs/extra/4k_10000001.mrcs
> 00015: Sending environment to 1
> 00016: 'xmipp_coordinates_noisy_zones_filter' have failed for 4k_10000001.mrcs micrograph. We continue...
> 00017: Sending environment to 1
> 00018: Traceback (most recent call last):
> 00019: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 149, in run
> 00020: self.step._run() # not self.step.run() , to avoid race conditions
> 00021: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 239, in _run
> 00022: resultFiles = self._runFunc()
> 00023: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 235, in _runFunc
> 00024: return self._func(*self._args)
> 00025: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pwem/protocols/protocol_particles.py", line 233, in extractMicrographStep
> 00026: self._extractMicrograph(mic, *args)
> 00027: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/xmipp3/protocols/protocol_extract_particles.py", line 212, in _extractMicrograph
> 00028: self.runJob('xmipp_transform_filter',
> 00029: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/protocol.py", line 1337, in runJob
> 00030: self._stepsExecutor.runJob(self._log, program, arguments, **kwargs)
> 00031: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/protocol/executor.py", line 352, in runJob
> 00032: runJobMPI(programName, params, self.comm, node,
> 00033: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 90, in runJobMPI
> 00034: send("env=%s" % dumps(env), mpiComm, mpiDest, TAG_RUN_JOB+mpiDest)
> 00035: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 77, in send
> 00036: raise Exception(str(result).encode(encoding='UTF-8'))
> 00037: Exception: b"a bytes-like object is required, not 'str'"
> 00038: Protocol failed: b"a bytes-like object is required, not 'str'"
> 00039: FAILED: extractMicrographStep, step 1, time 2020-08-17 13:51:08.570543
> 00040: ------------------- PROTOCOL FAILED (DONE 1/10)
>
> The errors come from the:
> 00035: File "/home/miniconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/mpi.py", line 77, in send
> 00036: raise Exception(str(result).encode(encoding='UTF-8'))
>
> I tested adding ".encode(encoding='UTF-8')" part to the original code, which clearly did not solve the problem. From the traceback, the error seems to come from the string sent by the mpi communications, which I am not acquainted enough to solve.
> On a related note, what is the recommended procedure for updating the scipion packages (not plugins)? I updated from the latest available from PyPI.
>
> Best regards,
>
> Gabriel R. Schleder
> Universidade Federal do ABC (UFABC)
> Laboratório Nacional de Nanotecnologia (LNNano/CNPEM)
>
> Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.
> Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.
> _______________________________________________
> scipion-users mailing list
> sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scipion-users