[scipion-users] Out of memory error on GPU when running 2D classification with Relion

[Veli-Pekka K. <sc...@gu...>]

Hi,

I have problem when running Relion 2D classification with setting of:

When running with option: Mask particles with zeros? No, fill with 
random noise

Run ends with out of memory error on GPU:
00112:   KERNEL_ERROR: out of memory in 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/src/acc/utilities_impl.h 
at line 253 (error-code 2)
00113:   in: 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/src/acc/cuda/cuda_settings.h, 
line 81
00114:   in: 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/src/acc/cuda/cuda_settings.h, 
line 81
00115:   slave 2 encountered error: === Backtrace  ===
00116: 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) 
[0x447ce1]
00117: 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/bin/relion_refine_mpi(_Z36globalThreadExpectationSomeParticlesR14ThreadArgument+0xe8) 
[0x5cbb48]
00118: 
/data/opt/sci/scipion-2.0/software/em/relion-3.0/bin/relion_refine_mpi(_Z11_threadMainPv+0x3f) 
[0x48fbcf]
00119:   /lib64/libpthread.so.0(+0x7e65) [0x7f6e6b492e65]
00120:   /lib64/libc.so.6(clone+0x6d) [0x7f6e6b1bb88d]

Run works when selecting: Yes, fill with zeroes (but the result isn't 
what is hoped for).

Is there known fix for this situation? There was similar bug in Relion 
bug tracker, but it was closed without explanation.

Specifications are:
System: CentOS Linux release 7.7.1908 (Core)
GPU: 2 x Tesla P100
Memory: 128GB
CPU: Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz

Scipion is v2.0 (2019-04-23)
Relion 3.0 is compiled against cuda-8.0

Greetings,
Veli-Pekka Kestilä