Re: [Scalablecr-discuss] User configuration file format
Brought to you by:
kathrynmohror,
moody20
Menu
▾
▴
Re: [Scalablecr-discuss] User configuration file format
|
From: Adam T. M. <mo...@ll...> - 2016-01-26 00:54:19
|
Hello Maksym, FYI, I just pushed a commit to fix the error you uncovered in the scr.user.conf example file. Thanks for pointing this out. In addition to everything Kathryn mentioned, you'll also want to use GROUP=NODE for all of your checkpoints in your scr.user.conf file below. Here, the GROUP field refers to the group of processes that are likely to fail at the same time. SCR uses this information to pick processes for its redundancy sets. If you specify GROUP=NODE, that tells SCR that all procs on a compute node are likely to fail at the same time (all processes that share the same string returned by gethostname). -Adam Maksym Planeta wrote: > Hello, > > I decided to try out SCR. I compiled it and installed as specified in > the manual. No I try to specify checkpoint descriptors in use > configuration file. > > It turns out that the documentation describes other format, from what an > example at https://github.com/hpc/scr/blob/master/scr.user.conf shows. > > For example, file doc/scr_users_manual.pdf does not contain keyword > CKPTDESC whatsoever. > > Could you tell me what is the correct format? > > I tried to use the one which documentation specifies, but I get an > error which tells, that I probably don't have enough nodes: > > SCR v1.1.8 WARNING: rank 10 on taurusi6325: Failed to find partner > processes for redundancy descriptor 0, disabling checkpoint, too few > nodes? @ scr_reddesc.c:169 > > I definitely do, because I specify SET_SIZE=1 and create a job with 4 > nodes. > > I attach all the configuration files for completeness. > >------------------------------------------------------------------------ > >scr_srun: Started: Mon Jan 25 15:54:36 CET 2016 >scr_srun: prerun: Mon Jan 25 15:54:42 CET 2016 >scr_prerun: Started: Mon Jan 25 15:54:42 CET 2016 >scr_prerun: Ended: Mon Jan 25 15:54:42 CET 2016 >scr_prerun: secs: 0 >scr_prerun: exit code: 0 >scr_srun: RUN 1: Mon Jan 25 15:54:43 CET 2016 >SCR v1.1.8 ERROR: taurusi6325: scr_log_event: Missing username, jobname, or start time, disabling logging @ scr_log_event.c:204 >SCR v1.1.8 WARNING: rank 0 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 9 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 61 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 77 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 33 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 1 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 65 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 78 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 43 on taurusi6328: SCR v1.1.8 WARNING: rank 2 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 66 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 80 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 24 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 4 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 67 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 83 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 25 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 5 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 70 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 86 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 26 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 6 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 48 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 92 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 27 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 8 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 49 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 72 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 28 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 10 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 50 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 73 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 29 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 11 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 51 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 74 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 30 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 12 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 52 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 75 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 31 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 13 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 53 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 76 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 32 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 14 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 54 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 79 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 34 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 15 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 55 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 81 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 35 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 16 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 82 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 36 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 17 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 84 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 37 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 18 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 85 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 38 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 19 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 88 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 39 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 20 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 89 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 40 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 21 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 56 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 90 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 41 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 22 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 57 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 91 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 42 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 23 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 58 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 93 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 44 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 7 on taurusi6325: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 59 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 94 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 45 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 3 on taurusi6325: SCR v1.1.8 WARNING: rank 60 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 95 on taurusi6330: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 46 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 62 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 47 on taurusi6328: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 64 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 68 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 69 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 71 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 63 on taurusi6329: Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 0 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 1 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 2 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 3 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 4 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 5 on taurusi6325: SCR v1.1.8 WARNING: rank 6 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 7 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 24 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 8 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 87 on taurusi6330: SCR v1.1.8 WARNING: rank 25 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 0, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 26 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 27 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 9 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 72 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 28 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 73 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 29 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 74 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 30 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 10 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 75 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 76 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 77 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 78 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 11 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 79 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 48 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 80 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 12 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 49 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 81 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 50 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 82 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 51 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 13 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 52 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 53 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 54 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 55 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 14 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 56 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 57 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 58 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 59 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 15 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 60 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 16 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 31 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 32 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 33 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 17 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 18 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 34 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 35 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 36 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 37 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 19 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 38 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 39 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 40 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 41 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 20 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 21 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 22 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 83 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 84 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 85 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 86 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 23 on taurusi6325: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 87 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 88 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 89 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 90 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 91 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 61 on taurusi6329: SCR v1.1.8 WARNING: rank 92 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 42 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 62 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 93 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 43 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 63 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 94 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 44 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 64 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 95 on taurusi6330: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 65 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 66 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 67 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 68 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 69 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 70 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 71 on taurusi6329: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 0 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 1 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 2 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 3 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 4 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 5 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 45 on taurusi6328: SCR v1.1.8 WARNING: rank 46 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 47 on taurusi6328: Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 6 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 24 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 25 on taurusi6328: SCR v1.1.8 WARNING: rank 7 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 26 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 48 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 27 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 8 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 49 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 28 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 50 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 51 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 52 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 9 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 53 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 54 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 55 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 10 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 11 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 56 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 1, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 57 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 58 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 72 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 73 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 12 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 74 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 75 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 29 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 13 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 76 on taurusi6330: SCR v1.1.8 WARNING: rank 30 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 77 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 31 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 78 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 79 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 14 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 32 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 33 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 15 on taurusi6325: SCR v1.1.8 WARNING: rank 34 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 35 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 36 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 37 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 16 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 38 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 39 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 17 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 18 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 19 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 20 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 59 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 80 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 60 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 81 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 21 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 61 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 82 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 62 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 83 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 63 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 84 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 22 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 64 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 85 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 40 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 65 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 86 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 41 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 23 on taurusi6325: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 66 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 87 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 42 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 67 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 68 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 88 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 69 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 89 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 70 on taurusi6329: SCR v1.1.8 WARNING: rank 90 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 71 on taurusi6329: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 91 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 92 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 93 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 94 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 95 on taurusi6330: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 > >SCR v1.1.8 WARNING: rank 0 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 1 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 2 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 3 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 43 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 44 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 4 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 45 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 46 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 47 on taurusi6328: Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169SCR v1.1.8 WARNING: rank 5 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 > >SCR v1.1.8 WARNING: rank 24 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 25 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 6 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 7 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 8 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >Failed to find partner processes for redundancy descriptor 2, disabling checkpoint, too few nodes? @ scr_reddesc.c:169 >SCR v1.1.8 WARNING: rank 48 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 49 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 9 on taurusi6325: SCR v1.1.8 WARNING: rank 50 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 51 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 52 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 10 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 53 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 54 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 11 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 26 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 27 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 12 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 55 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 72 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 28 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 56 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 73 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 13 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 57 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 74 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 75 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 14 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 76 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 29 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 77 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 30 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 15 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 78 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 31 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 79 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 32 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 33 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 16 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 34 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 35 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 36 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 17 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 18 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 19 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 20 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 21 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 22 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 37 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 58 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 80 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 38 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 23 on taurusi6325: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 59 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 81 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 39 on taurusi6328: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 60 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 82 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 61 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 83 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 62 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 84 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 63 on taurusi6329: SCR v1.1.8 WARNING: rank 85 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 64 on taurusi6329: Failed to find partner processes for redundancy descriptor 3, disabling checkpoint, too few nodes? @ scr_reddesc.c:86 >SCR v1.1.8 WARNING: rank 86 on taurusi6330: Failed to find partner processes for redundancy descriptor 3, disab... [truncated message content] |
