You can subscribe to this list here.
| 2013 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(4) |
Oct
|
Nov
(4) |
Dec
(6) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2014 |
Jan
(7) |
Feb
|
Mar
|
Apr
(6) |
May
|
Jun
|
Jul
|
Aug
(16) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
(6) |
Feb
(4) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Kathryn M. <ka...@ll...> - 2014-08-14 17:13:01
|
Hi Wadud, Does the information in section 6.2 of the User¹s Guide have what you are looking for? The CACHEDESC variable sets the location(s) for the caches. I¹ll look into how we can make that information easier to find. Thanks! And let me know if that doesn¹t help, Kathryn _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA On 8/14/14, 8:52 AM, "Wadud Miah" <w....@qm...> wrote: >Hello, > >Could someone please tell me the path to the cached checkpoint data? I >know the SCR_PREFIX variable is used to specify the path to checkpoint >data on the parallel file system, but is there a similar variable for >cached local checkpoint > data? I imagine it will live somewhere in /tmp and I can¹t find this >information anywhere in the documentation. > >Thanks in advance, > >------------------------------------------- >Wadud Miah >Research Computing Services (HPC) >020 7882 8393 > > |
|
From: Wadud M. <w....@qm...> - 2014-08-14 16:53:49
|
Hello, Could someone please tell me the path to the cached checkpoint data? I know the SCR_PREFIX variable is used to specify the path to checkpoint data on the parallel file system, but is there a similar variable for cached local checkpoint data? I imagine it will live somewhere in /tmp and I can't find this information anywhere in the documentation. Thanks in advance, ------------------------------------------- Wadud Miah Research Computing Services (HPC) 020 7882 8393 |
|
From: Teranishi, K. <kn...@sa...> - 2014-04-28 21:36:29
|
Hi Adam, After a careful investigation, I found the test program can handle two 8 Mbyte checkpointing files per process, but it failed with 16 Mbyte files at the second checkpoint. I just run a 16 process MPI job to get the space for /tmp. The amount of the space is 512M per node but only 66% is available, translated to 21.3 Mbytes per process. Is this the reason for failing 16 Mbyte checkpoint in the second trial? Regards, Keita (Here is the output from df) Filesystem 1K-blocks Used Available Use% Mounted on /dev/ram 512000 172280 339720 34% /ram On 4/24/14, 11:16 AM, "Adam T. Moody" <mo...@ll...> wrote: >Hi Keita, >I agree. It's most likely running out of space in node-local storage. >In fact, I think the error is coming from the test_api code which only >writes to node local storage. > >Is it failing on the first checkpoint? > >Do you have SCR configured to write to /tmp, and if so, can you check >the size of /tmp on your compute nodes? For example, login into a >compute node and run "df /tmp". >-Adam > > >Kathryn Mohror wrote: > >>Hi Keita, >> >>Sorry for the delay. I am not sure why that would be happening. Adam do >>you have an idea? >> >>It might that you are running short of node local storage. SCR will >>first write to node-local storage and then to the PFS. However 8 MB >>seems a bit small to be causing that problem. Hopefully Adam will have >>an idea. >> >>Kathryn >> >>On Apr 21, 2014, at 4:18 PM, Teranishi, Keita <kn...@sa...> wrote: >> >> >> >>>Hi, >>> >>>I am still playing with test_api in example directory and found the code >>>throws an error when I set file size bigger than 8 Mbytes. I¹d like to >>>know (1) what is the root cause of this error and (2) any possible ways >>>to >>>mitigate this problem. >>> >>>I set SCR_PREFIX to be a scratch space (1 Pbytes) in the lustre file >>>system connected to the PC cluster. The rest of the parameters should be >>>set to the default. >>> >>>1 on chama33: ERROR: Error writing: write(12, 0x2aaaba61000a, 13443078) >>>errno=28 No space left on device @ test_common.c:86 >>> >>>Thanks, >>>------------------------------------------------------------------------ >>>--- >>>-- >>>Keita Teranishi >>>Principal Member of Technical Staff >>>Scalable Modeling and Analysis Systems >>>Sandia National Laboratories >>>Livermore, CA 94551 >>>+1 (925) 294-3738 >>> >>> >>>------------------------------------------------------------------------ >>>------ >>>Start Your Social Network Today - Download eXo Platform >>>Build your Enterprise Intranet with eXo Platform Software >>>Java Based Open Source Intranet - Social, Extensible, Cloud Ready >>>Get Started Now And Turn Your Intranet Into A Collaboration Platform >>>http://p.sf.net/sfu/ExoPlatform >>>_______________________________________________ >>>Scalablecr-discuss mailing list >>>Sca...@li... >>>https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss >>> >>> >> >>_________________________________________________________________ >>Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ >>Scalability Team @ Lawrence Livermore National Laboratory, Livermore, >>CA, USA >> >> >> >> >> >> >> >> >> >> >> >>------------------------------------------------------------------------ >> >>------------------------------------------------------------------------- >>----- >>Start Your Social Network Today - Download eXo Platform >>Build your Enterprise Intranet with eXo Platform Software >>Java Based Open Source Intranet - Social, Extensible, Cloud Ready >>Get Started Now And Turn Your Intranet Into A Collaboration Platform >>http://p.sf.net/sfu/ExoPlatform >> >>------------------------------------------------------------------------ >> >>_______________________________________________ >>Scalablecr-discuss mailing list >>Sca...@li... >>https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss >> >> > |
|
From: Adam T. M. <mo...@ll...> - 2014-04-24 18:16:32
|
Hi Keita, I agree. It's most likely running out of space in node-local storage. In fact, I think the error is coming from the test_api code which only writes to node local storage. Is it failing on the first checkpoint? Do you have SCR configured to write to /tmp, and if so, can you check the size of /tmp on your compute nodes? For example, login into a compute node and run "df /tmp". -Adam Kathryn Mohror wrote: >Hi Keita, > >Sorry for the delay. I am not sure why that would be happening. Adam do you have an idea? > >It might that you are running short of node local storage. SCR will first write to node-local storage and then to the PFS. However 8 MB seems a bit small to be causing that problem. Hopefully Adam will have an idea. > >Kathryn > >On Apr 21, 2014, at 4:18 PM, Teranishi, Keita <kn...@sa...> wrote: > > > >>Hi, >> >>I am still playing with test_api in example directory and found the code >>throws an error when I set file size bigger than 8 Mbytes. I¹d like to >>know (1) what is the root cause of this error and (2) any possible ways to >>mitigate this problem. >> >>I set SCR_PREFIX to be a scratch space (1 Pbytes) in the lustre file >>system connected to the PC cluster. The rest of the parameters should be >>set to the default. >> >>1 on chama33: ERROR: Error writing: write(12, 0x2aaaba61000a, 13443078) >>errno=28 No space left on device @ test_common.c:86 >> >>Thanks, >>--------------------------------------------------------------------------- >>-- >>Keita Teranishi >>Principal Member of Technical Staff >>Scalable Modeling and Analysis Systems >>Sandia National Laboratories >>Livermore, CA 94551 >>+1 (925) 294-3738 >> >> >>------------------------------------------------------------------------------ >>Start Your Social Network Today - Download eXo Platform >>Build your Enterprise Intranet with eXo Platform Software >>Java Based Open Source Intranet - Social, Extensible, Cloud Ready >>Get Started Now And Turn Your Intranet Into A Collaboration Platform >>http://p.sf.net/sfu/ExoPlatform >>_______________________________________________ >>Scalablecr-discuss mailing list >>Sca...@li... >>https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss >> >> > >_________________________________________________________________ >Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ >Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > > > > > > > > >------------------------------------------------------------------------ > >------------------------------------------------------------------------------ >Start Your Social Network Today - Download eXo Platform >Build your Enterprise Intranet with eXo Platform Software >Java Based Open Source Intranet - Social, Extensible, Cloud Ready >Get Started Now And Turn Your Intranet Into A Collaboration Platform >http://p.sf.net/sfu/ExoPlatform > >------------------------------------------------------------------------ > >_______________________________________________ >Scalablecr-discuss mailing list >Sca...@li... >https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss > > |
|
From: Kathryn M. <ka...@ll...> - 2014-04-24 18:04:05
|
Hi Keita, Sorry for the delay. I am not sure why that would be happening. Adam do you have an idea? It might that you are running short of node local storage. SCR will first write to node-local storage and then to the PFS. However 8 MB seems a bit small to be causing that problem. Hopefully Adam will have an idea. Kathryn On Apr 21, 2014, at 4:18 PM, Teranishi, Keita <kn...@sa...> wrote: > Hi, > > I am still playing with test_api in example directory and found the code > throws an error when I set file size bigger than 8 Mbytes. I¹d like to > know (1) what is the root cause of this error and (2) any possible ways to > mitigate this problem. > > I set SCR_PREFIX to be a scratch space (1 Pbytes) in the lustre file > system connected to the PC cluster. The rest of the parameters should be > set to the default. > > 1 on chama33: ERROR: Error writing: write(12, 0x2aaaba61000a, 13443078) > errno=28 No space left on device @ test_common.c:86 > > Thanks, > --------------------------------------------------------------------------- > -- > Keita Teranishi > Principal Member of Technical Staff > Scalable Modeling and Analysis Systems > Sandia National Laboratories > Livermore, CA 94551 > +1 (925) 294-3738 > > > ------------------------------------------------------------------------------ > Start Your Social Network Today - Download eXo Platform > Build your Enterprise Intranet with eXo Platform Software > Java Based Open Source Intranet - Social, Extensible, Cloud Ready > Get Started Now And Turn Your Intranet Into A Collaboration Platform > http://p.sf.net/sfu/ExoPlatform > _______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Teranishi, K. <kn...@sa...> - 2014-04-21 23:18:27
|
Hi, I am still playing with test_api in example directory and found the code throws an error when I set file size bigger than 8 Mbytes. I¹d like to know (1) what is the root cause of this error and (2) any possible ways to mitigate this problem. I set SCR_PREFIX to be a scratch space (1 Pbytes) in the lustre file system connected to the PC cluster. The rest of the parameters should be set to the default. 1 on chama33: ERROR: Error writing: write(12, 0x2aaaba61000a, 13443078) errno=28 No space left on device @ test_common.c:86 Thanks, --------------------------------------------------------------------------- -- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 |
|
From: Adam T. M. <mo...@ll...> - 2014-04-18 21:27:05
|
Hi Keita, If the application calls SCR_Finalize, the library records this fact in the "halt" file. The halt file is stored in the SCR_PREFIX directory. If the halt file specifies an active reason for halting, the SCR library will not restart the run. This message from your output indicates that's what is happening in this case: SCR: rank 0 on chama648: Job exiting: Reason: SCR_FINALIZE_CALLED. This feature is by design, as it serves a way for the user to specify that SCR should *not* restart the job. For example, if the application reaches the end of the simulation and exits, you don't want SCR to restart the job. Calling SCR_Finalize is one way for the user to signal to SCR that it should not restart the job. This behavior is desirable when running real applications, but it gets in the way of testing. To work around this, you can delete the halt file using the scr_halt command, like so: scr_halt --remove <prefix_dir> If you execute scr_halt from the prefix directory, you don't need to specify the directory as an argument. Also, you can shorten "--remove" to just "-r". Let me know if that helps, -Adam Teranishi, Keita wrote: >Hi, > >I have been trying to run OpenMPI program with SCR on a PC cluster. This >machine uses SLRUM for batch scheduler. In the interactive session, I >have been trying to run an example program test_api 3 times in the same >batch session (I allocated 2 extra nodes). However, the second and third >execution tries to pick up the same nodes that may have hanging processes >(processes terminated without MPI_Finalize). I¹d like to know if I have >to put any environment variables or something I have to do with slrum to >pick up spare nodes. Please let me know. > > > >Here is the error message for the second and third execution. > >SCR ERROR: rank 0 on chama648: Failed to read username or jobname from >environment, disabling logging @ scr.c:6451 >SCR: rank 0 on chama648: Job exiting: Reason: SCR_FINALIZE_CALLED. >-------------------------------------------------------------------------- >mpirun has exited due to process rank 122 with PID 171448 on >node chama1046 exiting without calling "finalize". This may >have caused other processes in the application to be >terminated by signals sent by mpirun (as reported here). >‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹ > > >SCR ERROR: rank 0 on chama648: Failed to read username or jobname from >environment, disabling logging @ scr.c:6451 >SCR: rank 0 on chama648: Job exiting: Reason: SCR_FINALIZE_CALLED. >-------------------------------------------------------------------------- >mpirun has exited due to process rank 11 with PID 148920 on >node chama648 exiting without calling "finalize". This may >have caused other processes in the application to be >terminated by signals sent by mpirun (as reported here). >------------------------------------------------------------------------- > >--------------------------------------------------------------------------- >-- >Keita Teranishi >Principal Member of Technical Staff >Scalable Modeling and Analysis Systems >Sandia National Laboratories >Livermore, CA 94551 >+1 (925) 294-3738 > > >------------------------------------------------------------------------------ >Learn Graph Databases - Download FREE O'Reilly Book >"Graph Databases" is the definitive new guide to graph databases and their >applications. Written by three acclaimed leaders in the field, >this first edition is now available. Download your free book today! >http://p.sf.net/sfu/NeoTech >_______________________________________________ >Scalablecr-discuss mailing list >Sca...@li... >https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss > > |
|
From: Teranishi, K. <kn...@sa...> - 2014-04-18 20:42:20
|
Hi, I have been trying to run OpenMPI program with SCR on a PC cluster. This machine uses SLRUM for batch scheduler. In the interactive session, I have been trying to run an example program test_api 3 times in the same batch session (I allocated 2 extra nodes). However, the second and third execution tries to pick up the same nodes that may have hanging processes (processes terminated without MPI_Finalize). I¹d like to know if I have to put any environment variables or something I have to do with slrum to pick up spare nodes. Please let me know. Here is the error message for the second and third execution. SCR ERROR: rank 0 on chama648: Failed to read username or jobname from environment, disabling logging @ scr.c:6451 SCR: rank 0 on chama648: Job exiting: Reason: SCR_FINALIZE_CALLED. -------------------------------------------------------------------------- mpirun has exited due to process rank 122 with PID 171448 on node chama1046 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). ‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹ SCR ERROR: rank 0 on chama648: Failed to read username or jobname from environment, disabling logging @ scr.c:6451 SCR: rank 0 on chama648: Job exiting: Reason: SCR_FINALIZE_CALLED. -------------------------------------------------------------------------- mpirun has exited due to process rank 11 with PID 148920 on node chama648 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). ------------------------------------------------------------------------- --------------------------------------------------------------------------- -- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 |
|
From: Kathryn M. <ka...@ll...> - 2014-01-28 22:04:23
|
Hi Arjun, I heard through the grapevine that the MVAPICH team is helping you out. Let me know if they get stuck and you need more help from our end. Kathryn On Jan 27, 2014, at 8:56 PM, Arjun J Rao <rec...@gm...> wrote: > I have write permissions on the location of the control directory. I have put the control directory as /home/username/Control and I own the directory. The same directory has been set in the scr.conf file as well in the CNTLDIR as well as in the SCR_CNTL_BASE variable. I have a /tmp too, and have write permissions on that directory (In fact, I have made my username own that directory) > My system runs Scientific Linux 6.4 > > > > > > > > On Tue, Jan 14, 2014 at 6:23 AM, Kathryn Mohror <ka...@ll...> wrote: > Hi Arjun, > > Sorry for the late reply -- I took some vacation. > > Just as a first guess, did you make sure that you have write permissions on the location of the control directory? I believe (and the error message supports this) that SCR_CNTL_BASE can't be set in the environment, but has to be set in your SCR configuration file. It defaults to /tmp. Do you have a /tmp? > > Kathryn > > On Dec 25, 2013, at 8:59 PM, Arjun J Rao <rec...@gm...> wrote: > >> I understand that SCR was built to be used with custom application codes that >> write their own checkpoints from within the application. However, MVAPICH2 >> >> claims to have integrated SCR such that the checkpoints written by BLCR can >> >> be written to the parallel file system in a scalable manner later. However, I >> am not currently writing out the checkpoints to a central location, but to a >> local disk for testing. >> >> I first installed BLCR and SLURM. Then I installed MVAPICH2 with the >> following options : >> ./configure --enable-ckpt --with-scr --with-pm=no --with-pmi=slurm >> >> However, taking a simple MPI program and compiling using mpicc and then >> running using srun yields the following errors : >> >> srun -N2 -n12 MPIExecutable >> SCR v1.1-8 ABORT : rank 1 on machine2: Failed to create store descriptor >> for control directory @ >> src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 >> In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 1) >> SCR v1.1-8 ABORT : rank 0 on machine1: Failed to create store descriptor >> for control directory @ >> src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 >> In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 0) >> . >> . >> . >> SCR v1.1-8 ERROR: rank 3 on machine1: SCR_CNTL_BASE cannot be set in the >> environment or user configuration file, ignoring setting >> SCR v1.1-8 ERROR: rank 1 on machine1: SCR_CNTL_BASE cannot be set in the >> environment or user configuration file, ignoring setting >> . >> . >> . >> slurmd[machine1]: ***STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL >> 9*** >> srun: Job step aborted: Waiting upto 2 seconds for job step to finish >> >> slurmd[machine1]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL >> 9**** >> slurmd[machine2]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL >> 9**** >> . >> . >> . >> >> There was actually a lot of output but i've just printed only one version >> of each of the message types in the output.I have set SCR_CNTL_BASE, SCR_RUNS, >> SCR_CACHE_BASE, SCR_PREFIX and SCR_FLUSH. What could be wrong with the >> configuration or the environment of SCR to yield such errors ? >> ------------------------------------------------------------------------------ >> Rapidly troubleshoot problems before they affect your business. Most IT >> organizations don't have a clear picture of how application performance >> affects their revenue. With AppDynamics, you get 100% visibility into your >> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! >> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_______________________________________________ >> Scalablecr-discuss mailing list >> Sca...@li... >> https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss > > _________________________________________________________________ > Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ > Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > > > > > > > ------------------------------------------------------------------------------ > WatchGuard Dimension instantly turns raw network data into actionable > security intelligence. It gives you real-time visual feedback on key > security issues and trends. Skip the complicated setup - simply import > a virtual appliance and go from zero to informed in seconds. > http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk_______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Arjun J R. <rec...@gm...> - 2014-01-28 04:56:29
|
I have write permissions on the location of the control directory. I have
put the control directory as /home/username/Control and I own the
directory. The same directory has been set in the scr.conf file as well in
the CNTLDIR as well as in the SCR_CNTL_BASE variable. I have a /tmp too,
and have write permissions on that directory (In fact, I have made my
username own that directory)
My system runs Scientific Linux 6.4
On Tue, Jan 14, 2014 at 6:23 AM, Kathryn Mohror <ka...@ll...> wrote:
> Hi Arjun,
>
> Sorry for the late reply -- I took some vacation.
>
> Just as a first guess, did you make sure that you have write permissions
> on the location of the control directory? I believe (and the error message
> supports this) that SCR_CNTL_BASE can't be set in the environment, but has
> to be set in your SCR configuration file. It defaults to /tmp. Do you have
> a /tmp?
>
> Kathryn
>
> On Dec 25, 2013, at 8:59 PM, Arjun J Rao <rec...@gm...> wrote:
>
> I understand that SCR was built to be used with custom application codes that
> write their own checkpoints from within the application. However, MVAPICH2
> claims to have integrated SCR such that the checkpoints written by BLCR can
>
> be written to the parallel file system in a scalable manner later. However, I
> am not currently writing out the checkpoints to a central location, but to a
> local disk for testing.
>
>
> I first installed BLCR and SLURM. Then I installed MVAPICH2 with the
> following options :
> ./configure --enable-ckpt --with-scr --with-pm=no --with-pmi=slurm
> However, taking a simple MPI program and compiling using mpicc and then
> running using srun yields the following errors :
>
> srun -N2 -n12 MPIExecutable
> SCR v1.1-8 ABORT : rank 1 on machine2: Failed to create store descriptor
> for control directory @
> src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299
> In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 1)
> SCR v1.1-8 ABORT : rank 0 on machine1: Failed to create store descriptor
> for control directory @
> src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299
> In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 0)
> .
> .
> .
> SCR v1.1-8 ERROR: rank 3 on machine1: SCR_CNTL_BASE cannot be set in the
> environment or user configuration file, ignoring setting
> SCR v1.1-8 ERROR: rank 1 on machine1: SCR_CNTL_BASE cannot be set in the
> environment or user configuration file, ignoring setting
> .
> .
> .
> slurmd[machine1]: ***STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL
> 9***
> srun: Job step aborted: Waiting upto 2 seconds for job step to finish
>
> slurmd[machine1]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL
> 9****
> slurmd[machine2]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL
> 9****
> .
> .
> .
>
> There was actually a lot of output but i've just printed only one version
> of each of the message types in the output.I have set SCR_CNTL_BASE, SCR_RUNS,
> SCR_CACHE_BASE, SCR_PREFIX and SCR_FLUSH. What could be wrong with the
> configuration or the environment of SCR to yield such errors ?
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
>
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_______________________________________________
> Scalablecr-discuss mailing list
> Sca...@li...
> https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss
>
>
> _________________________________________________________________
> Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/
> Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA,
> USA
>
>
>
>
>
>
>
>
>
|
|
From: Kathryn M. <ka...@ll...> - 2014-01-14 21:38:29
|
Hi Andrew, >> I am not aware of it, although you might be able to get it to work out of >> the box depending on how you plan to use the files. What I mean is, if it's >> OK with your application that SCR treats those files like checkpoints >> (meaning possibly deletes older ones when new ones are written), then it >> should be fine. Otherwise, you would need to copy each file out to your >> parallel file system, which would defeat the purpose of using SCR. >> >> That said, we are planning to add support for ordinary files into SCR in the >> future. It would be interesting to hear what your use case is and how SCR >> could best support that. > > Thanks, Kathryn. My interest is indirect. I don't have an app to use > SCR with, but I am a operating system, storage, and network researcher > that is interested in how file systems can be better designed for > users. I'm glad to hear that SCR will support ordinary files because > it seems like there's a good opportunity for it to take on the use > cases that traditional parallel file systems have difficulty with. That was our thinking too. > > What will the support for ordinary files look like? Will those files > persist on SSD between jobs? We're not sure what the support will look like at this point -- this is still in the early design stages. However, we imagine an API where applications can tell SCR what type of file is being handled and what to do with it, e.g. keep it in in-system storage, or only cache it in in-system storage, move it to the parallel file system in the background, and then delete the cached copy. As to whether files could persist on SSDs between jobs, I imagine that is would depend on the resource management policies. For example, if the same user/app doesn't get the same set of nodes in the next run you wouldn't want the files there necessarily -- for space reasons and maybe privacy/security. I'm glad to know of your interest! Kathryn > > Thanks again, > > Andrew _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Andrew S. <ag...@gm...> - 2014-01-14 02:34:52
|
On Mon, Jan 13, 2014 at 4:41 PM, Kathryn Mohror <ka...@ll...> wrote: > I am not aware of it, although you might be able to get it to work out of > the box depending on how you plan to use the files. What I mean is, if it's > OK with your application that SCR treats those files like checkpoints > (meaning possibly deletes older ones when new ones are written), then it > should be fine. Otherwise, you would need to copy each file out to your > parallel file system, which would defeat the purpose of using SCR. > > That said, we are planning to add support for ordinary files into SCR in the > future. It would be interesting to hear what your use case is and how SCR > could best support that. Thanks, Kathryn. My interest is indirect. I don't have an app to use SCR with, but I am a operating system, storage, and network researcher that is interested in how file systems can be better designed for users. I'm glad to hear that SCR will support ordinary files because it seems like there's a good opportunity for it to take on the use cases that traditional parallel file systems have difficulty with. What will the support for ordinary files look like? Will those files persist on SSD between jobs? Thanks again, Andrew |
|
From: Kathryn M. <ka...@ll...> - 2014-01-14 00:53:40
|
Hi Arjun, Sorry for the late reply -- I took some vacation. Just as a first guess, did you make sure that you have write permissions on the location of the control directory? I believe (and the error message supports this) that SCR_CNTL_BASE can't be set in the environment, but has to be set in your SCR configuration file. It defaults to /tmp. Do you have a /tmp? Kathryn On Dec 25, 2013, at 8:59 PM, Arjun J Rao <rec...@gm...> wrote: > I understand that SCR was built to be used with custom application codes that > write their own checkpoints from within the application. However, MVAPICH2 > claims to have integrated SCR such that the checkpoints written by BLCR can > > be written to the parallel file system in a scalable manner later. However, I > am not currently writing out the checkpoints to a central location, but to a > local disk for testing. > > I first installed BLCR and SLURM. Then I installed MVAPICH2 with the > following options : > ./configure --enable-ckpt --with-scr --with-pm=no --with-pmi=slurm > > However, taking a simple MPI program and compiling using mpicc and then > running using srun yields the following errors : > > srun -N2 -n12 MPIExecutable > SCR v1.1-8 ABORT : rank 1 on machine2: Failed to create store descriptor > for control directory @ > src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 > In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 1) > SCR v1.1-8 ABORT : rank 0 on machine1: Failed to create store descriptor > for control directory @ > src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 > In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 0) > . > . > . > SCR v1.1-8 ERROR: rank 3 on machine1: SCR_CNTL_BASE cannot be set in the > environment or user configuration file, ignoring setting > SCR v1.1-8 ERROR: rank 1 on machine1: SCR_CNTL_BASE cannot be set in the > environment or user configuration file, ignoring setting > . > . > . > slurmd[machine1]: ***STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL > 9*** > srun: Job step aborted: Waiting upto 2 seconds for job step to finish > > slurmd[machine1]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL > 9**** > slurmd[machine2]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL > 9**** > . > . > . > > There was actually a lot of output but i've just printed only one version > of each of the message types in the output.I have set SCR_CNTL_BASE, SCR_RUNS, > SCR_CACHE_BASE, SCR_PREFIX and SCR_FLUSH. What could be wrong with the > configuration or the environment of SCR to yield such errors ? > ------------------------------------------------------------------------------ > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Kathryn M. <ka...@ll...> - 2014-01-13 23:41:22
|
Hi Andrew, > I was reading through > http://www.csm.ornl.gov/srt/conferences/ResilienceSummit/2009/pdf/moody_slides.pdf > and I was wondering what people's experience with using SCR on local > SSDs or disks has been. Was it worth putting an SSD in each node of > Coastal? Our experience was that it was useful, especially for applications that use all their memory and can't store checkpoints in RAM disk. That said, overall we are still evaluating the usefulness vs. cost of attaching SSDs to compute nodes. The SSDs on Coastal were an early generation and not as long-lasting or efficient as newer models. > Also has anyone used SCR for non-checkpoint files? I am not aware of it, although you might be able to get it to work out of the box depending on how you plan to use the files. What I mean is, if it's OK with your application that SCR treats those files like checkpoints (meaning possibly deletes older ones when new ones are written), then it should be fine. Otherwise, you would need to copy each file out to your parallel file system, which would defeat the purpose of using SCR. That said, we are planning to add support for ordinary files into SCR in the future. It would be interesting to hear what your use case is and how SCR could best support that. Best, Kathryn > > Thank you, > > -- > Andrew Shewmaker > > ------------------------------------------------------------------------------ > CenturyLink Cloud: The Leader in Enterprise Cloud Services. > Learn Why More Businesses Are Choosing CenturyLink Cloud For > Critical Workloads, Development Environments & Everything In Between. > Get a Quote or Start a Free Trial Today. > http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk > _______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss _________________________________________________________________ Kathryn Mohror, ka...@ll..., http://scalability.llnl.gov/ Scalability Team @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Andrew S. <ag...@gm...> - 2014-01-13 18:38:46
|
Hi, I was reading through http://www.csm.ornl.gov/srt/conferences/ResilienceSummit/2009/pdf/moody_slides.pdf and I was wondering what people's experience with using SCR on local SSDs or disks has been. Was it worth putting an SSD in each node of Coastal? Also has anyone used SCR for non-checkpoint files? Thank you, -- Andrew Shewmaker |
|
From: Arjun J R. <rec...@gm...> - 2013-12-26 05:00:03
|
I understand that SCR was built to be used with custom application codes that write their own checkpoints from within the application. However, MVAPICH2 claims to have integrated SCR such that the checkpoints written by BLCR can be written to the parallel file system in a scalable manner later. However, I am not currently writing out the checkpoints to a central location, but to a local disk for testing. I first installed BLCR and SLURM. Then I installed MVAPICH2 with the following options : ./configure --enable-ckpt --with-scr --with-pm=no --with-pmi=slurm However, taking a simple MPI program and compiling using mpicc and then running using srun yields the following errors : srun -N2 -n12 MPIExecutable SCR v1.1-8 ABORT : rank 1 on machine2: Failed to create store descriptor for control directory @ src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 1) SCR v1.1-8 ABORT : rank 0 on machine1: Failed to create store descriptor for control directory @ src/mpid/ch3/channels/common/src/scr/scr_storedesc.c:299 In: PMI_Abort(0,application called MPI_Abort(MPI_COMM_WORLD,0) - process 0) . . . SCR v1.1-8 ERROR: rank 3 on machine1: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting SCR v1.1-8 ERROR: rank 1 on machine1: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting . . . slurmd[machine1]: ***STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL 9*** srun: Job step aborted: Waiting upto 2 seconds for job step to finish slurmd[machine1]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL 9**** slurmd[machine2]: *** STEP 195.1 KILLED AT 2013-12-24T10:11:23 WITH SIGNAL 9**** . . . There was actually a lot of output but i've just printed only one version of each of the message types in the output.I have set SCR_CNTL_BASE, SCR_RUNS, SCR_CACHE_BASE, SCR_PREFIX and SCR_FLUSH. What could be wrong with the configuration or the environment of SCR to yield such errors ? |
|
From: Kathryn M. <ka...@ll...> - 2013-12-06 02:20:47
|
Hi Rommel, Do you have SCR_RUNS set in your job environment? It looks to me like the scr_srun script is checking to see if the number of runs is exhausted, and since it thinks it is, it starts exit procedures. You might try setting it to 3 to see what happens. Kathryn On Dec 5, 2013, at 3:42 AM, AJR <ico...@gm...> wrote: > I had BLCR in mind when I asked the question. Yea, MVAPICH2 does claim that they have integrated SCR with MVAPICH. But, when I was previously able to easily run a Hello World MPI job using srun -N2 -n4 ./MPIHello, after I installed MVAPICH2's SCR support (Using the --with-scr flag during configuration) I get the following output while I run the same command. Maybe they have made every call to srun call scr_srun or something similar. My question is about what could be causing the following kind of error. Seems to be some kind of configuration glitch with SCR. > > scr_srun: Started: Wed Dec 4 19:06:23 IST 2013 > scr_srun: prerun: Wed Dec 4 19:06:24 IST 2013 > scr_prerun: Started: Wed Dec 4 19:06:24 IST 2013 > scr_prerun: Ended: Wed Dec 4 19:06:24 IST 2013 > scr_prerun: secs: 0 > scr_prerun: exit code: 0 > scr_srun: RUN 1: Wed Dec 4 19:06:24 IST 2013 > SCR v1.1-8 ERROR: rank 1 on qdr3: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting > SCR v1.1-8 ERROR: rank 3 on qdr4: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting > SCR v1.1-8 ERROR: rank 0 on qdr3: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting > SCR v1.1-8 WARNING: rank 0 on qdr3: Failed to record cluster name @ src/mpid/ch3/channels/common/src/scr/scr.c:367 > scr_srun: $SCR_RUNS exhausted, ending run. > scr_srun: postrun: Wed Dec 4 19:06:25 IST 2013 > scr_postrun: Started: Wed Dec 4 19:06:25 IST 2013 > scr_postrun: UPNODES: qdr[3-4] > scr_postrun: Looking for latest checkpoint set id > scr_postrun: Found no checkpoint set to flush on qdr3 > scr_postrun: Ended: Wed Dec 4 19:06:26 IST 2013 > scr_postrun: secs: 1 > scr_postrun: exit code: 1 > scr_srun: ERROR: Command failed: scr_postrun -p /home/arjun/Checkpoint > scr_srun: Ended: Wed Dec 4 19:06:26 IST 2013 > > > > > On Tue, Dec 3, 2013 at 2:01 AM, Moody, Adam T. <mo...@ll...> wrote: > Hi Rommel, > Most of the applications using SCR contain logic to write their own checkpoints, so we don't have a lot of experience yet with third-party checkpointing packages. Having said that, we know that the MVAPICH MPI team has integrated BLCR (Berkeley Lab Checkpoint Restart) with SCR. > > http://mvapich.cse.ohio-state.edu > > Also, there is a research-level checkpointing package at LLNL which implements a persistent heap (variables stored in this region are checkpointed with a call to msync). This package has also been integrated with SCR. > > http://e-reports-ext.llnl.gov/pdf/754806.pdf > > Is there a particular checkpointing package you have in mind? > -Adam > > From: AJR [ico...@gm...] > Sent: Sunday, December 01, 2013 11:44 PM > To: sca...@li... > Subject: [Scalablecr-discuss] Checkpointing using SCR > > Hi, > You mention in your manuals that SCR is meant to work well with checkpointing packages that write a file per MPI process. Could you give some examples of the kind of checkpointing packages that will work well with SCR ? > > Rommel Lucas > > > ------------------------------------------------------------------------------ > Sponsored by Intel(R) XDK > Develop, test and display web and hybrid apps with a single code base. > Download it for free now! > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk_______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss ______________________________________________________________ Kathryn Mohror, ka...@ll..., http://people.llnl.gov/mohror1 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: AJR <ico...@gm...> - 2013-12-05 11:42:35
|
I had BLCR in mind when I asked the question. Yea, MVAPICH2 does claim that they have integrated SCR with MVAPICH. But, when I was previously able to easily run a Hello World MPI job using srun -N2 -n4 ./MPIHello, after I installed MVAPICH2's SCR support (Using the --with-scr flag during configuration) I get the following output while I run the same command. Maybe they have made every call to srun call scr_srun or something similar. My question is about what could be causing the following kind of error. Seems to be some kind of configuration glitch with SCR. scr_srun: Started: Wed Dec 4 19:06:23 IST 2013 scr_srun: prerun: Wed Dec 4 19:06:24 IST 2013 scr_prerun: Started: Wed Dec 4 19:06:24 IST 2013 scr_prerun: Ended: Wed Dec 4 19:06:24 IST 2013 scr_prerun: secs: 0 scr_prerun: exit code: 0 scr_srun: RUN 1: Wed Dec 4 19:06:24 IST 2013 SCR v1.1-8 ERROR: rank 1 on qdr3: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting SCR v1.1-8 ERROR: rank 3 on qdr4: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting SCR v1.1-8 ERROR: rank 0 on qdr3: SCR_CNTL_BASE cannot be set in the environment or user configuration file, ignoring setting SCR v1.1-8 WARNING: rank 0 on qdr3: Failed to record cluster name @ src/mpid/ch3/channels/common/src/scr/scr.c:367 scr_srun: $SCR_RUNS exhausted, ending run. scr_srun: postrun: Wed Dec 4 19:06:25 IST 2013 scr_postrun: Started: Wed Dec 4 19:06:25 IST 2013 scr_postrun: UPNODES: qdr[3-4] scr_postrun: Looking for latest checkpoint set id scr_postrun: Found no checkpoint set to flush on qdr3 scr_postrun: Ended: Wed Dec 4 19:06:26 IST 2013 scr_postrun: secs: 1 scr_postrun: exit code: 1 scr_srun: ERROR: Command failed: scr_postrun -p /home/arjun/Checkpoint scr_srun: Ended: Wed Dec 4 19:06:26 IST 2013 On Tue, Dec 3, 2013 at 2:01 AM, Moody, Adam T. <mo...@ll...> wrote: > Hi Rommel, > Most of the applications using SCR contain logic to write their own > checkpoints, so we don't have a lot of experience yet with third-party > checkpointing packages. Having said that, we know that the MVAPICH MPI > team has integrated BLCR (Berkeley Lab Checkpoint Restart) with SCR. > > http://mvapich.cse.ohio-state.edu > > Also, there is a research-level checkpointing package at LLNL which > implements a persistent heap (variables stored in this region are > checkpointed with a call to msync). This package has also been integrated > with SCR. > > http://e-reports-ext.llnl.gov/pdf/754806.pdf > > Is there a particular checkpointing package you have in mind? > -Adam > > ------------------------------ > *From:* AJR [ico...@gm...] > *Sent:* Sunday, December 01, 2013 11:44 PM > *To:* sca...@li... > *Subject:* [Scalablecr-discuss] Checkpointing using SCR > > Hi, > You mention in your manuals that SCR is meant to work well with > checkpointing packages that write a file per MPI process. Could you give > some examples of the kind of checkpointing packages that will work well > with SCR ? > > Rommel Lucas > |
|
From: Moody, A. T. <mo...@ll...> - 2013-12-02 20:43:34
|
Hi Arjun, Right, you need to run scr_srun within a SLURM allocation, so an salloc command is first required to get the SLURM allocation. We'll work to clarify that detail in future documentation. As for the second error, SCR should default to set the control directory to use /tmp. You can override this during the configure step. If you did that, can you please cut-and-paste the configure line you used to configure SCR? Also, it would help to increase the debug verbosity, which you can do by setting the SCR_DEBUG environment variable to 1 or perhaps 2. Can you do that and send along the output? -Adam ________________________________ From: Arjun J Rao [rec...@gm...] Sent: Thursday, November 28, 2013 1:45 AM To: sca...@li... Subject: Re: [Scalablecr-discuss] Error: Invalid control directory INVALID Correction : It is not SCR_NODELIST, it is SLURM_NODELIST. And I also run it inside an allocation provided by SLURM.. salloc -N2 bash and then run scr_srun -N2 -n4 abc I also got confused because the way to run MVAPICH2 jobs in SLURM is to just run the srun commands which takes care of the allocation on its own. But here, with scr_srun, I guess we have to do our own allocations. On Thu, Nov 28, 2013 at 1:23 PM, Arjun J Rao <rec...@gm...<mailto:rec...@gm...>> wrote: I have a 2-node setup on which I intend to test out scalable checkpointing. I keep getting the error "Could not identify nodeset" when I issue the command scr_srun -N2 -n4 abc I found that the SCR_NODELIST environment variable needs to be set. What format must the nodeset be provided in ? I could not find it in the pdf manual or in the readmes. Setting SCR_NODELIST to abc[1-4] or to abc1,abc2,abc3,abc4 both of which silence the "could not identify nodeset" warnings. Which is the correct format to set the SCR_NODELIST variable to ? Also, after the "could not identify nodeset" warning was suppressed, I ran scr_srun -N2 -n4 abc to find this error ""ERROR: Invalid control directory INVALID" I checked the /etc/scr.conf file and it was set to /tmp. I tried changing it to /home/username/somedir but got the same error. I also tried setting the SCR_CNTL_BASE environmental to point to /tmp and to /home/username/somedir but got the same warning message. How can this issue be resolved ? |
|
From: Moody, A. T. <mo...@ll...> - 2013-12-02 20:31:16
|
Hi Rommel, Most of the applications using SCR contain logic to write their own checkpoints, so we don't have a lot of experience yet with third-party checkpointing packages. Having said that, we know that the MVAPICH MPI team has integrated BLCR (Berkeley Lab Checkpoint Restart) with SCR. http://mvapich.cse.ohio-state.edu Also, there is a research-level checkpointing package at LLNL which implements a persistent heap (variables stored in this region are checkpointed with a call to msync). This package has also been integrated with SCR. http://e-reports-ext.llnl.gov/pdf/754806.pdf Is there a particular checkpointing package you have in mind? -Adam ________________________________ From: AJR [ico...@gm...] Sent: Sunday, December 01, 2013 11:44 PM To: sca...@li... Subject: [Scalablecr-discuss] Checkpointing using SCR Hi, You mention in your manuals that SCR is meant to work well with checkpointing packages that write a file per MPI process. Could you give some examples of the kind of checkpointing packages that will work well with SCR ? Rommel Lucas |
|
From: AJR <ico...@gm...> - 2013-12-02 07:44:21
|
Hi, You mention in your manuals that SCR is meant to work well with checkpointing packages that write a file per MPI process. Could you give some examples of the kind of checkpointing packages that will work well with SCR ? Rommel Lucas |
|
From: Arjun J R. <rec...@gm...> - 2013-11-28 09:45:31
|
Correction : It is not SCR_NODELIST, it is SLURM_NODELIST. And I also run it inside an allocation provided by SLURM.. salloc -N2 bash and then run scr_srun -N2 -n4 abc I also got confused because the way to run MVAPICH2 jobs in SLURM is to just run the srun commands which takes care of the allocation on its own. But here, with scr_srun, I guess we have to do our own allocations. On Thu, Nov 28, 2013 at 1:23 PM, Arjun J Rao <rec...@gm...>wrote: > I have a 2-node setup on which I intend to test out scalable > checkpointing. > I keep getting the error "Could not identify nodeset" when I issue the > command scr_srun -N2 -n4 abc > > I found that the SCR_NODELIST environment variable needs to be set. What > format must the nodeset be provided in ? I could not find it in the pdf > manual or in the readmes. Setting SCR_NODELIST to abc[1-4] or to > abc1,abc2,abc3,abc4 both of which silence the "could not identify nodeset" > warnings. Which is the correct format to set the SCR_NODELIST variable to ? > > Also, after the "could not identify nodeset" warning was suppressed, I ran > scr_srun -N2 -n4 abc > to find this error > ""ERROR: Invalid control directory INVALID" > > I checked the /etc/scr.conf file and it was set to /tmp. I tried changing > it to /home/username/somedir but got the same error. I also tried setting > the SCR_CNTL_BASE environmental to point to /tmp and to > /home/username/somedir but got the same warning message. > > How can this issue be resolved ? > |
|
From: Arjun J R. <rec...@gm...> - 2013-11-28 07:53:57
|
I have a 2-node setup on which I intend to test out scalable checkpointing. I keep getting the error "Could not identify nodeset" when I issue the command scr_srun -N2 -n4 abc I found that the SCR_NODELIST environment variable needs to be set. What format must the nodeset be provided in ? I could not find it in the pdf manual or in the readmes. Setting SCR_NODELIST to abc[1-4] or to abc1,abc2,abc3,abc4 both of which silence the "could not identify nodeset" warnings. Which is the correct format to set the SCR_NODELIST variable to ? Also, after the "could not identify nodeset" warning was suppressed, I ran scr_srun -N2 -n4 abc to find this error ""ERROR: Invalid control directory INVALID" I checked the /etc/scr.conf file and it was set to /tmp. I tried changing it to /home/username/somedir but got the same error. I also tried setting the SCR_CNTL_BASE environmental to point to /tmp and to /home/username/somedir but got the same warning message. How can this issue be resolved ? |
|
From: Kathryn M. <ka...@ll...> - 2013-11-11 18:00:40
|
Hi Jorge, Is the environment variable SLURM_NODELIST set in your job environment? That's where SCR gets the 'nodeset' from. Also, the interpose library is not well supported anymore and may not be fully functional. We recommend using the test_api program in the examples directory for initial testing. Thanks, Kathryn On Nov 11, 2013, at 8:01 AM, Jorge Bellon <jb...@bs...> wrote: > Hi, > I am trying to check how SCR works in a test environment I've set up, which is > made up of two compute nodes and one controller running SLURM. > > First, I installed all the required software, by following the README and > INSTALL instructions. SLURM seems to work since I could queue some jobs and > they executed successfully. > > Then, after installing and reading the SCR documentation, I tried to launch > some examples to see how do the checkpoint perform. These examples are > included by default in "/usr/local/tools/scr-1.1/examples". I compiled all the > binaries through make command and then edited the scr_interpose.moab script in > order to fit my system architecture: > > #!/bin/bash > #MSUB -l partition=debug > #MSUB -l nodes=2 > #MSUB -l resfailpolicy=ignore > > # above, tell MOAB / SLURM to not kill job allocation upon a node failure > > # specify what the name of a checkpoint file looks like > export SCR_CHECKPOINT_PATTERN="rank_[0-9]+.ckpt" > > # specify where checkpoint directories should be written > export SCR_PREFIX=/home/jbellon/checkpoints > > # instruct SCR to flush to the file system every 20 checkpoints > export SCR_FLUSH=20 > > # exit if there is less than an hour remaining (3600 seconds) > export SCR_HALT_SECONDS=3600 > > # attempt to run the job up to 3 times > export SCR_RUNS=3 > > # run the job with scr_srun > /usr/local/tools/scr-1.1/bin/scr_srun -n16 -N2 ./test_interpose > > However, when I execute this script, the following error is shown: > root@frontend-0:/usr/local/tools/scr-1.1/examples# ./scr_interpose.moab > scr_srun: Started: Mon Nov 11 15:56:40 GMT 2013 > scr_srun: ERROR: Could not identify nodeset > > This is the output from 'sinfo' command: > PARTITION AVAIL TIMELIMIT NODES STATE NODELIST > debug* up infinite 30 down* compute-[3-32] > debug* up infinite 2 idle compute-[1-2] > > Is there anything I am missing? > > Thanks in advance, > Jorge > > > WARNING / LEGAL TEXT: This message is intended only for the use of the > individual or entity to which it is addressed and may contain > information which is privileged, confidential, proprietary, or exempt > from disclosure under applicable law. If you are not the intended > recipient or the person responsible for delivering the message to the > intended recipient, you are strictly prohibited from disclosing, > distributing, copying, or in any way using this message. If you have > received this communication in error, please notify the sender and > destroy and delete any copies you may have received. > > http://www.bsc.es/disclaimer > > ------------------------------------------------------------------------------ > November Webinars for C, C++, Fortran Developers > Accelerate application performance with scalable programming models. Explore > techniques for threading, error checking, porting, and tuning. Get the most > from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk > _______________________________________________ > Scalablecr-discuss mailing list > Sca...@li... > https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss ______________________________________________________________ Kathryn Mohror, ka...@ll..., http://people.llnl.gov/mohror1 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA |
|
From: Jorge B. <jb...@bs...> - 2013-11-11 16:21:18
|
Hi, I am trying to check how SCR works in a test environment I've set up, which is made up of two compute nodes and one controller running SLURM. First, I installed all the required software, by following the README and INSTALL instructions. SLURM seems to work since I could queue some jobs and they executed successfully. Then, after installing and reading the SCR documentation, I tried to launch some examples to see how do the checkpoint perform. These examples are included by default in "/usr/local/tools/scr-1.1/examples". I compiled all the binaries through make command and then edited the scr_interpose.moab script in order to fit my system architecture: #!/bin/bash #MSUB -l partition=debug #MSUB -l nodes=2 #MSUB -l resfailpolicy=ignore # above, tell MOAB / SLURM to not kill job allocation upon a node failure # specify what the name of a checkpoint file looks like export SCR_CHECKPOINT_PATTERN="rank_[0-9]+.ckpt" # specify where checkpoint directories should be written export SCR_PREFIX=/home/jbellon/checkpoints # instruct SCR to flush to the file system every 20 checkpoints export SCR_FLUSH=20 # exit if there is less than an hour remaining (3600 seconds) export SCR_HALT_SECONDS=3600 # attempt to run the job up to 3 times export SCR_RUNS=3 # run the job with scr_srun /usr/local/tools/scr-1.1/bin/scr_srun -n16 -N2 ./test_interpose However, when I execute this script, the following error is shown: root@frontend-0:/usr/local/tools/scr-1.1/examples# ./scr_interpose.moab scr_srun: Started: Mon Nov 11 15:56:40 GMT 2013 scr_srun: ERROR: Could not identify nodeset This is the output from 'sinfo' command: PARTITION AVAIL TIMELIMIT NODES STATE NODELIST debug* up infinite 30 down* compute-[3-32] debug* up infinite 2 idle compute-[1-2] Is there anything I am missing? Thanks in advance, Jorge WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer |
