scalablecr-discuss — Discussion of SCR topics

Re: [Scalablecr-discuss] Getting SCR running

[Kathryn M. <ka...@ll...>]

Hi,

> I am running SLURM and all the other stuff needed to run SCR on a single computer, as a way to evaluate SCR. I have a simple C counting program that I wish to run and take periodic checkpoints of. My executable name is "count" How can I get SCR to take periodic checkpoints ? I have specified two separate directories for the two kinds of checkpoint images
> 
> When I installed SCR, I don't have any command such as scr_srun on my system. 
> When I type in scr_srun, I get the error "Command not found"

Possibly you did these steps already, but just as a sanity check:
Did you do a 'make install'?
Did you set your path to point to the installation directory for SCR? 
setenv PATH /usr/local/tools/scr-1.1/bin:${PATH}

> 
> The executables are stored in /usr/local/tools/scr-1.1/bin/ and when I execute just scr_srun using ./scr_srun I get
> scr_srun: Started: Mon Sep 30 12:58:22 IST 2013
> scr_srun: ERROR: Could not identify node set

scr_srun assumes you are executing in an allocation given to you by SLURM and that the environment variable SLURM_NODELIST is set. You can get an interactive partition with salloc, e.g.
salloc -N 1 -p <queuename>

If I recall correctly, you may have problems running on a single node. SCR will want to find locations for storing redundant checkpoints that are not on the same node. That way if a node goes down, the checkpoints are protected on the other node. I think that it will exit with an error message if you run on a single node.

Hope that helps!
Kathryn


> 
> Nandaka Jojha
> 

______________________________________________________________
Kathryn Mohror, ka...@ll..., http://people.llnl.gov/mohror1
CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA

[Scalablecr-discuss] Getting SCR running

[Arjun J R. <rec...@gm...>]

I am running SLURM and all the other stuff needed to run SCR on a single
computer, as a way to evaluate SCR. I have a simple C counting program that
I wish to run and take periodic checkpoints of. My executable name is
"count" How can I get SCR to take periodic checkpoints ? I have specified
two separate directories for the two kinds of checkpoint images

When I installed SCR, I don't have any command such as scr_srun on my
system.
When I type in scr_srun, I get the error "Command not found"

The executables are stored in /usr/local/tools/scr-1.1/bin/ and when I
execute just scr_srun using ./scr_srun I get
scr_srun: Started: Mon Sep 30 12:58:22 IST 2013
scr_srun: ERROR: Could not identify nodeset



Nandaka Jojha

Re: [Scalablecr-discuss] Migration and restart from checkpointed images in the PFS

[Kathryn M. <ka...@ll...>]

Hi Arjun,
> If I have an older checkpoint flushed out to the parallel file system 
> and i suffer a catastrophic failure of nodes that violates the 
> redundancy scheme, is it possible in SCR from me to send the checkpoint 
> images from the parallel file system to one or more spare nodes to 
> restart the entire application from the previous checkpoint ?

Yes, SCR can do this. It will first attempt to restart the job with spare nodes using cached checkpoints. If it fails to be able to rebuild the cached checkpoints, it will restart using the most recent checkpoint that was pushed out to the parallel file system.

Hope that helps,
Kathryn

> 
> Arjun J Rao
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk_______________________________________________
> Scalablecr-discuss mailing list
> Sca...@li...
> https://lists.sourceforge.net/lists/listinfo/scalablecr-discuss

______________________________________________________________
Kathryn Mohror, ka...@ll..., http://people.llnl.gov/mohror1
CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA

[Scalablecr-discuss] Migration and restart from checkpointed images in the PFS

[Arjun J R. <rec...@gm...>]

If I have an older checkpoint flushed out to the parallel file system
and i suffer a catastrophic failure of nodes that violates the
redundancy scheme, is it possible in SCR from me to send the checkpoint
images from the parallel file system to one or more spare nodes to
restart the entire application from the previous checkpoint ?

Arjun J Rao