Hi Paul,
On Thu, May 22, 2008 at 12:17 PM, Paul Millar <p.millar@...>
wrote:
> On Tuesday 13 May 2008 12:11:09 gorva wrote:
> Yes, low efficiency could be due to poor programming but it could also be
> caused by external factors: bad planning by whoever submitted the job or
> plain "bad luck".
>
> [...]
Thanks for the explanation :) That's pretty much what I had understood.
> Now, I'd like to know how is the efficiency of jobs calculated in MonAMI.
> I
> > haven't checked the code in depth but I think it's using PBS API.
>
> Yes, partly. The code obtains the wallclock- and CPU- times from torque
> and
> simply takes the ratio of the two.
>
>
> > The reason I'm asking is because I've noticed either a wrong reported
> > value, or I've missed something.
> >
> > Let me explain : for some reason (memory issues that are not important
> > here), we've got a user ("turboult") that has submitted jobs using 2
> nodes
> > (each node has 2 CPUs), but the jobs only use 1 CPU per node.
> > Assuming the job uses 100% of that single CPU, the efficiency should be
> 50%
> > (because there is still one CPU on each node that is not used at all).
>
> Could you send the output from "qstat -f" for one of these job?
>
> As an aside, it sounds like the user's code is running as a single-thread.
>
> This isn't so bad if the job runs on a number of processors requested, but
> if
> the user requests (or thinks they are requesting) 2 processors, but each
> job
> instance is given two computer-nodes (each of which has two CPUs/cores),
> then
> something or someone is confused.
>
> Perhaps the user submitted the job incorrectly, or is confused how the
> underlying batch system is configured, or the job's code is not
> distributing
> the load across multiple cores when expected (due to submitting the wrong
> executable, broken configuration, etc), or ...
In fact, it's easier than that. The user needs only one CPU, and like 1,5Gb
of RAM. Our single CPU's nodes have only 1Gb of RAM.
So, to get this RAM requirement, the user submits his job on a parallel
queue (with more than 1,5Gb of RAM), even if he doesn't need the 2nd CPU. He
actually knows he's wasting one CPU but he couldn't do another way (or at
least that's what he told us)
> MonAMI reports an efficiency between 80 and 100%
>
> Looking at the MonAMI torque plugin code, there's definitely a bug in the
> efficiency calculations.
>
> The MonAMI plugin assumes a job only uses a single processor: if CPU-time
> exceeds wall-clock time, the plugin treats the efficiency as 100% (so the
> job
> goes in the 80%--100% efficiency bin).
>
> This needs fixing.
>
> > When I use the following command on the server where Torque/Maui are
> > installed, I got an efficiency of 50% :
> >
> > # showq -u turboult -r
> [...]
> > So what is wrong ? I think showq is a Maui command
>
> Yup, showq is a Maui command, I believe the (roughly) equivalent torque
> command is qstat.
>
> So, what would be interesting is the output from "qstat -f" specifically
> for
> one of these jobs.
Ok here it is (I've just changed some outputs as a privacy concern, but it
won't change the result) :
qstat -f 173688.serverhostname
Job Id: 173688.serverhostname
Job_Name = jobname
Job_Owner = turboult@...
resources_used.cput = 02:35:14
resources_used.mem = 932224kb
resources_used.vmem = 945668kb
resources_used.walltime = 02:35:28
job_state = R
queue = parall_3j_4
server = serverhostname
Checkpoint = u
ctime = Fri May 23 19:50:31 2008
Error_Path = serverhostname:/output/path.e173688
exec_host = n019/1+n019/0
Hold_Types = n
Join_Path = n
Keep_Files = n
Mail_Points = a
mtime = Mon May 26 11:41:53 2008
Output_Path = serverhostname:/output/path.o173688
Priority = 0
qtime = Fri May 23 19:50:31 2008
Rerunable = True
Resource_List.cput = 72:00:00
Resource_List.neednodes = 1:ppn=2
Resource_List.nodect = 1
Resource_List.nodes = 1:ppn=2
Resource_List.walltime = 12:00:00
session_id = 344
substate = 42
Variable_List = PBS_O_HOME=/home/turboult,PBS_O_LOGNAME=turboult,
PBS_O_PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games,
PBS_O_MAIL=/var/mail/turboult,PBS_O_SHELL=/bin/ksh,
PBS_O_HOST=serverhostname,
PBS_O_WORKDIR=/work/dir,var=13,
PBS_O_QUEUE=route
euser = turboult
egroup = r33
hashname = 173688.cli7
queue_rank = 26159
queue_type = E
etime = Fri May 23 19:50:31 2008
x = FLAGS:turboult.0
HTH :)
> > would that mean PBS and Maui don't use the same method to calculate the
> > efficiency ?
>
> No, torque doesn't provide an efficiency metric, maui must be calculating
> this.
>
> > Is the PBS API wrong ? or is that an error from MonAMI ?
>
> MonAMI is wrong: it assumes that a job-slot has access to a single CPU.
> Instead, it should look at how many processors the job has been allocated
> and
> divide by that number (it just needs fixing ;-)
Hope you can find a way of calculating correctly the efficiency without
relying on Maui.
Anyway, thank you for investigating the problem :)
Cheers,
gorva
|