Re: [Libmesh-devel] Ghosted vector bug fixed

[Tim K. <tim...@ce...>]

Dear Jed,

On Fri, 5 Jun 2009, Jed Brown wrote:

>>> Also, memory performance
>>> (especially bandwidth) is usually the overwhelming issue for implicit
>>> solvers (e.g. you are very lucky to get 4% of peak FPU performance for
>>> MatVec on Core 2 Quad).  Thus using more cores frequently does not help,
>>> and you need more sockets to improve performance.
>>
>> I must admit that I'm not familiar with these hardware issues at all, so
>> your explanations are all Greek to me.  Well, since you were talking
>> about sockets, I tried this (on one of the nodes):
>>
>> tkroeger@node033:~> cat /proc/net/sockstat
>> sockets: used 239
>> TCP: inuse 31 orphan 0 tw 21 alloc 39 mem 2
>> UDP: inuse 19
>> RAW: inuse 0
>> FRAG: inuse 0 memory 0
>>
> Heh, I was referring to hardware sockets, not network sockets.

Well, I told you I have no idea what I'm talking about.  (-:

> So most CPUs these days are multicore, but the cores share the same
> memory bus (at least partially).  When you say 8 CPUs on 3 nodes or 24
> CPUs on 4 nodes, I assume you mean 8 cores on 3 nodes, 24 cores on 4
> nodes.  This is a different number of cores per node, but the
> configuration of these nodes is relevant.  For instance, 4 dual-core is
> almost always faster than 2 quad-core (although the latter is probably
> cheaper and uses less power).

/proc/cpuinfo is attached.  Seems as if I've got 2 quad-core CPUs on 
each node, right?

> Knowing your hardware (/proc/cpuinfo and network) and MPI configuration
> (which MPI and which device it is using) is useful.

Hmm, how do I find out about the network hardware?

There is a pre-installed MPI version, that is:

ParaStation 5.0.9-1 (Wed Nov 12 18:27:53 CET 2008)
  config args: '--prefix=/opt/parastation/mpi2' '--with-confset=default'
  mpich2 version: 1.0.7

But for some cumbersome reasons I can't use it.  Well, perhaps I just 
haven't tried hard enough.  I installed another MPI myself, and that 
is mpich2 1.0.8, without any options (other than the installation 
path).  I'll send you its config.log off-list (180KB).  Well, now that 
I think about it, I wonder whether I could simply supply the above 
stated config arguments manually to my local MPI installation.  Would 
that be a promising idea?

Also, it might be useful to note that the cluster has two different 
queues, one of which is called "ethernet" and the other "infiniband". 
They use different nodes.  These names seem to indicate some 
information about the hardware (let's assume they have been chosen on 
purpose).  I understand that "infiniband" refers to the "better" 
hardware in some sense, but the "infiniband" queue has a much smaller 
number of nodes.  Also, the admin told me that using my manually 
installed MPI, I would not be able to benefit from the advantages that 
the "inifiniband" queue has.  Hence, I was using the "ethernet" queue 
all the time.  /proc/cpuinfo does not differ between the nodes of the 
two queues (except for changes in some minor decimal places of the 
values of "cpu MHz" and "bogomips").

Anyway, I'll send you my log output on Monday.

Best Regards,

Tim

-- 
Dr. Tim Kroeger
tim...@me...            Phone +49-421-218-7710
tim...@ce...            Fax   +49-421-218-4236

Fraunhofer MEVIS, Institute for Medical Image Computing
Universitaetsallee 29, 28359 Bremen, Germany