Re: [SSI] Scalability ?

[Bruce W. <br...@ka...>]

Greg,
  Sorry for the late reply.  I have been on "vacation" (finishing the rebuild of my house and
moving).  Should be more responsive now.

> Have you guys set a target for how many nodes you plan to be able to have in the cluster.

In the short term (this year), I don't expect to work well beyond say 40 nodes.   If we have
the interest in the scientific community (which there appears to be some interest), we will aim
for several hundred next year.

> 
> I have a single threaded brute force app that is taking 7 hours to process its nightly run.
> 
> I don't yet know the internal structure of the software, but it is currently cpu bound, and is not very disk intensive.
> 
> I've been told to come up with a way to make it handle 50x the data and to make it robust enough to handle a node failure so that the processed data is available first thing every morning.
> 
> My target "production" date is one year out but I only have to be able to support 10x on day one. (i.e. 4 quad-cpu nodes should work)
> 
> I'm thinking the SSI Linux might be a good base technology to use, but even with quad cpu machines I'll eventually need about 10 nodes and I would like to have a lot of headroom in case the data grows more than expected.
> 
> I plan to re-write it so it can use both multiple cpus on one node, and multiple nodes in parallel.
> 
> A related question is if a single multi-threaded app can have threads on different nodes?

Currently the answer is no, because of the the shared memory space and shared open file table.  Others have asked
about supporting cross-node threads so it might get done (if the shared data space is thrashed between the
threads, then spreading them out is a bad performace idea, which is why we never did in NonStop Clusters).
However, give CFS and it's cache coherency capability, we intend to support standard SystemV shared memory
and then could possibly extend that to shared data space.

bruce
> 
> Greg Freemyer
> Internet Engineer
> Deployment and Integration Specialist
> Compaq ASE - Tru64
> Compaq Master ASE - SAN Architect
> The Norcross Group
> www.NorcrossGroup.com
> 
> 
> _______________________________________________________________
> 
> Don't miss the 2002 Sprint PCS Application Developer's Conference
> August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm
> 
> _______________________________________________
> ssic-linux-devel mailing list
> ssi...@li...
> https://lists.sourceforge.net/lists/listinfo/ssic-linux-devel