Re: [SSI-users] ssi on podcast

[Vincent D. <di...@xs...>]

On Aug 26, 2012, at 9:35 PM, Brock Palen wrote:

> On Aug 26, 2012, at 3:16 PM, Vincent Diepeveen wrote:
>
>> OpenSSI doesn't work at HPC machines as it doesn't support  
>> Infiniband nor other HPC type connections.
>
> You are making the assumption that HPC only means massive parallel  
> MPI jobs or large OpenMP jobs.  On the other hand I know of a group  
> on campus that just needs to run many serial jobs and uses Mosix as  
> their platform on a few hundred cores.  What is the difference  
> between 1000 core parallel job, and 1000 serial jobs?

HPC = High-performance computing

OpenSSI is combining a number of machines together to 1 machine with  
shared memory.

If you run things serial or embarrassingly parallel there is no need  
for SSI.
All you need then is for example the free parallel shell to start at  
all machines
  at the same time the jobs. A good example of such shell is the pdsh  
shell. So there is no need for a SSI there.

As for Mosix, that ceased business in early 2008, so you must be  
speaking about something total obsolete.

What's the biggest advantage of a SSI over using MPI?

That's the fact that when using memory from a remote node that your  
code doesn't look different from
how it runs on a shared memory machine.

The reason to use HPC networks is because of latency and bandwidth.  
Typically your built in ethernet card
is not DMA, so every message received gives a big dang to the CPU.  
Its latency at best is 100-200 microseconds,
whereas the HPC cards are in the order of a few microseconds. Latest  
infiniband (FDR) is 0.85 microseconds.

Sure there is ethernet cards that are better than 100-200  
microseconds. Note they're usually more expensive than
a HPC card and still factor 3 worse in latency.

Fastest network card at 10 gigabit ethernet is solarflare right now.  
It's latency is around a 3 microseconds.
Its price double that of infiniband cards. In fact its latency is the  
same like HPC cards from 10 years ago.

TCP simply is not a safe protocol from several viewpoint. It's a slow  
protocol.

Infiniband gets used a lot for TCP as well, yet then it's 10 gigabit.  
A cheap QDR card is 2 x 40 gigabit using HPC protocols.
FDR with which supercomputers get equipped now is even faster.

If you do High Performance Computing, you typically see that at large  
clusters quite some software is not really professional optimized,
usually the experts who know what they want to achieve are not the  
best guys to also optimize the software.

Yet already in advance throwing away an advantage of factor 8+ in  
bandwidth is not a good idea as that removes the possibility
of optimizing the software very well as there is also a bunch of  
persons in HPC which does write the ultimate speed monsters.

HPC typically is the area where also the most optimized codes are. A  
SSI should not be in the way of that by just enforcing TCP.
TCP simply is not a suited protocol for HPC.

Typically you see then that latency plays a crucial role in some  
codes. A good example is my chessprogram.

For it a SSI is really interesting to have, yet it requires memory  
migration of shared memory.
Then i no longer need to program MPI commands myself and just leave  
it to the SSI.

Yet latency is total crucial then. Doing that over TCP is throwing  
away too much of a performance.

>
> Our local HPC resource that I admin is 12000 cores and only about  
> half of it has IB, the rest is Ethernet.
>

>>
>> AFAIK it just supports TCP.
>> Furthermore it's total outdated and not maintained.
>
> This would be a problem and thus makes it non-useful for new  
> adopters of the software?  We can pass on OpenSSI if it is dead.
>

There was first Mosix and OpenSSI, also SGI had its own SSI type  
software using Irix.

Irix is no longer, Mosix ceased business and OpenSSI no longer gets  
maintained.

One of the programs that really can use a good SSI is a chessprogram.  
Yet latency is total crucial to it.
It simply needs all features that the hardware also has, which  
includes migrating pages of memory, regardless whether that's
shared memory or normal memory.

Also you don't want to migrate the entire shared memory segment, you  
just want to migrate a page. Say 2 kilobyte at most or so.

So in itself a SSI is useful if it is delivering real performance for  
a limited number of cores.

>>
>> As for true SSI it also lacks crucial features like memory  
>> migration of shared memory.
>> Know another way to run parallel using multiple processes?
>>
>> Everyone uses shared memory for that of course.
>
> We did talk to the creator of ScaleMP, not open but this is what  
> our listeners asked to compare OpenSSi to: http://www.rce-cast.com/ 
> Podcast/rce-65-vsmp-scalemp.html
>
>>
>> So what do you mean with 'aimed' at HPC?
>
> Our listener base is probably mostly admins and some users and  
> developers mixed in there. We like informing them of anything that  
> could help them in their job.

They have grey hair and had a cluster 10 years ago that has a SSI?

As past 6+ years not much happened to OpenSSI nor Mosix, in fact  
mosix stopped business in 2008.

The last supercomputer that ran IRIX that got dismantled i'm not  
sure. Maybe around 2006 somewhere?

Of course a problem of SSI's is that it doesn't scale real well or at  
a far too high price.

Let me give you an example of SGI under Irix. Here in Netherlands at  
SARA one of the largest supercomputers at the time was
installed. Called Teras. It had 1024 processors in the year 2000.  
Split up in different SSI type partitions.

512 of them were in a single partition.

Now the first problem then is that how they solved this was by losing  
some processors.
So effectively the max you could use typically was 500. As a few were  
used for i/o and others for other centralized aspects.

This is a serious problem a SSI brings.

If i wanted to time the 500 processes it had to call the clock of  
course. I use GetTimeOfDay, a normal unix function to do it.
However the disadvantage of migration is that one never knows where a  
proces or its RAM are located, so time then gets
maintained in a centralized manner.

This was a major problem. The program slowed down exponential already  
at around a 100+ processors when getting for each
search the time. That caused the program to exponential slow down of  
course.

So SSI is very useful but there is a limit in its possibilities. Up  
to a core or 64 it's nice. Yet most will not want to pay big bucks for
this.

Programs that typically are interesting to scale to a core or 64,  
they usually need fast latencies and the software to do that
may not be expensive, let alone not be real expensive sorts of hardware.

Note majority of such software already cannot
work at different machines as such SMP coding is not simple for even  
the fastest latencies that the interconnects can deliver.

For example most chessprograms ugh out here at a tad older box here a  
16 core AMD opteron. It has 300 nanoseconds latencies.
That still is 3x faster than the fastest latency of the fastest  
network card.

We speak of latency here that is similar to 2x the 'one way ping  
pong' latency, or the RDMA latency that cards manage to deliver.

OpenSSI simply never delivered there.

>
>>
>> That's a different league you know...
>>
>> On Aug 26, 2012, at 8:49 PM, Brock Palen wrote:
>>
>>> We host a podcast aimed at the HPC community at rce-cast.com
>>>
>>> We had some listeners request that we cover OpenSSI on the show.
>>> Is this something a developer or two of OpenSSi would be  
>>> interested in doing?  If so contact me off list and we can set  
>>> this up.  It takes about an hour over phone or VoiP.
>>>
>>> If you have any suggestion for show topics please let me know!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> br...@um...
>>> (734)936-1985
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> ----------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond.  
>>> Discussions
>>> will include endpoint security, mobile security and the latest in  
>>> malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Ssic-linux-users mailing list
>>> Ssi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ssic-linux-users
>>
>