HPC

I’ve been test-flying a few datagrid/datafabric products lately and had a nice opportunity to try them out on gig-ethernet and infiniband. The thing I noticed with all distributed-data systems is that synchronous-replication is always a bottleneck, because you’re forced to wait on the replication-ack before proceeding with the next operation. Because of this, gigabit-ethernet’s latency puts a ~5k ceiling on any throughput rates you can achieve.

For any grid vendor it’s crucial to rollout modern, ultra-low latency interconnect support then, as its latency characteristics blow away any Gig-E numbers. For real-time price publishers, for algo-trading, latency is the key issue, more than throughput.

Rick Molloy just mentioned to me that he’s created a new blog on MSDN to coincide with Stephen’s talk at TechEd today about the Concurrency Runtime for Visual C++.

Parallel Programming in Native Code

Welcome to the Parallel Programming in Native Code blog.  I started this blog so that I and others on my team would have a place to talk about topics relating to native concurrency.  I want to use this blog to provide early looks into what we’re thinking about, give announcements about any publicly available content or CTPs and of course respond to feedback that we receive from readers and customers...

 

This is how Greg Wilson, Department of Computer Science at the University of Toronto, begins his presentation.

I do like that he points out

Software engineering for science has to address three fundamental issues […]

It’s a shame “getting the right answer” and “being productive” don’t make the list.

The Message Passing Interface (MPI) standard, and its .NET implementation, MPI.NET have been some of the cornerstones of development on compute clusters.  The standard supplies a simple yet primitive way of both sending and receiving data between running compute processes.

The large advantage of MPI has been a mix of its simplicity and speed.  A call to MPI Send on one node and MPI Receive on another block both callers until the operation is complete.  Some more complex calls, such as MPI Scatter and MPI Gather allow a single node to distribute data to a set of nodes or retrieve it from a set of nodes.  An MPI Barrier allows all nodes to stop until they have all reached the agreed upon place in code, then allowing them to continue.  Such primitives allow a distributed set of processes to communicate, do some work, and then share values that each needs to continue with eachother.  Because this is all done with some low level, bare metal socket tricks and/or shared memory, the result is blazingly fast communication.

With this simplicity however, comes a trade off.  MPI has been a standard for nearly 20 years and has changed very little since its inception.  The way we program today has changed drastically, especially with managed languages such as C#.  No longer do we tend to worry about memory allocation, or dealing with raw memory.  Today, most languages have a concept of automatic memory allocations, garbage collection and type safety.  Although the primitives in MPI are unparalleled in simplicity for allowing multiple processors to communicate about a shared set of work, some striking limitations are found once we dig a bit deeper.

When I started with MPI.NET, I found the interface very simple.  An Mpi.Send<T>(obj) would send an object to a waiting client.  Mpi.Receive<T>() would give you back that object.  Nothing could be simpler.  In my example, however T happened to be a class that contained a byte[] of an undetermined size.  Once run on the cluster, the size of the byte[] I was passing increased dramatically, and an unexpected exception occured:

AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory has been corrupted.

After lengthy investigation, I found that MPI.NET was attempting to pin some memory in the .NET GC heap and pass that memory location as a buffer to the underlying MSMPI stack.  In doing so, it did not allocate enough memory for my large byte[], causing the write to try to write into the GC heap, thus throwing the exception.  In my case, to solve this, I created a large enough buffer and passed it into an override of Mpi.Receive<byte>(byte[]).  This overload pins the entire array that was passed in on the GC, and then passes that to the MSMPI stack.  On the send side, I manually serialized my class, checked the length (to ensure I would not overflow the receive buffer) and sent the byte[] instead of the raw class.  This solution does not take into accound messages larger than my expected buffer.  For that, I would have needed to chunk down the data.

The moral of the story here is sending primitives, arrays of primitives or fixed-sized structs over MPI.NET (which is the most common scenario) is a great use of a very fast messaging interface.  Once your demands get more complex, the MPI stack gets less favorable, not because of its inability to send more complex messages, but because of the manual labor involved in serializing and chunking down data.

It is no wonder that the HPC community is moving away from the traditional methods of MPI and communication across a set of processors to a Service Oriented (SOA) model. The benefits of using existing components, such as WCF and its NetTcpBinding, the threading models, serialization and transport models, and other features already provided by these frameworks outweights the possible performance penalty.  Problems such as the one explained above simply do not happen with frameworks like WCF.  Furthermore, although the underlying concepts of MPI and its simple messaging model are very simple and appealing, the overall development, maintainence and debugging of a SOA application is much simpler than that of a MPI application.  The amount of code complexity and custom code drops when compared to an MPI implemenation.

The general industry trend seems to be towards SOA models.  Microsoft Windows HPC Server 2008 is a great example of this.  HPC Server uses WCF to distribute load across the cluster, and can even dynamically scale resources depending on demand of a particular service.  Platform, another industry competior has been building with a SOA model for some time now. 

I'm looking forward to playing with HPC Server 2008 and WCF more as time progresses.  I think that the WCF model will solve a whole bunch of headaches that one incurs when trying to communicate over perhaps over simple primitives such as MPI.  Many models and workloads simply do not require the type of communication MPI provides, and using MPI can be like fitting a square peg into a round hole.  This is not to say MPI does not have its place, many complex processes do require constant communication between a set of workers, however I believe many of the problems we use HPC for today can distributed using SOA in a much simpler fashion.

I ran across the following comments on an El Reg article about virtualization: Comments about “Virtualization: Nothing New”.

He’s right, isn’t he? Having the same company offer both virtualization and grid solutions is a truly virtuous pairing. First you tell people they need a grid solution to make a huge pool of computers look like a single one, then after it’s all set up you sell them virtualization software. It’s beautiful. It’s like if, for example, Nestle and Jenny Craig got together to simultaneously offer chocolate products and dieting solutions. Oh wait, they did: Nestle to buy Jenny Craig.
(more…)

Since EC2 appears to be the topic of the week, its worth looking at Scalr. Scalr provides the “elastic” to enable EC2 to provide a dynamic cloud. Google (BigTable) and Microsoft (SQL Server Data Services) both appear to be chasing Amazon (S3), but neither appear to be close to EC2 - curious given Microsoft’s […]