CEP/ESP

The explicit model works fine for my current POC – well as fine as anything can be for CTP2. However, I’ve been advised by MS StreamInsight that I might benefit by using the IObservable/IObserver development model for the problem I am trying to solve. Reading the documentation I thought it would be an easy change in my POC code base. Starting with the following code which works and produces data via an output adapter:

CepStream mds = CepStream.CreateInputStream(”mdi”);
…..
join d in mds
….

I though I should be able to do something like this:

MS StreamInsight kindly pointed me at a way to improve performance of my LINQ – push where clauses into join predicates. This worked fine for a simple where clause, but failed for multiple predicates i.e.

var equiJoin = from e1 in stream1
join e2 in stream2
where e1.i equals e2.i
select new { e1.i, e1.j, e2.j };

should really be

var equiJoin = from e1 in stream1
join e2 in stream2
on e1.i equals e2.i
select new { e1.i, e1.j, e2.j };

But the following

As a rumble down the bumpy StreamInsight road, I have the basics of a Pricing Engine running in CTP2. My current problem is that I need to use a market feed input stream in multiple LINQs coupled with the output of one LINQ being the output to an adapter as well as the input to another LINQ. Now the LINQ I came up with so far doesn’t work – no surprises here Thankfully the MS StreamInsight Team have offered a few thoughts which may help to solve my problem – or at least get me further down the road.

I finally (thanks to help from a colleague) got some LINQ running inside StreamInsight to generate Fly orders. I’ve attached an image from the Event Flow Debugger. Few features I’d really like in V1 StreamInsight:

• Ability to return event objects from User-Define Functions (UDFs)
• Integration of the Event Flow Debugger into Visual Studio

For the last 3 days I’ve been in CEP hell with messages like:

• Nested members not supported on EventTypes
• Microsoft.ComplexEventProcessing.Linq.QueryGenerateException: The Linq query is not supported by CEP engine, error message:h__TransparentIdentifier3.orderRequest.

For the last few weeks I’ve been attempting to write a strategy (bund/bobl/schatz fly) engine in StreamInsight. Adapters are the first item that needs to be decided. In the case of market data, probably the safest option is to TypedPointInputAdapter. The decision on the strategy adapter is a little harder as its probably not going to change that much, but you can assume that you’d want to trade many different variants of a strategy during the day. So I guess its TypedPointInputAdapter? However reading the MSDN docs on the three flavours of adapter, Strategy may come into interval model (a strategy is in play for a certain period during the trading day) or edge (a strategy is created and left open on the market).

I figured out what happened to StreamInsight CTP1. Looking back at Torsten Grabs TechEd 2009 presentation “Low-Latency Event Processing with Microsoft’s Platform for CEP”, there never was a TAP CTP1. CTP1 must have been a very private release.

A new twist on machine-readable feeds. Has StreamBase found a new opportunity to weld social networks and trading? Something for Microsoft to consider with Orinoco.

A few years ago we had CEP products coming from more vendors than sense, now it appears we have cloud’s coming from all directions. The latest appears to be Silver. Tim over at Cyberstrategics has compared Silver to EC2. However I suspect a better comparison is Silver vs Azure – especially as TIBCO Silver is available as a pre-built, pre-configured AMI (Amazon Machine Image) for the Amazon EC2 platform.

YouTube video available here – shame they don’t provide a better view of Silver in this video .Nice to see they have a blog – hopefully they will put something worthwhile on it soon.

Minimum requirements for Microsoft CEP if they want to enter the financial CEP space. One thing Microsoft has hopefully learn from their push of WPF/Silverlight in finance is that domain specific POC’s need to be used, and not NY Times Reader demos

• Integration with Velocity and Azure would be nice
• A decent cluster/HPC story would be useful
• Financial adapters such as Tibco RV, MQ, Kaazing, Lightstreamer as essential
• POC’s in the basket strategy, auto quoting space would help to sell the application into investment banks
• Good VS debugging and profiling support

Looks like Microsoft is going to release a CEP in 2010. This press release gives a little bit of information – build on top of SQL Server 2008. Maybe this is another Microsoft Research powered applications?

Having avoided reading any email/blogs for the last few days whilst on vacation I thought I’d spend a little time tonight while sitting in front of the fire and away from the snow (Norway) and catch up on the 1000’s of emails Google Reader postings. What follows is a list of some of the things I’ve read, together with comments were appropriate:

• Velocity reaches CTP3 – I’m glad Microsoft has got cache notification into Velocity. Maybe they should consider adding Velocity to Azure – Microsoft Data Fabric?
• Above the Clouds – Subscribed. Presentation available here

Sybase appear to be a little late to the CEP party.

It comes as no surprise to find two CEP vendors merging. I blogged a while ago that the space is to small for so many vendors. Will we see a merge of CCL and Spash? The sooner the two companies merge their CEP engines the better, as supporting two will cost Aleri from a development perspective, plus is leads to customer confusion. It’s going to be interesting to see how quickly Aleri manage this problem, and how quickly they resolve it.

The Azure SDK comes with a number of samples including the default Hello and the more interesting Distributed Sort sample.

The Distributed Sort is interesting since it provides an easy to follow example of an ASP.NET front end connecting to the work item (Azure service) via a queue (persistent messaging between services). AJAX is used to provide the “real-time” update on the web page. The sample itself is configured to use only one worker role instance, but changing the .cscfg file allows more workers to be created – when I run on my local Development Fabric I’m only running across my two CPU cores, how do the Worker’s get distributed in the Azure Services Developer Portal?

One 2U server consolidates a view of 180,000 orders per second with 99th percentile latency of approximately 3 milliseconds

Video, white paper and posting.

I added a video of the Aleri MLA tool on Lab49’s website here.