At a client-site I was privy to an impressive IBM presentation on their
Websphere MQ Low Latency Messaging platform (LLM), with some pretty exorbitant claims: 8-million messages per second on gig-ethernet, and 21-million per second on infiniband(!).
I’ll be interested to see if it stands up to scrutiny.
One 2U server consolidates a view of 180,000 orders per second with 99th percentile latency of approximately 3 milliseconds
I spent some time with our CEP friends at SIFMA this week - lots of new stuff going on…
Aleri - their booth was BUZZING over the last few days, mainly due to the very cool (Lab49 built) user interface for the Market Liquidity Analysis (MLA) tool. I spoke to a lot of partners, reporters and client prospects about this, almost all of whom were really excited and impressed by the offering. Aleri also have a number of feature upgrades.
Apama - announced version 4.0 that vastly improves latency, and some new offerings including a deal with NYSE Euronext’s Advanced Trading Solutions group to provide their algo platform as a hosted offering. They also announced a port to Solaris 10, and a new “real-time pricing accelerator”. Just as important, Apama had an open bar at the conference - those beers were needed late in the afternoon..
Coral8 - also announced a packaged offering with Wombat (ie NYSE Euronext ATS), and also got a nice client announcement in the form of market surveillance for LiquidNet.
Oracle - spoke to the folks at Oracle about their new combined Oracle-BEA offering that incorporates the Oracle Continuous Query language (CQL), the BEA Weblogic Event Server java container, Coherence for Distributed Cache, plus some new graphical improvements on the IDE front (finally!) and High Availability (HA) story. Clearly, an enterprise / scalability play.
Streambase - announced their new 6.0 version with an improved development & testing environment that now includes the ability debug CEP code through breakpoints, and support for unit testing (we got a demo and it is very cool). This should really help those nasty synchronization timing problems we have seen with presenting market data and complex derived data simultaneously to the user. Streambase are also narrowing their industry focus down to FS and Federal verticals, hiring more domain experts and refreshing their executive team. Good news all around.
New "Aleri Live OLAP" announced. I see they claim Excel pivot table integration and MDX support. Also Aleri/Activ seem to be targeting the Wombat/Coral8 duo.
Built on the Apama platform, it’s called the Detica Market Surveillance Accelator. Designed as a sift, to survey all trades booked in a system for bad and/or illegal behavior, detect errors, and to aid in development of a more proactive support model.
In light of the Société Générale debacle, I expect to see more of these in development. Apama strikes me as a near-ideal match for this kind of application.
The fifth installment of the CEP podcast series has been posted here and on the Lab49 website. This is the third and final part of the interview I did with Daniel Chait of Lab49 and Brad Bailey of the Aite Group.
All of these CEP podcasts and other audio/video sessions we have recorded can be found on the Lab49 website at: http://www.lab49.com/insidethelab/audiovideo
The fourth installment of the CEP Podcast series has been posted here. This podcast is the second part of the interview I conducted with Brad Bailey of the Aite Group and Daniel Chait of Lab49. In this part we are continuing to discuss the challenges facing capital markets firms and how CEP provides an opportunity to manage these market challenges in new ways.
All of these CEP episodes, and other podcasts we have done can be found at http://www.lab49.com/insidethelab/audiovideo
From my article:
Over the past decade, data volumes have skyrocketed, new sources of liquidity have mushroomed, sell-side firms have created internal crossing networks, trading has become increasingly sophisticated and trading volumes in complex derivatives have spiked, collectively far outstripping the stock markets. For these reasons, the old, batch-based technology model no longer suffices.
To read the main article you must be a subscriber to Automated Trader. Or you can grab the PDF from Inside the Lab (PDF download)
I had a meeting this morning with Al Moore, one of the founders of Fixnetix, a provider of ultra low latency market data and connectivity. It immediately brought to mind a conversation I had with Tom Groenfeldt earlier this week about hidden latency. It continues to baffle me when financial institutions will spend millions shaving microseconds off of their data handling processes by optimizing their code and implementing CEP solutions, and then, after all is said and done, they’ll take this newly optimized codebase and hook it up to something like Reuters to receive their data, which itself has a latency that is milliseconds more than a low-latency data provider. Why not pocket that money, save those man hours and just switch data providers? Or better yet, do both?
It’s a classic case of not seeing the forest for the trees. Optimizing a system requires looking at the entire system - not just diving into a piece of it. You might very well shave more latency off of your architecture by changing data providers or removing that one extra switch from your network architecture than spending man-years optimizing your event processing software. Financial institutions need to remember to focus on the not-so-hidden latency before diving into a search for hidden latency.
CEP engines try to keep state-management out of the developers’ hands (or, if you prefer, out of your way), letting you concentrate on just the events. Inevitably, though, we end up needing just a little more state than we have in the current event stream; or we have an event that’s needed by several monitors (e.g., benchmark prices); or the event data’s needed in several monitors, but doesn’t actually trigger any action.
Enter the cache-monitor: a common Apama pattern we’ve used in our projects. It’s a simple pattern, but I’m posting it to illustrate Apama’s programming model which seems to be not common knowledge. What throws people for a loop at first is that in order to retrieve a value from a cache-monitor, we mimic a function call from typical procedural languages by routing a Request-event to that monitor, and listening for the matching Reply-event routed back to us by the cache. To see what I mean, consider the 3 snippets below. Read the rest of this entry »
The idea is to respond to incoming price events, but always process the latest event, skipping intermediate events. Here’s the way you can express that in Apama’s monitorscript language.
You can stage processing of an external event by caching an internal-version of the event, and enqueuing a ‘process-this-price’ event (putting it at the back of the queue). As more 3Y prices come in, they overwrite the cached event, and enqueue more ‘process-this-price’ events. When the ‘process-this-price’ events are reached, the latest price will be removed from the cache and the internal-version routed as the ‘real’ price event. Subsequent ‘process-this-price’ events result in no events because the 3Y was already removed from the cache. Here’s the code: Read the rest of this entry »
Over at Aleri’s website, I contributed a piece helping customers sort out some of the decisions around CEP platforms:
When looking at the field of products and determining which to choose, I have seen many of my customers become confused and bewildered about how to even evaluate the various offerings. Meanwhile, the primary factor most often touted by vendors, journalists, and industry analysts is messages per second (MPS). However, MPS is not the right aspect to focus on. There are three basic problems with using MPS to drive purchasing decisions: first, most of the numbers touted in the press and in marketing literature are largely meaningless since they do not account for realistic or consistent use cases; second, at this point almost all of the major CEP products perform at adequate rates for all but the most demanding applications; and third, there are other, more important issues at play.
Read the full article here.
Having a little extra time on my hands, and maybe a bit too much egg nog, I set out to write a basic 3D rendering system that can run under WindowsCE (or “Pocket PC” or “Windows Mobile” depending on how long you’ve had your WindowsCE device). Attached is an ARM executable for your PPC that displays an animated tessellated Bézier Patch (optionally showing surface normals and mesh outlines).
mgraphics.exe (25.5 KB)
The code is also available on our public svn service.
The video I made when I visited the Microsoft Platform Adoption Center in Redmond a few weeks ago is now online. In the video I show off a demo done for a BEA conference by a team at Lab 49 that uses WPF and complex event processing (CEP) extensively. I wasn’t directly involved in this project, but I wish I was! Microsoft was very impressed with the demo.
Here at Lab49 the founders take pride in the fact that as a company we are very easy to work with. Being easy to work with is an important dimension for measuring potential technology vendors that is often overlooked. Recently I have been working with quite a few vendor products and one particular experience has reminded me of the importance of of this critical dimension.
In prior posts [1, 2] I wrote about my initial experiences with StreamBase’s complex event processing model. Here I will dive a bit deeper and talk about the actual computational tools StreamBase provides. Here I will concentrate on the perspective offered by their graphical environment. There is a second perspective offered by their StreamSQL language, which does not map exactly one to one with the graphical model (in that distinct graphical elements often combine into single StreamSQL statements and visa versa). I’ll blog about StreamSQL in a future post.
At the end you will find some use cases that bring things together based on my experience building a matching engine for an ECN.
This is a follow-up to my previous post about Streambase.
In order to test the capabilities of Streambase I decided to create the basic functionality of a matching engine for an ECN. I picked this application for two reasons. First it is an area that I have quite some experience with (in my past life at the NYSE) and second because it is somewhat atypical for a CEP application but not too far afield.
I just completed two days of training on Streambase v 3.7, one of the leading Complex Event Processing (CEP) frameworks. What follows are my initial impressions. I plan to try to develop a non-trivial application in the next few days and will repost based on that experience.
