Distributed Systems

August 27th, 2014 / Tales from a Trading Desk

Few interesting articles recently on distributed systems:

  • The Space Between Theory and Practice in Distributed Systems
  • Distributed systems theory for the distributed systems engineer

Of particular interesting:

How you decide whether an event happened before another event in the absence of any shared clock. This means Lamport clocks and their generalisation to Vector clocks, but also see the Dynamo paper.

And then there is the coolness of state machines:

Distributed state machine replication (Wikipedia is ok, Lampson’s paper is canonical but dry).

 

Legacy Data

August 27th, 2014 / Tales from a Trading Desk

Dave McCrory has an interesting presentation on InfoQ which discusses enterprise platform as a service.  Slide 23 onwards discusses legacy data, and the issues associated with data consumption.  Master Data Management is also discussed in the deck.  Which leads to a New York Times article, “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights”.

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

 

 

Capital Market Big Data Use Cases

August 26th, 2014 / Tales from a Trading Desk

Couple of interesting use cases in this InfoQ video:

  • Broker-Dealer transaction history
  • Client Review
  • Customer Relations and Prospecting
  • Enterprise Credit Risk
  • Cluster Pricing

Clearly the presentation didn’t have time to get into Lambda architectures, Spark and more

Open Bank Project

August 26th, 2014 / Tales from a Trading Desk

Interesting video on InfoQ on Open Bank Project – retail banking.  In many ways its similar to Bloomberg’s App Portal initiative.

Python and Big Data

August 19th, 2014 / Tales from a Trading Desk

Coupled of interesting video’s from EuroPython:

  • Python’s Role in Big Data Analytics: Past, Present, and…
  • Big Data Analytics with Python using Stratosphere
  • Scalable Realtime Architectures in Python

 

Dynamic Time Warp

August 15th, 2014 / Tales from a Trading Desk

Interesting read over on Systematic Investor, “Time Series Matching with Dynamic Time Warping”.  Would be interesting to apply DTW against trading recommendations.   Another paper on DTW also looking interesting: Trading Strategies based on Pattern Recognition in Stock Futures Market using Dynamic Time Warping Algorithm.  Likewise: Patterns In Financial Markets:dynamic Time Warping

Adding promises to Twitter Flight

August 14th, 2014 / Technical Tidbits

 Drafted in May 2013, but never quite finished, until now.

A good dose of Bootstrap is a wonderfully simple way to tart up your tired web site. So when Flight, an event based web framework, was open sourced early 2013 I was very interested to see what the talented engineers at Twitter had been up to.

Flight lets you construct your site with lightweight components. The key feature being that components cannot reference each other. All communication is via events delivered through the DOM.

As Zef Hemel asserts events are great, but often you really need a contract. He argues the answer is through dependency injection. By directly calling another object, you know immediately if the call succeeded or not.

Price Margining using Logistic Regression and Spark

August 11th, 2014 / Tales from a Trading Desk

Insight offered an interesting read on logistic regression and spark.  Which lead to the thought if one could track binary classification and predict a spread at which a client might hit a quote via an RFQ, RFS or ESP.   Interesting

Markov Model and Trading History

August 8th, 2014 / Tales from a Trading Desk

Interesting read over on Mawazo about smarter email marketing with Markov Model.  I wonder if instead of predicting when to send emails, one can predict the next trade, or RFQ?

Once we have the model, we will be able to predict, given the recent transactions of a customer, when the customer is likely to make the next purchase and how much he or she is likely to spend

Non-Technical Managers and Agile

August 7th, 2014 / Tales from a Trading Desk

Code Rant offers a view on coconut headphones.  Some great call-outs:

Current non-technical agile practitioners still don’t understand where the airplanes come from. They stand in their bamboo control towers with their coconut headphones on and wonder why their software projects still fail.

If your role is simply asking for estimates and enforcing the agile rituals: stand-ups, fortnightly sprints, retrospectives; then you are an impediment rather than an asset to delivery

 

Trading Research/Idea Recommendation using Alternating Least Squares (ALS)

August 7th, 2014 / Tales from a Trading Desk

Spark is generating a lot of interest.  Obvious ideas for using Spark in the Single Bank Platform space is around helping the sell-sides clients in identify trade ideas/research/commentary to assist them in their portfolio.  As a starting point, its worth reading the Spark movie recommendation with MLib article over on ampcamp, which provides details on collaborative filtering and alternating least squares (ALS).  Code is also provided :)

MLlib is a Spark implementation of some common machine learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives

Data Mining RFQ’s with SAMOA

August 6th, 2014 / Tales from a Trading Desk

Decided to throw a few RFQ’s into SAMOA.  Here’s the simple ARFF file I used:


@relation rfqs

@attribute CurrencyPair string
@attribute Notional numeric
@atrribute ClientName string
@attribute Quote numeric
@attribute RFQAccepted {0,1}

@data
'USDEUR', 5000, 'Client A', 43.33, 1

Firing up SAMOA with the following command line

bin/samoa local target/SAMOA-Local-0.2.0-SNAPSHOT.jar “PrequentialEvaluation -s (ArffFileStream -f rfq.arff) -f 10 -l (classifiers.trees.VerticalHoeffdingTree -p 4)”

Which I believe has trained the SAMOA classifier to identify instances that are either likely to be an accept or reject RFQ.  Not sure yet what the prediction would be based on further data – would I be able to predict a clients RFQ outcome?

Streaming Mining

August 5th, 2014 / Tales from a Trading Desk

Coupled of interesting articles worth a read:

Big Data – Getting Over The Hype

August 4th, 2014 / Tales from a Trading Desk

Big Data is one of those buzz words that unless applied to specific business solutions is often in the stratosphere on detail.  In the financial arena there has always been a large around of data in the house.  Lets start with a few real-world data insight solutions which hopefully the usual suspect sell-side organisations are working too.

Corporate Data Architecture – Fast Data and Big Data.

July 21st, 2014 / Tales from a Trading Desk

The VoltDB blog has a number of interesting articles on Corporate Data Architecture.  The Fast Data stack is somewhat different from a lambda architecture that has been discussed on various blog in the context of real-time big data.  Clearly Fast Data is a VoltDB database based on VoltDB’s view of the world. However, summingbird and other such stacks also have merit.  As does samza and Kafka.

Lets assume I want to serve knowledge and analytics from ingesting web site click data, pricing, execution and trade data, to power a salespersons trading cockpit.  What should/could I use?

Collateral Management view of Team Resourcing – Part 6

July 20th, 2014 / Tales from a Trading Desk

I’m back to coding, but this time in native Neo4j.  The aim to modelling teams, and time lines of teams, is wanting the ability to ask the system “What is the best team for x?”.  Which leads to a recent article on Moneyball.

Billy Beane analyzed the data on players’ performance and used the models’ outputs to select players

Food for thought

TDD Resistence

July 20th, 2014 / Tales from a Trading Desk

Ian’s talk on TDD: Where Did It All Go Wrong? is worth a watch.  The problem Ian outlines is known by I suspect a large percentage of development teams.

Avoid testing implementation details, test behaviors
– A test-case per class approach fails to capture the ethos for TDD. Adding a new class is not the trigger for writing tests. The trigger is implementing a requirement.
– Test outside-in, (though I would recommend using ports and adapters and making the ‘outside’ the port), writing tests to cover then use cases (scenarios, examples, GWTs etc.)
– Only writing tests to cover the implementation details when you need to better understand the refactoring of the simple implementation we start with.endencies

XP – Software Development Best practices

July 20th, 2014 / Tales from a Trading Desk

iMogital states quite simply that Scrum/Kanban will not tell your developers how to develop.  XP has 12 practices derived from software engineering best practices – TDD, Paired Programming, Coding Standards being a few.  CodeCentric touch on the same issue.  I suspect continuous integration, should really be continuous delivery these days.  Net out, if your development teams are not familiar with XP, maybe its finally time to help them get educated, and move down the XP road.

InfoQ has a recent article on engineering practices, offering the following equation:

Scrum + Extreme Programming (XP) = Agile

Agile Release Train

July 16th, 2014 / Tales from a Trading Desk

SAFe’s overview of the agile release train is worth reading.  Release trains aren’t new, they been around for years in both agile and non-agile organisations. Release trains aid in a consistent push to production, and work around certain black-out dates when production can’t be touched e.g. Christmas, end of month etc.  Sometimes certain train dates are also aligned to key business deliverables which although maybe anti-agile is some ways, are from a commercial perspective relevant.

FX Agency Model and FlowMonsters

July 14th, 2014 / Tales from a Trading Desk

Interesting read in Financial New this week on Faros Trading, and the agency intermediary model.  Even with the flowmonsters, there is opportunity for regional banks to specialise in home currency pairs. Breaking into the flowmonster space is costly from a technology perspective – problematic in post Basel III era.  The top four banks have a combined share of 53.5% of the FX market.