Apache Kafka – Multi Asset Class Usage

December 22nd, 2014 / Tales from a Trading Desk

InfoQ has an old, but relevant article on Kafka usage in financial services – specifically S&P Capital IQ.  Interesting comparisons to RabbitMQ and ActiveMQ.  “Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse” is also worth a read.

Innovation back in Financial Services?

December 20th, 2014 / Tales from a Trading Desk

Startups are giving financial services a good kicking from a retention and hiring perspective.  Everyone wants to be in a startup, live the dream, be involved in a cool product, and hopefully be rewarded appropriately.  Wall Street and Technology has had a number of articles this year offering insight into the innovation that financial services is driving to not only re-build the sector, but also I suspect to keep staff engaged.  BNY Mellon and Fidelity are the two recently called out, but a quick google will provide some insight into the money being invested into startups by the top 10 or so banks.  Interesting times both within financial services and outside.

Agile Frustrations

December 20th, 2014 / Tales from a Trading Desk

Not Just Code Monkeys presentation from Martin Fowler offers an interesting journey through his presentation.

Conversation stories – programmers should be part of story development.  BA’s and UX’s shouldn’t be left to define the backlog without engineering conversation.

Domain knowledge for a developer is king.

Engineers need to take responsibility for the code

Mr Fowler really doesn’t like finance.

Stream Processing

December 12th, 2014 / Tales from a Trading Desk

Coupled of interesting presentations on InfoQ:

  • High Throughput Stream Processing with ACID Guarantees
  • Mantis: Netflix’s Event Stream Processing System
  • Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse

First video is from CASK which has a number of products that look interesting. Anyone played with the CASK products in a financial application context?  There’s a vague financial use case hereTigon looks possibly interesting from a risk perspective, or depending on how low the latency is, maybe even pricing

Platform and Dog Food

December 10th, 2014 / Tales from a Trading Desk

Old but interesting read:

  • The Secret to Amazons Success Internal APIs
  • 6 Things Jeff Bezos Knew Back in 1997 That Made Amazon a Gorilla
  • Ex-Amazonian urges Google to sample Amazon’s secret sauce
  • Getting SOA Right – Thinking Beyond ‘The Right Angles’
  • Google Engineer Accidently Shares His Internal Memo About Google + Platform

 

TimeSeries AggregatoR – TSAR

November 26th, 2014 / Tales from a Trading Desk

InfoQ’s presentation “TSAR: How to Count Tens of Billions of Daily Events in Real Time Using Open Source Technologies” is worth viewing if you have time.  Apart from the beer trending (24:10), I’d be interested in knowing if anyone in capital markets has looked at user TSAR to understand customer trends, order trends, etc.

 

How Google Works

November 26th, 2014 / Tales from a Trading Desk

Recently just finished reading How Google Works.  Like so many book, the start of the book is better than the end in my view.  However, I still think its a great book, providing some good insight into practices used by Google that can be leverage elsewhere:

  • Page 28 offers a view on how how the Google culture works in solving problems – in this particular scenario, the result offending ads.  Culture
  • Page 30.  Culture.  Decision by committee vs autocratic or combative approaches.
  • Page 42. Rule of seven.
  • Page 45. Reorg day – get it done fast

Drive – Do you have it?

November 13th, 2014 / Tales from a Trading Desk

Can’t remember who recommended this book, but it was worth the read. Of note:

  • Page 32, “As organizations flatten, companies need people who are self-motivated”
  • Chapter 2, “Seven Reasons Carrots and Sticks (Often) Don’t Work…”
  • Page 51, “People will choose the quickest route” to the destination for an extrinsic reward
  • Page 57, “Companies pay a steep price for not extending their gaze beyond the next quarter”
  • Page 86, Results-Only Work Environment (ROWE)
  • Page 138, “The MBA Oath”
  • Page 162, “Try 20% Time With Training Wheels”
  • Page 169, “Turn Your Next Off-Site into a FedEx Day”

The Legacy of Ivory Tower Architects

November 12th, 2014 / Tales from a Trading Desk

Much has been written, comments and discussed on the interaction of Ivory Tower Architectures and project teams.  Certain organisations I’ve worked for have unfortunately built departments of architectures who are so disconnected from the business, with obvious consequences – the business builds their own IT mini department/team.  Relevance offers an old blog posting with some useful thought on capturing architecture decisions on an agile project – particularly useful on multi-year projects.

Large documents are never kept up to date. Small, modular documents have at least a chance at being updated

ELK Stack – OSS Splunk?

November 12th, 2014 / Tales from a Trading Desk

ELK has been gaining a good amount of blog press recently.  Which leads to the obvious questions, of can ELK replace Splunk?  No an uncommon question given again the blog entries by various people on leveraging ELK.  More interesting is recent events, with an Splunk Exec moving to ElasticSearch, which should help to push ELK forwards.

OpenStack, Mesos and Docker – PaaS

November 5th, 2014 / Tales from a Trading Desk

eBay’s 2014 posting on “Delivering eBay’s CI Solution with Apache Mesos” offer a hint of where we should possible be pushing with cloud computing:

Our new Mesos cluster is set up on top of our existing OpenStack deployment.

Our cluster is built on virtual servers in our OpenStack environment. Specifically, the cluster consists of virtual servers running Apache Zookeeper, Apache Mesos (masters and slaves), and Marathon services. This combination of services was chosen to provide a fault-tolerant and high-availability (HA) redundant solution. Our cluster consists of at least three servers running each of the above services.

Moving on from Scrum – Disciplined Agile Delivery

November 5th, 2014 / Tales from a Trading Desk

I recently picked up a copy of Disciplined Agile Delivery (DAD).  DAD is a hybrid approach that extends Scrum with proven strategies from Agile Modeling (AM).  Anyone who has used Scrum on large enterprise projects will know what Scrum/XP doesn’t assist in the modeling, documentation and governance arena.  Page 4 offers a simple diagram showing the differences of Agile Development, Agile Delivery and Agility@Scale.

Over the years I’ve heard various debates about how much intellectual work can be done in a working day.  Page 31 offers a view that 5-6hrs a day is the norm for achieving high-quality intellectual work.  Often overlooked, “Visualize Workflow” often aids teams in understanding blockers, and utilization.

Hadoop Tutorials

October 23rd, 2014 / Tales from a Trading Desk

Hortonworks seems to have done a good job on offering Hadoop tutorials, leveraging a Sandbox.  In particular, they offer a real-time event stream with Apache Kafka which is then persisted into HBase and Hive using Storm Bolt.  Food for thought

21st Century Pricing Engines

October 22nd, 2014 / Tales from a Trading Desk

In todays markets, there are a number of drives that influence the buy vs build of a pricing engine irrespective of asset class.  Those drives probably include the following:

  • Consistent pricing
  • Minimising the number of trade away %
  • Ability to on-boarding clients in a timely manner
  • High availability, low latency of both price construction, and price delivery

This leads to a number of requirements which i suspect maybe mandatory in certain organisations:

  • Liquidity steam per user (not client) sourced from a liquidity pool – historical 3-5 tier pricing probably isn’t good enough these days
  • High availability – consistent pricing from multi data centres

Definition of Ready

October 21st, 2014 / Tales from a Trading Desk

You’ll always hear about the Definition of Done (DOD) from agile teams.  However, what is not so often discussed is Definition of Ready (DOR) – sometimes referred to as DevReady in various organisations. Fabrique’s SXSW 2013 slide deck (53) offers a view of DOR.   The Agile Alliance crystallises why DOR is important:

Avoids beginning work on features that do not have clearly defined completion criteria, which usually translates into costly back-and-forth discussion or rework

Churn during iteration due to imprecise requirements is often the by-product of not having a strict DOR.

Hadoop Security – Role Based

October 20th, 2014 / Tales from a Trading Desk

Cloudera offer a number of interesting blog articles on Hadoop security – Impala using Apache Sentry.  From an enterprise perspective, creation of a data lake will inevitably require a clean security approach from day 1.

Shipping Quality Code

October 15th, 2014 / Tales from a Trading Desk

Scripting and Learning by Shipping both offer good reads on shipping quality code.  Its also nice to see that SAFe also has view on code quality, and the importance of Test First.

Test-first methods can be further divided into Test-Driven Development (TDD) and Acceptance-Test Driven Development (ATDD). Both are supported by Test Automation, which is required in support of Continuous Integration, team velocity, and development efficiency.

Beauty of Testing (Steven) offers a few home truths which unfortunately are seemingly lost in the rush to make a production release by people not aware of the complexity of software engineering (and it not being an exact science):

Single Dealer Platforms – Another Day Another View

October 15th, 2014 / Tales from a Trading Desk

WallStreet & Technology offers a view on Single Dealer Platforms (SDP).  Over the last few years there have been a number of articles stating that SDP’s will either take over the world, or are a dying breed.  The net out of this latest article is unsurprisingly a view that Single Dealer Platforms and Multi Dealer Platforms will be combined by either banks or clients to offer a subset of services based on the clients needs – best of breed (effectively a type of mashup).

Engineering Management

October 15th, 2014 / Tales from a Trading Desk

Great post from Slava @ReThinkDB.  Some great lessons.  32 – news doesn’t age well! 21 – Leadership is important

Further, Slava offers some view on project management – of particular note, treat big projects with care.  Of particular importance:

build the absolute most minimal version that solves the customer’s problem.

DDD – Part 3

September 30th, 2014 / Tales from a Trading Desk

Few relevant postings to read: