Complexity – Large-Scale Software Systems

March 3rd, 2015 / Tales from a Trading Desk

“Out of the Tar Pit” is an old paper, but worth a read never the less.  The paper was picked up in more recent years by Hacker School, and also “10 Technical Papers Every Programmer Should Read (At Least Twice), and more recently, Brian Gesiak.

Since Brian has offered a few interesting call-outs already, I’ll only offer a few additional thoughts/quotes:

Page 1: The biggest problem in the development and maintenance of large-scale software systems is com- plexity — large systems are hard to understand

Page 2: Complexity is the root cause of the vast majority of problems with soft- ware today

Event-Driven Architecture Pattern Topologies

March 2nd, 2015 / Tales from a Trading Desk

O’Reilly Radar has an interesting article on event-driven architecture, “Variations in event-driven architecture”.  Mark Richard’s eBook is available here if you want the full read.

Its probably worth pointing out that I’ve come across a number of people worried about the number of event queues when using the Mediator topology – a strange thought process in my view unless the architect is proposing to use a single queue for all event types :(

It is common to have anywhere from a  dozen to several hundred event queues in an event-driven architecture

Code Ownership

February 26th, 2015 / Tales from a Trading Desk

Jay Field’s has an interesting posting on code ownership, “Experience Report: Weak Code Ownership”.  “Too many cooks in the kitchen” is unfortunately a problem I suspect we have all seen to many times on medium to large projects.  Stating the obvious, encourage consistency on a large code base over n years is difficult – its unfortunate that stakeholders who haven’t come from a software engineering background don’t really grasp the issues of such problems.

User Story Map

February 19th, 2015 / Tales from a Trading Desk

Few useful links if your going down the User Story Map road:

Time to React.js?

February 18th, 2015 / Tales from a Trading Desk

Eric Florenzano offers an interesting read on “Facebook just taught us all how to build websites”.  Essentially he is talking React.js.  Given the noise level on the web around React.js, coupled with the recent React.js conference, I’d be surprised if readers don’t already have a view of React.js, and haven’t seen their peers/colleagues watching the various presentations.

A recent blog from the BBC offers insight into the road they are taking with the BBC Homepage refresh.  Unsurprisingly, the BBC has chosen React.js:

Bad Design Assures Bad Results

February 16th, 2015 / Tales from a Trading Desk

I’ve not read JavaScript Application Design yet, but was struck by the correctness of a statement made in “About the Book”:

The fate of most applications is often sealed before a single line of code has been written.

As I’ve blogged about before, all to often team lurch into code well in advance of actually understanding the problem they are attempting to solve, and lacking any sensible architecture and process.

Good design and effective processes are the foundation on which maintainable applications are built, scaled, and improved.

Technical Debt

February 12th, 2015 / Tales from a Trading Desk

Design Smells offer a single page on technical debt.  One of the key causes of technical debt often overlooked compared to the other items listed is “schedule pressure”.  Lack of understand of basic software engineering by stakeholders often means that teams are driven down a road where the only outcome can be technical debt, which in itself leads to other issues :(

Lean Enterprise

February 11th, 2015 / Tales from a Trading Desk

Got my hands on “Lean Enterprise: How High Performance Organizations Innovate at Scale” recently.  Found it a dry read.  However, the book has some classic comments:

The purpose of an organization is to enable ordinary human beings to do extraordinary things – Peter Drucker

Shareholder value is the dumbest idea in the world…[it is[ a result, not a strategy…Your main constituencies are your employees, your customers, and your products. – Jack Welch

Page 31, “establishing and trying to maintain effective communication within large teams consumes enormous amounts of capacity on large projects” – so so true.

Stop Writing Dedicated Server End Points – GraphQL

February 6th, 2015 / Tales from a Trading Desk

There are numerous interesting presentations from React.js Conference 2015. “Data fetching for React applications at Facebook” is one such presentation well worth watching.  Specifically the comment at 7:52!  Shaping data should also help with reduction of bandwidth usage – your only sending the data you need, and nothing more.  This blog posting provide a view on batching requests – cut down the chatter.

User Story Mapping – Reduce the Duplicate Stories, Improve Visibility

February 3rd, 2015 / Tales from a Trading Desk

Just finished reading User Story Mapping.  Although the concept isn’t new, the book offers a good read, which will hopefully improving the uptake of the practice by team in the construction and management of backlogs.  Few take aways:

  • Page 3 – “Stories get their names from how they should be used, not what should be written”
  • Page 23 – “Map for a product release across multiple teams to visualise dependencies”.  For me, “dependencies” is the keyword. Numerous times teams have failed to visualise dependencies of stories ending with discovery in game planning sessions and failed iterations :(

Event-Driven Applications in the Cloud – AWS Lambda

February 3rd, 2015 / Tales from a Trading Desk

Since the recent AWS Lambda launch, there have been numerous articles providing insight into the service.  What’s key from my perspective is that Amazon have enabled event from other services, thus making AWS Lambda a glue service for event-driven applications:

Lambda is launching in conjunction with a new Amazon S3 feature called event notifications the generates events whenever objects are added or changed in a bucket, and our recently announced Amazon DynamoDB Streams feature that generates events when a table is updated. Now developers can attach code to Amazon S3 buckets and Amazon DynamoDB tables, and it will automatically run whenever changes occur to those buckets or tables. Developers don’t have to poll, proxy, or worry about being over or under capacity – Lambda functions scale to match the event rate and execute only when needed, keeping your costs low.

Simple Made Easy – Complexity Will Kill A Project

January 23rd, 2015

Old, but great InfoQ presentation by Rich Hickey, “Simple Made Easy”.  The agile starting pistol (sprints) is very funny.

Simple = untangled

AWS Scale

January 21st, 2015 / Tales from a Trading Desk

High Scalability has an interesting article on Amazon, “The Stunning Scale Of AWS And What It Means For The Future Of The Cloud”, coupled with an older article, “Amazon Architecture”.  Both offer some great insight into AWS.  Although both articles, including the more recent article is packed full of interesting information, its worth calling out a few points:

  • “Testability. Their gear was better because they tested it better. Enterprise gear is hard to test at scale. They created a $40 million test bed of 8000 servers (3 megawatts). “

Stop Breaking Master

January 21st, 2015 / Tales from a Trading Desk

Another great InfoQ presentation, “Software Development & Architecture @ LinkedIn”.  10mins into the presentation offer software development process insight – tooling is key.

24mins in, Kafka Multi-Colo

Large projects are difficult!  There is no free lunch.

Sensitive Data in Hadoop

January 9th, 2015 / Tales from a Trading Desk

Jeremy Stieglitz presentation “The Big Data Imperative: Discovering & Protecting Sensitive Data in Hadoop” provides a good view of the data security issues when using Hadoop.  Compliance and security risk of data – data residence challenges.  Encryption vs masking isn’t discussed enough when thinking about data security.  Audit strategies – who accessed the data, reporting and tracking sensitive data.

Apache Kafka – Multi Asset Class Usage

December 22nd, 2014 / Tales from a Trading Desk

InfoQ has an old, but relevant article on Kafka usage in financial services – specifically S&P Capital IQ.  Interesting comparisons to RabbitMQ and ActiveMQ.  “Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse” is also worth a read.

Innovation back in Financial Services?

December 20th, 2014 / Tales from a Trading Desk

Startups are giving financial services a good kicking from a retention and hiring perspective.  Everyone wants to be in a startup, live the dream, be involved in a cool product, and hopefully be rewarded appropriately.  Wall Street and Technology has had a number of articles this year offering insight into the innovation that financial services is driving to not only re-build the sector, but also I suspect to keep staff engaged.  BNY Mellon and Fidelity are the two recently called out, but a quick google will provide some insight into the money being invested into startups by the top 10 or so banks.  Interesting times both within financial services and outside.

Agile Frustrations

December 20th, 2014 / Tales from a Trading Desk

Not Just Code Monkeys presentation from Martin Fowler offers an interesting journey through his presentation.

Conversation stories – programmers should be part of story development.  BA’s and UX’s shouldn’t be left to define the backlog without engineering conversation.

Domain knowledge for a developer is king.

Engineers need to take responsibility for the code

Mr Fowler really doesn’t like finance.

Stream Processing

December 12th, 2014 / Tales from a Trading Desk

Coupled of interesting presentations on InfoQ:

  • High Throughput Stream Processing with ACID Guarantees
  • Mantis: Netflix’s Event Stream Processing System
  • Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse

First video is from CASK which has a number of products that look interesting. Anyone played with the CASK products in a financial application context?  There’s a vague financial use case hereTigon looks possibly interesting from a risk perspective, or depending on how low the latency is, maybe even pricing

Platform and Dog Food

December 10th, 2014 / Tales from a Trading Desk

Old but interesting read:

  • The Secret to Amazons Success Internal APIs
  • 6 Things Jeff Bezos Knew Back in 1997 That Made Amazon a Gorilla
  • Ex-Amazonian urges Google to sample Amazon’s secret sauce
  • Getting SOA Right – Thinking Beyond ‘The Right Angles’
  • Google Engineer Accidently Shares His Internal Memo About Google + Platform