A recent internal conversation started relating to the infamous Good Agile Bad Agile article by Steve Yegge. We decided to share it, so here it is (part of it at least).
Bruce,

Great post. I read somewhere that a company had 99 lever arch files full of documentation so that they could be assessed and accredited by an external CMM auditor. CMMI is an acceptance by SEI that maybe its a bit too heavy weight for practical use; though many have moved up the levels of CMM with success. But agile is a bit more than iterative, as iterative is still around today in the forms of RUP and other BDUF methodologies that accept the canonical waterfall dictum is not practical. Iterative to me is a step towards realising that maybe the problems we are trying to solve are too big for us, and that embracing divide and conquer might be more wise; a positive realisation. Agile however embraces something different (to me), and that is Rittel’s notion of the wicked problem, that software is not only very complex, but also that by trying to solve some of the problem, we can learn more about what it is we are trying to solve in the first place – this new knowledge being feedback into understanding again what the problem really is. Of course, there is much more to the agile movement than just problem definition, but its still no use building for the wrong problem.

The more I work using agile practices, the more I develop an appreciation of the less agile/non-agile processes. People give BDUF a real bad-wrap, and I have to say that when I hear it, it makes me embarrassed to be in the sector. It wasn’t too long ago I used to hear that people dissed project management, that it was the root cause of failing projects, and that that this is why XP gives a royal finger to the PM world (not that I agree). Now, XP is definitely seen as being better complemented with Scrum, a project management framework; time to kiss and make up. RUP on the surface is as daunting and overwhelming as anyone looking at the ISO or CMM/I directions, and this is one reason for its bad press. The sad thing is is that most of the rhetoric is wrong – RUP is meant to be appropriātus, it is meant to be shaped and customized to meet the needs of the organization. Because of this fact, RUP isnt half as heavyweight as is believed by the misunderstood methodology-bourgeoisie.

There are also examples coming out that show that “agile” in the form of XP and Scrum are delivering on their promise, thats for sure (ACM have some good articles to this effect). At the same time there are also clear examples that unless there are Level 2 or Level 3 Cockburn style practitioners predominantly in an XP style team, then the rhetoric is fluff and nothing else, as these projects spin their wheels refactoring, rearchitecting and simply getting stuck in the mud of advanced engineering practices they have little or no skill in. Interestingly a project I worked on not too long ago feels today like such a project, and the introduction of agile practices will have offered some respite to the wheel spinning for sure, but I have doubts whether the team will achieve the levels of success attainable by other XP teams – simply because of the people on the team. In this case, some level of direction and rigour in applying less agile, more plan driven practices might have offered a better level of risk mitigation than just hoping quick deliveries will suffice – that is ofcourse if quick deliveries could ever be achieved.

Yegge’s comments are valid, but they are also naive in my view, for their lack of appreciation of the deeper truths behind the practice. Agile, in the sense of less heavyweight processes, where the team are constantly planning throughout the process to adapt to new knowledge, this type of agile is as much a candidate of choice as is any less agile option, but its based on context. I get miffed because Agile seems to be associated with things like XP, but I know of companies doing “Agile” who just do lightweight planning, use NUnit/JUnit and do TDD. I have a friend who is working for a company that is practicing Agile, and he sent me some code to review that had been crafted using test driven development. The code wasn’t good. Anyone can build crap code, but the very fibres that weave the elemental parts of eXtreme Programming into the methodology that it is are based in solid, quality driven software engineering practices, and this is one of the reasons for its ability to deliver. Not getting bogged down in project management can be more agile than a tanker PM process that cannot deviate quickly from its trajectory, if at all, but this type of agile is aeons away from XP. Granted, at the same time all are attempts at improvement in some form or another, yet I feel the industry knows pretty much most of the people issues and the planning issues. What the industry needs to do is push the envelope in the skillsets of the practitioners that extol the discipline, because I personally feel this is a place where the greatest travesties can occur.

As a closing anecdote, I started working with a team a few years ago on a web application. I started pairing with a guy who was ten years old than me, has been in the industry since I was a thought in my dads head, and apparently had been technical in every part of the IT practioner stack, from programmer to enterprise architect. Our first problem was to fire some data down to an instrument – the user pressed a button to say “send”, and this would trickle down and our Instrument Layer would be invoked to deal with generating the necessary data for the instrument. Within only a few moments my partner suggested that we needed to know some extra details, such as who is the user firing the send request, that type of thing. So I asked if he had any suggestions, and he told me he did – we should pop up a dialog to ask for the details. I first mentioned that we were building a low-level layer, and asked if he felt working with the UI from this level was appropriate – he didnt have a problem. I then asked him about the more pressing worry that we were building a web application, and popping up the MessageDlg from down here, if we could do that, would probably not be what we wanted to do anyway. Long story short, I failed to convince my partner, and he merrily went ahead building the pieces he felt were needed. Eventually he realised his error, but the deeper issue for me was “oh no!”, this guy has got over 20 years experience – he’s meant to be one of the good guys. There are indeed good guys out there, but you gotta look hard enough.

Nick Robinson
Lab49, Inc
http://www.lab49.com
http://blog.lab49.com

From: Bruce Fancher
Sent: Sat 21/10/2006 9:13 PM
To: Nick Robinson; Luke Flemmer; Ashley Whitney; tech
Subject: RE: “Good Agile, Bad Agile”
First of all, what the hell does Agile even mean at this point anyway? The original manifesto was pretty vague and the dude who wrote that blog post [yegge] is right that the term has become another buzzword for the usual suspects to pile onto. This is unfortunate because it obscures the original point of the so-called “Agile” methodologies (or what used to be called “Iterative Development” until they found out that no one could pronounce “Iterative”). And the result is that when guys like Steve whatsisname decides to rant about whatever it is he’s frustrated about at the moment (I couldn’t read far enough into the post to completely figure out that part) he blames it on “Agile.”

Software has always been difficult to write, and hard to deliver on time (or sometimes at all). For a long time the industry tried to address this by creating more and more elaborate systems and methodologies for upfront analysis, design, planning, scheduling, testing, measurement, etc. The thinking seemed to always be, well the last project was late because the customer didn’t tell us they needed feature X, and we didn’t think of technical problem Y, or anticipate bug Z. So, this time we’ll specify everything out in advance in even more detail so that our estimates will be more precise and the product will be finished on schedule. Oh, and while we’re at it, we’ll have one or a few architects at the top, who will tell the rest of the developers who everything should be designed (possibly with a few more layers of management in between).

And the amazing thing is, after all these years, and none of that stuff ever working, they’re *still* at it. Remember the CMM? I just checked and the Software Engineering Institute, which is apparently the North Korea of heavyweight software methodologies, has now replaced the CMM with the CMMI. Just one of the documents on their site is 573-pages of nonsense about how define your own ridiculously detailed set of policies and procedures. Here’s a random excerpt, taken from the “Process and Product Quality Assurance (PPQA)” section of the CMMI Product Team Technical Report (CPTTR):

Provide Objective Insight
Noncompliance issues are objectively tracked and communicated, and resolution is ensured.
SP 2.1 Communicate and Ensure Resolution of Noncompliance Issues
Communicate quality issues and ensure resolution of noncompliance issues with the staff and managers.
Noncompliance issues are problems identified in evaluations that reflect a lack of adherence to applicable standards, process descriptions, or procedures. The status of noncompliance issues provides an indication of quality trends. Quality issues include noncompliance issues and results of trend analysis.
When local resolution of noncompliance issues cannot be obtained, use established escalation mechanisms to ensure that the appropriate level of management can resolve the issue. Track noncompliance issues to resolution.
Typical Work Products
1. Corrective action reports
2. Evaluation reports
3. Quality trends
What the %*@*(#$ does that have to do with actually building working software?
Not surprisingly it turns out that trying to plan anything as complex as computer software, with instruments as limited as human brains, doesn’t work at a technical company any better than it did in the Soviet Union. If you check out a few books and articles by economists like Fredriech Hayek, Thomas Sowell, etc. you’ll find that just about everything they say about the limits of central planning, the importance of trial and error, having a feedback mechanisms, etc. is just as relevant to waterfall methodologies as to the socialist planning they were writing about. Planning anything too complex ahead of time doesn’t work, simply because we don’t know enough to do so, and usually the only way to accumulate the knowledge we need to do such planning is to actually attempt to build whatever it is we’re trying to build and refine it as we go along.
To me, that’s the main point of “Agile.” Simply acknowledging that you’re going to get a much better result if you distribute responsibility and authority to those closer to the problem (e.g. the folks writing the code, rather than a remote architect imposing his vision of how things should be), and build a feedback mechanism into your process. Accepting this insight doesn’t imply that Scrum, XP, or any other process or methodology is perfect. In fact, it almost means that it isn’t, since like everything else it should be subject to improvement and refinement based on ongoing experience. And whether or not the C3 project was a success or not doesn’t really have any bearing either. The question isn’t whether “Agile” (again, whatever the hell that means) is perfect, but how it compares to the alternatives. Given the absolutely appalling track record of the software industry, it’d be hard to do any worse.
Bruce