What We Learned About The Dangers Of “Headless” Agile

At a recent presentation selling the joys of Agile, a speaker began by pointing out the decisive weakness of Waterfall, a software-development process that sets out a logical progression of orderly steps. Complete one, move on to the next. if the original design is wrong, the final product will be wrong because there is no way to fix things during development. That truth being established, it’s easy to sell Agile on the basis that it allows repeated redefinition of the project (scope, features), so the deliverables evolve as the team learns more about the client’s needs, and as they run tests, experiment with different solutions, and awaken with brilliant thoughts in the middle of the night.

Listening to this description of such a magical methodology, I was suddenly seized by a horrible thought: What if Agility was used to simply enshrine the right of middle management to change their minds, to provide a methodological license for arbitrary management? What if the talk of agility was just a way of softening up developers for a life of methodologically sanctioned pliability? In short, what if Agile turned out to be worse than Waterfall? That, it suddenly seemed to me, was the likely outcome of using Agile tools in the absence of a governance regime to keep projects on target. Agile without governance is "headless" Agile.

When we formed LShift, a London-based software company, we sat with a group of engineers and asked them what the new company would need to do to make them want to join it. Their major request was that we eradicate a situation every developer is familiar with—mid-project, an account handler or program manager tells engineers that, in a fit of largesse during lunch with the client, they have agreed to add new features, swap out the core platform, or complete the project a month earlier than planned. Result: More late nights working on a fix or new design.

Our solution was to put lead developers in front of the clients as much as possible, immersing them in the business aspects of the project. And, to the extent the situation allowed, we let developers control the design, planning, and delivery of the project. Our mantra was (and is) to “disintermediate middle management to the extent it makes sense to do so.” If developers are working a late shift because of a rash promise they’ve made, they are the authors of their own misfortune, which is easier to bear. As it happens, quite apart from the moral advantage, this turns out to be a good way to run a project efficiently; projects tend to come in on time and on budget.

But even this super Agile approach to software development doesn’t magically solve all problems, because governance is always a concern. What if the developers can’t govern themselves? What if their freedom from middle management is, in fact, a manifestation of a “headless” Agile approach? It never happened to us at LShift—but it could happen.

AGILITY WITH GOVERNANCE

To understand the necessity of Agile governance, over and above Agile tools, think of your overall project (planning, designing, building, and delivering) as having a trajectory. You might think of the project’s trajectory as a vector path, aimed more or less accurately at the correct end state.

In this way of thinking, a Waterfall project has a single vector, determined at the project outset. I think of it like the 1970s game show, The Golden Shot, where contestants would give a blindfolded archer instructions (“Left… right… back a little… right a bit… stop! Fire!”) to get their crossbow bolt to cut through a thread. A successful shot severed the thread to unlock the prize. The point is that, once you told the archer to fire, there was nothing you could do to alter course: the bolt either hit the target or it didn’t. Everything hinged on how accurately you aimed to begin with. Very few people succeeded.

In any project, using any delivery methodology, there are such control points at which the direction of the project vector can be modified. These are the key points at which project governance is exercised, allowing you to steer the project. In a Waterfall project, as in The Golden Shot, the project vector is set at the beginning, where you have the project’s main control point, and where you set its course (the project vector) once and for all.

It is usually necessary to exercise a second control point at the project’s end (acceptance), when changes are accepted in order to bring deliverables into line with the client’s goals. Even if you correctly interpreted the requirements at the start and aimed with complete accuracy, you most often won’t hit the sweet spot because it is a moving target—clients also learn during the course of the project and change their expectations and requirements accordingly. You could define the client’s pain as being proportionate to the area of the (red) triangle created at the end of the project, determined by the amount you are off target, and the time and effort it takes to repair matters.

EXHIBIT 1: THE GOLDEN SHOT

How are matters improved by Agile? The benefit of Agile is that, by splitting the project into discrete sprints, you have more opportunities to tweak the project vector, to exercise governance, to alter the project’s course. Each sprint offers new control points (during both planning and acceptance) when you can amend direction.

This new situation is a far cry from our one-off ‘Golden Shot’ scenario, and more akin to that of someone working a remote-control drone. The wind may blow it off course mid-flight, but we now have the opportunity to steer into the wind for a while to get back on track (see “Points of Governance in a Series of Agile Sprints”).

EXHIBIT 2: POINTS OF GOVERNANCE IN A SERIES OF AGILE SPRINTS

Th crucial point is that Agile provides you with the ability and opportunity to correct course. However, while it allows you to steer, it does nothing to help us steer correctly. Using Agile methods, our pilot can redirect the drone, but those methods do nothing to help keep it on course. Agile tools provide a steering mechanism, not a homing device. Consequently, our Agile-empowered project manager is just as likely to be steering off-course rather than toward it.

This is a terrifying thought: In and of itself, Agile is just as likely to be a tool for empowering chaotic management as it is to be a mechanism for delivering successful projects! At least in Waterfall, if you did a good job of designing the project, and if you stuck to the spirit of the method, you had a shot at success. But the outcome in a badly governed Agile project is that each iteration takes you further off course. The result can potentially be even worse than a badly planned Waterfall project, and with more late nights worked along the way. The pain coefficient at the project’s end can also be greater than with the Waterfall.

ON GOVERNANCE: PLIABILITY IS NOT AGILITY

Agile enables pliability; but pliability in not Agility. The project-management team (or self-managed developers) must manage that pliability to create a truly Agile project. The key is to introduce effective project governance. This refers to the detail of the control mechanisms used to direct the project, how they are used to capture and incorporate the necessary information so as to guide the project direction.

If Agile tools don’t provide mechanisms for exercising governance correctly, everything now hinges on how this is going to be achieved. Dynamic Systems Development Method (DSDM), a powerful and widely applicable methodology, provides a context and methodology for ensuring that at each control / governance point, input that synthesizes project management, business, and technical inputs is synthesized into a coherent whole (see “Governance Inputs”).  

EXHIBIT 3: GOVERNANCE INPUTS

Using such multi-faceted governance, you can begin to effectively apply Agile’s pliability, steering your project nearer the goal with each iteration of the project. Now, each sprint and iteration offers not only the chance to exercise project management, but to ensure that it is exercised by capturing appropriate, timely information from all the appropriate stakeholders.

That’s all easier said than done, of course. Everything hangs on correctly applying governance tools and methods, and it can take a lot of failed experiments before you get that right. But the better you get at it, the more likely that you’ll really be in control of your project, in the sense of consistently steering it toward the optimal result.

Agile is better than Waterfall— but that is not a foregone conclusion. If it is “headless” and not properly managed, Agile could be worse than Waterfall.