Unraveling Agile: Overcoming Pitfalls in Software Development

In 2001, the Agile Manifesto revolutionized software development. Despite its brilliance, Agile faced setbacks. The culprit? The clash between principles like “Individuals and interactions over processes and tools” and the rise of rigid methodologies like Scrum. The irony: attempting to prioritize individuals led to building processes. In this blog, we dissect the manifesto’s values, exploring the pitfalls encountered, such as the tension between “Working software over comprehensive documentation” and the bureaucratic quagmire of documentation. Let’s delve into the challenges, seeking a trust-driven approach to overcome the myth of methodology.

The Manifesto for Agile Software Development was written back in 2001. What a seminal work of software development! In all the time since I haven’t actually found anything wrong with it. I’m a big fan, and so are my colleagues at 345. Throughout the ’00s it looked like agile development was becoming a thing, but then the wheels started falling off. “Agile” got itself a bad name for a variety of reasons and the vultures were already waiting to feast on its fall.

So what went wrong?

Individuals and interactions over processes and tools

Well, Scrum for a start. The first of the values is “Individuals and interactions over processes and tools”. So the first things was to….build a process and a set of tools so that we can control the individuals and interactions. Um, yeah.

I’m not saying that you go off half-cocked and just wang in whatever code you fancy on any given day. Nooooooo. But management (almost) always want to formalise process and in doing so they (almost) always place the needs of the process above the needs of the individuals. If you have a team of low-productivity automatons who are happy to sit at desks like corralled cattle all day you might not notice much difference, but good people don’t work like this. I mean, really good people. People who work off inspiration, who can come up with solutions in a couple of days that that are worth more than their entire annual earnings.

Working software over comprehensive documentation

The second value, “Working software over comprehensive documentation” is a corker as well. It really separates the people who deliver from the people who hide behind process as an excuse for non-delivery. If you value working software primarily, delivery of quality product is your focus.

Let’s put this another way. If you’ve worked on a large and heavily-documented system, how often were your documents read? I mean REALLY?  Did you deliver value, or did you make sure you followed the process?  Did you deliver quality 1, or did you ensure the checklist for the project’s “quality gate” was met?

What does get read, and read often by nearly all technical staff, are conceptual diagrams. Boxes and arrows. Yes, do these, and lots of them. In fact, why don’t you whiteboard this stuff and then take photos of it. Do this design activity often, refining all the time, but don’t spend your time making it into documents that nobody reads.

The best documentation I have left behind on a project was a 2-hour video of me and a colleague at a whiteboard briefing some new team members on our application. That video was watched by every new starter after that.

Put it another way: You have stuff that originates in the heads of your tech team, and needs to get into the heads of other team members (especially new starters) so that they can contribute to delivery. What is the most effective method of extracting and transmitting this information? Use that method. Don’t write documents because someone selling a methodology in the ’80s decided you should.

Focus on value. Value is working, high-quality software.

Customer collaboration over contract negotiation

Those of us who build software for clients need contracts. Contracts specify how we get paid. At a fundamental level they need to happen.

Often a strange metamorphosis happens with contracts. Ideally, a contract would say something like “hey, build us a great system, work as good as you can, and we’ll pay you based on the effort you put in”.   You then set off on the best route you can see, but you learn along the way, you get new ideas, you get feedback from users when they see prototypes. You incorporate the feedback, the product improving with every iteration. Everyone’s happy, the developers are motivated, the product is great, minimal time is wasted. Great job everyone.

Or, back in the real world, management (especially financial management) step in. They want a contract that says “hey, build us a system that does X and Y, and deliver it on Z date and we’ll pay $$”. Off they walk, smug in their ignorance of how things work in the real world. This leaves much unanswered, such as what does X mean? If you don’t give us the information on Y then we can’t make date Z. What happens if this is more complex than we thought?

Next, you’re staring down the barrel of a “process” that’s disguised as project management but underneath is subterfuge designed to provide commercial cover, creating a contractual imperative to adopt bad behaviour.

Responding to change over following a plan

Before long you’re in a much darker place than you were before. You worry about delivering the letter of the contract rather than delivering a great system. Collaboration gets frowned on, because improvements you find along the way would require contract change, and the contract is already agreed. Do you need to get the legal bods in again? Do you want to ask the CFO for more money? No. Then pipe down.

Inability to respond to change is almost always closely tied to the commercial conditions you work under.

It doesn’t have to be this way. The difficult thing is that approaching in the right – agile – way takes trust.

To be Agile is to trust

Accept that when you start your knowledge is incomplete. Accept that the product you need is not the one you’re able to describe when you start. Accept that your priorities will change. Get good people together who are motivated to get the job done.

Then trust them to do it.

1 For those who don’t understand the philosophical implications of Quality, I recommend reading Lila by Robert M Pirsig.

Share this post