Software development is the art of transforming vague requirements into precise statements executible by a machine, resulting in a working software system. Almost by definition you can’t begin a project with perfect requirements; if you did so they would be executible or translatable on their own with no need for further development. The reason “Agile” development methodologies have persisted is that they are designed around this basic truth.
Introduction
Legend has it that in the (usually unspecified) past development used a “waterfall” methodology , meaning development flowed in one direction: Specifications, coding, testing, QA, release. Process heavy methodologies elaborated on this bframework, requiring elaborate diagramming tools, documentation style guides, rigid coding and test plans.
Then came the “Agile Manifesto” in 2002. The “Agile” principles were simply these
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
From these four bullet points a mini-industry was born.
You can see these principles are a reaction to the “waterfall” approach. In reality, there were always many styles of development since the beginning of the craft of computer programming. Much of the best, most impressive software was developed by one or two people not conciously following any methodology at all. Think of most games from the 70s up to the early 2000s. During the same period large engineering teams followed rigid, heavily planned schedules because that’s what large engineering firms do, and also because they were trying to be extremely careful not to make mistakes.
Nevertheless, by the end of the 1990s many developers on larger projects grew disillusioned with the state of software development. They felt the process heavy “waterfall” style was becoming too pervasive in the industry, and hence, the manifesto.
The Essence of Agile
Agile stresses short iterations. “Agile” comes in many flavors but they all have frequent iteration in common. The typical description boils down to a series of short cycles following some initial planning:
- Developers / project manager talks to customer to get requirements for a small feature – a small part of the larger product
- Developers build the feature
- Customer reviews the feature. Go back to step (1) if it isn’t right. , or pick a new feature to build and return to (1) with the new feature.
Particular brands of agile development introduce lots of ceremony and process around those three steps, to sell trainings and books, and to empower management. Developers have gotten pretty tired of the Agile methodology fads, process heavy “Agile” tools and practices, and the trainings and workshops. Yet, we haven’t abandoned the basic concept because it works,
Why Agile Works
Software development creates new knowledge and records implicit knowledge: How to make a playable game, how to simulate flow of air around a wing at supersonic speeds, or how to automate a business process, just for instance.
During the planning phase of any software project one thing is certain: The planners don’t know everything. Usually, they don’t know as much as they think they do.
As development proceeds gaps in knowledge are revealed. “What did the customer mean by that?”, “We didn’t know the customer had so many different customer databases!” “Nobody told us the reports needed to run in five seconds!”
It took actually trying to build a planned feature to uncover these issues. They will be solved, or the project is stuck. Getting unstuck requires getting or creating new knowledge. By the end of the whole project, many,many issues will have come up and gotten solved. Sometimes they get solved through investigation, sometimes through direct experimentation, often communication with the customer to get clarity.
These issues were always going to arise, it’s simply a question of when, and when the issues get resolved. Under the somewhat mythical “waterfall” approach, issues get filed away, show up in testing and QA and eventually get resolved if serious enough or the project fails. With the rigid schedule of a waterfall project in place, that means extending the schedule beyond the planned delivery date. So almost inevitably waterfall style projects are late and failure prone.
The truth is lots of software projects come in late, over budget, and some fail. Methodologies are only one part of the story. Methodologies exist to ensure quality and success, and are a means to control when the project completes.
The obvious difference with agile methodologies is that they respond to change quickly: The specify/build/demonstrate loop is all about responding to change. By iterating on small features agile methods ensure the cycle is short. The “change” in “responding to change” is usually “change in what we know or thought we knew.”
In the worst case, when using agile methods, by learning early on in a project that they have to adjust their plans, project managers can set customer expectations sooner than later. In the best cases, with any luck, the discovery of imperfect information early means nothing more than developers take a slightly different approach than expected and little time is lost. If they couldn’t adjust until late in the schedule, a lot of the existing code might need to be re-done.
To put the case more strongly,if you believe developing new knowledge is the essential work of developing software, than any methodology that rejects incorporating new knowledge early and often is, in essence, not a true software development methodology. It would be deferring or procrastinating the real job while re-arranging the desk. Perhaps it would qualify as an anti-development methodology.
Developing New Knowledge
We can refine the concept of “developing new knowledge” to the more narrow definition of “changing requirements.” Some of the changes come from users, some from the developers themselves.
Requirements fall into two buckets: The “what” and the “how.” The “what” is What the user imagines they’d like the software to do and look like; the “how” is how developers will accomplish this. The two aren’t perfectly partitioned though. If user requirements are vague enough many implementations would satisfy the requirements; but users may specify details such as “notifications must come by text” or “reports must include three dimensional data visualization in the browser.”, so that the “how” is constrained to a set of solutions.
During the course of a software project new information is uncovered that wasn’t accounted for in planning. It happens every time. Discoveries late in a project make the project later with little notice. These discoveries are inevitable, but the timing of these unfortunate discoveries is somewhat controllable. We should get feedback and incorporate it into planning early and often. That is the value of frequent iterations with customer feedback.
Changes to projects happen for a number of reasons:
- Refinements to initial user requested requirements After looking at demos users realize they forgot something important
- Discovery of miscommunicated requirements After looking at a demo users correct designs or software functionality misinterpreted by developers
- Unanticipated development due to less than perfect developer planning This always happens; sometimes the scope is trivial, sometimes monumental when it’s discovered late in the game that the wrong tool was chosen initially.
Unanticipated technical challenges
When requirements change, simplifying assumptions may be broken; implementation may turn out to be more difficult: We may be solving a harder problem.
Other times changes just result in doing a different thing. Often estimates rely on specific existing features of APIs or libraries and new features require non-existent library or APIs.
The Bottom Line
Course correcting early is easier than changing direction at the last minute.