Since the 1970s, waterfall was the de-facto project management approach. Until agile came along in the early 2000s and changed the way that individuals and teams build products or solve problems.
Some leaders are asking whether agile is just hype, whereas others are approaching the adoption of agile more practically. At the end of the day, both agile and waterfall are approaches to creating plans and delivering on them—even if they help organizations to achieve this in very different ways.
Which causes them to ask one question… Can agile and waterfall co-exist?
Agile and waterfall program/project management can co-exist. The reality in many organizations is that they already do. Agile is better-suited for projects with a high degree of change and uncertainty, whereas waterfall is best for projects where the scope and timeline are foreseeable and stable.
In this post, we’ll explore what are the key differences between agile and waterfall that cause this conflict in the first place—and what that means for initiatives, programs, and projects where choosing one, the other, or both is complicated to do.
The conflict between agile and waterfall comes from two key differences:
- Agile is an empirical approach to managing projects and waterfall is a predictive one;
- Agile prescribes an iterative and incremental approach to getting things done. Work in waterfall is done in a linear and sequential manner.
Before we talk about how agile and waterfall can co-exist, let’s take some time to see why that co-existence is a challenge in the first place.
Empirical vs. Predictive Process Control
The premise of waterfall is that a project team can determine what work needs to be done before the start of the project—and commit to delivering a specific scope by a specific date. Waterfall also assumes that the surrounding environment of the project team is going to allow them to protect the scope and date from change.
Agile is based on empiricism, the theory that knowledge can only come from experience. Empiricism began in the 17th century as scientists sought out to build knowledge by making hypotheses for how the world worked, then observing and measuring how it actually worked to prove these hypotheses right or wrong.
Since the agile movement started in 2001, to say that agile vs. waterfall is a heated debate is a major understatement. Depending on which camp you talk to, they’ll either tell you that one or the other is completely counterproductive for managing projects effectively.
I’m going to make the unpopular statement in both camps and tell you that there’s pros and cons to each. Which approach is more suitable for your organization and for your team will depend on a number of factors. Here’s why…
When talking about agile vs. waterfall project management, I often compare agile to a compass and waterfall to a map.
A map is pretty much useless in unexplored territory. When there’s no beaten path to walk on, you need a compass that helps you keep your course as you make your way through the unknown.
A compass is highly impractical for finding your way around in a crowded city. You need a map that allows you to navigate the boroughs, streets, and buildings.
Let’s see what happens when you apply the same reasoning to managing projects instead of navigating territories.
Waterfall is the better project management approach for projects where it’s clear what needs to be done—and that definition is not likely (or supposed) to change by the time the project is over. Waterfall works especially well in stable and regulated environments where what needs to be done and how it should be done are, to a large extent, determined in advance.
Agile is more suitable for projects where the vision, mission, goals, and/or objectives are clear, but the implementation approach and deliverables are not. This is especially true for volatile, uncertain, complex, and ambiguous (VUCA) environments that add a high degree of variability to how the outcomes are going to be achieved.
In the early days of Scrum, the most popular agile framework today, its creators Jeff Sutherland and Ken Schwaber collaborated with a group of industrial process control specialists at DuPont.
Industrial process control is the science of monitoring and controlling machines, systems, and processes at production plants across a large number of industries. Industrial process control is especially important in the chemicals industry, where the consistency and safety of the production process is often manufacturers’ number one concern.
Founded in Delaware in 1897, DuPont is one of the leading chemicals companies in the world.
In the 20th century, the company experienced a number of incidents in its chemical production plants, the biggest of which was the Bhopal disaster. On the night of 2-3 December 1984 at a pesticide plant in Bhopal, India, a gas leak caused an explosion that killed 3,787 victims and exposed 500,000 local residents to methyl isocyanate gas.
After the Bhopal explosion, DuPont hired some of the world’s best industrial process control specialists to prevent disasters of this magnitude from happening again.
According to the first principles of industrial process control theory, there are generally two types of processes: predictive processes and empirical processes:
- Predictive processes were complicated, but could be run as an assembly line. Every step of the process had a highly predictable outcome with little deviation. Predictive processes could be run with a predictive control system where every step, its inputs, and its outputs, could be planned in a linear and sequential way.
- Empirical processes were complex and their outcomes were difficult to predict. These processes, like mixing chemicals in a vat and exposing them to high temperatures to trigger chemical reactions, required close monitoring and continuous adjustment. Empiric processes had less predictable outcomes with high deviation. They required a different, empirical control system.
“Back in the early days when you think about Scrum, we were told that the leading process experts in the world were at DuPont—and they were hired after Bhopal to stop chemical plants from exploding,” Jeff Sutherland recalls in a talk about Scrum at GOTO 2011.
So Scrum co-founder Ken Schwaber went down to the DuPont headquarters and talked to these specialists. These specialists told Schwaber about the difference between empirical and predictive processes, and why each process type required a highly specific control system.
“They had two types of process. They had what we call predictive processes you could run like an assembly line. (…) But there were other processes in a chemical plant where you put chemicals in a vat and the vat bubbles and boils—and if you don’t watch that vat, the whole chemical plant could blow up. So you need a different control system; an empirical control system.”
In every case of an exploding chemical plant, somebody had applied a predictive control system to an empirical process.
Jeff Sutherland
The industrial process control specialists then asked Sutherland and Schwaber what kind of control system waterfall project management was, and what kind of process software development was.
Waterfall project management is a predictive control system. Software development, however, is more often than not an empirical process. Studies show that as many as 65% of a software project’s original requirements change throughout the course of the projects.
Iterative/Incremental vs. Sequential/Linear Approach
The second source of conflict between agile and waterfall is in how work is done using each of these two approaches.
Agile is iterative and incremental. Iterative because agile teams work in short cycles, usually 1 to 4 weeks long (a.k.a. iterations). Incremental because they build the products in small chunks and deliver working software at least once per cycle (a.k.a. product increments). There is no final product, only version 1, 2, 3, …, n.
Waterfall is sequential and linear. Sequential because a waterfall team works in sequences (a.k.a. phases). Each sequence inherits inputs from its predecessor and provides outputs to its successor, usually through handovers. The project is scoped from the start and the final product is only completed once the project is finished.
Clearly, there’s a big difference in the way things are done in an agile vs. a waterfall project.
The iterative and incremental approach of agile allows the project team to welcome change at any stage of the project. An iteration starts with a goal, for example “Enable credit card payments with Stripe on the checkout page.”
The project team makes a plan for how to achieve that goal in the iteration by committing to specific work items on the product backlog, usually in the form of user stories. Then, it sets out to build a new version of the product that delivers the functionality that’s needed. As soon as the iteration is over, plans are flexible and can change as needed.
The sequential and linear approach of waterfall requires the team to protect themselves from change at every stage of the project. The project starts by collecting requirements and defining what needs to be done, usually in the form of a Software Requirements Specification (SRS) document and a project plan laid down on a Gantt chart.
The team then begins to build the full and final product step by step, in separate phases. Most waterfall software development projects tend to have an “Analysis,” “Design,” “Development,” “Testing,” “Deployment,” and “Maintenance” phase.
Can Agile and Waterfall Co-Exist?
When wondering whether to take an agile or a waterfall approach to a project, program, or initiative in your organization, consider these two questions:
- Is it clear and known what needs to be done?
- Are the requirements expected to change or to stay the same?
- Does a specific project scope need to be committed by a specific delivery date?
Use the answers to these three questions as the result of a litmus test that can help you gauge if the situation requires an empirical (agile) or a predictive (waterfall) process control system.
When it comes to getting the two to co-exist, the reality in many organizations is that they already do.
Just like chemical companies use empirical process control systems for parts of their production process and predictive process control systems for others, certain aspects of your company’s operations will require certain types of management systems in place.
The question that people like you and me should really be asking is, “Where is the middle ground between the two?”
An agile team can work toward goals, objectives, and target/delivery dates, as long as these dates are simply timeboxes—and don’t represent commitments to deliver specific scope.
An agile team can also estimate their work in story points (a relative estimate based on the Fibonacci sequence) and measure their throughput in velocity (the amount of story points of completed work in an iteration; where “completed: means work that meets the team’s Definition of Done).
Contrary to popular belief, an agile team can also do forecasting, as long as everyone in the organization that relies on their forecasts understands that they are indicative. Forecasting is less accurate in newly-formed teams and freshly-started projects, where the level of ambiguity and degree of change are high—and more accurate in seasoned teams on mature projects, whose scope and environment have achieved a certain level of pace and stability.
Similarly, a waterfall project team can keep their scope small and collaborate closely with the customer.
In doing so, they can keep the sequential and linear way of getting things done while becoming more responsive to change by shortening the feedback loop. Collaborating closely with the customer helps to ensure that the team is working on the right goal/s and work item/s.
The idea that agile and waterfall can, in a way, be combined has given birth to “hybrid agile.”
Depending on who you ask, hybrid agile either takes the best or the worst from both worlds and merges them in a semi-linear, semi-iterative approach where:
- A plan for a long timebox (usually several quarters or years, with a preference for the shorter timescale) is made in a linear and sequential way;
- The plan is delivered using an iterative and incremental approach.
Some organizations use hybrid agile as the first step to transition to agile. Simply said, hybrid agile could be the set of “training wheels” that your company needs to take the leap to strategizing and delivering in agile ways.
Other organizations use hybrid agile to bid and deliver on fixed-price projects in an agile way. This approach is common among government contractors who need to determine the scope of their projects before they start, but want to benefit from the efficiency gains of agile delivery toward a known product backlog.
I’ve heard stories of organizations that run waterfall and agile projects in parallel to deliver complex programs. Every now and then, the timelines of these projects meet in the form of milestones/deadlines for waterfall and target/delivery dates for agile. As long as the waterfall teams communicate their critical-path dependencies as highest-value and the agile teams work on them first, a small number of people I know claim that this hybrid system can work really well.
The Bottom Line
Can agile and waterfall co-exist?
Whether die-hard fans of the Project Management Book of Knowledge (PMBOK) and agile purists who swear by the Manifesto for Agile Software Development like to admit it or not, agile and waterfall co-exist in many organizations today.
The questions that organizations, leaders, teams, and individuals should be asking instead is, “What makes that necessary?” and “How can we make this work?” There’s no one-size-fits-all approach to organizational management, and this level of introspection will help you determine how to approach solving the challenge for your organization’s unique and specific case.
