In many organizations, “we’re agile” is all too often used as an excuse to not have any answers instead of an aspiration to come to the answers to the best of one’s knowledge and in an empirical way.
Deadlines and target/delivery dates are a particularly difficult topic among many agile practitioners. And, despite the fact that agile can be a perfectly good approach to work iteratively and incrementally toward a deadline, the general belief is that agile and deadlines can’t coexist.
Can agile have deadlines?
Many people think that agile projects can’t have deadlines. But that’s not entirely true. Agile teams can work toward target dates and hard deadlines. They simply do so in an iterative and incremental way — acknowledging these dates as boundaries and maximizing the delivery of value through customer feedback and measurement as they get closer to them.
Agile projects can have deadlines. Unlike waterfall projects, where the timeline, budget, and scope are fixed, a hard deadline is simply a target date by which the agile team will deliver a product or solve a problem. The agile team works on a product backlog and forecasts their delivery based on estimates and velocity.
Since 1995, research company The Standish Group has been publishing a report on the success rate of IT projects. The report is powered by data and insights from The Standish Group’s so-called CHAOS database of over 50,000 project profiles of IT projects.
The most recent report, which covered IT projects from 2013 and 2017, found that agile projects had a 42% success rate and an 8% failure rate, compared to a 26% rate and 21% failure rate for waterfall projects.
Successful projects were ones that were delivered on time, within the budget, and with the forecasted scope. Failed projects were canceled before they were completed, or were completed but not used. The projects in-between were neither successful nor failed, which means they were delivered with compromises in time, budget, and/or scope.
Just looking at the statistics, you come to one counterintuitive conclusion: between agile and waterfall, agile is the better project management approach to help you meet a deadline.
This happened to American farming machinery maker John Deere at the beginning of the previous decade. In September 2010, John Deere had a product launch planned for January 2011 — and its software development team realized that they were not going to be able to deliver must-have features for the launch without a major change in their ways of working.
At the time, John Deere was using waterfall project management, where the requirements were set in advance and programmers got to work to produce deliverables to meet milestones and deadlines, Computer World magazine reports. The magazine interviewed Tony Thelen, then-Director of John Deere’s IT operation, who said that the demands of their business required a faster pace of development.
John Deere had started experimenting with Scrum for a few teams in 2007 and had seen success, so management placed a bet on the Scrum framework in an attempt to save the troubled product launch. As then-Program Manager Chad Holdorf told InfoQ in a 2012 interview, “We broke down the functional barriers, organized into agile teams, introduced Scrum to 150+ people in one week, and went ‘all-in’ with agile on that program.”
The agriculture equipment maker’s bet on agile for the product launch was a success. John Deere embarked on an agile transformation and its agile journey continues to this day.
How Agile Can Work With Deadlines
Say that a Scrum Team is starting a project on February 1, 2021. They have 6 months to build a product, giving them a target date to complete their project on August 1, 2021.
The Scrum Team consists of a Product Owner, a Scrum Master, and 4 members of the Dev Team. All team members have the necessary skills, experience, and accesses/tools to build the product and will be working on the project full time.
The team agrees to work in two-week sprints. This gives them 15 sprints in total as there are 30 weeks and 2 days from 2/1/2021 till 8/1/2021. At the beginning of Sprint 1, they invest time to talk about the product vision and the problem that the product (and the project to build it) are trying to solve.
In the conversation, the Scrum Team uncovers the business goals that the company has for the product, the customer persona the product is for, and the job that the customers want to achieve by using this product and its alternatives. The team learns that the target date for this project comes from a budget constraint. At this stage, the company can only invest this much in the product, so making it easy and affordable to operate and support will be key.
The Scrum Team reflects on their learnings and incorporates them in their Definition of Ready (DoR), Definition of Done (DoD), and working agreements. They roll up their sleeves, create and estimate the first user stories in the product backlog, set up their access and tools, and spend some time building a technical backbone that will help them to work efficiently in the coming sprints.
The Scrum Team estimates user stories using story points, a relative estimate based on the Fibonacci sequence (0, 1, 1, 2, 3, 5, 8, 13, 21, …), and measures their pace using velocity, or the number of story points completed in a sprint. In the first 3 to 4 sprints velocity varies until the team achieves a sustainable pace of 21 story points/sprint.
During each sprint, the team builds working software that a small group of pilot customers can use and give feedback on. At the end of each sprint, the Product Owner holds a Sprint Review where the team demos the new version of the product, seeks out feedback from pilot customers and project sponsors — and talks about how their projected delivery as they get closer and closer to the target date.
The team also conducts a Sprint Retrospective, which allows them to practice continuous improvement. At the end of each sprint, the team discusses what went well, what could have gone better, and agrees on a set of small improvements that each team member can make to their ways of work over the course of the following sprint. Each improvement is small by itself. Cumulatively and over the course of time, these improvements help the team achieve an unrivaled customer satisfaction and employee engagement.
Halfway through the project, the Dev Team has achieved a sustainable pace of 21 story points/sprint and the Product Owner has refined a backlog of the highest-value features and enhancements. This allows the team to make strategic decisions and tactical trade-offs for what to build next and which features or enhancements to drop (because their original assumptions and the customer’s original requirements eventually change).
Close to the end of the project, the team has been building a product iteratively and incrementally for almost 30 weeks. The product is built based on feedback from the customers and in close collaboration with the project sponsors at each Sprint Review. Some of the original ideas for the product stuck, others were no longer valid by the time it’s almost complete.
From day 1 of the project, the target date was not an unmeetable or arbitrary deadline. It was simply a high-level timebox that helps the team and its stakeholders to focus on making better choices along the way. Every sprint resulted in working software, which allowed the team to collect customer feedback and continuously improve their ways of working.
By the end of the project, the customers are satisfied with the product and it’s built in a way that takes into account the business goals and organizational constraints. The product is scalable and requires little-to-no operations and support, which allows the company to invest more in marketing as it seeks out product/market fit — and decide where to take the product next.
Why Agile Projects Have a Higher Success Rate
In the early days of Scrum, co-founder Ken Schwaber collaborated with process control scientists at DuPont, one of the leading chemical companies in the world, to understand what made the framework work so well.
In the 20th century, DuPont experienced a number of disasters in its plants worldwide. After the Bhopal disaster, a gas leak that exposed 500,000 residents of a city in India to methyl isocyanate gas, DuPont hired the world’s best scientists to prevent problems of this magnitude from happening again.
The reason why Scrum worked so well, Schwaber found out, was in one of the first principles of industrial process control theory. According to industrial process control theory, there were generally two types of processes: predictive processes and empirical processes.
- Predictive processes were complicated but could be run like an assembly line. Every step of the process had a highly predictable outcome with little deviation. Predictive processes could be run with a predictive control system where every step, its inputs, and its outputs, could be planned in a linear and sequential way.
- Empirical processes were complex and their outcomes were difficult to predict. These processes, like mixing chemicals in a vat and exposing them to high temperatures to trigger chemical reactions, required close monitoring and continuous adjustment. Empiric processes had less predictable outcomes with high deviation. They required a different, empirical control system.
“In every case of an exploding chemical plant,” Scrum co-founder Jeff Sutherland recalls his and Schwaber’s learnings, the process control scientists had found “that somebody had applied a predictive control system to an empirical process.”
From the lens of industrial process control theory, software development is an empirical process. Research has shown that as many as 65% to 70% of a software development project’s original requirements would change throughout the course of the project. Yet waterfall is a predictive control system that assumes little to no change will happen.
Can Agile Projects Be Fixed Price?
Yes, an agile approach can be used for fixed-price projects. Though the fixed-price model of work negates some of the benefits of using agile (such as welcoming change at any stage of the project and collaborating closely with the customer), agile can nevertheless be a highly efficient approach for delivering a fixed-price project.
Case studies from the government sector show how some defense & aerospace contractors use agile approaches to bid and execute on government contracts with higher degrees of accuracy and profitability than their traditional peers who use a waterfall approach.
In a particular defense & aerospace software development company, an agile team would take the requirements from a Request for Proposal (RFP) document, break them down into a product backlog, estimate the effort to complete the work items on the backlog in story points, and use team size and velocity statistics from comparable projects and technologies to provide a bid.
So yes, agile can be used to deliver a fixed-price project. However, the nature of fixed-price projects constrains agility to a tool that can help teams and organizations achieve efficiency (build the product in the right way), and not necessarily effectiveness (build the right product and features in the first place), at scale.
The Bottom Line
Contrary to popular belief, agile projects can have deadlines. An agile approach to software development, however, requires leaders and teams to give a different meaning to the term “deadline.”
No longer is a deadline a date by which a project must be delivered with a specific scope and in an exact way. Instead, the deadline becomes a target date that helps the agile team make strategic decisions and tactical trade-offs on the scope and implementation of a project.
As the practice of some of the leading agile consultancies in the world shows, an agile approach can even be used to deliver fixed-price contracts. However, the predictive nature of fixed-price projects negates much of the value that organizations, teams, and individuals can achieve through agility.
