Here’s an interesting question that I got asked recently: “Can agile and Scrum be used interchangeably?”
Based on how much you know about agile and Scrum, and who you’re reading, listening to, or watching, the difference between the two might be confusing. But, by the end of this post, it’s going to be crystal clear for you.
Agile and Scrum can’t be used interchangeably because agile is a mindset and Scrum is an agile framework, or one of the ways in which organizations, teams, and individuals can practice agile. You can therefore adopt an agile approach to building products and managing projects without using the Scrum framework.
Agile is a mindset that consists of 4 values and 12 principles as described in the Manifesto for Agile Software Development (agilemanifesto.org).
The agile mindset—along with its 4 values and 12 principles—gives birth to an unlimited number of agile practices, like working in small teams, shipping in short cycles, seeking out feedback early, welcoming change at any stage of the project, and many others.
Agile frameworks, like Scrum and Extreme Programming (“XP”), are sets of roles, events, and artefacts that make it easier for organizations, teams, and individuals to work in agile ways. Agile frameworks make the proliferation and adoption of agile repeatable and scalable throughout the organization.
When coaching individuals and teams, I often compare agile to the act of playing games.
Agile practices are similar to the best practices that any player can use to play any game, like knowing the rules, observing other players, and making calculated bets along the way.
Agile frameworks resemble the different types of games out there, like football, soccer, or basketball. Subsequently, the organizations that promote the adoption of these games and govern their rules, like NFL, FIFA, and the NBA are not that different from the Agile Alliance and the Scrum Alliance.
Can you play a game if you don’t play football? And can you practice agile without using Scrum? When you look at it from that point of view, the difference often becomes clearer.
What Is Agile?
Agile is a mindset and approach to developing products in an iterative and incremental way. Iterative as work is done in short cycles called “iterations” and incremental as each iteration results in one or more chunks of working product called “increments.”
Organizations, teams, and individuals adopt agile ways of working to become more competitive, innovative, and adaptive. Working in short cycles and shipping small chunks of products frequently allows them to seek out feedback early and continuously improve their ways of working, which are one of the biggest benefits of an agile approach.
The agile movement began in 2001, when a group of 17 software development thought leaders got together on a ski retreat in the mountains of Snowbird, Utah. As the story goes, a blizzard came and the members of the group got snowed in. With little skiing to do, they started talking about how each built software and managed projects with their clients instead.
The group found out a surprising amount of similarities in their ways of working. Most of them were based on the principles of lean manufacturing as well as a new approach to customer collaboration and product development that some of the leading product companies in the world, such as 3M, Canon, Brother, and Fuji-Xerox, had used to become highly performant in the past two decades.
The group wrote these similarities down on a whiteboard, creating the Manifesto for Agile Software Development (or the “agile manifesto” as most people know it today). The manifesto is a 68-word document that describes the four values of agile software development:
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Agile is grounded in the principles of empiricism. Empiricism is a school of thought in philosophy which claims that the only way to obtain knowledge is through “sensory experience.” An empiricist knows that the only way to know if a plan is going to work is to go out and try to make it happen—and observe what actually happens as an outcome.
This is why agile is all about small teams, focused on customer value, working in short cycles, and shipping products in small chunks, so that they can seek out customer feedback as early and as often as possible, learning lessons and deciding what to do next based on the lessons learned. Agile is the art of maximizing value in a complex world.
The day after the group wrote the Manifesto for Agile Software Development, the blizzard had passed by. So the group’s members went on to the outside activities that brought them there in the first place, like skiing. Later that they, they got back together and expanded the manifesto by describing 12 principles behind it:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity–the art of maximizing the amount of work not done–is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Agile started out as a movement among software developers—and quickly spread, across large companies and fast-growing startups, as the de-facto way to build products and manage projects. Over the course of time, agile spread beyond software development teams and the IT function and has now seen adoption in virtually any industry and department.
In recent years, agile was picked up by some of the world’s leading consulting firms, like McKinsey & Co, Bain & Co, and Boston Consulting Group, and entered in corporate boardrooms worldwide. Many companies are right now going through agile transformations as they eliminate hierarchical structures and create networks of agile teams that self-organize toward a common vision, mission, and goals.
At its heart, agile remains a mindset that can help anyone in any line of work keep things simple, stay open to change, focus on delivering value, and continuously improve their ways of working.
What Is Scrum?
Scrum is an agile framework for developing and maintaining products.
One thing about Scrum that surprises most people I talk to is that Scrum is an agile framework, but it predates agile. Scrum was created in the 1990s by Jeff Sutherland and Ken Schwaber, who are also two of the 17 co-signers of the Manifesto for Agile Software Development.
Scrum is an agile framework that individuals, teams, and organizations can use to build products of any kind and to solve complex problems in an iterative and incremental way. Scrum helps its practitioners to focus on customer value, work in short cycles, ship in small chunks, and continuously improve.
Scrum is described in the Scrum Guide (scrumguides.org), a 14-page document that’s intentionally lightweight and purposefully incomplete. Every now and then, Jeff Sutherland, Ken Schwaber, and their teams at the Scrum Alliance, Scrum.org, and Scrum Inc. make updates to the guide based on their lessons learned from case studies of companies and teams using the framework.
In Scrum, there’s a Scrum Team. The team consists of a Scrum Master, a Product Owner, and Developers. One thing that most people misunderstand about Scrum is the meaning of the word “developers.” Scrum is not a software development framework per se, it’s a product development framework. In that sense, a developer is any team member who is developing the product.
A Scrum Team consists of at least 3 and at most 9 members (this number includes the Scrum Master and the Product Owner). The Scrum Team works in iterations called “sprints,” shipping chunks of a product called “increments” during each sprint. In Scrum, the Scrum Master fosters an environment where:
- The Product Owner orders the work into a product backlog.
- The Scrum Team turns a selection of the work into a product increment.
- The Scrum Team and its stakeholders (sponsors, customers, users) inspects the results and adjusts for the next sprint.
At the start of each sprint, the Scrum Team has a Sprint Planning session. The Product Owner outlines a sprint goal and explains to the team why the sprint that’s about to start is valuable to the team, the organization, their customers, and their broader set of stakeholders.
The team then has a conversation about which work to pull from the product backlog into a sprint backlog based on their understanding of the sprint goal and their past performance, upcoming capacity, and Definition of Done (“DoD”). Once a sprint backlog is agreed on, the work begins.
The Developers self-organize and meet every day at the Daily Scrum. Each of them discusses what they did yesterday, what they plan to be doing today, and whether they’re facing any organizational impediments that they need removed to complete the sprint goal.
As the team works throughout the sprint, they convert work items on the sprint backlog into product increments. An increment is a stepping stone toward the completion of the sprint goal. It can be a feature, a change, or anything else that contributes to the team’s achievement of it.
At the end of the sprint, the team presents the results of their work to their stakeholders in a Sprint Review. This usually means demoing working software and getting feedback from their stakeholders, so that the team can decide on what goal to focus on for the next sprint.
Finally, the Scrum Team has a Sprint Retrospective. It’s conducted toward the end of the sprint and concludes it. During that meeting, the team inspects how their sprint went with regards to individuals, interactions, processes, tools, and their Definition of Done (DoD).
They agree on small and concrete actions to continuously improve their way of work for the next sprint. Over time, these small changes contribute to significant improvements in effectiveness (working on the right things) and efficiency (doing things in the right way).
Where Agile and Scrum Come From
Agile and Scrum are essentially founded on empiricism and lean.
In the early days of Scrum, Ken Schwaber collaborated with process control scientists at DuPont, one of the leading chemical companies in the world, to understand what made the framework work so well.
In the 20th century, DuPont experienced a number of disasters in its plants worldwide. After the Bhopal disaster, a gas leak that exposed 500,000 residents of a city in India to methyl isocyanate gas, DuPont hired the world’s best scientists to prevent problems of this magnitude from happening again.
The reason why Scrum worked so well, Schwaber found out, was in one of the first principles of industrial process control theory. According to industrial process control theory, there were generally two types of processes: predictive processes and empirical processes.
- Predictive processes were complicated, but could be run as an assembly line. Every step of the process had a highly predictable outcome with little deviation. Predictive processes could be run with a predictive control system where every step, its inputs, and its outputs, could be planned in a linear and sequential way.
- Empirical processes were complex and their outcomes were difficult to predict. These processes, like mixing chemicals in a vat and exposing them to high temperatures to trigger chemical reactions, required close monitoring and continuous adjustment. Empiric processes had less predictable outcomes with high deviation. They required a different, empirical control system.
“In every case of an exploding chemical plant,” Jeff Sutherland recalls his and Schwaber’s learnings, the process control scientists had found “that somebody had applied a predictive control system to an empirical process.”
Sutherland and Schwaber concluded that software development (and most product development projects, software or non-software) was a complex process that required an empirical control system.
Building products is not a predictable process. You can hypothesize that a feature is going to create value for your product’s customers, but the only way to verify your hypothesis is to get that feature into the hands of these users—and determine through feedback and measurement whether it was true.
This empirical mindset and open-ended approach to building products was the topic of a research paper that, to a large extent, informed Sutherland and Schwaber’s definition of Scrum.
In 1986, Harvard Business School professor Hirotaka Takeuchi and organizational theorist Ikujiro Nonaka published an article in Harvard Business Review called “The New New Product Development Game.”
The article looked at a new approach to product development, which leading companies in Japan and the United States at the time were using to innovate quickly and bring products to market faster than their competitors. The list included 3M, Fuji-Xerox, Honda, Canon, NEC, Epson, Brother, and Hewlett-Packard.
“From interviews with organization members from the CEO to young engineers,” Takeuchi and Nonaka wrote, “we learned that leading companies show six characteristics in managing their new product development processes.”
These characteristics included:
- Built-in instability
- Self-organizing project teams
- Overlapping development phases
- Multilearning
- Subtle control
- Organizational transfer of learning
Built-in instability meant that the project teams at these companies would rarely work on a clear-cut project plan given to them by management. Instead, they would be tasked with complex problems and challenging timelines that forced them to rethink the status quo and find new solutions to existing or unsolved customer needs.
“Top management creates an element of tension in the project team by giving it great freedom to carry out a project of strategic importance to the company and by setting very challenging requirements,” the article said. In all organizations that Takeuchi and Nonaka studied, the project teams acted like small startups and companies within the company. They took initiative and risk and drove their own plans and agendas.
The teams were self-organized in three ways: they were autonomous, self-transcendent, and cross-fertilizing.
Autonomous because the involvement of management was only to the extent of giving guidance, money, and moral support (and not providing approaches and solutions).
Self-transcendent in a way where the teams kept challenging their own beliefs for what was possible, often helping them to achieve multiple technological breakthroughs in a single project.
Cross-fertilizing as the teams were cross-functional and consisted of members with various backgrounds, specializations, thought processes, and behavior patterns that allowed them to communicate and collaborate on a broad set of ideas.
Takeuchi and Nonaka found that the project teams playing the “new new product development game” would, over time, achieve a unique dynamic or rhythm. “Although the team members start the project with different time horizons with R&D people having the longest time horizon and production people the shortest—they all must work toward synchronizing their pace to meet deadlines.”
“As a result,” the Japanese management theorists concluded, “the team begins to work as a unit. At some point, the individual and the whole become inseparable. The individual’s rhythm and the group’s rhythm begin to overlap, creating a whole new pulse. This pulse serves as the driving force and moves the team forward.”
The notion of a dynamic or rhythm that Takeuchi and Nonaka described in 1986 is very familiar to the idea of iterative and incremental work that agile teams work toward today. One of the first tells that a team is getting agile right is when they achieve a sustainable pace of delivery, usually measured as velocity, or the amount of work in the form of story points completed per sprint.
Over the course of time, mature agile teams come to a reasonable accuracy of estimation and a sustainable pace of delivery that allows them to become good at product roadmapping and measuring progress toward target/delivery dates—while responding to change and adjusting to customer feedback and measurement insight.
An important conclusion from Takeuchi’s and Nonaka’s article is that the project teams at these high-performing product companies worked in overlapping development phases. Just like waterfall software development, the product management approach used by the majority of product firms in the 1980s was linear and sequential.
Multilearning meant that teams stayed connected to their surrounding environment and responded quickly to changing market conditions. Similar to the concept of empirical process control that Ken Schwaber and Jeff Sutherland discovered from their conversation with DuPoint scientists, Takeuchi and Nonaka noticed that the project teams at the product companies that they studied learned primarily by doing.
The management teams at these companies also exercised “subtle control.” Though project teams were self-organized and acted largely on their own, they were not left uncontrolled. In most companies that the Japanese management theorists studied, management had “established checkpoints to prevent instability, ambiguity, and tension from turning into chaos.”
Subtle control was practiced through selecting the right team members for project teams in the first place, creating an open work environment, encouraging engineers to go out into the field and listen to what customers had to say, and establishing a rewards and incentives system based on group, and not just individual, performance.
Last but not least, the project teams and the organizations that they were part of practiced rigorous organizational transfer of learnings. The individuals and teams used knowledge from previous projects to new product development initiatives. They took their lessons learned and helped other functions and business units to apply them for solving their own unique challenges. And they turned individual knowledge into organizational know-how by creating playbooks and standard practices.
