Why Agile Goes Awry — and How to Fix It

In the spirit of becoming more adaptive, organizations have rushed to implement Agile software development. But many have done so in a way that actually makes them less agile. These companies have become agile in name only, as the process they’ve put in place often ends up hurting engineering motivation and productivity.

Agile software development

Frameworks for adaptive software development like Agile, have been around for a long time, and have manifested in many forms. But at the heart of most of these models are two things: forming hypotheses (e.g., what is a feature supposed to accomplish) and collaborating across domains of expertise on experiments, all in the spirit of driving learning and not careening down a path that proves to be incorrect.

When Agile software development was born in 2001, it articulated a set of four critical principles to elevate the craft of software development and improve engineering and product manager motivation.

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Over the last three years, in our research on human motivation, we have analyzed the practices of engineers across over 500 different organizations using a combination of survey-based and experimental approaches. We’ve found that what happens in practice wildly departs from these stated principles.

For example, in common practice, processes and tools have become the driver of work, not individuals and interactions. In one large Fortune 100 company, the head of digital products said to us, “we’re not allowed to question the Agile process.” In another Fortune 500 organization, product managers and engineers communicate exclusively through their tools, which are used primarily for the former to issue commands to the latter.

Similarly, documentation often trumps working software. In one large tech company, their product team focused significant upfront time writing small requirements (called “user stories”). These requirements were put into a ticket queue as tasks for the next available engineer to start working on. The bar for documentation to keep the queue moving became high. Ultimately, this process became one of many small “waterfalls,” where work is passed from a product department to designers to engineering. This process is exactly what Agile was meant to eliminate. It is no wonder that the CTO of this company said, “my engineers feel like short order cooks in the back of a diner.”

When it comes to “responding to change over following a plan,” this often gets misinterpreted to mean “don’t have a plan.” For example, in one fast growing tech company, the Agile teams did not try to understand the broader strategy of the organization. As a result, their attempts to iterate often focused on low-value or strategically unimportant features. Without a plan, teams won’t know how to prioritize actions, and how to invest in those actions responsibly. This principle has gone so far as to let engineers believe that it is not appropriate to have timeboxes or common milestones.

It would be one thing if these misapplications actually improved engineering motivation and performance, but we have found that in practice, the opposite happens. Agile, when practiced as described above, reduces the total motivation of engineers. Because they’re not allowed to experiment, manage their own work, and connect with customers, they feel little sense of play, potential, and purpose; instead they feel emotional and economic pressure to succeed, or inertia. They stop adapting, learning, and putting their best efforts into their work.

For example, one venture capital partner shared with us a story of how a video game development company continued to build a product for a year, despite every engineer feeling like the game was not worth playing. The company realized they wasted a lot of time and money.

Agile processes go awry, because as companies strive for high performance, they either become too tactical (focusing too much on process and micromanagement) or too adaptive (avoiding long-term goals, timelines, or cross-functional collaboration).

The key is balancing both tactical and adaptive performance. Whether you’re an engineer or product manager, here are a few changes to consider to find this balance, so you can improve your engineering (or any) team’s motivation and performance.

1. Software development should be a no-handoff, collaborative process.

Rather than a process where one person writes requirements (even small ones) while another executes them, all without a guiding strategic north-star, a team striving for true agility should have a no-handoff process versus a process where one person writes requirements while the other executes them. In a no-handoff process, the product manager and the engineers (and any other stakeholders) are collaborative partners from beginning to end in designing a feature.

First the team, including executives, should articulate the team’s strategic “challenges.” Challenges take the form of a question, always focused on improving some kind of customer outcome or impact. Think of them as a team’s detailed mission in question-form to trigger expansive thinking. The challenges themselves are developed and iterated by the whole team, including its executive sponsors (and customers). Every single person on the team (or any team for that matter) is asked to contribute ideas to each challenge whenever they want.

For example, in one bank, a challenge was, “how can we help customers be better prepared for possible financial shocks?” Another was, “how can we make it more fun and less of a chore for customers to maintain healthy financial habits?” These challenges produced dozens of ideas from many different people.

Then, instead of someone writing requirements while another person executes, these teams develop and mature an idea collaboratively, from rough draft to testable hypothesis.

2. The team’s unit of delivery should be minimally viable experiments.

Teams often find they waste time by adapting too much. To avoid this, not only should ideas be formed for a strategic challenge, but they should also be executed with fast experiments aimed at learning just enough to know what works for customers. In other words, they should be maximizing their “speed to truth.”

In order to reduce wasted effort and increase the team’s decision rights, experiments should be short in nature. If possible, an experiment should be no longer than a week.

Sometimes this requires the team to minimize a feature to what is absolutely needed to test its weakest assumption. Sometimes it means that the team doesn’t code but instead completes an “offline” experiment through research.

3. The team’s approach should be customer-centric.

The process of building software (even internal-use software) should be squarely customer-centric.

At the simplest, these principles should hold:

“Challenges” are always framed around customer impact.
Problem solving meetings always start with a customer update, and representatives from the frontline are included frequently in these discussions.
Every experiment is built around a customer-centric hypothesis. That way, the team can hold themselves accountable to the outcome predicted by the experiment.

However, even more important is that engineers see with their own eyes how customers use their products. This requires the frontline and the engineers working together to see if the product is creating customer impact.

4. Use timeboxes to focus experimentation and avoid waste.

Interestingly, adaptive software development encourages timeboxes as a way to ensure an experiment is given the investment that is justified and to signal the acceptable quality level of given feature. On the other hand, typical Agile practitioners avoid timeboxes or deadlines, for fear that the deadline will be used to create emotional pressure. One of the worst feelings for a software developer is spending a few months working on something that ends up being not useful. This fills you with emotional pressure (“I let everyone down”) and a sense of inertia (“why am I even doing this?”).

To avoid this outcome, you want to be clear on how far an engineer should go before they check to see if the direction is still correct. The greater the uncertainty on a team’s hypothesis, and the greater the risk, the shorter that runway should be. With that in mind, the timebox isn’t a deadline. It is a constraint that should guide the level of depth and quality for an experiment before a real test. In this way, timeboxes can increase total motivation.

5. The team should be organized to emphasize collaboration.

To make sure you end up with a no-handoff process, the various stakeholders involved should function as a single cross-functional team, also known as a pod. The goal of the pod is to drive collaboration. Each pod should contain the full set of experts needed to deliver a great product. This may include senior executives. In one organization, for example product pods include a product manager, front-end engineer, back-end engineer, designer, a quality engineer, and part-time representation from customer service, and a senior executive from a control function.

In many organizations, there are tell-tale signs of “faux pods” — teams that call themselves pods but don’t actually operate that way. Signs of faux pods include:

Experts are in separate “aligned” teams, not the same team. For example, a product team has dedicated engineering “sprint teams.” These are not pods.
The team uses tools that prevent real collaboration. For example, while asking one engineering team why they chose the Agile software tools they are using, they said, “these tools will prevent executives from engaging in our work.” All this does is perpetuate a cycle of mistrust.
Engineering and Product functions actually have different goals from the top. Executives in both functions use their hierarchical power to get their people to prioritize the function’s goals above all others, including their pod’s goals. These conflicts ultimately result in clashes in the working teams that prevent true teamwork.
Rigidly hierarchical talent processes, like performance ratings, hierarchical titles, pressure to get promoted, and up-or-out systems destroy the teamwork required to make pods function well. These systems will either make team members more beholden to their boss than their team’s customer or they will put team members in competition with each other. Either way they will not function as a team.

Put differently, the stronger an organization’s silos, the more people will solve for the needs of their silo, versus the needs of their team. This makes collaboration and consensus very difficult to achieve without constant escalation.

6. The team should constantly question their process.

A famous maxim of engineering design is known as Conway’s Law. It states: any organization that designs a system will produce a design whose structure is a copy of the organization’s communication (i.e., process) structure. In other words, if you’re a monolithic organization, you’ll produce monolithic designs. If you’re organized by user segments, your product will optimize for that structure.

If you want to defeat Conway’s Law, the better practice is to constantly adjust your structure and processes to suit the problem at hand. This requires teams that have simple, lightweight processes and structures that they constantly question and tweak.

Thus, rather than building “Agile” as a religion that cannot be questioned, engineering teams should be in the habit of constantly diagnosing and iterating their own team’s operating model. In the best examples we’ve seen, on a monthly basis, teams diagnose their operating model and decide if it needs changing to produce a better product.

***

The ability to attract, inspire, and retain digital product talent is becoming mission critical for organizations. Most organizations have fallen prey to a simple message — implement Agile as a series of ceremonies and everything gets better. Unfortunately, this is often not the case when the human-side of the equation is lost. By getting back to the basics of motivation and adaptive performance, you can build an organization that is truly agile.