Incremental and Iterative Development

In my last few posts (here and here), I have been talking about Iterative and Incremental Development (IID). This is where the solution under development is built in small chunks or increments. These chunks are integrated and reviewed for feedback on fit with the customer need and quality of the product. This feedback informs future develop as we iterate through the solution. Development teams have been applying IID for over 50 years. IID calls for progressive elaboration in support of iterative and incremental delivery in the context of delivering value. It helps reduce product risk by facilitating change associated with learning throughout the project. In this way IID delivers on two values of the Agile Manifesto – working software and responding to change.

What Makes IID Agile

The Agile Manifesto also identifies two more values – individuals and interactions and customer collaboration. These are social aspects of software development. And while good teams have probably been doing this as long as IID has been around, Agile explicitly calls these out. Now the team is continually getting feedback on the viability of the product and capabilities of the development team. So in addition to IID, Agile drives transparency in the project status, viability of the product, and performance of the team. It also compels collaboration and learning about both the product and the delivery capabilities of the team.

But Wait, There’s More

In addition to addressing the four values of the Agile Manifesto, the team has to be developing using effective techniques. This means using automated testing, continuous integration, effective design and refactoring. In my CSM class, Jeff Sutherland stated that he had seen no hyper-productive development teams that were not applying these Agile engineering practices. These practices enable the team to reliably deliver on the four values from the Agile Manifesto.

Agile is more than Iterative and Incremental Development

So, Agile is more than IID. In addition to the ongoing product feedback from IID, Agile calls for ongoing feedback on the capabilities of the team, the viability of the product, and the knowledge developed by team members. And that is what is “new” in the last 15 years. All five areas of focus are explicitly called out. Teams that are moving to Agile or struggling with Agile adoption should look at each of these areas of focus:

• Progressive elaboration in support of iterative and incremental delivery in the context of delivering value
• Reduce risk by facilitating change associated with learning throughout the project
• Drive transparency in the project status, viability of the product, and performance of the team
• Compel collaboration and learning about both the product and the delivery capabilities
• Enabling software development engineering practices

If these areas of focus are not in place, or they don’t interoperate together to continually improve the performance of the team, the team will probably not be achieving the benefits of Agile.

Last week I wrote a post on Iterative and Incremental development. In that post I pointed out that you can do iterative and incremental development without having time-boxed iterations. Mike Cottmeyer asked in a comment to clarify when time-boxed iterations made the most sense and when continuous flow made the most sense. This post answers that question.

When Time-boxed Iterations Make the Most Sense

Time-boxed iterations serve several purposes that are not well served in continuous flow. They create an opportunity for the team to stop and get feedback from the customer on their product.  They can help with chunking up work to support the planning process and validation of the design of the solution. There are times when this is really important.

So, time-boxed iterations work best:

1. When there is a lot of opportunity for feedback on current deliverables to significantly update subsequent work

2. When the planning horizon is stable enough that a subset of the backlog can be prioritized for at least the duration of the iteration

3. When the backlog items can fit in the iteration time-box

4. When dependent resources (for example, planning and testing) can not be available to support continuous flow

For example, time-boxed iterations work best at the outset with a new product, new technology, or substantially new set of features.

When Continuous Flow Makes the Most Sense

Continuous flow has some advantages over time-boxed iterations. Overtime, the level of resources available to do work is higher and the backlog is much more flexible and adaptive to variation in the arrival of work. For example, once the product architecture is stable and well defined and work is arriving in bursts with various levels of urgency.

So, continuous flow works best:

1. When the work is arriving continuously

2. When the planning window can be (or must be) very small

3. When the outcome is pretty clear and feedback is not as likely to impact subsequent work

4. When there are multiple work streams operating at different cadences

For example, continuous flow when working on defects and small enhancements in a stable product or working with technical support issues.

Non-relevant factors

Effective planning. Data can be gathered from both iterations (velocity) and continuous flow (lead time) to determine when a product can be delivered and to establish visibility into the current status of the project.

Retrospectives. Iterations create a natural break for retrospectives. There is no reason why retrospectives must be tied to iteration releases. Continuous flow teams can have retrospectives on either an event driven or calendar driven basis.

Motivation to get to done. Iterations create pressure to complete work in a specific time-box. Continuous flow creates pressure to met lead-time commitments (policies).

How they can work together

We are making a standard practice of combining Kanban at the strategic planning level to support continuous flow of strategic initiatives across the organization and Scrum (or Kanban) at the development team level. It doesn’t make sense to us to create time-boxed deliverables at the strategy execution deliverable level. We generate plenty of pressure to move forward from the time commitments on the tasks on the Kanban board. At the development level – when there is a lot of opportunity for feedback to update subsequent work – we (may) use iterations to keep the team focused. For defects and minor enhancements we continue to use a continuous flow model – maybe within the same team.

Iterations versus Continuous Flow?

I believe this is a hammer versus saw discussion. Each approach is suited to fit different scheduling and feedback needs. In many cases, time-boxed iterations make a lot of sense but would result in a less than optimized approach then in other cases. I believe that early on in a product (when there is a lot to be learned and a lot of risk to manage) iterations probably make the most sense. At some point, when the product has switched primarily to support, continuous flow makes the most sense. I hope this answers Mike’s question from my previous post.

One of the key parts of Agile Software Development is the concept of iterative and incremental development (IID). I here people say this frequently – but when I ask more specific questions I hear things like, “We we develop small chunks and we have two week iterations.” We there is an entire group of Agile methods becoming mainstream that are iteration-less – instead they follow a continuous flow model. So if Agile requires IID, and Lean and Kanban are iteration-less, does that mean they aren’t Agile. Well, no, because iterative development doesn’t mean you are doing iterations.

In incremental development various parts of the system are developed at different times or rates, and integrated as they are completed. You can do this in a waterfall project or an iterative project. The alternative to incremental development is to develop the entire system with a “big bang” integration.

In iterative development the team plans to iterate, or go back over parts of the system, to revise and improve the system. In iterative development testing and/or user feedback is used to revise the targets of the successive deliverables. You don’t have to have iterations to be doing iterative development. The practice of iterations arises from a desire to coordinate feedback from increments with revising a future deliverable. Iterative and Incremental Development work well together. A hard stop, or a time-boxed iteration, has historically been useful to coordinate feedback with deliverables.

One of the powerful and interesting things that has happened in the last decade in software development is Continuous Integration. This is a practice that arose from Martin Fowler and Kent Beck in the Extreme Programming community. Continuous Integration means we can deliver increments of software at any time. I have worked with a team whose continuous integration environment was so robust they were prepared to responsibly deliver updates to production several times a day. When you can continually deliver working software then you can eliminate the hard stop. Just perform the user feedback of the application whenever it makes sense. You can use this feedback to revise the successive deliverables without creating a hard top for everyone on the team.

No iterations in IID?

There are several benefits of going iteration-less. A big benefit of single piece flow is not stopping the entire team every week or two weeks. Reducing batching to a single item results in an increase in the amount of time development can be focused on writing code. For example, if the entire team has to stop for two days (one for delivery one for planning) every two weeks, you are effectively eliminating 20% of the time developers can work. A second benefit of going iteration-less is the elimination of an artificial constraint that an item must be small enough to fit into the time-box. While it is always better to deliver small pieces at a time – value cannot always be shaped into the time-box. A third benefit of going iteration-less is that demand on related resources has a smoother pattern. Time-boxes batch related work such as testing, requirements grooming, and deployment. The pace is more sustainable for the non-development resources.

There are several drawbacks to going iteration-less if the team doesn’t address them. First there can be challenges with coordinating feedback into successive deliverables. If the developers just keep writing code – when do we update the backlog with the input we would historically receive at the iteration review. Also, since there is no time-box, there is also a perceived lack of pressure to get to done. Additionally, by not enforcing effective decomposition there can be a lack of pressure to design well. Also, there isn’t a fixed time to stop and review the development processes to keep the team in synch. Finally, continuous delivery can result in continuous demand on related resources like analysts and quality assurance.

There are effective strategies for dealing with each of these drawbacks. For example, the developers that just delivered a feature can stop to get feedback from the customer without disrupting the entire team. You can create visibility into delays by tracking the days of effort against each story as it moves across the board – which will result in pressure to get to done. Enforcing norms that work must be small enough to get through in less than two weeks – but allowing for exceptions when it makes sense to the team will help ensure good design. You don’t have to come to a hard stop on everything in process to do a team retrospective (operations review). Finally, the related resources timing can be decoupled from the teams cadence. For example, business analysts can still groom backlog in weekly batches to create some buffer.

History if IID

One of the interesting things for many current agilists to understand is that we have been doing IID in software development since the late 50’s – that’s right – half a century. Here are some highlights.

1957: Gerald Weinberg runs a project at IBM using IID practices.

1960: Gerald Weinberg starts teaching IID at the IBM System’s Research Intitute

1969: Robert Glass published a paper on current practices in IID in ACM Computer Surveys

1975: Vic Gasill and Albert Turner publish Iterative Enhancement: A Practical Technique for Software Deveopment in IEEE Transactions on Software Engineering

1980: Gerald Weinberg publishes Adaptive Programming: The New Religion

1984: Tom Gilb recommends two week iterations as a best practice

1985: Barry Boehm promotes the spiral model – using risk as the key prioritization method

1986: Fred Brooks publishes No Silver Bullet which articulates the benefits of IID over waterfall

1988: Tom Gilb publishes Principles of Software Engineering Management showing an evolutionary approach to development

1990’s: Scrum, DSDM, RUP, XP, and FDD all become mature practices

It is just in this decade, fueled by improved development tools, lower cost computing and network, and in response to pressure to rapidly deliver solutions with a high level of unknowns that IID, in the form of Agile methods, has become common place.

I have an interesting paradox I have been listening to in the Agile community. On the one hand the craftsmanship discussions resonate with me. After years of experience, apprenticeship, and study one can develop the art and craft of being a great developer. There were points in my life where I have experienced this – in the 80’s with RPG (laugh if you want to) and in the early-mid 90’s with C++.  I was a craftsman of both languages. Since then I have worked with real craftsman – and have established the craft involved in helping development teams perform above their own expectations.

On the other hand, I hear talk of cross functional teams. How the developers should be involved in all aspects of deciding what the product is, how to run the project, where the market is going, how to best optimize profit in an industry, and how to position and sell the product.

And this is paradox. Some developers believe that there is a craft that takes years to develop to what they do. But they don’t recognize craft in enterprise analysis, project management, product management, market analysis, or sales and marketing. I believe this to be a common form of social bias that intelligent people are particularly prone to. The belief that what one person does is very complicated – but that what other people do is very easy.

I had a meeting once with the CFO, COO, CIO, and head of HR. They were having a debate. Each executive was relatively limited in realizing the complexity of the other’s job. At one point, the CFO was telling the CIO how he should run the support organization. The CFO was coming from a simplistic and limited view. I mentioned to the team I had some ideas about how the business should handle non-performing assets from the recent spate of acquisitions. The CFO asked me what I was doing – that there was no way I understood the complexity of his domain. I smiled and told him that since he was directing the technology team without a clear understanding – I was ready to direct the accounting operations on behalf of the the CIO. Luckily, I had established enough trust that the CFO thought about it and laughed. He realized that what appears simple on the outside is difficult when you get down to brass tacks.

So, while I believe that co-locating teams is important to successful teams – and I believe that we need to generate a shared understanding across everyone involved in the project. I also believe that in many jobs there are fine details that separate the very competent from the capable. There is room for specialization in most jobs in the product development organization. But, when we try to push to far in the direction of the generalizing specialist we get dilution across the board.

Most of the studies that discuss failed software development projects find misunderstood requirements and inadequate change management among the leading causes of failure. These failures can’t be adequately addressed just through more rigorous documentation or web based tools. Generating shared understanding is a social act – therefore, one of the elements of improving requirements is through improving the related social interactions or conversations.

But how do we go about improving conversations? Where are the engineering practices or process improvement techniques we can apply to consistently achieve improved performance? John Searles has written several books that break down conversations into a series of speech acts and even has a notation for describing a conversation. I found Searles work on Speech Acts to be very interesting. There is value in classification and understanding of patterns in conversations. But I am not sure this translates to an easily digestible approach. I also believe that conversations have flows and states, but I am also not sure that a process flow is an effective way to represent a conversation.

I can’t break the process of the flows and states a cake baking in the oven goes through down to detailed steps. I can tell you when it might be appropriate to bake a cake, when we are ready to put the cake in the oven, what happens during the baking process, and I can tell you how to check to see if a cake is done. So the approach I am advocating here is to be able to know when a specific conversation is called for, what it takes to be prepared for a conversation, how to approach the conversation, and how to check and see if the conversation has been performed.

Conversations for Shared Understanding

A conversation for shared understanding often involves one person on the project learning something new from someone else on the project. Conversations for shared understanding are important between different functional groups and when defining expected outcomes. For example, requirements documents define what needs to be done on a project. The requirements documents by themselves are typically not adequate. The gap in understanding leaves room for the development organization to build something that is not optimal either from a development or a requirement standpoint.

I love college football. One Saturday when I first got married, I was watching a game on TV. I planned to head over to my friend Steve’s house later in the day to watch the afternoon game. My wife called down from upstairs to ask me to take out the trash. I agreed to do it, thinking I could grab it out of the kitchen on the way out the door after the current game before I went over to Steve’s house. Her context was very different from mine. She meant, right now – all the trash in every room of the house – and clean the bathroom’s while you are at it. So, while I agreed to her requirements, we didn’t have a shared understanding of the request.

When is a Conversation for Shared Understanding called for?

Anytime a lack of shared understanding slows down the project or creates rework or other waste. A lack of shared understanding happens all the time in projects – particularly between functional groups and early in a project. So a conversation for shared understanding is important early in a project. Typically, these early conversations should be around the overall context and objectives of the project. A conversation for shared understanding is also called for with each specific feature or request. On software development projects it is important everyone involved in delivering, verifying, or accepting a feature or project deliverable has a shared understanding. Conversations for shared understanding should be at the outcome and context level. They are not intended for the technical implementation details to be explained to business stakeholders. The point is to establish context and understanding of the outcomes necessary to optimize the performance of the project performers.

What does it take to be prepared for a Conversation for Shared Understanding?

The mood and perspective of the participants in the conversation will impact the ability to successfully perform these conversations. So each participant must have an intention to have a conversation that results in a shared understanding. They should be willing to put in the effort to review or understand any artifacts produced to seed the understanding. They must bring a belief that the other person’s context matters. Additionally, participants need to put some thought into the boundaries of the conversation. The performer will want to think about what they need to understand to be most effective in delivering this request. The requestor will want to consider what parts of the context, what outcomes, and what language is particularly important to ensure they get what they intended to ask for.

How do we approach a Conversation for Shared Understanding?

1. The participants present their expectations and boundaries for the conversation.
2. Each participant explains their understanding of the context, targeted outcomes, and significant language. The other participant(s) will note where their understanding varies.
3. The discrepancies should be discussed, evaluated and resolved. Sometimes, the details of one participant will not be important. Sometimes, more specific discussion is necessary to gain clarity.
4. The participants will agree when they have a shared understanding.

How can we check to see if the Conversation for Shared Understanding has been successful?

At the end of the conversation the participants should be able to present an understanding of context and outcomes in common language. As soon as either participant identifies a gap in understanding they should revisit the conversation. Over time, the participants establish a common background that reduces the effort required to establish a shared understanding.

Today, when my wife asks me to take out the trash I understand what she wants. We also have identified when it may be important to have a conversation for shared understanding. For example, when she sends me to the store for milk I ask her to take a minute and think about what else I should get while I am there. My experience is that she has a lot of background information that I am not aware of. If I go to get milk and don’t get the eggs and butter she needs I will be heading back to the store. Taking the time to have conversations for shared understanding will almost always accelerate the effective pace of the project.

I have been reading a lot about of the recent group discussions and papers regarding Kanban and Lean Software Development. One of the things I have noticed is that there is not a consistent application of the concepts regarding time. One of the reasons we establish a body of knowledge is to create a common language that helps us communicate based on a shared understanding. This inconsistency in the use of terms around time create a gap in understanding. In an effort to help the conversation here are the appropriate definitions for each type of time.

Durations

Duration is simply a measure of elapsed time.

Lead Time: The average time it takes for one unit to go through the entire process – from start to finish – including time waiting between sub-processes. Lead time can take different perspectives.

Development Lead Time is the time it takes for a unit to go through development from release from the backlog until it is done (which includes analysis, development, test, acceptance, deployment – but maybe not release to production). It is interesting to the development manager.

Order Lead Time is the time that elapses between when the customer places an order and when the customer receives what they ordered. In cases where requests are queued in a back log and releases are decoupled from development completion this can be far longer than the Development Lead Time. This is interesting to customer.

Order to Cash Lead Time is the time from when the customer places the order to the time cash is received by the business. While this is often longer than the Order Lead Time it can be shorter than the Order Lead Time when the customer prepays. This is interesting to the person paying the bills.

Process Lead Time: The time the unit is being worked on in a specific process before it is passed on to the next process. For example, within Development Lead Time this could be analysis, development, automated test, acceptance test or deployment time. This is interesting to the person performing the specific process or activity.

Processing Time: The time the unit is actually being worked on. It would be the sum of all the Process Lead Times. This is interesting to use as we try to improve our Lead Time.

Value Add Time: The Processing Time spent on a unit that transforms the unit in a way that the customer is willing to pay for. Not all Processing Time is Value Add Time – although non-Value Add Time can be necessary due to compliance requirements, to optimize performance of value add resources, and to minimize rework. This is interesting as it has been the subject of much of the lightweight process effort in Agile development. When taken to far it results in increased Rework Time and increased Process Lead Time.

Rework Time: The Processing Time that results from poor quality.  Failure Demand is the amount of resource that is consumed by Rework Time. This is interesting as it is expensive.

Queue Time: The time a unit sits between sub-processes waiting for someone to work on it. This is interesting to us as we try to improve our Lead Time. This is no cost to improving flow to reduce Queue Time but it reduces Lead Time.

Cadences

Cadence is a measure of time per unit.

Cycle Time: The average time between units coming out the end of the pipe. This is interesting because it helps us predict Lead Time.

Takt Time: How often units need to come out the end of the pipe to meet customer demand. This is interesting because it reflects the rate we need to produce to optimize investment against revenue.

You would expect Lead Time to be the sum of Processing Time, Queue Time, and Rework Time – and it is in a purely serial process flow. But in cases where work moves in parallel the sum of Processing Time, Queue Time and Rework Time will be greater than the Lead Time. Lead Time is interesting because it is what we use to make commitments.  Improving Lead Time can be done by reducing Queue Time, shortening Process Lead Time, eliminating Rework Time, minimizing non-Value Add Time, and performing processes in parallel.

The concept of Value Add Time is interesting and has had strong impact on Agile processes. At some point things like documentation, planning, testing, and adding additional features are not adding value in a way the customer is willing to pay for. Reducing these items to the bare minimum time that is necessary to meet the needs of the customer is helpful. Reducing them to the point they increase rework or reduce the productivity of the value added performers is not helpful.

The biggest inconsistency I have noticed is the improper use of Lead Time (a duration), Cycle Time (a cadence), and Takt Time (a cadence). In summary, Lead Time is how long it takes us to build and deliver it, Cycle Time is how often a new one is produced, and Takt Time is the rate of customer demand. In a stable system, Duration and Cadence are related by Little’s Law. The cadence of Cycle Time will equal WIP divided by the duration of Lead Time.

I believe that consistent use of these terms will help us have more meaningful conversations and continue to improve the ways that we deliver software to our customers. Part of my focus over the last decade has been on reducing Lead Time through reducing WIP, Queue Time, and non-Value Add Time, improving quality to reduce Rework Time, and improving architecture, automation, and shared understanding to reduce Processing Time. Having a shared understanding for each type of time leads to productive conversations around specific techniques and improves the teams ability to focus on the right types of improvements.

Knowledge based projects, like software development,  are performed by people.  So the way people learn, collaborate and interact is more impactful in knowledge based projects than in traditional projects like building a bridge or a satellite. If Agile software development and Project Management 2.0 have introduced anything new, it is an explicit focus on the social aspects of performing projects. Traditionally project management methods focused explicitly on the processes, tools and techniques of projects.

The traditional tools and processes are still important to the success of projects. We still need to clearly define scope, coordinate project resources, and manage commitments and acceptance of work. But these are social endeavors – and they can’t be accomplished with tools and processes alone. They require an intentional focus on the social aspects of the project. The shift toward the social focus is reflected in the Agile Manifesto’s first value, “Focus on individuals and interactions over processes and tools”. This shift in explicit focus is not an abandoning of processes and tools – rather an intentional focus on achieving these purposes to include people and interactions.

Project Conversations

How do we manage these social aspects? It is through conversation. Conversations are the use of language to exchange thoughts, ideas, or information. Understanding, Coordination, and Commitment within the team arise through conversations. John Searles wrote about how conversations can be defined with clear outcomes and broken into specific series of acts. Gordon Pask wrote in Conversation Theory about how understanding arises based on our interpretation of another person’s behavior.

Using these approaches we can construct ways to improve project performance through improving conversations. But, this isn’t a touchy- feely effort. For each specific conversation there is a specific set of outcomes and a specific set of “speech acts”. Intentionally deciding what conversations need to occur, when they need to occur, and what they look like when they have been completed will  improve the performance of interactions on Agile teams. Over the next week I will be discussing various conversations around understanding, committing, and coordinating. I will present a starting point you can use within your teams to have the meta-conversation to improve these conversations. Then I will build on each of these conversations at each order of scaling (small team, multi-team, program, enterprise).

There is a nice summary of Kanban at http://www.kanbandistilled.com/. It shows Feature Requests going into the Software Development Pipeline and Improved Software coming out.The article does a great job of showing how Kanban helps improve the flow of software development. Getting better at software development is important and Agile methods have been helping teams achive this goal. This is the first bargain of Agile-delivering software faster and with much higher quality then ever before. In most organizations however, this is only a small part of the business equation. Even when development is successful, a narrow focus on software  development narrows the value an organization will achieve. We also need to align of development with the most important needs of the business, balance development with adoption, and ensure improved business performance is achieved.

Feeding the Software Development Pipeline

Unless a company is a software development service provider, its business is not about software development. Software likely enhances the businesses ability to create value for it’s customers. So, while Agile development helps teams get better at software development – the goal is to get better at creating value for it’s customers. The focus of Agile software development is upside down. The focus should be on the capabilities of the business. Aligning development with the needs of the business requires understanding  which business capabilities will benefit from improvement and prioritizing these needs based on business value.

After using a business capability approach to need identification and prioritization, the business needs must be appropriately scoped and communicated to development. Over the last month I have been writing about Making Agile Requirements Work. need to make sure the requests that go into the Software Development Pipeline are clear and appropriately defined. Following the six principles of Agile Requirements That Work help with this feeding of the Software Development Pipeline.

1. Articulate Purpose and Outcome
2. Establish a Shared Language
3. Provide a Stable View of the Business
4. Support Progressive Elaboration
5. Testable
6. Prioritized and Reflect Business Value and Risk

Gaining Business Benefit

The other part of the equation is to ensure the organization adopts and supports the improved software. Once the Software Development Pipeline has produced Improved Software, it still isn’t valuable until the Business Capability has achieved the intended performance improvement. Just as the elicitation and analysis of requirements has to match the cadence and needs of Software Development. Software Development has to match the cadence and needs of the organizations ability to adopt and support Improved Software. Part of the overall feedback to the organization includes ensuring the capability improvement is achieved.

Risk

At every step in this equation different risks may exist. identifying, prioritizing, and scoping feature requests have risk associated with a lack of understanding. Software Development has risk associated with the Ability to Execute and Verify the delivered software. Benefit realization has risks associated with achieving adoption and being able to sustain and support change in the organization. These risks should be explicitly managed and feedback mechanisms should be in place to recognize the realization of risks. Failing to understand and manage risks in an integrated fashion will also diminish the organization’s ability to maximize benefit from it’s investment.

The Real Agile Bargain

The real agile bargain is the maximizing of economic outcomes by increasing the speed and focus an organization can improve and respond to the market. The Agile Business Value Equation is solved at the Business Capability Level. The ability to rapidly develop high quality software moves Technology from a change obstacle to become an change enabler. This happens at the intersection of Feeding the Software Development Pipeline, Developing the Software, and Gaining Business Benefit. Solving any of these parts independent of each other does not result in optimizing business agility.

On many software development projects, teams that are interested in getting started writing code early start by building the things they understand first. While this gets code started and in front of the customer as soon as possible, it isn’t necessarily the shortest time to value. This approach also sets the team up for the later stages of the project to be wrought with uncertainty and pressure from the business.  Two keys to successful Agile projects are focusing on early learning (mitigation of risk) and then rapid time to business value.  This is achieved by prioritizing the order that work is performed by the team.  When requirements analysis doesn’t clearly communicate the areas requiring risk mitigation and clearly identify those requirements that are most valuable to achieving the business outcome, then the team can’t appropriately prioritize the work.

Requirements reflect business value and risk

On most software development projects, not every requirement is equally valuable to the business. Some requirements are core to achieving the business outcome expected for the project and some are critical because they are  deliver on the brand of the business. But others, while valuable to the business, may address edge conditions or be nice to haves. Requirements need to be prioritized so that unknowns are addressed early, then higher business value requirements are addressed, then the remaining requirements. Part of the requirements analysis process should include an assessment of business value and risk.  If the prior principles have been applied, but the requirements are not worked with an eye toward risk mitigation and business value priority, the team will not be optimizing investment or time to value.

Because software development has some level of uncertainty, it is likely that a project will reach the end of its budget or time estimate before delivering on all the requirements expected. If you haven’t prioritized business value and risk, you may have to report, “We are really, really close. For just a little more investment or time, we can deliver something that meets your needs”. It is far easier report at the end of the time constraint or budget and say, “We have delivered on the most critical features and addressed the primary risks. At this point you can use the system we have delivered to address (most of) your business need. However, if you want to continue to invest, we can add additional capability.”

Many projects wait until the project is in danger to begin the trade-off analysis. Early work is done to a great deal of detail and late work is rushed through. The time to triage the requirements is at the beginning of a project. Then at each stage of elaboration, the detailed requirements should be further triaged. Requirements must reflect business value and risk – and the work should be prioritized based on these factors.

I hear a lot about emergent outcomes, and progressive elaboration, and self organization on Agile projects. These are slippery slope concepts, and far too often are applied irresponsibly. Emergent outcomes don’t mean we have no idea what the business is paying for – they have a pretty clear idea of the problem they are investing to address. Progressive elaboration doesn’t mean the team will figure out what they are doing as they go along – it means they will gain clarity on mitigating risks and meeting the technical and customer experience aspects of how to solve the business problem as they move forward. Self organization doesn’t mean the team can work on whatever it wants – it means that the team collectively decides the best way to focus their efforts and skills on a specific effort.

The business is willing to make an investment, to solve a specific problem, within a specific timeframe. In the first four posts of this series we discussed that the way that requirements are captured should express the purpose and outcome, they should establish a shared language that communicates context and intent, is should be a stable view of the problem, and it should support progressive elaboration. In addition, Agile requirements include clarity on what "done" looks like. Agile requirements also include identifying the impediments to getting to "done".

Agile Requirements should be Testable

Not having a clear outcome does not make a development effort Agile, it makes it irresponsible. If we don’t communicate requirements in a way that we can ensure the requirement is accomplished then we are being unclear about the value proposition we are trying to achieve. By testable, we don’t mean the “how” of implementation, we mean understanding what is not available or performing adequately what will be different after the requirement is delivered. If the business can’t communicate to the developer how to make sure they know when they are done – the project will meander. Even with lots of customer involvement during development – it is likely that development will be less focused than it could be.

What does it mean to be testable? I have a client that had its development team develop a “data-warehouse” to provide management reports. The developers spent eight months defining all the data in the business, developed programs and processes to consolidate the data into a central repository, and started building reports against the data. Eight months later the manager’s biggest problem is that they don’t have the information the need to manage the business. How is this possible? The business wanted a data warehouse and they got one. They have been intimately involved in the data gathering process. They have been asking for reports through the system for months.

The problem is that no one stopped to understand how they would test the requirement. How would they know if the business need was met? The business need is met when the manager had the information needed to manage the business. When the requirement is communicated as a granular “how” – and there is no way to verify it has met the need of the business – the business is unlikely to get what they expect.