Last week I wrote a post on Iterative and Incremental development. In that post I pointed out that you can do iterative and incremental development without having time-boxed iterations. Mike Cottmeyer asked in a comment to clarify when time-boxed iterations made the most sense and when continuous flow made the most sense. This post answers that question.

When Time-boxed Iterations Make the Most Sense

Time-boxed iterations serve several purposes that are not well served in continuous flow. They create an opportunity for the team to stop and get feedback from the customer on their product.  They can help with chunking up work to support the planning process and validation of the design of the solution. There are times when this is really important.

So, time-boxed iterations work best:

1. When there is a lot of opportunity for feedback on current deliverables to significantly update subsequent work

2. When the planning horizon is stable enough that a subset of the backlog can be prioritized for at least the duration of the iteration

3. When the backlog items can fit in the iteration time-box

4. When dependent resources (for example, planning and testing) can not be available to support continuous flow

For example, time-boxed iterations work best at the outset with a new product, new technology, or substantially new set of features.

When Continuous Flow Makes the Most Sense

Continuous flow has some advantages over time-boxed iterations. Overtime, the level of resources available to do work is higher and the backlog is much more flexible and adaptive to variation in the arrival of work. For example, once the product architecture is stable and well defined and work is arriving in bursts with various levels of urgency.

So, continuous flow works best:

1. When the work is arriving continuously

2. When the planning window can be (or must be) very small

3. When the outcome is pretty clear and feedback is not as likely to impact subsequent work

4. When there are multiple work streams operating at different cadences

For example, continuous flow when working on defects and small enhancements in a stable product or working with technical support issues.

Non-relevant factors

Effective planning. Data can be gathered from both iterations (velocity) and continuous flow (lead time) to determine when a product can be delivered and to establish visibility into the current status of the project.

Retrospectives. Iterations create a natural break for retrospectives. There is no reason why retrospectives must be tied to iteration releases. Continuous flow teams can have retrospectives on either an event driven or calendar driven basis.

Motivation to get to done. Iterations create pressure to complete work in a specific time-box. Continuous flow creates pressure to met lead-time commitments (policies).

How they can work together

We are making a standard practice of combining Kanban at the strategic planning level to support continuous flow of strategic initiatives across the organization and Scrum (or Kanban) at the development team level. It doesn’t make sense to us to create time-boxed deliverables at the strategy execution deliverable level. We generate plenty of pressure to move forward from the time commitments on the tasks on the Kanban board. At the development level – when there is a lot of opportunity for feedback to update subsequent work – we (may) use iterations to keep the team focused. For defects and minor enhancements we continue to use a continuous flow model – maybe within the same team.

Iterations versus Continuous Flow?

I believe this is a hammer versus saw discussion. Each approach is suited to fit different scheduling and feedback needs. In many cases, time-boxed iterations make a lot of sense but would result in a less than optimized approach then in other cases. I believe that early on in a product (when there is a lot to be learned and a lot of risk to manage) iterations probably make the most sense. At some point, when the product has switched primarily to support, continuous flow makes the most sense. I hope this answers Mike’s question from my previous post.

Most of the studies that discuss failed software development projects find misunderstood requirements and inadequate change management among the leading causes of failure. These failures can’t be adequately addressed just through more rigorous documentation or web based tools. Generating shared understanding is a social act – therefore, one of the elements of improving requirements is through improving the related social interactions or conversations.

But how do we go about improving conversations? Where are the engineering practices or process improvement techniques we can apply to consistently achieve improved performance? John Searles has written several books that break down conversations into a series of speech acts and even has a notation for describing a conversation. I found Searles work on Speech Acts to be very interesting. There is value in classification and understanding of patterns in conversations. But I am not sure this translates to an easily digestible approach. I also believe that conversations have flows and states, but I am also not sure that a process flow is an effective way to represent a conversation.

I can’t break the process of the flows and states a cake baking in the oven goes through down to detailed steps. I can tell you when it might be appropriate to bake a cake, when we are ready to put the cake in the oven, what happens during the baking process, and I can tell you how to check to see if a cake is done. So the approach I am advocating here is to be able to know when a specific conversation is called for, what it takes to be prepared for a conversation, how to approach the conversation, and how to check and see if the conversation has been performed.

Conversations for Shared Understanding

A conversation for shared understanding often involves one person on the project learning something new from someone else on the project. Conversations for shared understanding are important between different functional groups and when defining expected outcomes. For example, requirements documents define what needs to be done on a project. The requirements documents by themselves are typically not adequate. The gap in understanding leaves room for the development organization to build something that is not optimal either from a development or a requirement standpoint.

I love college football. One Saturday when I first got married, I was watching a game on TV. I planned to head over to my friend Steve’s house later in the day to watch the afternoon game. My wife called down from upstairs to ask me to take out the trash. I agreed to do it, thinking I could grab it out of the kitchen on the way out the door after the current game before I went over to Steve’s house. Her context was very different from mine. She meant, right now – all the trash in every room of the house – and clean the bathroom’s while you are at it. So, while I agreed to her requirements, we didn’t have a shared understanding of the request.

When is a Conversation for Shared Understanding called for?

Anytime a lack of shared understanding slows down the project or creates rework or other waste. A lack of shared understanding happens all the time in projects – particularly between functional groups and early in a project. So a conversation for shared understanding is important early in a project. Typically, these early conversations should be around the overall context and objectives of the project. A conversation for shared understanding is also called for with each specific feature or request. On software development projects it is important everyone involved in delivering, verifying, or accepting a feature or project deliverable has a shared understanding. Conversations for shared understanding should be at the outcome and context level. They are not intended for the technical implementation details to be explained to business stakeholders. The point is to establish context and understanding of the outcomes necessary to optimize the performance of the project performers.

What does it take to be prepared for a Conversation for Shared Understanding?

The mood and perspective of the participants in the conversation will impact the ability to successfully perform these conversations. So each participant must have an intention to have a conversation that results in a shared understanding. They should be willing to put in the effort to review or understand any artifacts produced to seed the understanding. They must bring a belief that the other person’s context matters. Additionally, participants need to put some thought into the boundaries of the conversation. The performer will want to think about what they need to understand to be most effective in delivering this request. The requestor will want to consider what parts of the context, what outcomes, and what language is particularly important to ensure they get what they intended to ask for.

How do we approach a Conversation for Shared Understanding?

1. The participants present their expectations and boundaries for the conversation.
2. Each participant explains their understanding of the context, targeted outcomes, and significant language. The other participant(s) will note where their understanding varies.
3. The discrepancies should be discussed, evaluated and resolved. Sometimes, the details of one participant will not be important. Sometimes, more specific discussion is necessary to gain clarity.
4. The participants will agree when they have a shared understanding.

How can we check to see if the Conversation for Shared Understanding has been successful?

At the end of the conversation the participants should be able to present an understanding of context and outcomes in common language. As soon as either participant identifies a gap in understanding they should revisit the conversation. Over time, the participants establish a common background that reduces the effort required to establish a shared understanding.

Today, when my wife asks me to take out the trash I understand what she wants. We also have identified when it may be important to have a conversation for shared understanding. For example, when she sends me to the store for milk I ask her to take a minute and think about what else I should get while I am there. My experience is that she has a lot of background information that I am not aware of. If I go to get milk and don’t get the eggs and butter she needs I will be heading back to the store. Taking the time to have conversations for shared understanding will almost always accelerate the effective pace of the project.

Knowledge based projects, like software development,  are performed by people.  So the way people learn, collaborate and interact is more impactful in knowledge based projects than in traditional projects like building a bridge or a satellite. If Agile software development and Project Management 2.0 have introduced anything new, it is an explicit focus on the social aspects of performing projects. Traditionally project management methods focused explicitly on the processes, tools and techniques of projects.

The traditional tools and processes are still important to the success of projects. We still need to clearly define scope, coordinate project resources, and manage commitments and acceptance of work. But these are social endeavors – and they can’t be accomplished with tools and processes alone. They require an intentional focus on the social aspects of the project. The shift toward the social focus is reflected in the Agile Manifesto’s first value, “Focus on individuals and interactions over processes and tools”. This shift in explicit focus is not an abandoning of processes and tools – rather an intentional focus on achieving these purposes to include people and interactions.

Project Conversations

How do we manage these social aspects? It is through conversation. Conversations are the use of language to exchange thoughts, ideas, or information. Understanding, Coordination, and Commitment within the team arise through conversations. John Searles wrote about how conversations can be defined with clear outcomes and broken into specific series of acts. Gordon Pask wrote in Conversation Theory about how understanding arises based on our interpretation of another person’s behavior.

Using these approaches we can construct ways to improve project performance through improving conversations. But, this isn’t a touchy- feely effort. For each specific conversation there is a specific set of outcomes and a specific set of “speech acts”. Intentionally deciding what conversations need to occur, when they need to occur, and what they look like when they have been completed will  improve the performance of interactions on Agile teams. Over the next week I will be discussing various conversations around understanding, committing, and coordinating. I will present a starting point you can use within your teams to have the meta-conversation to improve these conversations. Then I will build on each of these conversations at each order of scaling (small team, multi-team, program, enterprise).

There is a nice summary of Kanban at http://www.kanbandistilled.com/. It shows Feature Requests going into the Software Development Pipeline and Improved Software coming out.The article does a great job of showing how Kanban helps improve the flow of software development. Getting better at software development is important and Agile methods have been helping teams achive this goal. This is the first bargain of Agile-delivering software faster and with much higher quality then ever before. In most organizations however, this is only a small part of the business equation. Even when development is successful, a narrow focus on software  development narrows the value an organization will achieve. We also need to align of development with the most important needs of the business, balance development with adoption, and ensure improved business performance is achieved.

Feeding the Software Development Pipeline

Unless a company is a software development service provider, its business is not about software development. Software likely enhances the businesses ability to create value for it’s customers. So, while Agile development helps teams get better at software development – the goal is to get better at creating value for it’s customers. The focus of Agile software development is upside down. The focus should be on the capabilities of the business. Aligning development with the needs of the business requires understanding  which business capabilities will benefit from improvement and prioritizing these needs based on business value.

After using a business capability approach to need identification and prioritization, the business needs must be appropriately scoped and communicated to development. Over the last month I have been writing about Making Agile Requirements Work. need to make sure the requests that go into the Software Development Pipeline are clear and appropriately defined. Following the six principles of Agile Requirements That Work help with this feeding of the Software Development Pipeline.

1. Articulate Purpose and Outcome
2. Establish a Shared Language
3. Provide a Stable View of the Business
4. Support Progressive Elaboration
5. Testable
6. Prioritized and Reflect Business Value and Risk

Gaining Business Benefit

The other part of the equation is to ensure the organization adopts and supports the improved software. Once the Software Development Pipeline has produced Improved Software, it still isn’t valuable until the Business Capability has achieved the intended performance improvement. Just as the elicitation and analysis of requirements has to match the cadence and needs of Software Development. Software Development has to match the cadence and needs of the organizations ability to adopt and support Improved Software. Part of the overall feedback to the organization includes ensuring the capability improvement is achieved.

Risk

At every step in this equation different risks may exist. identifying, prioritizing, and scoping feature requests have risk associated with a lack of understanding. Software Development has risk associated with the Ability to Execute and Verify the delivered software. Benefit realization has risks associated with achieving adoption and being able to sustain and support change in the organization. These risks should be explicitly managed and feedback mechanisms should be in place to recognize the realization of risks. Failing to understand and manage risks in an integrated fashion will also diminish the organization’s ability to maximize benefit from it’s investment.

The Real Agile Bargain

The real agile bargain is the maximizing of economic outcomes by increasing the speed and focus an organization can improve and respond to the market. The Agile Business Value Equation is solved at the Business Capability Level. The ability to rapidly develop high quality software moves Technology from a change obstacle to become an change enabler. This happens at the intersection of Feeding the Software Development Pipeline, Developing the Software, and Gaining Business Benefit. Solving any of these parts independent of each other does not result in optimizing business agility.

On many software development projects, teams that are interested in getting started writing code early start by building the things they understand first. While this gets code started and in front of the customer as soon as possible, it isn’t necessarily the shortest time to value. This approach also sets the team up for the later stages of the project to be wrought with uncertainty and pressure from the business.  Two keys to successful Agile projects are focusing on early learning (mitigation of risk) and then rapid time to business value.  This is achieved by prioritizing the order that work is performed by the team.  When requirements analysis doesn’t clearly communicate the areas requiring risk mitigation and clearly identify those requirements that are most valuable to achieving the business outcome, then the team can’t appropriately prioritize the work.

Requirements reflect business value and risk

On most software development projects, not every requirement is equally valuable to the business. Some requirements are core to achieving the business outcome expected for the project and some are critical because they are  deliver on the brand of the business. But others, while valuable to the business, may address edge conditions or be nice to haves. Requirements need to be prioritized so that unknowns are addressed early, then higher business value requirements are addressed, then the remaining requirements. Part of the requirements analysis process should include an assessment of business value and risk.  If the prior principles have been applied, but the requirements are not worked with an eye toward risk mitigation and business value priority, the team will not be optimizing investment or time to value.

Because software development has some level of uncertainty, it is likely that a project will reach the end of its budget or time estimate before delivering on all the requirements expected. If you haven’t prioritized business value and risk, you may have to report, “We are really, really close. For just a little more investment or time, we can deliver something that meets your needs”. It is far easier report at the end of the time constraint or budget and say, “We have delivered on the most critical features and addressed the primary risks. At this point you can use the system we have delivered to address (most of) your business need. However, if you want to continue to invest, we can add additional capability.”

Many projects wait until the project is in danger to begin the trade-off analysis. Early work is done to a great deal of detail and late work is rushed through. The time to triage the requirements is at the beginning of a project. Then at each stage of elaboration, the detailed requirements should be further triaged. Requirements must reflect business value and risk – and the work should be prioritized based on these factors.

I hear a lot about emergent outcomes, and progressive elaboration, and self organization on Agile projects. These are slippery slope concepts, and far too often are applied irresponsibly. Emergent outcomes don’t mean we have no idea what the business is paying for – they have a pretty clear idea of the problem they are investing to address. Progressive elaboration doesn’t mean the team will figure out what they are doing as they go along – it means they will gain clarity on mitigating risks and meeting the technical and customer experience aspects of how to solve the business problem as they move forward. Self organization doesn’t mean the team can work on whatever it wants – it means that the team collectively decides the best way to focus their efforts and skills on a specific effort.

The business is willing to make an investment, to solve a specific problem, within a specific timeframe. In the first four posts of this series we discussed that the way that requirements are captured should express the purpose and outcome, they should establish a shared language that communicates context and intent, is should be a stable view of the problem, and it should support progressive elaboration. In addition, Agile requirements include clarity on what "done" looks like. Agile requirements also include identifying the impediments to getting to "done".

Agile Requirements should be Testable

Not having a clear outcome does not make a development effort Agile, it makes it irresponsible. If we don’t communicate requirements in a way that we can ensure the requirement is accomplished then we are being unclear about the value proposition we are trying to achieve. By testable, we don’t mean the “how” of implementation, we mean understanding what is not available or performing adequately what will be different after the requirement is delivered. If the business can’t communicate to the developer how to make sure they know when they are done – the project will meander. Even with lots of customer involvement during development – it is likely that development will be less focused than it could be.

What does it mean to be testable? I have a client that had its development team develop a “data-warehouse” to provide management reports. The developers spent eight months defining all the data in the business, developed programs and processes to consolidate the data into a central repository, and started building reports against the data. Eight months later the manager’s biggest problem is that they don’t have the information the need to manage the business. How is this possible? The business wanted a data warehouse and they got one. They have been intimately involved in the data gathering process. They have been asking for reports through the system for months.

The problem is that no one stopped to understand how they would test the requirement. How would they know if the business need was met? The business need is met when the manager had the information needed to manage the business. When the requirement is communicated as a granular “how” – and there is no way to verify it has met the need of the business – the business is unlikely to get what they expect.

We have proposed that Agile Requirements must be built at a purpose and outcome level, must leverage shared language that communicates context and intent, and must be based on a stable view of the domain. But even with these in place requirements can still fail to support the way people solve problems and the way people learn. When we try to be to prescriptive and communicate too much detail initially it will reduce effectiveness – not improve agility.  Agile requirements must support progressive elaboration.

Don’t Support Progressive Elaboration

One of the faulty assumptions made in Requirements Elicitation is that we believe we can know up front exactly what we want. It is also faulty to believe that we should document requirements in too granular detail. In fact, neither of these is accurate. Requirements should support the concept of progressive elaboration. That is, they should be documented so that more specific detail unfolds during the course of the project. This addresses the flaw in both of assumptions. First, documenting what we know at a high level of detail allows us to establish a consistent context that we can communicate in more detail the more we learn. Getting to too much detail early can result in rework and inconsistency in communication.

Agile teams may break requirements into Themes, Epics, Stories, and then Tasks. This may be hierarchical, follow a story-map, or be connected in a mind-map.  Themes are be defined early on and directly connected to business value propositions. These are then progressively elaborated into Epics and Stories as needed to feed the team. As Stories move into a ready state for development they can be further broken down into tasks by the development team. This progressive elaboration allows the team to apply what it learns about the best way to deliver the project as the project unfolds. Critically, it also supports the way that people learn about the problem. Too much detail early on can be overwhelming and result in a disconnected or inconsistent understanding of the effort. Not enough detail later on results in a lack of shared understanding of the problem. Progressive elaboration supports effective communication, collaboration in solution finding, and the application of what the team learns as the project unfolds.

The Gilb’s have started a series of posts against Agile. The first on is Wrong Focus.

“It is not about working software. It is not about your preferred working process. It is not about user stories. It is not only about your customer or user. It is all about delivering, to your set of Stakeholders, value improvements to them, to what they care about, what makes their world better, within agreeable, minimum or pre-determined costs. If that is not the main focus, if that is not clear to everyone on the team, you will eventually loose! If your methods are not focusing on delivering that value, as todays most popular Agile methods are not, they will fail to deliver, they will become last years fad.”

They claim Agile software development is flawed because it focuses on improving the process and environment of the team developing software. These improvements result in improved customer collaboration that ensures the team better understands what the customer wants. From my experience, the combination of improved environment, appropriately sized process, and customer collaboration results in mature agile teams doing a great job of delivering the projects and products. But I understand their point. There is a difference between understanding the customer and focusing on delivering customer value.

Wrong Focus

Whether the software development organization is building software for external clients or to enable the organization’s business processes, there are always more things that can validly be improved than there are resources to do the work. Not all of the improvements will result in the same return of business value to the overall organization. Some benefits are more valuable to the organization than others. If you don’t pick the one that is most valuable to the business it is not the right investment for the business. In any of these improvements, there is more improving that is valid than is necessary to achieve the benefit. If the team works on improvements that aren’t the focus of the effort, “while they are there”,  they are not focusing on business value. This delays delivery of the effort and is not the right investment for the business.

Nothing in the Agile Manifesto, nor most of the Agile methods, give the tools to pick the right improvements to focus on. They also create an environment where it is extremely likely to work on more then is the minimum necessary to achieve the targeted business benefit. In many organizations, close customer collaboration can lead to “gold-plating” of features. If you offer the customer whatever they want – they will ask for it.

Wrong Focus in Practice

I have a customer right now that is going to build some software to support their field employees. The business goal is to achieve a significant cost reduction in the field through improving quality and efficiency in the field. The developers are extremely interested in developing mobile applications, distributed collaboration, GPS interfaces, image capture and markup, mapping, barcode, and training training. They have started a new business to build this product with the hope of selling it outside the business in the future. The challenge here is how do they clearly identify those improvements in the field and in dispatching that will deliver the significant reduction in field expense. If they go and and talk to the dispatchers they might find 30 things that could be done better. All of these improvements would result in an improvement in work conditions or performance. But they may not be aligned with the current business focus. The same will happen in the field.

Just because it will improve satisfaction or make someone’s job easier doesn’t mean it will help with the current business focus. How do they align absolutely the minimum amount of technology to deliver on these improvements? Especially when their interests are in this bigger, more robust solution. Especially when this limiting of features is not in line with their future business objectives. Nothing in Agile addresses how to deal with this conflict.

Where’s the Answer

I address this in Feeding the Agile Beast. This is the domain of Business Analysis. As the Gilb’s point out, most Agile methods abstract this from the team. They focus on improving their processes and collaboration. In Scrum, Business Analysis is abstracted behind the Product Owner. In XP, it is abstracted behind the Onsite Customer. I don’t agree with the implication from the Gilb’s that this abstraction is a deficiency in Agile. It is just outside the scope. I also don’t agree with Agile teams that close collaboration with the customer is sufficient to ensure business value is achieved. This is not an OR conversation. This needs to be handled as an AND conversation. This is the next thing we need to learn – how to keep an Agile team laser focused on business value AND maintain the benefits of understanding the customer that comes with Agile software development.

Dictionary.com defines a requirement as “that which is required; a thing demanded or obligatory”.

BABOK® is consistent with this. It defines a requirement as:

1. A condition or capability needed by a stakeholder to solve a problem or achieve an objective
2. A condition or capability that must be met or possessed by a solution or solution component to satisfy a contract, standard, specification, or other formally imposed documents.
3. A documented representation of a condition or capability as in (1) or (2).

The purpose of documenting requirements is not to document detailed steps that someone performs – it is to communicate the outcome needed to achieve an objective. One problem that arises is when requirements are expressed as a document of the existing process – typically through a review of existing process documentation or through observation of people doing the job. As I have discussed in previous posts, just documenting process can lead you into the How Trap and it can fail to generate a useful shared understanding. These problems arise when the focus is on how stakeholders are currently doing the work and not on understanding the challenges, problem or objective.  But, focusing on process is a problem in another way. It is an unstable lens.

Unstable

Modeling processes early in the elicitation process is not helpful to creating an appropriate understanding of the effort. Process reflects how people say they ought to do a task in perfect circumstances. The work is almost never consistently performed based on process documentation – or even how you observe the process being performed. Exceptions, reorganizations, changes in personnel within or adjacent to process, new tools and technologies, and innovation will change the process view. Even just asking five different people what the process is will likely produce five different results.

The process view can also propagate flawed understanding on the part of the people doing the work. You have likely heard the story about the recently married young women who was cooking her first Thanksgiving dinner in her own home. She had invited her family over the beautiful house she shared with her husband to show her parents how successful she had become and to demonstrate her gratitude to her parents. As she brought out the turkey for dinner her mother noticed the young women had cut the tail off of the turkey. The mother asked her daughter why she had cut the tail off the turkey. “Because that’s they way you always did it, Mom.” The mother smiled and replied, “That’s because the only pan we owned was too small for the turkey.”

A process based view hides assumption, complexity and is an unstable representation. The real requirement doesn’t change based on who you ask, who is doing the job, or the most recent reorganization. Basing requirements on unstable views doesn’t reflect the intent of the solution. This may result in an expression of the solution that doesn’t meet the needs of the business. As you observe a process or review process documentation ask why each step is important and what the intent of each step is. Consider what would happen if the steps were done in a different order or something was left out. Creating a stable, purpose driven view of the objective will provide valuable input to the development organization that will result in an improved solution.

The BABOK® says the purpose of eliciting requirements is to identify and understand the stakeholders needs and concerns, and understand the environment in which they work. This goes beyond documenting stated or superficial desires, it involves ensuring that the stakeholders actual underlying needs are understood. This is all great, but it is not useful if the requirements are elicited and documented in a way that a shared understanding is generated. Just getting the requirements documented and signed off on doesn’t help create the solution.

Failure to Establish a Shared Understanding

The myriad of stakeholders involved in an effort all have different perspectives. Their language, context, and intent are based on their particular experience. When my wife asks me to take out the trash she really means, “Talk out the trash right now, from every room in the house, replace the bags, and pick up any dirty laundry you find in the rooms you are in”. I didn’t share this understanding when we first got married. I do now – but it was more expensive than it would have been to take the time to get clarity on her context and intent. When requirements are elicited without establishing a basis of shared understanding of language, context, and intent there is the high likelihood that the requirements will be misinterpreted. This is particularly true when requirements are passed from stakeholder to business analyst to developer to quality assurance. This lack of common understanding and context often obscures the rationale for the requirement. When people don’t understand what they are doing and why they are doing it the potential for missing the mark is very high.

Establish a common language that communicates context and intent. This takes some time up front – but it is less expensive than the rework and delay associated with misunderstanding. Focus on context and intent more than detailed documentation. Verify the understanding of requirements between various stakeholders – not just a single sign-off. Clarify why each requirement is important in business language everyone agrees upon and can understand. Taking a minute to get clarity from my wife on intent would have resulted in a faster time to the desired outcome.