Feeding the Software Development Pipeline

Unless a company is a software development service provider, its business is not about software development. Software likely enhances the businesses ability to create value for it’s customers. So, while Agile development helps teams get better at software development – the goal is to get better at creating value for it’s customers. The focus of Agile software development is upside down. The focus should be on the capabilities of the business. Aligning development with the needs of the business requires understanding  which business capabilities will benefit from improvement and prioritizing these needs based on business value.

After using a business capability approach to need identification and prioritization, the business needs must be appropriately scoped and communicated to development. Over the last month I have been writing about Making Agile Requirements Work. need to make sure the requests that go into the Software Development Pipeline are clear and appropriately defined. Following the six principles of Agile Requirements That Work help with this feeding of the Software Development Pipeline.

1. Articulate Purpose and Outcome
2. Establish a Shared Language
3. Provide a Stable View of the Business
4. Support Progressive Elaboration
5. Testable
6. Prioritized and Reflect Business Value and Risk

Gaining Business Benefit

The other part of the equation is to ensure the organization adopts and supports the improved software. Once the Software Development Pipeline has produced Improved Software, it still isn’t valuable until the Business Capability has achieved the intended performance improvement. Just as the elicitation and analysis of requirements has to match the cadence and needs of Software Development. Software Development has to match the cadence and needs of the organizations ability to adopt and support Improved Software. Part of the overall feedback to the organization includes ensuring the capability improvement is achieved.

Risk

At every step in this equation different risks may exist. identifying, prioritizing, and scoping feature requests have risk associated with a lack of understanding. Software Development has risk associated with the Ability to Execute and Verify the delivered software. Benefit realization has risks associated with achieving adoption and being able to sustain and support change in the organization. These risks should be explicitly managed and feedback mechanisms should be in place to recognize the realization of risks. Failing to understand and manage risks in an integrated fashion will also diminish the organization’s ability to maximize benefit from it’s investment.

The Real Agile Bargain

The real agile bargain is the maximizing of economic outcomes by increasing the speed and focus an organization can improve and respond to the market. The Agile Business Value Equation is solved at the Business Capability Level. The ability to rapidly develop high quality software moves Technology from a change obstacle to become an change enabler. This happens at the intersection of Feeding the Software Development Pipeline, Developing the Software, and Gaining Business Benefit. Solving any of these parts independent of each other does not result in optimizing business agility.

BABOK® is consistent with this. It defines a requirement as:

1. A condition or capability needed by a stakeholder to solve a problem or achieve an objective
2. A condition or capability that must be met or possessed by a solution or solution component to satisfy a contract, standard, specification, or other formally imposed documents.
3. A documented representation of a condition or capability as in (1) or (2).

The purpose of documenting requirements is not to document detailed steps that someone performs – it is to communicate the outcome needed to achieve an objective. One problem that arises is when requirements are expressed as a document of the existing process – typically through a review of existing process documentation or through observation of people doing the job. As I have discussed in previous posts, just documenting process can lead you into the How Trap and it can fail to generate a useful shared understanding. These problems arise when the focus is on how stakeholders are currently doing the work and not on understanding the challenges, problem or objective.  But, focusing on process is a problem in another way. It is an unstable lens.

Unstable

Modeling processes early in the elicitation process is not helpful to creating an appropriate understanding of the effort. Process reflects how people say they ought to do a task in perfect circumstances. The work is almost never consistently performed based on process documentation – or even how you observe the process being performed. Exceptions, reorganizations, changes in personnel within or adjacent to process, new tools and technologies, and innovation will change the process view. Even just asking five different people what the process is will likely produce five different results.

The process view can also propagate flawed understanding on the part of the people doing the work. You have likely heard the story about the recently married young women who was cooking her first Thanksgiving dinner in her own home. She had invited her family over the beautiful house she shared with her husband to show her parents how successful she had become and to demonstrate her gratitude to her parents. As she brought out the turkey for dinner her mother noticed the young women had cut the tail off of the turkey. The mother asked her daughter why she had cut the tail off the turkey. “Because that’s they way you always did it, Mom.” The mother smiled and replied, “That’s because the only pan we owned was too small for the turkey.”

A process based view hides assumption, complexity and is an unstable representation. The real requirement doesn’t change based on who you ask, who is doing the job, or the most recent reorganization. Basing requirements on unstable views doesn’t reflect the intent of the solution. This may result in an expression of the solution that doesn’t meet the needs of the business. As you observe a process or review process documentation ask why each step is important and what the intent of each step is. Consider what would happen if the steps were done in a different order or something was left out. Creating a stable, purpose driven view of the objective will provide valuable input to the development organization that will result in an improved solution.

Prioritize Requirements

Organize Requirements

Specify and Model Requirements

Define Assumptions and Constraints

Verify Requirements

Validate Requirements

Agile Requirements Analysis
All of the tasks in Requirements Analysis are important on Agile projects. The primary distinction in this Knowledge area is the support for the emergence of the solution over time. Particularly interesting is Define Assumptions and Constraints. While many Agile teams focus on the impediments to delivering code, they fail to manage the bigger risks and constraints associated with getting to business value.

Requirements Management and Communication: BABOK® Tasks

Prioritize Requirements

Organize Requirements

Specify and Model Requirements

Define Assumptions and Constraints

Verify Requirements

Validate Requirements

Agile Requirements Management and Communication

All of the tasks in Solution Verification and Validation are important on Agile projects. The primary distinctions in this Knowledge area are the support for the emergence of the solution over time. Particularly interesting are Assess Organizational Readiness and Transition Requirements as these are often overlooked in Agile projects (many software development projects). Remember, the product isn’t done-done until the customer is using it and achieving the benefits expected by the product owner.

So, What’s New?

Agile methods have emerged from a few simple concepts. Change happens and people trump process (Thanks to Ken Schwaber via Steve Adolph). In response to this, substantially all of the Agile methods describe a way of delivering software that support the following attributes:

• Progressive elaboration a little ahead of demand (for planning,  requirements,  architecture, testing,  etc)
• Incremental and iterative delivery (though not time-boxed in Kanban/Lean continuous flow)
• Ongoing learning through close customer engagement regarding product fit/business value that informs requirements and prioritization
• Ongoing learning regarding development/delivery approach and interactions that informs ongoing improvement efforts
• Self organization around the work (where the people performing the work decides how best to do their jobs within organizationally appropriate constraints)

In traditional software development, this identification and prioritization of what you are going to build falls in the role of the Business Analyst. However, traditional business analysis methods do not operate in harmony with the iterative and incremental cadence of Agile methods. For the most part, this is different than traditional plan driven projects.  Over the next week or so, I am going through the capabilities associated with Business Analysis and discuss the how Agile impacts how business analysis is performed.

The Guide to the Business Analysis Body Of Knowledge

I am going to build this discussion on work from The Business Analysis Body of Knowledge® (BABOK®) from the International Institute of Business Analysis. The IIBA® is the independent non-profit professional association serving the growing field of Business Analysis.  The IIBA has collected the current generally accepted practices within Business Analysis in The Business Analysis Body of Knowledge® (BABOK®).The BABOK® Guide describes Business Analysis areas of knowledge, their associated activities and the tasks and skills necessary to be effective in their execution.

The BABOK has six Knowledge Areas.

• Business Analysis Planning and Monitoring: Determine which business analysis activities are necessary to support an effort.
• Elicitation: Understand the underlying needs rather than stated or superficial desires.
• Enterprise Analysis: Understand what the enterprise is capable of performing.
• Solution Verification and Validation: Determine which solution best fits the business need – includes assessing the performance and effectiveness of the solution.
• Requirements Analysis: Organize, prioritize, and model requirements and risks.
• Requirements Management and Communication: Ensure knowledge gained through business analysis activities throughout the effort is shared among the team.

Within each Knowledge Area are several tasks. I will focus on how progressive elaboration, incremental delivery, close customer engagement, ongoing learning, and self organization impact who performs business analysis tasks and how business analysis tasks are performed.

Agile Business Analysis

Just to set my context. All of the Capabilities that are reflected in the BABOK happen on all projects – traditional or Agile. They may or may not be done by a specific role. They may or may not happen in a specific phase. They may or may not be done with any ceremony or intention. They do occur. Since the identification, prioritization, and scoping of requirements is so important to success, they need to be done well. And on an Agile project they need to be done so they enable – not constrain or impede – the delivery of value in projects leveraging progressive elaboration, incremental delivery, close customer engagement, ongoing learning, and self organization.

Enterprise Agility

Many companies are dependent on technology to be competitive. Either because they deliver technology as their product or because technology enables the processes they depend on. For those companies it is important to be able to operate faster than their competition.  This requires not only the ability to rapidly deliver technology – but the ability to adapt their organization.  This is the concept of operating within your competitors decision cycle. The model I use to demonstrate this concept is John Boyd’s O-O-D-A model.

This is Enterprise Agility, when the enterprise has learned to leverage technology and change management to develop an energized workforce frequently delivering value that meets the emerging needs of the customer.

Three Concepts for Achieving the Agile Enterprise

I am going to build on three concepts in showing how to build Enterprise Agility.

1. Theory of Constraints. You can’t get there all at once. It is necessary to decide where to focus. You have to develop the underlying capabilities from the bottom up. If you can’t produce quality product, the best ideas in the world will run into problems. The way of thinking about problems covered in the management system the Theory of Constraints provides the method for deciding where to focus. Identify the current constraint, get the most out of the capability that is the constraint, subordinate the system to the constraint, elevate the constraint, and then continue to pursue this cycle. This “Process of On-Going Improvement” is applied to the flow of value to the customer.

2. Creative Solution Finding. Once we understand where to change we need to understand what to change to. We have to apply a method to lead the overall adoption effort that supports aligning the capabilities with the overall goals of the organization. The stages of Creative Solution Finding are useful for identifying the situation specific approach to developing this road-map. There is no magic sauce for developing this road-map. By applying the seven principles, along with a knowledge of applicable best practices, a useful road-map that fits both the process and social context of the Enterprise can be developed.

3. The Learning Organization. Technology development efforts are knowledge based efforts. As such, they are performed through a social construct. To improve the performance of software development we have to not only improve the methods of development, we have to improve the performance of the social construct. First the individuals on the team must be willing to change. While the People Design principle in Creative Solution Finding will help prepare individuals for change, their personal fears and concerns must also be addressed. Once the constraint is identified and a road-map for improvement is developed, the learning organization concepts in the Fifth Discipline are relevant. The right individual competencies must be developed on the team to do the job (Personal Mastery and Mental Models).  The team has to be functioning well together (Shared Vision and Team Learning).  And the Team has to be functioning in a way that is aligned with the goal of the overall system (Systems Thinking).  The primary tool used to influence a social construct to create change and to improve performance is the conversation.

Discussing Big Agile

So the approach is to take the Big Agile capabilities, (1) provide insight to identify when a specific set of capabilities at a level of scaling are the current constraint, (2) leverage the solution finding model and best practices to define/refine the roadmap to move toward,  (3) and then identify the competencies to develop and conversations that are necessary to execute the change.  Then continue the cycle continuously.

This is the Big Agile improvement approach. It takes a model that I have been applying for two decades implicitly and evolving explicitly for the last decade. Clear direction combined with appropriate best practices and an effective adoption approach. All that’s left now is to build out the details of the capabilities, when they are a constraint, how to select appropriate best practices, and the conversations and change approach at each level of scaling for the model.

I have gone to dictionary.com to gain clarity on what it means to scale.  I pulled eight definitions to determine which one we intend.

Scale (skeyl)

1. one of the thin, flat, horny plates forming the covering of certain animals, as snakes, lizards, and pangolins
2. a balance or any of various other instruments or devices for weighing
3. a succession or progression of steps or degrees; graduated series:
4. a succession of tones ascending or descending according to fixed intervals
5. the proportion that a representation of an object bears to the object itself
6. Australian Informal. to ride on (public transportation) without paying the fare
7. Dentristy scal•ing. The removal of calculus and other deposits on the teeth by means of instruments
8. a certain relative or proportionate size or extent

We don’t mean what comes off fish, what we weigh things with, a way of measuring temperature, a musical construct, making a model of something, riding without paying the fare, or cleaning teeth.

So we must mean taking Agile to a proportionate size. That’s to point of the different orders of scaling. That Agile will be expressed differently at each order. But what are we actually taking to a proportionate size? Do we want to have an Enterprise wide stand-up with thousands of people? Do we want to have every job done by two people? What are we scaling?

Small team agile, as guided by the Agile Manifesto for Software Development, is a set of values and principles. Agile, as practiced in many development teams, is a set of management, leadership, and engineering practices resulting in an energized workforce frequently delivering software that meets the emerging needs of the customer. For other definitions of Agile, I went back to dictionary.com.

Agile (aj-ahyl)

1. quick and well-coordinated in movement
2. marked by an ability to think quickly

Well, we aren’t hoping that every person in the Enterprise learns to write code. It isn’t the specific practices that we want to see scale. It is quick and well coordinated movement we see in small team agile. It is the ability to think quickly and respond to emerging customer needs.

What do we mean by scaling agile to the enterprise? We are scaling the benefits of agile – the benefits of an energized workforce frequently delivering value that meets the emerging needs of the customer. We want to take this up from small development teams  so that the entire organization can be more adaptive and responsive.

It turns out that the capabilities (not the specific practices) that make the benefits of small team agile possible can be expressed at each order of scaling. And when layered together, the benefits of agile can be scaled to the enterprise level. Over the next few weeks, I am going to go through the management, leadership, and engineering capabilities of small team agile and discuss how the capabilities scale to deliver benefits at the different orders of scaling.

First off, The Agile Manifesto really applies to software development. Small Team Agile is about getting teams together to develop software in a way that meets certain criteria. Agile teams operate as a healthy team and they reliably deliver a continual stream of high-quality valuable software. Valuable means that it serves some purpose for the customer. There are engineering practices, management practices, and leadership practices that combine together to enable Agile teams to be successful. A key benefit of Agile teams is that you can produce the most valuable features first, and continue to invest until the economic value of the code has been realized. Agile teams are different then most traditional software development teams because of the focus on small incremental delivery of always releasable code and the focus on pushing decision rights down to (empowering) the team.

Scaling Agile

I believe there are four distinct levels of Agile adoption. This is not four maturity levels. It is distinct levels where the engineering, management, and leadership capabilities express themselves differently to support the expanding context. In most organizations the scaling will not be linear – it will be implemented across these levels of scaling based on the needs of the business.

Horizontal Scaling is the first level of scaling. This is the ability to get multiple teams to deliver a continual stream of high-quality valuable software. What happens when you begin to have the ability to spread Agile across teams is you begin to have the ability to strike a new bargain with the business. You can help product management interface with the business and customers in a new way where you can be responsive to individual customer requirements and changes in preferences in the market.

First Order Scaling is where multiple teams are coordinated across a common project or product. Once you have teams reliably performing, a new level of coordination arises. Your constraint is no longer in the pace of development. New opportunities to exploit speed and responsiveness arise across product development are possible. Taking advantage of these new opportunities requires the engineering, management, and leadership practices to be expressed in new ways. For example, Architecture becomes more important from a common perspective. Agile, this raises the ability to strike new bargains with the customer. Complex projects can be developed in a more responsive fashion.

Second Order Scaling is where multiple teams are coordinated across multiple projects and products. This is where you may begin to see economic benefits from reuse across products. The new bargain you strike with the business is now at the portfolio level. The business can be more flexible in investment into products and projects and then shift to other efforts when the economic benefit has been maximized.

Third Order Scaling (the Agile Enteprise) is where the Agile Enterprise exists. This is where the business leverages the ability to continually deliver high quality valuable products beyond the product development effort. New bargains can be struck in Customer Service, Sales and Marketing, Invoicing, and every customer facing part of the business. Faster learning from the market, customer preferences, and other environmental inputs is leveraged to rapidly produce products that profitably adapt to what we learn.  Other internal functions are also impacted. In fact, third order scaling often comes into play in even Small Team Agile. Once you can benefit from (or have to take into account) making changes to the way value is delivered to your customer beyond the product development organization, you need to think about Third Order Scaling.

Big Agile and Orders of Scaling

Scaling Agile has different meanings depending on the respective scale. From effective small teams, to reliable small team adoption, through products, portfolio’s to the organization -the focus is on achieving scale appropriate outcomes and purposes while maintaining focused value creation at the Agile team level. You can’t execute an Agile strategy from the top if you can’t first operate in a reliable agile fashion at the small team level.

We don’t want to try to present the one right path you have to follow – because we think each situation will require distinct strategies.  But we also want to get beyond the intuitive approach. Just saying, “Eliminate waste, identify and remove impediments, and write code better” is not enough guidance to create a reliable and sustainable approach that can be scaled .  So, what is a structured way to think about being good at Agile that supports getting big? Part of the answer is to focus on outcomes and purposes using capabilities. But how do we apply these and where do we start?

First off, teams are the organizational units that actually produce value.  So  getting small teams highly productive is the right place to start.  Then, as we scale, the principles that make the small team function well scale into different instances of the same principles. The objective is to add the least amount of overhead possible at each order while maintaining the productivity of the small teams.

Today, I present the (draft) Big Agile Learning Framework. This reflects the approach  I have intuitively used when approaching this problem. The three dimensions are Value, Velocity, and Context.  The capabilities of the organization operate together in an interdependent way.  Understanding how each one plays in this model, and how they impact each other, is a key to developing situation specific strategies for taking Agile to the Enterprise.

The Big Agile Learning Framework

Value, Velocity, and Context

Value: What the product development team provides back to the business that the business might sell to an actual customer.

When we look at value, we want to identify what Jack Welch called the value nub – the intersection where low-cost and just-the-right-features intersect. Moving up on the value axis means moving towards the nub. In Agile, we leverage the ability to get a working product to the customer frequently to continue to narrow in on the value hub. This helps us reduce the costs and risks associated with a granular up  front requirements effort.

Velocity: The rate that work is moving toward its objective.

Confusion over the distinction between speed, utilization and velocity is a common problem. Velocity is speed in one direction. The car driving 60 mph on a straight road has much higher velocity than the one going 60 mpg on a winding road.In Agile we increase velocity through finishing what we start, coordinating hand-offs to remove delay, improving quality, and continually removing waste from our processes.

Context: The environment that the team operates it. It is shaped by the team, the organization, the goal of the effort, and the sphere of influence of leadership.

Software development is a knowledge-based endeavor.  People need room to think and innovate.  Difficulties in the context can compromise efforts to improve Value and Velocity. In Agile we improve context through commitment to a productive environment, empowered teams, disciplined compliance with the team standards, working toward individual mastery, bringing in external knowledge, and managing expectations.

Each of the dimensions in the learning  framework are interdependent. Let’s take the capabilities around requirements as an example. In once case, we put in an up front focus on detailed requirements. We think they are perfect and they take a long time. So the capability is high on value but low velocity. As we move along on the project, errors in the original assumptions and evolving customer preference degrades the actual value, moving the capability down on the value axis. If we take an Agile approach, Agile recommends a lightweight, progressive elaboration approach to requirements. This would be farther right on the Velocity axis but potentially not as high on the Value axis. Over time, as we uncover specifics and move towards the value nub, the capability raises on the value axis. We also know when to stop adding things not valued by the customer – so we don’t begin to slide down the value axis again. Experience in software development shows this is effective, but the organizational context has to allow for this approach.

The Thinking Framework is a way to frame the development of strategies as we scale. As we develop situation specific strategies to scale Agile the way that we can impact Value and Velocity become more complex But keep in mind that we must not sub-optimize small team productivity. Top down approaches have not been consistently successful in scaling agile up into the enterprise. Just replicating the same practices up through the organization hasn’t worked either. Basing our work on capabilities and leveraging the thinking framework, we are able to develop appropriate strategies.

Mike and I need to write about 157,000 words over the next 9 months to produce the content for the book. We will be blogging most of the ideas as we write the book and really look forward to your comments and feedback on our approach. Learning to tell the story so it resonates is really important to achieving our goal of writing a book that is very valuable to managers in organizations who are facing the Scaling Agile challenge.

Common Capability Model

A Capability is “an ability or capacity the organization relies on for a specific purpose or outcome”. We focus on capabilities because we are initially interested in What and Why, not on How. Getting caught up in How without clarity on What and Why is common in methodology debates. We Call this the “How Trap”. Using a capability analysis approach we have been able to get people out of what we call the “How Trap” and get them thinking about purposes and outcomes.

As a core part of communicating how managers can develop situation specific strategies for scaling Agile, we are building out a common capability model. Why not use an existing model? Well, we are working from the understanding that there is no single model that is outcome/purpose focused that can be used to build out the scaling. We need to build that base – not by recreating it from scratch but by building on top of traditional models that have worked at the enterprise level, like CMMI and PMI, as well as Agile approaches that have worked at the team level like Open RUP, Scrum, and XP.

As I introduce this capability model, I will start from a Scrum/XP perspective even though those methods are based on practices and not Capabilities. For example, iterations in Scrum and short releases in XP are about Limiting Work in Progress. So is incremental development in Open RUP. I will introduce the Big Agile Capability Model in terms of  Agile practices. I will show how they are complementary to the traditional models and how this outcome-based understanding helps us develop a situation specific approach to scale. Finally, I will talk about common dysfunctions in organizations that limit effectiveness.

Capability Analysis

I have been working on this consolidated capability model for a few years. At a number of clients, I have used an approach based on a consolidation of CMMI, OPM3, XP, and Scrum. More recently I have included Kanban and the Open RUP in the model. At higher levels of scaling I rely on some other capability models like the Process Classification Framework, ITIL, Pragmatic Product Management, and the Business Architecture work I did at Microsoft. The goal is always to find a way to talk about outcomes and purposes before we talk about implementation.

The trick is in translating these jargon specific detailed models into something straight-forward that facilitates the development of situation specific strategies for scaling Agile. This is a challenge that Mike and I are working through now. Before everyone pukes on the idea of basing our approach on a capability based model, I want to cover a couple of important things.

First, I am not a big fan of staged models. I get the concept of CMMI’s levels of maturity – it is important to establish maturity in the lower level processes to gain benefit from the higher level processes. But I believe an organization can make progress on higher-level processes while they don’t have all the lower level processes in place. The capabilities are interdependent and so can’t be addressed in a linear manner.

In OPM3, the model is not staged. You pick an area of focus, such as project cost or program risk, and the model will walk your though capabilities related to that strategic area. Like OPM3, our model is not staged or prescriptive in nature. We talk about these in an order, but in reality, the team needs to be focused on improving capabilities based what makes the most sense for a specific situation.

Second, I don’t like the idea that Product Development, Project Management, and Software Engineering (http://www.dennisstevens.com/2009/03/12/project-management-product-management-and-agile/) are evaluated separately and independently of other the rest of the organization. They are too tightly integrated in real life once you get past the small team to be treated separately. At the end of the day, there is no point in getting better at any of these areas if it doesn’t result in an improvement in delivery of value to the customer.

Finally, one problem I have with formal capability models in adoption is that these models come off as overly complicated, prescriptive, and aren’t typically treated as situation specific. Typical models drive the conversation, “What do I have to do to pass an audit”. We want to change the conversations in the business to, “What is the next most important thing we can focus on to improve our ability to deliver value to my customers”.

The Big Agile Capability Model

The model that Mike and I are working on starts with Small Team Agile. We build this out to Horizontal Scaling, where you have multiple teams doing Agile development on distinct outputs. Then we build out to First Order Scaling, where multiple Agile teams are working together on a single product. At Second Order Scaling, multiple Agile teams are working on multiple products. At Third Order Scaling, we are working to leverage the ability to rapidly deliver working software across the Enterprise. Wrapped around this is a set of common capabilities that apply to every level of scaling. Over the next several days I am going to describe the capability model we have derived from our experience and research. As with everything else in the book, this model is subject to change.

I am going to describe the Small Team agile capabilities over the next few days. Over the next few weeks, I will go into the capabilities associated with Horizontal scaling, then First Order, Second Order, and Third Order scaling.

As we go through our approach, please share your thoughts – good and bad – with the approach and how it is being communicated. We need to make sure this is valuable contribution to the body of works hoping to improve our ability to deliver high quality products that meet the needs of customer.