We are Doing QA All Wrong

Over the last month I have presented or participated in several user groups and conferences. There was Southern Fried Agile in Charlotte, Agile 2010 in Orlando, a PMI Atlanta professional growth event, meetings of Agile Atlanta and the Atlanta Scrum Meet-up, and Product Camp Atlanta (not surprisingly in Atlanta). These conferences give me an opportunity to meet with people doing, or attempting to do agile in the enterprise. There are a lot of common patterns that I am seeing. One of the most common patterns is that we are doing QA all wrong in most companies.

Imagine if someone approached you up front and said “I have an idea. Let’s design our product development system like this. First, development is going to take in less then precise requirements. Then development is going to build something from the requirements. Then we are going to deliver this to a separate group that will then figure out how to test it for accuracy. Then that team will test it. When the product doesn’t pass the test we just defined we will return it to development for them to fix it.”   Is this the way that your test organization operates with your development organization? Does the relationship between QA and development seem reasonable? Why are challenges between QA and development so prevalent in our organizations?

We Don’t Understand Quality Assurance

Quality Assurance and Quality Control are different things. We have Quality Assurance organizations – but most often they are involved in Quality Control control activities.

Quality Control: Find defects. These activities are focused on validating the deliverable. They are performed after the product (or some aspect of the product) has been developed. It also enforces fit to specification. This protects production and our customers from getting defective deliverables that do not meet the specification. But when done on a completed product it isn’t doesn’t enable flow and defects found downstream of the deliverable result in rework in development.

Quality Assurance: Prevent defects. These activities are focused on the process used to create the deliverable. They are performed as the product is being developed. So, Quality Assurance should be helping us improve our processes so we don’t create defects in the first place. A key point to this is that creating a common understanding of the specification and how we will validate it up front is really important to eliminating defects.

If your Quality Assurance activities are primarily focused on finding defects and ensuring fit to requirements after the fact then your QA group is primarily doing Quality Control. Getting QA involved in improving the specification communication process through is a good place to start. A specific technique might be exploring communicating acceptance criteria to development with the same examples QA will use to perform acceptance tests.

We Don’t Understand the Cost of Rework

Finding defects after the product has been delivered ensures that you will generate rework. Rework from failed QA tests are extremely expensive to flow. Sending work back into development interrupts work in process. This has a dramatic effect on cycle time across the system – which results in increased cost of every item in the system.

After the fact testing is also typically done in batches when a release or iteration is delivered to QA from development. Defects uncovered in these batches create bottlenecks at development.  Just like when traffic backs up on the highway, bottlenecks in product development result in a long term cost to the overall system.

We can dramatically reduce the cost and adaptability of the system by reducing this rework. One way to address this defect bottleneck is to perform quality control frequently – not in batches. Another way is to prevent defects through effective quality assurance in specification understanding and in development. This will reduce the number of defects produced that escape to Quality Control.

We Have Designed the Product Development System Wrong

We used to teach that there needed to be a separation of testing and development. Having them both report to the same person was like having the person that wrote the checks in accounting also reconcile the check book. It was too easy to cheat.  We created QA teams that were completely separate from development – and that reported up through different management structures. I was talking to the director of development at a company where the QA and development chains of command meet at the CEO. When you design the organization to QA and development in different silo’s you will make it very difficult to have effective Quality Assurance involvement. You will ensure Quality Control behavior.

This separation made sense at one point (I think). It doesn’t make as much sense today – especially as the resulting silos result in slower delivery times and increased costs. What has changed in the last 10 years is:

  • the cost of setting up testing environments is much lower
  • our ability to rapidly and frequently produce testable deliverables for these environments is much greater, and
  • our ability to perform acceptance, integration, regression, performance, load, and stress testing frequently is enhanced.

Quality Assurance needs to be part of the development team and process. We need to change our thinking around this.

So we need to integrate Quality Assurance very early in the process. We need to ensure early arrival of acceptance test examples to development. But, we still need independent test teams when we are doing large or complex projects that require integration of many team’s outputs. Independent test teams can be running parallel testing or test-immediately-after delivery. When they are getting high quality components, they focus on user experience, performance, load, and stress testing.  Move everything you can into the development teams (enhanced with quality assurance) so you can produce low defect escape rates. Then eliminate the defect bottleneck and increases in cycle time by treating most of the defects at the independent test team as new specifications to development.

We have the Wrong Expectations

We Expect Perfection – Not Improved Value

Our QA organizations tend to be focused on defect identification and perfection in delivery. The focus of QA, and everyone in the organization, needs to be on producing value-add and delighting the customer – not on defect-free code. This doesn’t mean that defects are okay – but every application has some defect or limitation. Unless you are building aircraft navigation systems or other critical systems you are going to deliver some defects. Make sure the defects don’t block business value – but there is a point where it is more important to the customers and business to deliver value.

We Expect Local Optimization – Not Flow

Rather than perfecting Quality Control, QA can add a lot of value helping the organization enhance the flow of value through the entire product development system. Delivering a unit of value requires analysis, development and testing. There is no value delivered internally. The local optimization problem of improving one area at the expense of another is not useful. Crossing the boundaries between analysis, development, and testing is needed to optimize flow of value. The various teams need to figure out how to do it together – not just improve their area.

Expect our Current Constraints to be Permanent

Organizations labor under the belief that their constraints are fixed. Find a way to break down the barriers that lead to your constraints. Don’t accept the current constraints as permanent. Increase thinking at the system level to find ways to improve the product development process.

We Don’t Use the Tools Available to Us

There is an impressive amount of automation and related techniques available at very low cost to help build the QA environment that allows for fast feedback, building on progressively tested levels, and improving opportunities for frequent testing. Developers should be using tools that support automated unit testing and only checking in code that passes all their unit tests. This provides a basis for validation and refactoring. Test driven development or test  just after development should be ubiquitous – but it is not. Continuous Integration environments that ensure that each check-in results in a valid and testable platform help teams perform integration and build validation. This helps everyone work on a stable build and ensures that no one gets way off track. There are automated tools for acceptance testing and some levels of UI testing. When the tools are combined you can build a very powerful regression testing environment that can also rapidly perform performance, load, and stress testing.

The lack of general adoption of these tools probably has multiple roots. Sometimes it can be difficult to leverage all of them with legacy systems. They require new skill sets and generate new artifacts that have to be maintained. On the surface, they automate tests that are currently performed by QA groups and so their is fear and organizational threat. But some subset of these tools should be broadly adopted by each team. QA teams can focus their efforts on valued added items like Exploratory Usability Testing and Release Validation.


Most organizations are doing QA all wrong. It is time to start doing it right, the cost of low quality in dollars, time, customer satisfaction, and the engagement of employees depends on it.

  • Switch their primary focus to Quality Assurance
  • Help the organization reduce re-work
  • Integrate QA and QC with analysis and the development organization and work to eliminate defects – not just identify themF
  • Enable value, flow, and ongoing improvement
  • Leverage proven tools and techniques across the Product Development Organization to increase the value delivered

The status quo is not sufficient to meet the needs of most organizations. While not all of these approaches will be right for every situation, the rate of adoption is too low to be justified. There is far too much latent value being tied up in organizations because we are doing QA all wrong.. What obstacles do you see in your organizations?

Tags: , , , ,

17 Responses to “We are Doing QA All Wrong”

  1. Lisa Crispin says on :

    Well said! I wish I could get everyone in every business that has a software organization to read this post!

  2. Dennis Stevens says on :

    Ladies and Gentlemen. The Amazing Lisa Crispin (applause, whistles, applause). Lisa is one of the coaches who is out there actively implementing these practices.


    I think it is clear that QA is a key leverage point as we move Agile into the Enterprise. You just can’t do Agile well without effective integration of QA with analysis and development. And while there are challenges to moving in this direction – the benefits far far out weigh the costs.

    Thanks for your kind comment.

    Dennis Stevens

  3. Jon Bach says on :

    One thing: No one can assure quality. Can I convince you to change the reference to “Quality Assistance?”

    I can’t guarantee how the product will seem to the user I have in mind — even if they are part of the process — because things change. (Once had a customer who was involved, but then replaced toward the end of the project. New bugs suddenly emerged because the person saw it through different eyes.)

    What I can do is assist with *notions* of what quality *might* be.

    (I know, I know, I’m tilting at windmills, aren’t I?)

  4. Michael Bolton says on :

    One of the bigger problems, it seems to me, is that we keep using completely shopworn ideas about quality assurance and quality control from manufacturing, rather than from other disciplines. What if we viewed a software development shop as a newspaper—with writers, editors, researchers, librarians, compositors, designers, and so forth—rather than as a factory?

    Another problem is that a good idea of an independent test team for very high-risk projects (originating with IBM for its work with NASA on the Mercury program; the team was led by Jerry Weinberg) morphed into two not-so-good ideas: that every development effort should have an independent test team, and that independent testing should be the only kind of testing on the project.

    Yet another severe problem is the idea of using the inflated title “quality assurance” for testing (far worse than the title inflation that resulted in “developer” or “architect” for “programmer”). Let’s call us all developers, with programming and testing and user interaction and graphic design and documentation and business analysis specialities. And let’s make it a conscious effort to learn from each other as much as we can about each other’s fascinating domains (because make no mistake: each one of those domains can be fascinating).

    My immodest contribution to the discussion is here:


    —Michael B.

  5. Luisa Baldaia says on :

    Thanks for this post.
    You were straight and very clear about something that has been so hard to understand and implement.

    In my company we have two separated teams, DEV and QA, reporting to distinct managers. This year we gave the first steps to approach these two teams, make them collaborate with each other. We believe Quality is everywhere and everyone is responsible for it. We believe that working more closely we achieve quality results with less effort. In the future we would like to see QA team spending more time in, like you said, valued added items.
    But, as I mentioned, we are giving the first steps and there is a long road to ride. This is not an easy task to do.

    I will make sure people in my company will read your post.


  6. Lisa Crispin says on :

    Jon and Michael make some interesting points.

    Personally I would like it if testing and coding were seen as simply two components of the software development process, and not separate efforts.

    And also, I prefer ‘testing’ and ‘tester’ to ‘quality assurance’ and ‘QA engineer’ or whatever. But these are nits. The message of Dennis’s post is the point – focus on value, improvement, delivering a better product.

  7. Dennis Stevens says on :


    The unfortunate parallel to manufacturing leads the names and an understanding of a lot of things in software development being overloaded. However, I believe we can make better progress by teaching what we mean by the words – rather than changing all the words. To your point, I believe we can do a much better job of ensuring we are delivering on customer value and business value by using effective techniques on the analysis front end. This is absolutely in scope for a Quality Assurance effort.


    Dennis Stevens

  8. Dennis Stevens says on :


    Great comment. I agree with all of your points – and particularly like the view that everyone is a developer with different specialties. I really like the mind shift you recommend in the post you recommended as well.


    Dennis Stevens

  9. Dennis Stevens says on :

    Good Luck Luisa. I hope it goes well for you. Done well, the outcome is worth the effort.

  10. Jon Bach says on :

    OK, then, I’ll bite… let’s talk about the meaning of the term QA. What does it mean to assure quality?

    Even if we can do a better job of talking about what assuring (or ensuring) means when we deliver, all it would take to defeat that is any reasonable change in the marketplace.

    Your cell phone is likely not the cell phone you had three years ago. Did it suddenly suffer from low quality? More likely it was replaced by something that had more value.

    Is anyone at Motorola trying to assure that their cell phone from 2003 still has quality? Nope, they’ve moved on, and so have we.

    So, let’s assist our clients in understanding what they value, and discover whether we’re delivering that.

  11. Adam Geras says on :

    The wording in this post is clearer and more approachable than the wording in the CMMI PPQA process area.

    As I’ve said in public before, this is partly why I introduce myself to teams as the “public health nurse” for their project, and then subtly place ‘quality assistance’ in my email signature. Sometimes helds reduce the friction, if there is any, on including people like me early-on in the project.

  12. Dennis Stevens says on :

    Thanks Adam,

    That’s what I am going for. Pragmatically and clearly communicating what capabilities we need to develop to be able to deliver technology that makes a difference for the business. I am glad you liked this.


  13. Dennis Stevens says on :


    If there is risk the marketplace is going to change – we need to have an approach in place to sense and adapt to that. This is stretching a bit from typical Quality Assurance and into the realm of Business Analysis. But that’s okay, because I see the distinction between these two areas blurring in effective Enterprise Agile. How do we know we are building the right things? How do we know we are delivering business value? How do we optimize flow so the time from start to delivery is short enough that we reduce the risk of irrelevance?

    You can see some of my thoughts using capability mapping from my Agile 2010 workshop and presentation about how to manage this. The short answer is we make business value explicit, we associate specific outcomes with achieving it, and we have traceability all the way through Product Development. When we sense our business value is shifting, we adapt our priorities to the changes and continue to deliver high value products.

    Dennis Stevens

  14. Michael Bolton says on :

    Thanks for the kind words, Dennis.

    If I haven’t overstayed my welcome, there’s one more thing that I’d like to add. If we’re going to get any better at testing, we have to understand that there are actual skills of software testing, and we have to set about learning them. Most of the discussion around testing comes unencumbered by mention of key testing skills like critical thinking, systems thinking, scientific thinking. There is no mention of ontology (what we think we know), or epistemology (how we know what we know), or cognition. I can read through (and attend) presentation after presentation without ever hearing about risk, oracles, coverage, context, learning, exploration, heuristics, or skepticism. Other than from a handful of students of Jerry Weinberg (and it’s a small handful), I almost never hear about serious discussion about quality, or value, the idea that we develop software for people to use, that decisions about quality are political and emotional. I hear quite about about processes and tools, but these are made central to the conversation. We’re told to applyu them, but there’s typically little reference to judgement and skill and context in applying them. By contrast, the Agile Manifesto itself de-centres processes and tools, negotiated contracts, and following a plan. I wish people would pay more attention to that.

    For example, for those who thought that “exploratory testing” is a fancy name for “playing around with the computer”, I’d ask you to look here: http://www.developsense.com/resources.html#exploratory

    All that adds up to the fact that testing is a complex, organic, cognitive activity, rather than a rote, bureaucratic, clerical one. Testing is, as James Bach puts it, far more than a bar glowing green or red with news of the trivial.

    To those who are interested, I’d also like to suggest joining us at the Conference of the Association for Software Testing in (probably July) 2011. Watch the site http://associationforsoftwaretesting.org for details.

    —Michael B.

  15. Dennis Stevens says on :


    I have a post scheduled early next week that talks about this exact issue. There were comments at Agile 2010 from people I respect about the lack of technical content at the conference. The thing that really struck me was the implication that software took years, or a lifetime, to master. But that the other “related” practices (Quality Assurance, Business Analysis, Project Management, Product Ownership, Change Management) could be easily absorbed and applied. This statement is an example of the Fundamental Attribution Error. What I do is hard, what you do is easy.

    I am pretty lucky. I was a developer for a decade. Since then, in addition to software development teams, I have had the opportunity to manage and coach Projects, Programs and Products; develop and apply some significant Business Analysis techniques; owned large transformation projects and, managed and coached QA teams. What I recognize is that all of the practices around software product development, including development, require the right knowledge, skills, and aptitudes. All of these take time to master. Respect for this is one of the critical components of creating a successful team. Making them flow together to create value is awesome.

    BTW, I first saw James Bach present on Heuristic Testing probably 15 years ago at an ASQ conference in New Orleans. His passion and wisdom have been influential in my coaching ever since.

    Thanks for your comments,

    Dennis Stevens

  16. Ian Savage says on :

    Thanks for the contribution to the cause, Dennis. For people who like this post, please also see Crosby’s “Quality is Free” (yes, straight from the manufacturing floor… we can learn lots from that domain).

    When Crosby’s prevention mentality is coupled with effective retrospetives, we – all of us developers – can indeed *assure* quality.

    Best regards, Ian

  17. Best. Comment. Ever. | testology says on :

    [...] Best. Comment. Ever. Michael Bolton‘s brilliant description of why software testing is a rare skill From http://www.dennisstevens.com/2010/08/23/we-are-doing-qa-all-wrong/ [...]

Leave a Reply