How To Prepare For, And Survive, Ship/No-Ship Decisions

Reality Bites

One can, sometimes even fruitfully, engage foundational issues of QA process, tools, mindset, ethos, and expertise. Which is great. These topics are necessary mental heavy lifting for any true QA professional or professional in training. They also make for good TED talks, so there’s that in their favor too. Because you can post those videos on your social media until Ragnarok arrives.

But sometimes it’s also necessary to engage the nitty-gritty realities of the QA rollercoaster, even if engaging those realities is purely an exercise in raw pragmatism, requiring no quantum leaps in process understanding, no thinking outside the box. 

But exactly the contrary: Getting down in the mud and drama of precisely what that box is way too full of can’t be avoided by thinking outside of it. Because it’s not a box, to begin with. It’s a Thunder Dome.

So this is what this article is about. It has nothing new or “game-changing” to offer, just very practical reflections and advice based on my own nail-biting experiences as a QA leader over several decades, and lessons I have extracted from them. My own QA executive realness as it were. Lessons which, of course, you are free to ignore/contradict/mock as you see fit.

It’s clear to anyone who has been in QA leadership for any length of time that the central trauma of that experience is negotiating the notorious “ship-no ship” meeting. And surviving it. 

Because this is where everything comes to a head, and QA is grilled like a murder suspect, and the detectives are all the other “stakeholders” (a euphemism if there ever was one). And, like all detectives, all they want from you is for you to sign the confession.

But it doesn’t have to be like that. All you need is an airtight alibi. I am writing this article to provide you with one.

Story Time

The biggest mistake QA leaders make going into a ship-no-ship meeting is a lack of preparation. Or rather, lack of effective preparation. 

Most QA leads or managers go into this meeting “prepared” primarily, and often exclusively, with bug metrics. A number of open issues, number of open Category A issues, number of open Category B issues, blah blah blah. Sometimes, if the rest of the team is lucky, the QA lead will be able to give some hazy, impressionistic sense of what test coverage QA has achieved up to that moment. Though usually, the data they use to do this is simply an inventory of tests and test scripts they have run. Which is not at all a measurement of actual test coverage. These are effort metrics, not true quality metrics.

The fallacy behind this strategy goes beyond using ambiguous or meaningless metrics to make (some kind of?) case for shipping or not shipping. The mistake being made here is far more fundamental.

Because the QA decision strategy outlined above is based on an obvious category mistake. The QA leadership in this scenario is assuming the purpose of the ship-no ship meeting is to present data for others to consume and make a decision on themselves. Nothing could be further from the truth, or more damaging to QA’s authority.

What the rest of the product team is expecting from QA leadership in this meeting is not yet more data. It is expecting a clear and authoritative verdict on whether to ship the product or not. Now. Today. 

Of course, testing and defect data will be necessary to defend and get buy-in for whatever QA’s expert verdict happens to be. But an authoritative verdict is what QA must provide. 

Otherwise, QA is abandoning its unique responsibility to, and authority within, the product delivery organization. It is inherently avoidant and passive-aggressive. Which does nothing to enhance QA’s standing within that same organization. 

It is, over time, a death spiral for QA’s chances to become an equal voice within the development process. A situation QA complains endlessly about, yet often is the author of its own irrelevance for this very reason.

This means that QA leadership must go into the ship-no ship meeting with a clear and compelling story that leads to an unambiguous recommendation. In other words, you have to go into that meeting ready and willing to sign on the dotted line, in digital blood, for whatever recommendation you are making. No hedging your bets. No hiding under the skirts of plausible deniability. 

Because if you don’t, you and your team will lose all credibility in the organization. And, what is worse, set yourselves up for taking all the blame if others have to make that decision for you, and the result is a quality disaster in the field. Which, is, let's be honest, exactly what the other functions want to happen. They want to blame you for everything and use your own waffling on product quality as evidence against you.

One key enabler of this passive-aggressive approach to the ship-no ship decision is the conviction — often opportunistically deployed —  that “It doesn’t matter what I say, they are going to ship anyway”. This mind-set, this strategic adoption of powerlessness, is precisely what perpetuates the organizational powerlessness of QA. Talk about a self-fulfilling prophecy! 

Politics and self-defeating psychology aside, this strategy is based on a fundamental misunderstanding of what QA is being asked to provide in that meeting. For it is not in fact being asked to render an irrevocable ship-no ship decision, even though the decision you are being asked to publicly make takes that public form. 

And, yes, you do still have to present an authoritative verdict. But to do that, you have to understand the question you must actually answer authoritatively.

The question QA leadership is really being asked to answer is this: What are the risks — their likely prevalence and severity in the field — of shipping today? As opposed to, say, next month instead? In other words, you are being asked to provide a risk assessment to enable rational risk-taking by upper management at that moment. 

This means that your definitive answer in the ship-no ship meeting is a definitive answer ultimately about risk. Not about whether or not to push the button that launches the rocket into outer space.

Which means in turn that ultimately the quality story you tell in that meeting must be scenario-based. That is to say, it will not be a simple yes or no, no matter how badly everyone else wants that in order to wash their hands of any responsibility for the decision (which is never the case, right?). To schematize a bit, your presentation should have two parts:

• First, an authoritative, *meaningfully* data-driven, statement of the current state of product quality. With metrics to back that up. Spoiler alert: You’re going to need more than bug metrics to do this.
• Second, an authoritative appraisal of the latent quality will be left on the table if the decision is to ship the product now. Is it so great that it’s worth adding time to the release schedule to collect on that quality debt? If so, what are the time/quality tradeoffs QA recommends? And what specific areas of the product’s functionality would need to be targeted during this extension, and how would we know that the latent quality had been realized?

This is not in fact a way of dodging the question being asked. It is a way of answering it in the full context of all the parameters, variables, and costs associated with that decision — today, next week, or next month. 

Because, and this is a truth you need to burn into your QA consciousness, your real audience in that meeting is not the other functional leads. It is upper management. Whether they are physically present in the meeting or not. What you say will be conveyed to them post-haste. Trust me on this.

And this only works to your advantage. Because what CEOs and COOs like more than anything else are being presented with meaningful options, backed by concrete facts and data. What they hate more than anything else is just being told, “Uh, we’re not ready to ship. And we can’t tell you exactly why. But we might in four months.” This is why CEOs go hunting for new acquisitions — to get a whole new product delivery team. 

The perennial complaint from product delivery teams is that upper management dictates, by fiat, a hard release date, that they refuse to compromise on. You might want to ask yourself why they do this so consistently. 

It’s because they can never get a definitive answer to the state of product quality, and if their favored ship date can’t be met, what is the date when it can. So they impose their own. Because the product team has left them with no meaningful alternatives.

So you must go to the ship-no ship meeting with a compelling, 3-D, fact, and scenario-based story to tell, that leads to an authoritative set of recommendations, which themselves can include meaningful quality cost/benefit options around the decision to ship — or not. Mumbling for a few minutes about bug metrics, then leaving the decision up to everyone else, is not that story.

Of course, to do this you will have to have your own QA methodology, training, and metrics in tip-top condition. I have written detailed articles on these subjects elsewhere (i.e., on LI), but, being process-theoretical, they are outside the pragmatic focus of this article.

A Happy Ending

What I have written above may seem forbidding, and perhaps a bit contradictory — on the surface. So let me end this article with my experience with a very successful ship-no ship meeting. One that went a bit differently from what I describe above, and differently from even how I expected it to go.

My team had been testing a brand new product. Not just a new product, but one that addressed a product niche the company had no previous experience in. So it was a marathon of hidden dangers, unknown unknowns, and unpleasant surprises from the start.

The time came for the ship-no ship meeting. I had, at the beginning of my tenure as QA Director, instituted a strict regime of how we would define quality metrics, including metrics for test coverage that was requirements-based, not a module or test script-based. We also had not just bug metrics, but bug trend metrics, and code change velocity metrics in place to bolster our story. 

And all these metrics were undeniably all positive. All systems were gone. No, if/then/else scenarios were necessary. I saw no reason, for once, not say loud and clear, “Ship it!”, without qualification. 

Yet, technically, it was not my role to do that. Because the project had been led by one of my QA Managers, and it fell to that person to render that authoritative verdict. This manager accurately summarized all the metrics and acknowledged they were all positive. 

Then, to my surprise, hemmed and hawed about whether it was ready to ship or not. Surprised because, of course, we had discussed what QA’s position would be before going into that meeting. (Important Note: Don’t surprise your boss in a meeting. Ever.)

I asked the manager, “What would it take to make you feel comfortable in recommending a ship decision? If not today, in the future?”

Again to my surprise, the manager could not — or would not — answer my question. It became clear to me that this manager simply did not want to take responsibility for making that decision, for signing on the dotted line. Which I found beyond disappointing. 

It also was an insult to the engineering manager on the project, who had been endlessly supportive of the QA effort and had engineered a first-class product. He did not deserve to be blindsided like this. Neither did I.

So I stepped in and said, “I have seen, and have heard, no good reason not to ship this product TODAY. That is QA’s verdict, and I, as QA Director, take full responsibility for making this decision.”

I thought the engineering manager was about to kiss me. But after that meeting, QA received massive street cred and mad respect from engineering, project management, and product management. 

Because the path to being taken seriously in our world is publicly taking responsibility. Which I felt totally safe in doing at that moment because our story had been prepared for from the very beginning. And it had been written with a happy ending in mind.

The product, once released, showed great quality in the field, And was a big success. Delaying its release for purposes of avoiding responsibility to make a ship-no ship decision would only have resulted in a minuscule increase in product quality at an unreasonable cost in money and time-to-market. 

How we did that is another story for another day.

As always, best of luck in your QA efforts.

Related Podcast: TEAMWORK, AI, AND CONTAINERIZATION (WITH NASA’S MICHAEL RITCHSON)