Leadership In Test: How Much Testing is Enough?
Editor’s Note: Welcome to the Leadership In Test series from software testing guru & consultant Paul Gerrard. The series is designed to help testers with a few years of experience—especially those on agile teams—excel in their test lead and management roles.
In the previous article, we outlined a risk manifesto to help managers. In this article, we shall ask the time-honored question “How much testing is enough?”. Spoiler: it’s down to the stakeholders.
Sign up to The QA Lead newsletter to get notified when new parts of the series go live. These posts are extracts from Paul’s Leadership In Test course which we highly recommend to get a deeper dive on this and other topics. If you do, use our exclusive coupon code QALEADOFFER to score $60 off the full course price!
How much testing is enough? This is the classic, unanswerable, philosophical question that all testers ask because they themselves are asked by their stakeholders.
Stakeholders want to know because they want confidence that systems have been adequately tested but, as they are paying for it and have deadlines to hit, they also want to know the potential cost of testing and how long it will take.
Therefore, it is for the stakeholders to judge how much testing is enough. Your job as a test manager is to provide as much value as you can to them by helping them make a decision. In this article, we’ll be covering:
- The Value Of Testing For Stakeholders
- Quantum Theory And The Theory Of Relatively (Not Physics)
- Using The Right Language
- Estimates, Budgets, And Negotiation
Let’s begin.
The Value Of Testing For Stakeholders
Every test that we run should have value to stakeholders in that it provides evidence to support their decision making in four ways:
- Evidence that the system will meet the business goals of the project.
- Evidence that the system will not fail, or if it does that the impact of failure is bearable.
- Evidence to reproduce and diagnose failures and repair and re-test the failed system.
- Evidence to support decision-making in the context of a project (to accept, to release, to reject etc.)
Our goal in testing is to create tests that incrementally increase test coverage of the system with respect to a recognizable test model. Our tests should demonstrate that the system will meet the business goals and that the risk of failure is known and, hopefully, acceptable.
Of the four types of evidence above, the first three underpin the fourth. Ultimately, the stakeholders need to make an evidence-based decision. It is their decision to make, not the testers, so it is for them to judge whether they have enough to be confident. At any rate, the value of testing is in the eye of the stakeholder.
Now, every tester makes choices on what to test by using a formal model or even gut feeling. These choices are made on the basis of some perceived value.
Unless the tester is also the stakeholder, the tester usually judges value on the basis of some measure of completeness or coverage. Or, if they know the minds of stakeholders, on whether the test will support some acceptance decision.
The above tenets have some important consequences.
Firstly, if you do not know the mind of the stakeholder, your perception of test value is unlikely to be the same as theirs. If you select tests without reference to their values, when the time comes to present your results, your stakeholders may find they have insufficient data in some areas and an excess in others. They certainly won’t feel as confident as they should.
Secondly, when you design or run a test, what is its contribution to the stakeholder decision-making? If a test doesn’t provide some incremental information supporting a decision, or if stakeholders don’t care whether your test passes or fails, then it has no place in a test plan.
We said in article 3 on models that the test models you use must be relevant to stakeholders. If your test results map to models that the stakeholders understand and value, then they will value your contribution.
Quantum Theory And The Theory Of Relativity
There are two further principles relating to the value and significance of tests. I call one the Quantum Theory and the other the Theory of Relativity. These labels sound pretentious (they are tongue in cheek for sure) but they describe two phenomena that underpin all discussions of test value, prioritisation and scoping.
When we run a test, we usually interpret the outcome as a pass or a fail. The pass/fail judgement is a binary outcome – a true or false, yes or no, a one or a zero. That test outcome generates a discrete quantum of evidence. Evidence builds up as we execute more and more tests. Whatever the outcome, that test incrementally increases the coverage of our test model and the knowledge we have of the behaviour of the system. Tests that don’t add to your knowledge don’t add value.
If a test does not incrementally increase coverage in some way, it has little value.
The second aspect is the value of a test. What is the value of a test? Could you really put a dollar, pound or euro value on a single test? Probably not. But what we can do – and often rather easily – is say, ‘this test is more valuable than that one’.
Suppose we use a code coverage model like statement coverage. We could run a test that exercises five lines of code or five thousand lines of code. What is the value of each? It’s hard to say. But if our goal is statement coverage, the second test has more value.
We cannot put an absolute value on any test, but we can usually compare tests and infer a relative value of each. That is, if we are under time pressure, we can usually say one test has less value than another and therefore de-scope the first test if we have to.
We can compare the value of tests but only if they derive from the same model.
We need to stress, however, that these comparisons are really only meaningful if they share the same model. A test that covers a large range of extreme conditions in a process probably has more value than a test of the ‘straight-through’ path. End-to-end tests of a complex process can’t be compared directly with the unit tests of critical components, for example.
While the theories of quantum and relativity may not directly apply to testing, the principles of adaptability and perspective do. Find a test management tool that aligns with these principles to optimize your testing strategy/
Using The Right Language
Now that we’ve established who is responsible for deciding ‘how much testing is enough’ how can we, as conscientious testers, support this decision-making?
Note that our treatment of the value of tests is very similar to our assessment of risks. As discussed in the previous article, it is very hard to put numeric values on the scale or exposure to risk. However, it is usually possible to compare one risk with another and rank them to make choices on the risks in scope for testing.
By using the language of risk, testers will be heard by management.
Development team leaders often seem to be listened to by managers, even when they talk in technical terms. The difference is that they present the technology as exciting and beneficial. When testers talk in their own technical terms – of the ‘clerical’ details of tests, such as incident statistics – the message tends to be negative, and managers can become bored or irritated.
Management may already think testers are slightly dull as a species, but this is arguably because many managers don’t really understand what testers do and what value they add. So testers should raise their language to management level.
Risk-based testing talks to user management and project management in their language.
These are people who think very much in terms of risks and benefits. If testers address them in similar terms, management is more likely to listen to testers and thereby make better decisions. By relating tests to the goals of the project we can focus attention on the deliverables of most value to stakeholders, so we spend our time testing the most important things.
Further, as progress is made through testing, we can demonstrate that the most valuable benefits are now available. The release decision is ultimately a judgment of whether the benefits gained outweigh the risks, so testing will be providing better data to management for them to base decisions on.
Use the language of risk (and benefits) to scope, plan and to report progress on testing.
Testers obviously need to talk technically with developers and other technical staff. For example, the quality of incident reports is a key factor in getting faults corrected quickly and reliably. I am simply saying that, when talking to management, the tester talks in terms of risks addressed and outstanding, rather than tests, incidents and faults.
Estimates, Budgets, And Negotiation
Early into a new project your project manager asks you “I need to schedule and resource the testing in good time, please can you give me an estimate for how many people and how long you need to do the system testing?”
You do some thinking and go to talk to the boss.
“I need six testers for eight weeks.”
The project manager thinks for a moment and consults his draft schedule and resource plan.
“You can have four testers for six weeks and that’s all.”
You object, and say “But it’ll take longer than six weeks! It will cost more than you’ve allocated to test this system. The system is bigger than last time. It’s more complicated. It’s certainly too risky for us to skimp on the testing this time. It’s just not enough”
But the manager is adamant, muttering something about other dependencies, higher authorities and so on…
What do you think was the point of doing an estimate if the manager knew what the budget must be all along? What relevance does an arbitrary budget have to the job in hand? Why don’t they ever take testing seriously? The developers always get the time they ask for, don’t they? It doesn’t seem fair.
You might feel aggrieved and that your professional judgment is undermined. But the problem is, and always was, that the test budget in this situation was fixed. All you can do is figure out what is the best or most valuable testing you can squeeze into your budget.
But, often, the PM really does want to know how long things will take so the plan can be adjusted. If you think you are in a negotiation, you need some chips to bargain with. You also need to discuss the outcomes of the plan, not the inputs. Scope is an outcome of the planning, the effort you apply is an input.
You need to negotiate scope.
Negotiation of test budgets should always be about scope, not effort.
Scope might be in one or several formats, and the way you present scope and discuss it will vary, but here are some common patterns. Whatever scope you have, you will use that as the basis of your estimate and your defence of it:
Scope as an inventory of requirements or features
When the estimate is cut by 30% ask, “Which 30% of the system should I not test?”
Scope as an inventory of risks
When the estimate is cut by 30% ask, “Which stakeholder risks shall cut from the plan?”
Scope as a tabulated or graphical model
When the estimate is cut by 30% ask, “Which paths/journeys/items shall I tell the stakeholders won’t be tested?”
I hope you see what’s going on here.
- The scope of testing is based on some model that has been discussed and agreed with stakeholders first. This scope is provisional, subject to resources and time being available, and you should make stakeholders aware of that.
- You estimate on the basis of that provisional scope. Use the model (risks, requirements, business process or any other model) to set a coverage target and count coverage items and estimate on that basis.
- Have a discussion with the Project Manager. If the estimate is too high then use the responses above to trigger the discussions with stakeholders.
As the tester or test manager, you are not in a good position to negotiate testing with the project manager. Stakeholders have the concerns and, if you share the models that define testing scope they will have opinions, will buy into the scope and can defend it. Stakeholders are also responsible for justifying the budget for their system, so they are best placed to balance the cost of testing with the need to address their concerns.
Some Food For Thought
Have a think about who, in your projects, takes responsibility for the amount of testing to be performed:
- Do you as the tester define the scope and the amount of testing?
- Does the project manager set a budget and you do the best you can with the time you are allocated?
- Is the scope negotiated with the project manager and stakeholders to agree a balance between effort and the scope of testing to be performed?
One of the key responsibilities of a tester is to ensure the project is aware of the product risks being taken and to document them. Only if these risks are visible can management recognize the risks they take by squeezing testing.
There is no formula for the right amount of testing. In most environments, the level of testing can only be determined by consensus between project management, customer sponsors, developers, technical specialists and the testers – tests are deemed in scope if they address the risks of concern.
Enough testing should be determined by consensus, with the tester facilitating and contributing information to the consensus process.
Sign up to The QA Lead newsletter to get notified when new parts of the series go live. These posts are extracts from Paul’s Leadership In Test course which we highly recommend to get a deeper dive on this and other topics. If you do, use our exclusive coupon code QALEADOFFER to score $60 off the full course price!
Related Articles:
