Testing For Compliance—Or For Quality?

The world is full of software that perfectly complies with all its documented requirements but still fails to create value for its users.

When I studied software engineering - ages ago - quality meant compliance, i.e. conformance to requirements. I also learned that quality assurance means all the measures taken to enable, create, and verify quality, and that testing is a key method of quality assurance. 

Today, Wikipedia defines testing as the act of examining the artifacts and the behavior of the software under test by validation and verification. Testing is defined as an activity, leaving its purpose open. The definitions I learned at engineering school are very compatible with the Wikipedia definition of compliance testing: testing that determines whether a process, product, or service complies with the requirements of a specification, technical standard, contract, or regulation. Curiously, neither of those definitions use the word quality at all. 

What is Quality?

Nowadays, I avoid the temptation to define quality except indirectly through four dimensions: benefits, functions, process, and experience. Let’s think about something everybody knows, say Paypal. People use it for transferring money online; this is its function. The benefits they are seeking are ease, security, and low cost. Using PayPal involves various user processes and the software guides the user through them. The feelings, such as happiness, frustration, or accomplishment, the user feels while using Paypal comprise the experience. Whatever quality is, it can hardly be there without all these four dimensions.

Testing usually focuses on functional compliance, i.e. checking that the software does what its requirements state and does not do anything else. There may be non-functional requirements, too, such as response times, the number of parallel transactions handled, and the security of user’s data. Standards and regulations, such as SOC2 or GDPR, bring additional requirements, as do Paypal’s agreements with credit card companies. These may have little to do with the actual functions and value of the software but complying with them is a prerequisite for joining the business.

Quality goes beyond mere compliance. To achieve both, consider our curated list of top software testing platforms

The ideal world

Great testers apply both a compliance-based view that focuses on satisfying the requirements and an outcomes-based view that focuses on the realization of the benefits. In an ideal world, satisfying the documented requirements would imply the realization of the user’s benefits, but the world is rarely ideal. Even if the software were perfect, the users are not. The realization of the benefits very much depends on the user’s actions, and the user’s actions depend on how the software interfaces with the user and guides her. These, in turn, affect the quality of the whole experience. These characteristics of the software can hardly be captured in compliance requirements or formal test cases. Again, in an ideal world, all these challenges are resolved early enough through user-centered design - but the world is far from ideal, and a tester needs to play the proxy of the real user.

Compliance Testing vs. Outcome Testing

Compliance testing and outcome testing require different mindsets. Compliance testing is like setting up traps: reading the requirements, imagining what could go wrong, and designing test conditions under which the software is likely to fail. Outcome-based testing is more like an act of curiosity: understanding what the user needs to accomplish, imagining all those things she might do, figuring out how the software should react, and then trying it out. 

Compliance is a pre-requisite of quality. In many industries, meeting formal compliance requirements is even a pre-requisite for participating in the business. One may also mitigate liability risks by producing a documented proof that compliance has been properly tested. I tend to think that compliance testing helps you reduce the risk of failure and outcome-based testing helps you improve the probability of success.