Leadership In Test: Test Planning

What is a Plan Anyway?

A project plan combines the project tasks, estimated durations, dependencies, and deliverables together into a scheduled model of reality.

If you regard the plan as a prediction of the future you would be very wary of relying on it, but that’s what we usually do.

You might have encountered project managers who treat their plan as their personal reality – their own delusional world. But we should not be so harsh with project managers – they are often motivated to create a plan and deliver against it no matter what.

The plan is not reality; it is a model of reality requiring constant change.

In this article, I’ll cover every stage of test planning including:

• Deliverables
• How
• Resources
• Your Support Network
• Estimates
• Dependencies, Risks, and Assumptions
• Communication, Commitment, and Progress Reporting

But first, let’s get a firm idea of the usefulness of a plan 

Why Plan?

The purpose of planning is usually to create some agreed approach, set of commitments, dependencies, costs and schedule to deliver a project. The plan is an understanding between project stakeholders, suppliers and participants that typically sets out the following:

• What resources are required and when
• When tasks need to start and end and who will perform them
• The skills required to complete tasks
• The tools and technologies that support the plan
• The deliverables and when they will be delivered
• The costs of effort and resources required
• The process for advancing the project/process through its stages
• The risks that threaten delivery.

Some of these aspects might be specified in a strategy or already be in place. It might be known, for example, which suppliers are involved and what they will do for the project. The tools and technology to be used may already be known and in place. The people, test environments and data may be ready. There may also be a strategy defining the process, approach or techniques to be used.

But, whereas the strategy sets out the principles or the theory, the plan describes the practicalities or logistics for how the project will be delivered in reality.

A strategy sets out how a project will be executed in principle, the plan defines and confirms how the project will be executed in practice.

For many project managers the plan is what is held in Microsoft Project, but a viable plan depends on all the project participants knowing what they are doing and how. To achieve this, the plan and knowledge of how things will be done need to be communicated and committed to by all involved.

Planning is a Journey, Not a Task

To quote former US President Dwight D. Eisenhower “planning is everything. The plan is nothing.” This sentiment is so important it bears repetition in almost every context of work in systems projects. But what does it mean to say the plan is nothing? What is the point of planning if the outcome is a worthless plan? 

The plan you end up with is never worthless, but Eisenhower’s sentiment relates to the act of planning and its value compared with the plan. Let’s quickly compare how planning works in longer structured and agile/continuous projects separately.

Join our Newsletter

Discover how to deliver better software and systems in rapidly scaling environments.

• Your email*
• By submitting this form you agree to receive our newsletter and occasional emails related to the CTO. You can unsubscribe at anytime. For more details, review our Privacy Policy. We're protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
• Comments
  This field is for validation purposes and should be left unchanged.

Structured vs Agile Planning

In a longer-term project, your business, suppliers, and internal IT staff need to know what commitment is required of them so they can schedule the availability of people and physical resources.

It takes precious time to gather the required information to create a plan. With all the dependencies on resources and people, and the commitment and performance of suppliers as well as internal people, many things can go wrong. Some things will go wrong. Because of this, the plan, as a prediction of the future, is fraught with difficulty.

The day after a plan is published, and every day subsequently, new information comes to light and some adjustment needed. Requirements get scrapped; suppliers deliver late; environments, test data or tools are not ready in time. The list goes on. Planning is never a one-off task, it is a continuous — almost daily —activity.

Often unplanned events are treated as noise and don’t get much attention. But, later, some of these minor glitches become major problems. One of the challenges of waterfall projects is that this continuous adjustment can be burdensome because any changes are viewed as unnecessary tinkering. Projects often steam on regardless, hoping that things will be ‘alright on the night’. But this is rarely the case and it has been said many times:

How does a project get to be a year late? One day at a time.

The agile approach is partly a reaction to the frustration of fixed or inflexible plans. Agility is the proven alternative to the inertia of cumbersome, staged approaches. But does this mean that agile projects don’t do planning? No. 

Even in agile some up-front planning is necessary to structure the work, to marshal resources and to schedule — at least at a high level — the release process over the coming months. But iteration by iteration, and often day by day, the overall plan is continuously adjusted to take account of events and new information that comes to light.

Planning is a continuous learning journey, not a task with a deliverable.

Test Planning

So far, we’ve looked at project planning in the round and suggested it is very much a continuous activity. Now we’ll focus specifically on planning testing in projects. 

Test planning is very similar to project planning – it’s just a smaller scale plan, really. Of course, the test plan must also integrate with the larger project plan because it is dependent on other project activities and deliverables (and other tasks depend on testing). Frequently, test plans are disrupted because these dependencies are not met.

Test plans are relatively simple in overview. There may be several activities that have dependencies on the parent project. These are what often appear as tasks in the larger project plan. But these activities are distributed through a team, with perhaps a large inventory of test items to plan and execute tests for. 

This level of detail is not so relevant to the project manager’s schedule and is usually defined and managed at a local level within the test team.

Let’s look at some of the core elements of your test plan.

Deliverables

What are the deliverables of testing?

What a question! Surely they’re the test specifications, the tests, the results and reports? The deliverables are essentially contained in documentation, so all we need worry about is getting the paperwork out and ticking some boxes. 

But this approach, although common, is partly to blame for testing (and testers) gaining a reputation for being expensive, flat-footed bureaucrats who have little value. After all, who reads these voluminous documents?

Suppose there is no intention of producing test documentation in an agile project. This is more than likely, so what does testing actually deliver in those situations? What value does testing have at all? It’s a bit late to be asking these questions, isn’t it?

When a component, sub-system, or the entire system is tested, the output is evidence of how the system behaves in some situation or context. Evidence of system behaviour is collected and collated to be presented to stakeholders (developers, users, managers etc) for them to make a decision: to fix bugs, to integrate a component, to accept or reject functionality, to release or deploy a sub-system or the system as whole.

The deliverable of testing is evidence of system behaviour used by stakeholders to make a decision.

Now, in some projects, it is essential to provide test documentation to record how a test was scoped, designed, implemented and executed. But it is the evidence of system behaviour that is the ultimate deliverable of value to stakeholders.

The value of testing is the level of confidence stakeholders have in making decisions based on test evidence.

This evidence might be systematically collected and tabulated and analysed in sophisticated test management solutions and presented to stakeholders in elegant graphical formats. Or the handwritten notes might be used by a tester to present the testing story verbally to a product owner in a stand-up meeting.

However collected and presented, the final task of testing is the handover of evidence.

How

Regardless of project scale or methodology, testing depends on a certain series of activities. We’ll look at a generic process through the eyes of the tester (or team) and then look at some of the variations that occur. 

There are what might be called ‘test activities’ and there are ‘test-support’ or ‘logistical’ activities. To be sure you include all activities in the plan you might find the tables below useful checklists.

One activity in the above table you might not recognise so much is the feedback activity. You must already have had to work with poor requirements. 

The feedback, review and challenge activity is where testers, having thought through and modelled a requirement, discover issues and can use examples to demonstrate where requirements are missing, ambiguous or contradictory.

If you view requirements as poor, it is definitely worth making an allowance for this activity.

Test Logistics

Test logistics support the test activities above. Whereas the test activity list I’ve given is reasonably comprehensive, test logistics vary with every project and organisation. The list below is therefore not comprehensive. In your project, take the trouble to think through every activity or dependency required to deliver testing.

Resources (human, physical)

We often use the word resources to refer to people on our projects and, for some, that’s not comfortable. People are not things, they are human beings of course. In smaller projects, it might be possible to use names but, in larger organisations when projects teams are involved and might not yet be staffed, a number of resources is simply shorthand for a number of people.

More importantly, it’s not necessarily the number of people that count – it’s their capabilities that really matter. Of course, people work in different ways and usually at different speeds so you will need to take this into account in your planning.

Beyond people, you’ll need, at different stages, different physical resources for the testing to proceed. These range from the mundane to the highly specific and lacking any of these may threaten the testing mission.

You might already be lucky enough to have access to a fully equipped, managed, dedicated test lab. If you don’t, you may need to specify everything – from physical space and furniture down to post-its and erasers – to define your working environment.

In the table below, there are some typical resources required. Your list will differ considerably, no doubt. 

When you have identified all the resources required to deliver your plan, consider whether you need additional activities in your plan to acquire them.

Your Support Network

If the people and skills you need to do the testing were under your control, projects could be much simpler! But few teams have all the skills they need, or authorisation and access to the necessary physical resources. Many projects are staffed and run in the style of matrix management. Key members of the team actually report to managers of other specialist departments and you’ll get a limited commitment from them.

You might need to specify a set number of hours per day or schedule times to access the specialist skills above. Sometimes, you’ll be asked what level of service is acceptable e.g. ‘high-priority requests will be responded to in thirty minutes and so on.

Estimates

Estimation in software projects is a tricky business. Much has been written about how estimation is difficult or even impossible. But estimates are required in all projects and the bigger the project the more we depend on them.

We need estimates to create a schedule, but the problem is that estimation does not give you accuracy. The best we can ever do is calculate an effort or elapsed time with some level of confidence or probability.

Some people regard estimation as a black art and there is even a #NoEstimates movement with a sizable following. There is much dispute over whether estimation can ever be accurate enough or is even a good practice in software projects.

Estimation will never be an exact science. But, from my experiences, some I feel comfortable sharing these principles:

• Requirements are fallible and imprecise.
• People are more or less competent, conscientious and hard-working.
• The smaller the item of work, the easier it is to estimate; break large tasks into smaller units wherever possible and roll-up the estimates to the larger task.
• Estimation is based on experience. If you don’t have any, find where the experience of others is relevant and can be adapted.
• Your work item is unique, so look for patterns of work in other known situations where you have experience.
• Ask others to estimate and compare. Discussing discrepancies exposes differences in expectations, confidence and thoroughness.
• Estimate the best-case situation and then the worst-case situation. A good estimate is somewhere in-between.

Estimate today and start work; tomorrow and every subsequent day, with the knowledge gained, your estimate to completion will improve.

Dependencies, Risks and Assumptions

In a previous article, we discussed risk and testing’s role in risk management. Product risks relate to whether the product meets the needs of users with respect to functional or technical requirements. Here, the focus is on the risks to the actual delivery plan.

Things go wrong in projects. In your planning activity you need to make it clear which risks of failure you have considered and allowed for. There are three aspects to this: dependencies, risks and responses.

Dependencies

Dependencies represent a list of human and physical resources required, and predecessor activities, that must complete for your planned activities to succeed and deliver. 

There are always dependencies for any test activity, whether it is a large-scale system-level test or an exploratory test session of a feature. Dependencies cover three main aspect.

Risks

The risk is the likelihood of your plan failing during execution. Risk relates to things like:

• Your estimates: what could cause your estimates to be wrong? The system being more complex, defective, incomplete or constantly changing could all affect your estimates.
• People not being available, lacking experience or skills, or not being fully committed to your phase of the project. These might be team members or people in your support network.
• Infrastructure such as test environments, tools (or training in tools use) or office space not being available, being improperly prepared or defective.
• Predecessor activities not completing on time (or being abandoned, delivering partial or defective products or without delivering at all).

Responses

For each of these risks, you need to assess the likelihood, impact and where significant, an appropriate response or expected consequence. These responses tend to take one of these forms:

• Assumption: the risk is deemed low enough to ignore or regard as inconsequential. But it is on the radar, and recorded as an assumption (of availability, completeness etc.)
• Adjust: the risk is significant, but there is some action that can reduce its impact on the plan. For example, if the system under test is partially delivered, with features missing, then the plan can be adjusted to test only those features that are available.
• Consequence: some risks cannot be assimilated. If the system is delivered late, then the testing cannot start until it is delivered.

Communication, Commitment and Progress Reporting

Lastly, we come to crucial but often overlooked aspect of planning. You might produce a project plan with all tasks, participants, resources, responsibilities, dependencies, timings, effort and costs included. Or it might be a verbal understanding between the members of an agile team. 

Either way, the plan needs to be effectively communicated to everyone so all know what is required and when.

An agreed-upon plan is a contract between participants. If a team manager signs off on a plan, it is implicitly or explicitly a commitment to fulfill their team’s side of the contract.

The plan is a contract between participants.

In less formal projects there may be no written plan or commitments at all. In these circumstances, the plan is an ongoing conversation between the members of the team. The team meets daily and the active tasks in the plan are discussed in real-time. What are people working on? Is progress going according to expectations? What problems are people experiencing? What problems are blocking progress?

In all projects, the purpose of progress reporting is twofold. Obviously, there is a need to communicate where everyone is in terms of their current project activity. But the progress report is also a continuing periodic check that progress can be maintained and to verify that the commitment of participants can be relied on.

Thanks for reading, join us next time for… you guessed it, execution!

Sign up to The QA Lead newsletter to get notified when new parts of the series go live. These posts are extracts from Paul’s Leadership In Test course which we highly recommend to get a deeper dive on this and other topics. If you do, use our exclusive coupon code QALEADOFFER to score $60 off the full course price!