Testing Essentials: What to test when you run out of time?

This question came from a job interview.

The answer can vary greatly depending on the application and project.

From the 7 rules of testing, we know that testing everything is not possible (skip the memes) unless it is a very simple system or functionality. In practice, however, rather than attempting thorough testing, you should target your testing efforts appropriately to the application of risk analysis, testing techniques, and prioritization.

A few days ago Michael Bolton wrote on Linkedin:

As testers, it’s our job to ask “What could possibly go wrong?” and then to perform experiments to show that it can happen. It’s not really our job to show that “it works on my machine”; any programmer worth her salt has already seen the product working.

Michael Bolton

And this is what we do – we think about the quality, we try to break as much as we can before the software comes out of our comfort zone – the dev environment. We test, try, and create the new test cases or explore risk areas.

But what if, apart from the complex system, the tester is also very limited in the time spent on testing? What if tested functionality is critical or extremely important from the perspective of the entire system?

1st – You don’t have the Time Turner

Not many of us have a Time Stone like Dr. Strange. Perhaps only he had it. Therefore, if you hear the sentence “this functionality is critical – you have 3 hours for testing” – it usually means that you have 3 hours for testing and this time cannot be stretched.
Count how many people are in your team. Divide the tasks. Plan your assignments and get them done!

Only you – Tester – know your system and just you can assess what is the most critical in a given situation.

2nd – FOCUS

Many factors distract us in our daily work – e-mails, matters not related to the project, private problems, smartphones. If you are in a time-limited and high-risk situation – your attention must be focused.
It takes forever for a minute if you concentrate on NOW.

3rd – first things first

Prioritize areas of the software that have the biggest impact on your users. What will you be testing, and why exactly THIS is important for your application?

What happens if you don’t test it?

What will be the impact of errors in this area on the user?

4th Design a set of tests to check the functionality

Of course, I don’t mean the 3-hour planning- rather a division of tasks in time.

5th Log results

Record what you are doing – just like exploratory testing. Note down questions, doubts, errors, their criticality, and possible inaccuracies with the documentation. Record defects in the tool your team uses (Jira, Bugzilla, HPALM etc.).

6th Prioritize defects based on their severity

In a situation where time for testing and fixing defects is short, it is important to determine which of the errors found are critical to the operation of the application and which can be resolved later.

It is worth starting with the so-called – low hanging fruits – meaning important bugs that are relatively easy to fix, then move on to the most valid but more complicated ones to repair and test.

7th Have the highest severity defect fixed


In one of the projects I participated in, a defect about the incorrect shade of green on the “Send” button, reported by the test team as “Enhancement”, was considered “Critical” by the client, because the success of the entire sales process, that the user has been through on the website, depending on it .

So, make sure what the real priority of found bugs is and then decide which should be fixed first.

And the most important – don’t forget about RETESTING.

8th Collect lower-impact defects to resolve them later

Many times I have encountered the situation that if the manager / PO / PM says – “Test this critical functionality in no longer than X” – and it is Friday afternoon or the end of an important development phase – usually it has “and do not report any defects” OR in the alternative version” and we will fix the reported bugs later. “

Even the ISTQB syllabus convinces us that the belief that finding and fixing a large number of defects will ensure a successful implementation of the system is incorrect. For example, very thorough testing of all specified requirements and fixing all defects found may still not save us from building a system that is difficult to use. It may be an application which will not meet the requirements and expectations of users or will have worse parameters than competing solutions.

You have to accept the fact (I know it’s hard, breathe) that not all defects will be fixed.

Some of them will not even be even found! It’s important not to give up and NOT to blame each other for failures.

My final thoughts

I understand terms such as “contractual penalty”, “good of the project”, “bonus for the manager for delivering the project on time”, but I am a software tester, usually quite stubborn in my views on product quality. That’s why I often suffer from selective deafness and report defects, I insist on fixing them, and negotiate the time that is not there.

I encourage you to care about the quality of the software that passes through your hands. Perhaps it will not be you who will be ashamed if errors appear in production.

Perhaps, the end user will suffer from crashing application.

Do you have experience in testing under a herd pressure?

About agile recruitment, agile organizations and employees trying to keep up

The word Agile has become so fashionable that the LinkedIn community cannot live without it anymore. Still, many job advertisements say that the company is looking for an agile tester and developer (if not currently looking for a ninja). It may sound posh and fashionable, but as long as we do not want to employ a circus troupe, agility shouldn’t be the main recruitment criterion. At least in my opinion.

When browsing the feed on LinkedIn, you get the impression that regardless of the organization – not being agile – means not keeping up. Even when the organization is a large iceberg that is effective in its navigation, it tries to give the impression of a little jumping bunny. We can observe daily glitches of: agile development teams and agile testers (quite accidentally filling reports in Excel and tools that can be dated C14), agile projects (with Test Managers and Project Managers), and overall environmental agility. And fruity Wednesdays.

We also observe a growing group of Agile Coaches and a multitude of online courses and webinars on this subject, which, like a fairy, can fly into the world of ancient projects, wave the wand, and from that day on, everyone will work efficiently, happily and deliver twice as much.

I’m afraid it can’t be that way.

Does this mean that truly agile organizations don’t exist?

It depends.

Let me move to the familiar topic of software testing. Twelve years have passed since Lisa Crispin and Janet Gregory published the book Agile Testing. It is already a full-fledged teenager. Since its continuation – More Agile Testing – 6 years.

Are the conclusions from reading about agile processes taking place in organizations valid also in 2020? Today, an agile approach to software development is no longer a novelty, but rather a standard. At the same time, like every common issue, it has had its modifications and pathologies.

The understanding of Agile has changed and organizations have started to interpret it in a way that suits one another. Unfortunately, as my project experience shows, the term Agile has become a convenient replacement for the lack of complete or unambiguous customer-side requirements, lack of documentation in the project, or simply organizational clutter.

The phrase “we work in agile” repeated like a mantra during recruitment, often means “we do not know which way we are going, but we row with all our strength“.

On the other hand, agility is nothing more than a way to reduce the cost of change and uncertainty.

Like I usually do, I decided to raise this issue in a wider group by asking a question on Twitter:

So many different views popped up on the same issue. An interesting discussion resulted from the question, ended with a live debate with the organizers of Agile Testing Days, the conclusions of which I would like to present to you.

Agile – what does it mean?

Constant improvement and constant adaptation.

A Feedback culture.

A space for personal development.

Change and adaptation.

The most memorable, however, are two sentences – Agility has different shades – and Agile means no fear of failure. This is probably what agile work is at the heart of today. It is not only following the pattern of XP, Scrum, or Kanban, nor is it the famous “we work in Agile because we have Dailies“, nor is it a colorful board in Jira. Agility is a different way of thinking.

How was it before?

In the traditional design model, you have done your best from start to finish to achieve the goal flickering in the distance. It was far, lots of dots to combine, but all were going in the same direction. The problem arose when, after a few months of work, it turned out that the client meant something completely different, and your dedicated work was…. in short, not needed.

In such situations, there is usually a wave of mutual accusations in the project team: “You should have written the requirements better”, “You should have coded better”, “You should have tested differently” etc.

The whole team became frustrated.

How can it be?

In the agile model of work, you encounter such situations every day, but you are mentally ready for them, you can adapt them as something normal, and accept the change.

A bit like in this joke:

Two friends meet. One confesses to the other that he wets himself at night. A friend recommends him a great psychologist, a well-known specialist. After some time of therapy, colleagues meet again and the guy recommending psychological help asks:

  • Have you been to a psychologist?
  • I was.
  • How are you now?
  • Great!
  • Did you stop getting wet at night?
  • No, but now I’m proud of it!

And that’s a bit like Agile. The point is not that after adopting a new way of working, it will magically change the project. On the contrary, not only the team, but most of all the client will learn to live with small failures and constant change. Sometimes expensive ones.

The agile approach to work can help both – the project teams and the clients – understand that no one can predict market needs. No matter how hard we try – often the planning effect differs significantly from the final product, and it is influenced not only by our willingness or the ability to work in a team but above all by the budget, software, or market limitations.

We have to accept it and learn to live with it. At the same time, agility does not have to apply only to the team, but also to the entire company and all its employees.

In the above context, is the sentence I heard in one organization true:

“We are an agile organization because we switched to remote work during the pandemic.”

Survival – This is not agility.

Agility comes from willingness, from trial and error, not from a panicked attempt to save what you can. Moreover, adapting new things to evolve, maintaining positive trends, learning from mistakes, and adapting, not blindly returning to “what was before” – this is what, in my opinion, agility is.

Besides, it takes time to implement a different organizational culture and way of working.

If a given organization, a given project team, decides to evolve from a traditional model to a lighter one, they need time. It is not to be expected that the Agile – Fairy will come and use a magic spell to heal all the problems that have plagued the project for months or years.

Is it then worth adapting the change?

In 9 out of 10 cases I will say yes.

However, it cannot be done by force, or just because it is fashionable. A different approach to software development, recruiting, or treating an employee should be the result of evolution and the real need of the company and not a trick.

How is your organization like? Do you work in Agile?