Testing Projects: Expectation vs Reality
Every day, Quality Assurance testers spend much of their time transforming code to ensure programs are as bug-free as possible and running smoothly. But this journey isn’t without its ups and downs.
In this article, we’ll shed some light on the potential pitfalls that a novice tester may encounter at various stages of the process.
Hello! My name is Nastya Mordvintseva, and I lead the testing department at Red Collar. We develop solutions for companies across a wide range of industries, working with a variety of products, which sometimes leaves us short-staffed. When we hire new people, we have to quickly get them up to speed with our workflows.
Among our recruits are those who have memorized ISTQB and excelled in class work, but fall a little short when it comes to real-world task creation and application.
To prevent this, I’ll explain how actual testing differs from textbook knowledge and how we tackle the challenges that arise from this discrepancy. Keep in mind that the testing and documentation processes at your company might differ from ours. Ready for the adventure? good. Let’s begin then.
Requirements
Expectation: Requirements gathered and structured by managers and analysts are immediately passed to testers for static testing. All defects are identified in the structure of the future product, before even writing a single line of code.
Reality: Testers independently gather all project information. Analysts may not always have time to fully structure the future product, so testers often have to supplement it based on other sources.
Where do we get reliable project information? Mostly the usual channels that are available to all testers- calls, chats, and comments on every task.
It’s also important to inquire about any verbal agreements clients had with managers that, for some reason, may not have been documented. After such in-depth research, we conduct a team sync to share findings, highlight discrepancies in requirements, and clarify roles within the project.
Product Structure
Expectation: Analysts put together the project structure, outlining the main functionality step by step. There can be several structures: for TMS (Test Management System), metrics (to identify areas with more bugs), understanding product readiness, and creating tasks that require verification as well as test documentation writing.
Reality: Testers write the structure, dissecting various user scenarios, for example. Analysts, on the other hand, may offer tips and feedback, but the bulk of the work rests with testers.
Test Plan and Test Strategy
Expectation: The QA team comes up with a testing strategy, and prepares a master test plan and a separate test plan for the first iteration. Each step of the way is coordinated with the client, and the entire process takes anywhere from a few hours to a week.
Reality: Due to the abundance of small projects, there’s often no time left for high-level strategy development. Therefore, we write a test plan that encompasses both the master plan and the first iteration plan.
The test plan should answer the following questions:
What?
What test documentation needs to be ready and what we’re doing to meet deadlines. The types of testing to employ and whether we will require developers to precisely match the mockups, or if the manager will propose alternative options to the client.
Where?
Here, we outline where the product testing will take place. We specify the platform it’s being prepared for — be it desktop, mobile devices, or both. At this stage, it’s crucial to compile a complete list of browsers and operating systems the product will operate on to understand how many times the testing cycle needs to be run.
When?
Next, we determine the time frame when tasks will be ready for testing and when they are promised to the client. Here, we always remind the manager that testing takes time. After all, we can all agree that handing over something for testing on a Friday evening-the same evening the client expects the product, is much worse than a faux pas!
Writing Test Documentation
Expectation: We aim for maximum test coverage. For critically important sections, we write test cases, and for less critical ones, checklists. We create a large traceability matrix of requirements, striving to check everything possible.
Reality: Often under tight deadlines, we write test cases only for the core functionality. In other cases, we limit ourselves to checklists.
Actual Testing
If the previous stages are done correctly, testing itself is relatively easy: we calmly follow checklists and test cases, fill out results, log bug reports, and then retest the fixed bugs.
We prioritize testing functionality first, then layout and element display come after. Time is calculated by the formula: test run time on one configuration (OS + browser) x number of configurations = total testing time.
Added risks include time for logging defects, discussing them, and retesting. Also, clients may report defects, and we need to check if we’ve already logged them. If not, we log them (after verifying the defect, of course) or include them in the task.
On average, testing takes about as much time as the test design process, but the former often takes a bit longer.
Expectations vs. Reality: During the testing period, design and requirements may change, forcing us to revert to the previous stage — test design.
Analyzing Results
At this stage, the QA lead checks test coverage and the status of runs and calculates metrics. The time allocated for this can vary.
For instance, we can identify product weaknesses based on the number of defects or simply compile a large coverage matrix.
Expectations vs. Reality: In an ideal world, we refine bugs after analyzing results, but in reality, this can turn into an endless cycle. The reason is simple — exhaustive testing doesn’t exist. Fully testing every aspect of a product is unrealistic and impractical as it requires a lot of time and resources too.
When to Stop Testing?
Determining the end criterion for testing is challenging, as testing principles dictate that we can never be entirely certain a product is 100% defect-free. Therefore, other conditions are used:
- Pre-set deadlines.
- Completion of all planned test cases.
- Reaching a certain level of test coverage.
- When new bugs or critical defects are scarcely found after a certain point.
- Management decision.
It’s also important to understand that bugs will always exist, even if we haven’t found them. But if they are of low priority, they’re unlikely to be fixed.
Expectations vs. Reality: The harsh truth is that we often stop when the budget runs out or upon a manager’s decision.
Testing Report
This document reflects all the work we’ve done: data gathering, test writing, testing, bug finding, and fixing.
Internally, we use our own templates, an approach we recommend adopting as it greatly simplifies the testing process.
Expectations vs. Reality: In this aspect, there are no significant discrepancies — reports are made, they are useful and showcase the work done by the testing side.
For Non-Testers
What should you do if you don’t have a tester in your in-house team? Leverage your user experience. It’s the simplest tool available to everyone, and can effectively reveal the most elusive bugs. Forget everything you’ve seen “under the hood” of your product for a moment and try to “drive” it.
For instance, if you’re developing an online store, try to register and make a purchase. Any non-working scenarios you encounter along the way will be the bugs or workflows that need fixing.
This approach in software testing is called exploratory. Unfortunately, as powerful as it is, it’s not a cure-all, and it certainly won’t assure you of the product’s state for certain.
But if you imagine yourself as a tester, then feel free to go through all the previous points we’ve shared, and perhaps, you’ll achieve something.
Optionally, you could just hire a tester for your project, right? 🙂
If You Don’t Have a QA
Theoretically, your entire team can test the product. Such a decision should be coordinated with the manager, who also chooses the approach and leads the process.
Depending on the testing stage, other key members that could be involved are the product manager, system and business analyst, developer, and designers.
So, How to Test After All?
Armed with my recommendations and your own experience, you can try to navigate the path of testing.
But remember, these practices are well-suited to a small testing department and a large number of urgent and creative projects. Transferring these methods to other project formats and teams may pose challenges. That’s all for today.
