Testing when things are chaotic
There are many pieces that stitch a product experience together. On Josh Skills, we have users who spend close to 7 hours every day either practising with partners or taking lessons. Imagine if something goes off in the product where people spend that much time in a day, it will certainly disrupt their flow of learning. It would be downright frustrating for them and distressing for us. But sometimes, unforeseen risks do appear and even the best of things go haywire.
Since we stick to promised timelines, we do not want to perform exhaustive testing all the time, but we are definitely focussed on avoiding any such situations. So, we have adapted a Testing approach that helps identify and curb any major disruptions before the product release.
Identifying the risks
In tech-first companies, the QA team is generally trusted with the task of delivering a high-quality product to the end users. This process can become challenging very quickly at hyper-growth startups where you need to ship features fast without bugs — the complexity of the product increases by the day, the testing time however remains the same. This creates major bottlenecks to releases and exacerbates quality issues.
While testing during chaos, there’s a risk of an unforeseen event leading to problems/delays. There could be —
- Bug leakages
- Product performance issues
- Tested feature not merged/untested feature merged
- A/B tests not shutdown or not enabled etc.
Reducing the risks
In times of chaos, the QA needs to take a bird’s eye view, adapt and come up with an approach which involves test analysis, planning, estimating, designing and executing based on the identified risk prone areas rather than spending time testing the entire product.
So in times of chaos when we’re already dealing with a time crunch, this is how we perform risk-based testing. The process usually involves creating a plan for risk-based testing and trying to address each of the following questions:
- What features should be covered?
- What are the priorities for testing? What needs to be tested for sure? [the priority and the order in which it has to be tested]
- How can we effectively utilise the time available for testing? [Reduce the testing efforts for stabilised features and focus more on features with high business impact]
- Is there anything that doesn’t need to be tested? [Identify the features with less business impact (comparative analysis )]
The first step is usually to identify the features which are crucial to give/retain the best user experience and which have a high functional and business impact. Product Managers/Dev Team/QA Team work closely to achieve this.
We then categorise the test scenarios according to impact areas. For example- if something is interrupting the learning practice for our users, it might be impacting them in a greater way and we certainly don’t want to put that at risk. So, whichever feature has the greatest impact on user experience is always the highest priority.
The prioritisation of the features is made in the order of P0, P1, P2, P3, P4 (P0 being the highest priority and P4 being the lowest priority) and the features which do not fall into the prioritisation categories are not tested. This way we are able to increase the coverage as well as reduce the time taken to conduct testing.
Retesting and regression testing
After we’re done prioritising, the next step that we follow during round 1 of execution is Defect Triage — due to the impending chaos or time crunch, only the defects under P0(Showstopper) + P1(Critical) + P2(Major) are opted for fixing whereas P3(Minor) + P4(Trivial) issues are deferred for the next release.
Once the Dev team completes fixing the identified defects, the second step is to retest those fixes and proceed with the corresponding regression testing. The regression testing cases are selected in such a way that it covers the areas of high business impact, integration points of the features and the features with the current defect fixes. This way we effectively use the available time to achieve stability and quality.
Prod sanity testing
We don’t stop there, our team members are also the users of Josh Skills — so we go ahead and start our learning , re-learning and thereby complete Prod sanity testing.
Calming the chaos
In the end, I believe that the QA team’s job goes far beyond finding bugs in the product. Some people say — a product is as good as the testing team behind it but when the testing is not accompanied by the right expertise, experience, and a commitment to excellence, the chaos tends to become a disaster, worsening the user experience. So, with the right minds, right approach and with a great team, we are focussed to deliver the best experience.
We’ll be covering more about our processes for Testing and the tools we use in future blogs, so stay tuned!
If you liked this, you may also like —
When to rebuild a feature and how to do it
