Improve your app quality with simple QA processes
The example of BlaBlaCar Daily, BlaBlaCar’s commuting app
In this article, we’ll go over how BlaBlaCar Daily significantly improved its quality thanks to simple QA processes.
This article was written for companies, small businesses and applications that don’t have quality processes in place, or that do have quality processes in place and are interested in discovering ours. We’ll go over 3 steps that will help you reach and keep a certain level of quality. For the sake of simplicity, all tests mentioned in this article are done manually. None of them are automated.
Before jumping into the 3 steps, let’s quickly look at what Quality Assurance is about and how we see it at BlaBlaCar.
The Quality Assurance team (QA team) is responsible for delivering quality products and features. It’s composed of QA analysts and QA engineers. At BlaBlaCar, the QA team works closely with everyone involved in the development of new features or any modifications that would impact the code. By this, we mean that we help product owners and designers shape future features of the app in the best way. We also help developers make sure the full scope of the features was implemented, during and after the development. To sum up, our job is not simply to test the app after a new increment was added, but to help the whole team deliver the best services for our users. Indeed, here is one great definition of Quality Assurance :
“The maintenance of a desired level of quality in a service or product, especially by means of attention to every stage of the process of delivery or production.”
Now that we defined the QA job, let’s go over the 3 steps that will help you deliver new features with no fear:
1 — Know your product and create an efficient regression tests suite
2 — Create Test Strategies for all the new features you’re developing
3 — Organize efficient test sessions
Step 1: The very 1st step is a one-time task that can sound a bit fastidious at first glance, but that will be the foundation of your QA job. The idea is simply to write down all the possible scenarios of your service, and then to select the most critical ones.
The goal of this task is to:
- better organise the QA work;
- make sure you don’t forget use cases when you’ll test our app, and especially new features (in the future it can also be very useful to onboard new collaborators and share knowledge)
To write and update our test cases, we use Zephyr Scale, one of JIRA’s plug-ins. If you don’t use JIRA, and don’t have lots of resources to dedicate to a Testing Software, you can find free tools online (TestRail, PractiTest, and others).
For you to better visualize this step, here are some examples of use cases for BlaBlaCar Daily. Let’s try with the sign up and login flows:
- As a member, I can sign up with a phone number
- As a member, I can log in with a phone number
- As a member, I can sign up with BlaBlaConnect
- As a member, I can log in with BlaBlaConnect
As you can see, on BlaBlaCar Daily, you can sign up and log in by using a functionality that we called “BlaBlaConnect”, which allows you to quickly create a BlaBlaCar Daily account from your BlaBlaCar one. In case you don’t have a BlaBlaCar account, you can use your phone number to sign up and login.
The idea of this 1st step is to do it for ALL the possible scenarios of your service. Inside each of them, you will detail all the steps (i.e. screens and actions) a user will perform when going through the flow. You may even add the needed setup and test material to perform the tests. That way, anyone inside your organisation is able to test it, even without knowing the app or the new feature.
To give you an idea, after 3 years of development, BlaBlaCar Daily counted 120 test cases. To compare with BlaBlaCar’s 800 test cases at the same time, after 12 years of development.
Now that you’ve written all your service’s test cases, you should identify the most critical ones, which we call the Regression Tests. The Regression Tests will constitute a subset of all your tests, that you’ll go over every time you want to check for regressions. The idea is that If one of the test cases identified as Regression Test doesn’t pass (meaning that the test fails), then the new app version that is tested shouldn’t go to production. By focusing only on these test cases, we secure the most critical flows, while saving time and energy, since all the tests are performed manually. Please note that depending on your project scope, your product or your industry, the rate of Regression Tests (# regression test cases / total # of test cases) may go from 0 to 100%. Nuclear or pharmaceutical companies for instance will have a very high rate of Regression Tests among all their tests. At BlaBlaCar Daily, about 40% of our test cases are regression ones.
For you to better visualize the Regression Tests step of this process, I’ll give you examples of Regression and non-Regression Test Cases.
At BlaBlaCar Daily, it’s crucial that our members can book a carpool. Thus, the following tests cases are identified as Regression Tests :
- As a passenger, I can send a trip request to a driver
- As a driver, I can accept a trip request
However, some of our test cases defined in the 1st step are not defined as Regression Tests, such as :
- As a member with a profile picture, I can modify it.
Here, let’s imagine that the test fails because of a bug, and that our members can’t modify their profile picture. The fact that they can’t modify it is obviously not optimal, but it’s not critical either.
Now that we’ve been through the definition of all your service Test Cases, and that we’ve identified the Regression Tests among them, let’s jump right into …
Step 2: The creation of test strategies.
Once the 1st step is completed, you’re going to go to the 2nd one, which consists in creating Test Strategies for all the new features you’re developing. At BlaBlaCar, it is the job of our QA Analysts. The idea is very simple, and close to the 1st step explained previously.
Right after the product owner has written the Product Specifications of the future feature, the goal is to come up with all the possible test cases with the upcoming feature.
For instance, at BlaBlaCar Daily a year ago, we launched a new feature that lets users modify the time of a booked trip. Thus, the Test Strategy was containing the following test cases (and many others) :
- As a driver with a booked trip, I can modify its time
- As a passenger, I am notified when my driver modifies the time of our booked trip
Depending on the complexity of the features, Test Strategies can be very small (only 3 or 4 test cases), but can also considerably impact the application/website, generating dozens of test cases.
Adding to this, if some of the new test cases are considered as critical, you must mark them as Regression Tests, in order to keep your regression test suite up to date.
The Test Strategies don’t only contain new Test Cases. They must also contain hints regarding the feature testability. Meaning, all the necessary information that will help testers test the feature. The dataset, as well as the eventual needed materials, will have to be shared with the testers, thanks to the Test Strategy.
An example of this for BlaBlaCar Daily could be when we opened our app to border countries, such as Switzerland, Belgium and Luxembourg. For trips made between these countries, we do not offer our usual 4 free trips for passengers, as we do for trips from France to France. In that case, we must make sure that all testers are aware of it, so they can test properly. Also, given passengers have to pay from the first trip, we must make sure to share Test Credit Card information. As a matter of fact, we’re not testing with our personal credit card. All this additional information must be shared with everyone involved in the testing process, and is part of Step 2: constituting the Test Strategy.
At BlaBlaCar, once a Test Strategy is finished, it is presented to the rest of the QA Team, in order to identify missing test cases and review the strategy all together. During this weekly meeting, there is always at least one test strategy presented, and it’s always leading to the test strategy improvement. Indeed, a QA Analyst who just joined the team may not know all the app’s complexity. That is the reason why it is beneficial to have everyone’s visions on all Test Strategies.
Once the Test Strategy is validated, we share it with the Product Owner, to make sure that we understood the specs correctly. We also share it with the developers who will work on the feature, for them to have all the scenarios in mind, even before starting coding the new feature.
At the end of the day, Test Strategies are initiated by QA analysts, but are built by every team member. It is a living document that should be adapted whenever a change in the specs happens or a forgotten test case is discovered.
When the Test Strategy is validated and shared with everyone, the QA team is ready to test the feature. While we wait for the development to be done, it’s a good time to start new test strategies, until Step 3 …
A few days after the Test Strategies is validated, the developer(s) will come to you and say “Hey, I’m done with the new feature, let’s test it!”.
As a QA analyst at BlaBlaCar, we would then organise a test session. During this session, which can go from 15 min to generally 1 hour, we go over all the test cases generated by the new feature. By “we”, we mean:
- The developer(s) who worked on the feature
- The QA Analyst that is following the project and sometimes other QA members
- The Product Owner who worked on the feature
- The Designer who worked on the feature
- … And sometimes special guests, such as Data Analysts or Marketing Experts if the scope of the feature requires it
It’s a great moment for everyone to be involved in the testing efforts, making sure it was visually and technically developed as intended.
To perform the tests, the QA Analyst that is responsible for the feature must previously allocate the test cases to the testers. This means distributing each test case to the participating testers. It facilitates the test session and helps make sure we don’t forget test cases.
During this test session, especially the first one, we may find bugs and create bug tickets for developers to fix them afterwards. Once the bugs found are fixed, we organize a new test session, to go over all the test cases of that new feature again. And this until all the bugs were found and fixed.
After that, and only when all test cases pass successfully, we consider the feature validated. We can then jump to the final and last step, which consist in making sure that the new version doesn’t contain regression.
The idea here is to organize a new test session, which consists in making sure that the integration of the new feature and/or of bug fixes, doesn’t break the existing parts of our application.
To do so, we test :
- All the test cases of the app that are identified as Regression Test Cases
- All the new test cases created by the new functionality(ies) that are in the tested version
- All the other test cases related to code modification, and that wouldn’t be Regression Test Cases nor related to the new functionality(ies). For instance, if the bug mentioned above regarding the profile picture modification was fixed, we’re going to integrate the following test case to the list :
“As a member, I can modify my profile picture”
During the Regression Test Session, we invite the following people :
- The full platform team. Let’s say we’re testing an Android release, then all Android developers will take part in the Regression Test Session. Inviting developers to the test sessions will increase their ownership around quality.
- The full QA Team
We generally don’t bother anyone else (such as Product Owners or Designers), as they already helped to validate the new features during the Test Sessions.
During the Regression Test Session, and with the same idea as for Feature Test Sessions, we create JIRA tickets when bugs are identified. Once all tests have been performed, we look at the bugs found and take a decision on whether or not the tested version is ready for production.
The release is blocked if the following is true :
- At least one blocker bug is found on the new feature
- We found a regression bug on an existing flow of the app (one of the Regression Test Case failed)
Here, there is usually a decision to take. At BlaBlaCar, it is the QA job, with the help of Product Owners and developers, to decide whether or not a bug is acceptable regarding its criticality for our business. Let’s imagine that there is only one bug found, still regarding profile picture modification. We will not hold the release, as the bug is still minor, but we’ll make sure to fix it for the next Regression Test Session.
If the release is blocked, all the blocking issues must be fixed before we conduct another Regression Test Session, exactly in the same way as for the Feature Test Session process. Here is a diagram of the full testing flow:
By implementing this rather simple process, we were able to considerably optimize the quality of our commuting app. We were also able to explore our product even more, which was really critical for us in a period of high growth of our service and our user base. The BlaBlaCar Daily dev team now fully owns the quality of the app. Eventually, test sessions get better and better as all testers get to know the app in depth, and also thanks to improvements we continuously make regarding test cases definition, testability, and many other aspects of the test phases.
We hope this article was useful, and that you’ll reach higher levels of quality thanks to it. Please do not hesitate to reach out to us if you have any questions!
Thanks a lot!
Clovis for the Quality Assurance team at BlaBlaCar and BlaBlaCar Daily
Contact: qa@blablacar.com
