API exploratory testing: an opportunity to improve your API testing
Why explore a new test approach?
Innovation and continuous improvement are important for companies to adapt to the challenges. As Staff QA, I am always challenging the status quo. Especially, with my team, we are looking for improvements to our test strategy and practices.
Our context: ManoMano has opted for a microservices architecture. Of course, our testing strategy includes (automatic) API testing. However, we had several questions. How can we be more user-focused regarding API testing? How can we continue to detect new defects, avoiding the pesticide paradox test principle (now called tests wear out)? These questions are why our story of API exploratory testing was born. Let me tell you about it in this article.
Birth of an API exploratory testing at ManoMano
According to Wikipedia, Cem Kaner, who coined the term in 1984, defines exploratory testing as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”.
The book Explore It! by Elisabeth Hendrickson is a must-read if you want to learn more about exploratory testing. If you have ever read it, you may have noticed that she addresses the subject of API exploratory testing a bit. However, if you want to go further I recommend the book Testing Web APIs by Mark Winteringham. After reading these books, the idea of performing API exploratory testing grew within our team.
This is why in 2022 we prepared our first session. Our Lead QA, Fred Vigna, initiated a workshop. We had a lot of questions, such as: What to explore? What do we want to discover? How to prepare? Do we have to record the session, and all the requests sent? After this workshop, he initiated the first charter.
I prepared the technical aspect. Such as selecting the endpoints, and preparing the tooling configuration. Then I had to make sure that we were able to have at least one successful response and an unsuccessful response. I also verified that we all had access to the GitLab project(s) related to the API, including Open API documentation, the code source, and all steps of the pipeline’s results.
One afternoon, we started the first API exploratory testing session. This is how four QA explorers embark on an adventure for two hours. The scope of the first session was to discover behavior not documented in the official documentation. Using a Mob Testing approach with the Pomodoro technique, each one of us had the chance to propose and test his/her ideas. The coolest part was learning and teaching others: why did you try this ‘.’ value? Isn’t it supposed to be a 403 instead of a 401?
But this is not the only output, here are some KPIs that we decided to follow for this first session: 4 endpoints tested, 9 bugs created (mainly concerning documentation and Edge cases).
Is it a good result? Let me give you some important metrics:
- Even if all these endpoints were in production we found several bugs: 8 out of these 9 bugs were reproduced and therefore fixed.
- Tester’s satisfaction: we did a retro after the session and 67% of our feedback was on the “good” side.
For us, our results were extremely promising. Our idea was to make some improvements and then onboard teams to perform exploratory testing before deploying an end-point in production and improving our defect escape rate.
Next, the doubt phase
We organized other API exploratory testing sessions. We onboarded new people (QA). Anurag Pandey and I created a rotation for the preparation of the technical part, such as finding relevant data for authentication, request body, and initialization of the tool(s) used. Each time we tried to improve the process a little bit, helped by the discoveries we had made.
Unfortunately, five sessions later we did not progress as much as we had expected. We performed exploratory testing mainly with QA and the process was not improved that much. We were in a kind of a comfort zone, always executing the same kind of test: trying the same inputs, with and without authorization, with an expired token, etc.
If we always perform the same tests: why do we do it manually? Exploratory testing allows the person doing the test to think creatively and think out of the box, it was not the case anymore. Even if ‘testing is not a phase’, the doubt was a phase and we were full of doubts. We needed an innovation trigger.
Let’s rock… and roll!
A few months ago, ManoMano decided to launch the first Hacking Weeks, some specific days for innovation. If you want to learn more about the Hacking Weeks, Stefan Toma wrote a medium article. For us, it was obvious that we had to propose a subject around API testing. It was even the perfect moment to level up our exploratory testing practices on APIs. We were convinced that API exploratory testing had enormous potential. And finding how to reboot was not so complicated. What is the ‘new’ IT technology that is making the mainstream press talk about? Artificial Intelligence (AI) and this also affects API testing. To follow what we had as an idea we will have to go back in time a little.
So, let’s go back to the first half of this year, when I was included in the Early Access to Postbot (Postman’s New AI Assistant) beta. I tested it with Maria Avramescu, a QA colleague. We thought it was promising but it wasn’t an open beta so we didn’t get as far as using it regularly. Now that Postbot is accessible on every account, it changes the game.
So the idea was to leverage exploratory testing using PostBot, Postman’s New AI Assistant. An extract of my first draft of the pitch reads like “For this hackathon let’s combine the best of both worlds. The IA will allow us to find other kinds of tests and create a new list of possibilities. Why is it innovative? As far as I know, no company merged IA and exploratory testing of APIs.”. On this subject, I was joined by Anurag Pandey and Stefan Toma. In our mind, we had many ideas to investigate. Can we use AI to assist us in finding new tests to perform? Can it allow us to go faster? And why not reopen our minds to think out of the box?
Hacking week step-by-step
The three of us decided to start with an API exploratory testing session, to see the pros and cons of the current process. The result was not surprising:
- Good: we are always discovering new bugs. After one session a new participant is able to perform it. New participants are learning a lot.
- To improve: after a few sessions, we are not learning anymore. We are doing repetitive tests. We are thinking only as “testers”. We are, mostly, trying to find bugs in edge cases, security issues, and documentation that is not up to date. Even if this is relevant, the testers are not the users. Shouldn’t we think and act like users would?
First action: Act like a user 🙂
During a brainstorming session, we thought about a lot of API users! Such as an internal Developer, a junior Developer, a senior Developer, a QA, a PM. But also a business person who will need to use the endpoint to perform some actions (mostly via a front interface), the hacker, the developer from another company (for public APIs). As an action, we decided to design the main relevant users with personas. We created six persona cards, one for each of them.
Each exploratory testing session can use these personas or a new one can be created for specific needs.
Second action: reducing (eliminating) the redundant tasks
As former testers, we said to ourselves: let’s automate them! Of course not automate exploratory testing but the way we are testing. Our first task was to look at existing templates and to create new ones adapted to our needs. In parallel, we created a Postman API workspace. Then the selected templates have been added to the workspace and documented. Last, we experimented with the templates on some end-points from different feature teams to validate their usage and their documentation.
As an example, we created one to perform fuzzing testing using the big naughty list. You can copy-paste your endpoint, add a variable, and run it 👍 And if you want to add it to your pipeline? We also created a GitLab template project.
N.B. to perform exploratory testing on API we can use other tools but we chose to focus on Postman for this Hacking week.
Third action: how to find other ideas?
This was the moment we focused on Postbot. I would take an entire article to describe in detail this AI, so I’m not going into deeper details but let’s say we used the prompt and with a little imagination (and knowledge) we explored new tests to perform. If we want to use it, be very careful, we always need to understand the tests created, and sometimes they aren’t relevant at all. A human review is still essential.
Our exploratory test session benefited greatly from Postbot AI, enabling us to dynamically create API test cases and let us quickly make different kinds of tests for our system and see how it reacts.
Last action and not least: sharing with the company
Our last main subject was to start spreading the process. For that, we reviewed and completed the charter and the documentation. And prepared a demo shared with the whole company. Now we are starting to onboard feature teams with mob API exploratory testing on their endpoints. The whole ManoMano tech adoption is on its way.
“Explore, dream, and then explore some more” 🗺️
If you plan to perform API exploratory testing remember: exploratory testing is a mindset, not a testing technique. Explore new paths, always.
The first Hacking Weeks finished a few weeks ago. Wait for it, finished, is it? Not really! For us, it is a new start.
Having had the opportunity to participate along my colleagues to the inaugural ManoMano Hacking Weeks and on such an exciting subject was definitely a breath of fresh air. I’m already looking forward to seeing what other topics we will be working on the next time we will do Hacking Weeks. At least from a QA perspective, we’re already thinking about what other subject we could explore next
We ❤️ learning and sharing
If you’d like to get in touch or just talk about QA in general, I’m always reachable through my LinkedIn profile. Drop me a line! Whether you had a similar or different experience, I’d love to hear about it.
