Applying Session Based Exploratory Testing to Gaming
Bring Them On!
Introduction
“Anyone can cook” was the saying that Cheff Gusteau from Pixar’s animated movie “Ratatuille”, used to mention. And that, indeed, can also be applied to testing. “Anyone can test” is something that is heard in many companies, from different actors and players. And that is true. Testing is one of those disciplines that does not necessarily need a specific background or knowledge to start working and obtain a decent performance while doing it. However, if you want to be a professional and get the best results, you need two basic things: to be prepared and to train and gain expertise in certain abilities that will help you excel in your task. When it comes to game testing, these abilities are even harder to acquire. It is easy to deviate from the true QA objective, and say “Ok you only need to play the game”. And that is a common misconception. Testing in Gaming is both hard and exciting. On today’s games, testers need to cover multiple edges, from visual hitches that are subtle to the eye to sound and graphics, without leaving playability aside. Multiple edges that easily can get lost in the middle of the passion playing an exciting game.
In this article I explore a new approach, based on Session Based Exploratory Testing to enhance the team’s capabilities to find bugs in this challenging environment.
Do you need to be special to be a game tester?
As said before, “Anyone can test”, and that is true. But when it comes to gaming, in addition to having a Tester’s Mindset, you need to be passionate about gaming. Tester’s Mindset includes having attention to detail, an unbreakable spirit to keep finding those hard-to-find bugs, questioning and interpersonal skills, among others. But if you have all these skills, in order to be a successful game tester, you also need to have a unique appeal to play the same bloody game over and over again, without losing focus in all the important aspects of the game itself, and all the multiple types of bugs that may be found in a game. We are talking about dozens and dozens of different types of bugs, which are easily overlooked if you do not have a tester mindset, in addition to having a unique appeal for playing games.
“Testing is a skill. While this may come as a surprise to some people it is a simple fact.”
Fewster, Graham: “Software Test Automation”
A special challenge: the limited human attention.
Humans have a limited ability to pay attention to multiple stimuli. And this is a key fact when dealing with games. In this article we explore a particular First Person Shooting (FPS) game. In these kinds of games, a tester needs to pay attention to multiple stimuli that are an essential part of the game while playing it. Just to mention a few: changes in the environment (while wandering around the map, user needs to pay attention to the different characteristics of the map, in order to check possible paths to follow, or parts of the map where an enemy may come or hide), sounds (player must be alert in order to detect possible enemies according to the perceived sounds), items that are fetchable from the environment (such as weapons, health kits, etc.), hidden parts of the map that may lead to hidden treasures or paths), enemies and Non Primary Character (NPC) that come across the player’s route, among many others. These stimuli may capture a player’s attention, and prevent him/her from finding valuable bugs due to not having a full attention to the different characteristics of the environment, music, special effects, movements and other types of characteristics that may contain bugs. Furthermore, as human attention is limited to a particular number of stimuli, it is important to know how attention works in humans, in order to take a good approach when defining a gaming test strategy.
Image 1 — Attention (Watterson, 2015)
So let’s start with defining what attention is:
“Everyone knows what attention is. It is the taking possession by the mind, in clear and vivid form, of one out of what seems several simultaneously possible objects or trains of thought.” William James
Attention is an adaptive process which objective is to filter the information, given the processing limitations that human beings have. Thru history, there are two different conceptions about attention:
- As a quality of perception
We cannot pay attention to all stimuli, and attention is the process to select only those relevant. - As control mechanism
All cognitive processes need supervision in order to be adequate to an objective.
Image 2 — Sohlberg & Mateer Attention levels (Scott & McCall, 2020)
From bottom to top, we start with a focused attention where we can direct the attentional window to a unique source of information. As levels go up, the attention is more diverse and less focused. When we are in a divided level of attention, we can respond to multiple tasks or demands simultaneously. It is a limited capacity. As the number of sources of information increases, or the demands for a task are higher, the execution gets worse. The quality of the execution in multiple tasks and simultaneous, depends on how automatic those tasks are (Universidad Intercontinental — Facultad de Psicología, 2020).
So, if we consider the gaming exercise, it will really depend on the type of game we are playing. If we go back to the 80s, we may find different games that had a fixed amount of stimuli for the player, like Green Beret (see Image 3), Saboteur (see Image 4) or Ghost n’ Goblins (see Image 5).
Image 3 — Green Beret (Future PLC, 2020)
Image 4 — Saboteur (Indie RetroNews, 2020)
Image 5 — Ghost n’ Goblins (Crider, 2020)
In such games, the player used to have a limited number of enemies, with a fixed map, with only one direction to take (generally from left to right), and with the same amount of enemies to confront, and they usually came in the same moment, and following the same pattern. Today games are totally different from those original ones. Taking as an example First Person Shooter games, the player faces multiple and complex maps, with kilometer-sized structures to wander and roam into. For example the most updated Doom version (2016, see Image 6) gives the user several maps to dive into, with many different types of enemies, which are powered by Artificial Intelligence (Ortiz, 2020).
Image 6 — Doom FPS game
These types of games make attention more difficult to make, as multiple stimuli are given at the same time for the player. Now, maps are more complex, with a 360° freedom for the user to dive into, many different types of enemies which are freely wandering around the map, and many options for the player to have in hand. For example, multiple ammo to select, multiple paths to hide or run to, objects to step over and structures to climb, among others. All of these combined with multiple sound effects from the environment, makes a unique experience for the player. These visuals and sounds combos makes the player have a divided level of attention, making it difficult to focus on what to look at.
Bug Gaming taxonomy in today’s gaming industry
In today’s gaming industry, we may have a very diverse variety of bugs that may be caused by different factors. If we think of FPS, we can have more than 30 different categories for bugs, ranging from Art related bugs, UI, Sound, Stability, AI among others (see Image 7). This will easily reach 100+ different types of bugs, summing all categories.
Image 7 — Bug Taxonomy for Gaming
So if we consider that a game tester has to always bear in mind how addictive the game is, how playable it is, and therefore the player needs to get into the game with all his/her senses to deal with the adrenaline of the game. Additionally, as a tester, the player needs to focus on all the 100+ possible combinations of bugs, as stated before. Considering the limited level of attention that Humans have, as analyzed before, it is really a very difficult task to carry out. In terms of testing, this may easily lead to undiscovered bugs. Imagine you are fighting with 5 to 6 aliens that come towards you, and you need to aim them with the proper gun, take care of your health indicator, and in the path you are following to not step over a mine or some lava that is there in front of you. You may easily not notice that a particular texture of an item is not correct (Art / Incorrect/Wrong Texture UV), or that a specific box or any other item that is some inches away from the central view of the screen is actually floating instead of laying in the ground (Model / Scale Placement), and we can continue with the rest of the 100+ types of bugs. So under this scenario, we are facing a bug detection problem, not related with the tester itself, but with the type of application we are testing, and the Human capability regarding attention.
So, what do we do?
First let’s do a brief summary of the problem that we are facing: when talking about game testing, we have mainly two items in sight: as in any other testing discipline, we have a goal to find bugs. The more bugs we find, the less bugs the end user will encounter, so more robust our application will be. On the other hand, a very important part of our testing will be focused on the playability and how addictive our game is, as this is also a key element when talking about game testing. In this article we are focusing on the first item: how to maximize the use of testing time in order to find more bugs. And as said before, our main limitation is the human attention, as we have a vast diversity of bugs in sight (remember we have 100+ types of different bugs that can be extracted from Image 7).
So in order to cope with this situation, we decided to try a Proof of Concept (POC) with Session Based Exploratory Testing (SEBT). SEBT is basically a testing approach that maximizes the testing time used by a tester in a limited amount of time (called Session), where the tester focuses only in a specific part of the problem (called Charter) and uses the session time exclusively to test that charter (not logging bugs, nor testing other parts of the application). You can get a thorough understanding of SEBT by reading “Why you should use Session Based Exploratory Testing?”.
How do we implement Session Based Exploratory Testing to gaming?
One of the most important things about implementing SBET is to have a clear definition of the Charter. It is the fundamental stone where the SBET is built on. As our main problem regarding game testing is the large amount of different types of bug that we may encounter while we are testing, it is utterly important to correctly define the focus of every session, that is to say, the Charter.
As in any other testing activity, planning of every session is key to success, and we will go several centuries ago, and use a very old approach: Divide and Conquer. Imagine you have 20 levels in a game that need to be tested. And you have 100+ types of bugs that may arise in any moment, in any part of every one of those maps. Unless you have a team of testers like the one in Image 8, it will be a very difficult task to achieve.
Image 8 — The multi-eyed tester (Art, 2020)
In order to apply SBET to gaming, we need to define the Charter. Our focus will be to find bugs, and as we have plenty of bugs to consider, we need to Divide and conquer. So, in order to define our Charter, we will focus only on a certain number of bugs for a particular session. We can easily implement a spreadsheet that allows us to define a single charter, as shown in Image 9.
Image 9 — A sample Charter definition
What we define in this Charter is just one bug type, or set of bug types for a specific Area (in the sample provided, Art) and Sub Area (Physics in this case) that apply only for a specific Map (and Mode/Difficulty in this sample). So, in this session, the tester will only focus on bugs about Character Physics / Ragdoll Error. Tester will not look for any other types of bugs. And as this is SBET, the tester will only pursue bugs belonging to the charter. If tester finds a bug regarding any other bug type (see Image 7), he/she will just add a note for the next session (ie “there is a Lightning/Missing Lights error in this screen” — and will add a Screenshot), and will continue looking for bugs belonging to the charter.
By doing this, we will ensure that we have a 100% pure session dedicated to the Charter in scope.
As Managers, how do we manage these sessions?
It will really depend on the team that will be taking care of the game testing. A useful approach that has been implemented with pretty good success, is to divide the team, and assign a specific Charter grouped by similar types of bugs, in order to cover as many bug-types as possible. A sample is shown in Image 10.
Image 10 — Sample of Charter management
In this sample, we have decided to have some testers focusing on only one type of bug (like Tester 1, 2, 4, 5, 6 and 10), while the rest of the team would be focusing in several types of bug at the same time, as those types of bugs are easily clustered (like Text bugs, Screen Elements, or certain Physics bugs that can be checked together (as Tester 8 & 9).
Also, on Image 11 we can see all the 80+ types of bugs, and which of them are covered in a single session (in this sample we are not including bugs of type Stability and Sound as they were not covered in the selected Charters). Those covered are marked in green.
Image 11 — Session Coverage
So, are you ready?
Implementing a good and effective bug-hunting experience in gaming is not a simple task, mainly due to the vast different types of games. Furthermore, many devices make this a very challenging task (consoles, PC, mobile, etc). So the task of having a solid test management strategy with the focus of bug-finding is critical. Of course, there are many other aspects of the game that need to be attacked properly, as playability, the addictive factor, appealing UI, and many others. But, all of these will surely fail if the game has so many bugs that makes the application unmanageable, or bugs from the different key areas of the app are so noticeable that the player just gives up and finds another similar game in the store.
I have already listed many game-sections where a gamer may find bugs, and there are more than a hundred categories to check. So it is utterly important to plan a proper bug-hunting strategy. I truly believe this approach is not only necessary but also imprescindible in any game-developer’s vault.
Don’t you think it is worth it giving a try?
So run, before the game is over!
Bibliography
360 Logica. (25 de July de 2020). Game Testing. Playing Games does not define Game Testing! Noida, Noida, India: 360 Logica. Obtenido de https://www.360logica.com/blog/playing-games-does-not-define-game-testing/
Art, C. (2020). Alien with many eyes. Alien with many eyes. Cool Art, New York. Obtenido de http://www.mascotdesigngallery.com/many-eyes-alien-mascot/
Crider, M. (16 de 09 de 2020). Capcom plans to bring classic Ghosts N Goblins, Ghouls N Ghosts, Commando, and 1942 arcade games to Android. Obtenido de Android Police: https://www.androidpolice.com/2017/03/02/capcom-plans-to-bring-classic-ghosts-n-goblins-ghouls-n-ghosts-commando-and-1942-arcade-games-to-android/
Future PLC. (16 de 09 de 2020). Retro Gamer. Obtenido de Retro Gamer: https://www.retrogamer.net/retro_games80/green-beret-2/
Indie RetroNews. (20 de 09 de 2020). Saboteur! — A brilliant ZX Spectrum game gets a florinthedwarf review. Obtenido de Indie Retro News: http://www.indieretronews.com/2017/03/saboteur-brilliant-zx-spectrum-game.html
Rivero, S. (2020, July 10). Bug Taxonomy. (D. Marin, Interviewer)
Scott, C. A., & McCall, A. (16 de 09 de 2020). Cognitive Rehabilitation: We All Can Help! Obtenido de Rainbow Rehabilitation Center: https://www.rainbowrehab.com/cognitive-rehabilitation-we-all-can-help/
Universidad Intercontinental — Facultad de Psicología. (2020, 09 16). Intervención funcional para el desarrollo de funciones ejecutivas. Retrieved from Neurociencia Cognición: https://neurocienciacognicion.files.wordpress.com/2013/06/atencic3b3n.pdf
Watterson, B. (29 de January de 2015). Calvin and Hobbes by Bill Watterson for January 29, 2015. Obtenido de GoComics: https://www.gocomics.com/calvinandhobbes/2015/01/29/
