Testing parties and why you might need them
Here at Fullscript, we don’t have dedicated testers nor test teams. Rather, developers test at the same time as they review the code of others.
Our teams have an average of 4–6 developers, and we develop new pieces of functionality in cycles of 6 approximately weeks. This means at the end of each 6 week cycle we’ve likely built some functionality that is at an MVP stage.
Six weeks is long enough that the team has had the chance to work on many pieces of the functionality in the bigger feature. Towards the end, pieces that worked well at the beginning of development might need to be re-tested end-to-end to make sure it all fits and works together well.
This post describes how we tackle testing in the final stages of a development cycle here at Fullscript, a process that we’ve found very successful, with the hope that others can also reap its benefits. If you want to ship better functionality into the hands of your users, and create a customer centric culture in your product engineering teams, keep reading!
Disclaimer: Testing parties are not an invention of Fullscript, and in my career, I’ve seen them used in at least two other places with different names and scale, but the idea is similar.
What’s a software testing party?
Testing parties are used at Fullscript to test functionality just before it is about to be released. The objective is to catch any major bugs or misses that the feature may have before it lands in the user’s hands.
Actually, it’s not a party at all — or not what you would expect. It can be fun though, and definitely has a strong social aspect. But the main objective is to test features that are about to be released before they are, with the goal of shipping more robust, less buggy pieces of functionality.
Testing parties also give you a chance to know, interact, and build relationships with people at the company you wouldn’t otherwise meet. It’s fun building new relationships while trying to break something remotely and in-person. It also allows the development team to receive kudos and feedback about what they just built.
Since members of other teams get to join in on these parties, they also feel heard and know their opinions matter, and that they have a say on what gets shipped to users. This is an amazing opportunity to boost morale and inject a product focused, user first approach into the culture of your company.
The objective of testing parties
We believe that any piece of functionality that has been developed for an extended period of time or that touches something that affects many areas of the application (and has the potential to introduce important regressions), should be end-to-end tested towards the end of the dev cycle. This means it should be tested as the user would see it, with a user focus, and by as many people as possible.
Anyone who’s built software for long enough will have stories of major bugs being discovered the night before the release, or worse, right after it has been released. The team lead or manager sends the announcement of the feature that is now shipped, closes the milestones, the team starts to receive kudos about how great of a feature it is and what a great job they have done. Then an actual user tries out the feature it doesn’t work. Yikes!
The goal of the testing party is to never have one of those situations again. This way, you’re more confident about what you’re shipping because several people have tested the functionality and main flows end-to-end, just before release.
A testing party will answer the question: Is this feature really ready for prime time?
When to do it
Short answer, as close to release time as possible. You want to do it when all functionality has been implemented. So when the project is wrapped up, and all that’s left to do is some copy changes and tiny UI adjustments. That would be a great time to do it. Last thing you want is to do a large testing exercise like this, using up lots of people’s time, and then build another big piece of functionality that needs to be tested all over again.
Since we develop in 6 week cycles, we usually book testing parties mid-week in the sixth week.
How long should it be?
This can vary. In a large organization I worked at in the past (that built an enterprise product in six month cycles) a testing party took the whole day, and involved 50+ people. However, most agile teams, specially those working in cycles that last weeks rather than months, should be looking at spending one hour (max two hours) in a testing party.
What’s important here is not that the testing lasts for a long time, but that you invite as many people as possible to the party.
You want to aim for the quantity of testers, not the length of the party.
The reason why you don’t want to spend more than one or two hours at a testing exercise is because people get distracted after a while, conversations shift to something unrelated, and the efficiency of the exercise decreases. Having lots of people testing in a short period of time will increase your chances of finding major problems, while using people’s brain power in the most efficient way.
Who is cool enough to come?
Anyone, REALLY. You start with the team that built the feature, of course. But the most important people that should be there are those that were not directly involved in the development or planning of the feature. And if you can bring in people that have never heard or seen it, even better.
When you release a feature to your users, they will see it for the first time. That’s what you want to mimic when you bring in people to your testing party that are unfamiliar with the feature. These new testers you bring in will ask better questions, come up with better test cases, and just stumble into more problems than the team that built it.
You want those unfamiliar with the feature giving you all the feedback possible, not just about bugs, but about the functionality in general. Does it make sense to them? Would they do something different? Did they think anything was hard to find in the UI? Those are all questions you should encourage them to answer during the testing party.
So when you plan a testing party, try to reach out to people in other departments as well, not just product team members. We are all users, and we are all intelligent apes using applications for the first time. We should all be able to provide great insight into the decisions the team made during the design phase of the project, and see real users navigate through them.
Setting it up
There are a few things you need to consider when setting up a successful testing party. Essentially, you want to create an event that can find major problems, but that is also fun and people will want to come again (those external to your team) instead of feeling like it’s a waste of time.
Here are a few pointers, based on my experience, that will help you be successful:
- Set up a channel for feedback (a new Slack channel for example) and invite all attendees to that channel.
- Set up a meeting and encourage everyone joining remotely to have their cameras on. Body language helps when describing problems and frustrations that these testers might have. The intent for this is to be a bit of a social event, so presence in all aspects is important.
- Write a document that can be shared ahead of time with attendees. This document should describe what the test is for (basically what the feature does). Don’t try to give them overly detailed test cases, just define major flows, and why the feature is built. Let your testers figure out what to test and how, you will get better results this way.
- If you need to set up testing environments, with different kinds of users, do it ahead of time. The actual testing party should be for testing, and you want to be ready from the beginning for people to start testing. It’ll really suck if they have to watch you set stuff up on the fly.
- If you can, arrange for some snacks, treats and bubbly drinks. This obviously depends on your company and budget, and it’s not a must. But giving people some other reason to show up means they will keep doing so, and even encourage others to come.
- If testing a web application, ask people to test in different browsers, and, if possible, on different devices.
- Communicate at the start how bugs and feedback will be collected. More on this in the next section.
Receiving feedback on bugs and suggestions
While the testing party is happening, you need a way for people to share their findings so that you can follow up with them during the party and afterwards.
Your company probably already has a messaging tool, so use that. If using Slack for example, encourage everyone to post one message per issue they find, and to attach a screenshot or video of what they are seeing. Pictures are good to give people good context on the issue, and for reference later.
It’s also useful to assign at least one person who is familiar with the feature to follow-up on issues posted by the testers in real-time. Let’s call this person the tracker. The tracker could be a team lead or product manager in the project. They likely won’t be able to test much themselves but that’s okay, a person very familiar with the functionality being tested likely has tested it before many times during the project development and won’t be very useful finding bugs anyways.
The tracker should follow the messages or conversations happening during the testing party and ask for more context, if necessary, to understand what the issue is. A message posted per issue here is useful for following up as they can start conversations under threads in the message and get to the bottom of the problems without bloating the channel with messages related to a single issue.
Testing party is done, now what?
When the party is done, it’s time to clean up! Set up a time with your immediate team (development team) to go through all the bugs and concerns raised and create issues for all of the ones you deem necessary to fix. This is a good time to separate bugs from general suggestions.
Take suggestions seriously, especially coming from people that where not familiar with the feature, as there is a good chance users will have the same issues when using the functionality for the first time. Put suggestions in backlogs or whatever you use for it to be looked at after.
This is also the time where you decide which bugs are blocking your release, and which can be fixed after you’ve released the feature. Obviously anything crashing the page, blocking, or being super confusing to the user should be addressed before shipping, but that’s up to you and your team to decide.
After the party, you should have a pretty good idea if you are actually ready for shipping the feature (spoiler alert, you are likely not ready yet as some bugs will be blocking) or need to spend few more days polishing things. A big blocker at Fullscript means we might have to shift the release date because quality and our users come first, timelines can change.
Your burn-down chart, if you use one, will look ugly right after the testing party as a bunch of new bugs and issues have been created in your project. Don’t sweat it too much, bugs and issues of this matter can usually be fixed pretty quickly. Remember, the important thing is that you fix what is important for shipping, and that you have now removed the surprise effect that comes from declaring something as shipped without having an actual user testing event before doing so.
Now you know where you stand. Before, you didn’t.
Conclusion
If you read through all this text you probably have a pretty good idea of what we mean by a testing party, why we have it, and how they can help you and your team build more robust features, with fewer bugs out of the gate. This is something you can implement or help implement within your company and make your users trust your product and the things you build much more.
