Getting to know your potential future colleagues a little better can be a valuable thing. Seeing how they react to and interact with other people in “realistic” situations somewhat resembling the work they will be expected to do can give insights that change uncertainty into “hell yeah!”… or thumbs-up to thumbs-down. While the process was costly in terms of people’s time (on both sides of the table) we found the “Sprint In A Day” format to provide a lot of useful information we could use to make a more balanced decision.
As a successful candidate was expected to participate and contribute throughout the SDLC we tried to capture as much of this as reasonably possible in the SIAD. We would have them do a bit of testing, a bit of planning, prioritising, and story writing, and they would pair up with a developer for implementing a few things:
Paired Testing
We would start off with a session of “paired” testing and exploration of a small web app pulling in information about football matches and scores from a 3rd party source. The app provided some summary info and allowed navigating through dates and to do some basic filtering. While it was of course the candidate who drove the testing, the paired tester would be chatting along and asking questions about why we did this or that if the candidate went silent. We would ask them to think out loud so we could get a feel for how they worked, as well as giving us the chance to nudge them in the right direction if they were going off into the weeds or was getting close to some juicy bug but started to veer off.
Part of the paired testers “mission” was to make the candidate relax a bit, feel welcome and that we did this together. We didn’t want an interrogation, more of a discovery. Even though a bit of healthy stress can sharpen the senses most people tend to think more freely when they feel at least a little bit comfortable. In general we wanted to learn how they worked in a healthy environment, not how they would react to waterboarding at Gitmo.
This session was about an hour and forty-five minutes and like the homework it was fairly open for the candidate to show their strengths with a few limitations; we would not spend time on installing complex tools or build automation, but they were encouraged to describe how and why they would like to do this if that was the case.
There were probably about as many approaches to this task as there was candidates so it was quite interesting to be part of. While it is difficult to give a “correct” answer to this one I would be hope the candidates would approach it roughly like this:
- Tour the app to start building a mental model of what it might be
- Use the model to ask about the intended use (we had a backstory), target audience, what kind of testing has been done etc.
- Explore the functionality
- Look into correctness of data (we explicitly told what website we pulled the data from so it could be used as an oracle)
- Evaluate accessibility and general usability
- Look at the code
If focusing on only one or two of these, hopefully the candidate would mention at least some of the others to show a more holistic view of testing, and of course there are security, performance, cross browser, mobile, back-end etc. as well that could be mentioned but two short hours flies quickly.
For the sake of time we kept bug-reporting to one-liners in a git repo, except for one “most important” issue of the candidate’s choice (bug or improvement) where they would show what they considered essential information for sending a bug report e.g. to another team in a different time zone.
Planning
After a short break to stretch our legs we would meet up with a PO and dev for some planning. The candidate would do a quick debrief on what they had done and found during the testing and then select the three (-ish) most important findings which was converted into stories for the backlog. This could be bugs, suggestions for improvement, or important things they felt was missing entirely. These stories were then combined with some additional stories/features we want to add according to a backstory we had fleshing out what and who the app was for and when it would be released (tomorrow/ASAP of course). At this point we wanted some input from the candidate regarding prioritisation of this backlog. While this was not strictly the testers responsibility on their own, we were looking for people who considered the big picture and wanted to engage with what we were building and why. Not just checking if it was technically correct. I might get back to some more details about how we worked in a later post.
The world rarely plays along with even the best laid plan, so of course the PO would throw a curveball or two to see if the candidate had any questions or just accepted it. One classic would be that the CEOs spouse had a feature request and this would have to be prioritised. Usually the feature made some sense, so it wasn’t totally ridiculous, but pushing back a little and raising some concerns around the spouse having a prioritised channel into the backlog would usually give you a bonus point or two. Asking if the spouse was at all working for the company would be another welcomed approach.
This session which lasted about an hour would end with the candidate in collaboration with the dev decide what stories to commit to for the pairing session. As this was mostly a matter of picking according to the prioritisation that was just done it was kind of a judgement call on how much can you realistically cover in an hour and a half, or so. Those less familiar with programming this would be an opportunity to ask supporting questions to the dev to uncover any risks or unknowns while still having the dev’s expertise lead the estimate. As always, we tried to keep it open and let the candidate contribute in whatever way they could.
Pair programming/Developing
After taking the candidate out for lunch and a bit of fresh air we got back into the office to implement the stories selected in the planning session. As for most other stages in this process we wanted to evaluate the candidate on their terms (i.e. there is not much of a point testing somebody on something they have already stated they are note any good at. If the lack of a certain skill is a problem, the time to decide is before calling them in to the SIAD). This meant that some candidates grabbed the keyboard to happily show off their coding skills while others preferred to ask questions and suggest test ideas, or something in-between. For those who were not too enthusiastic about taking the keyboard the dev might still ask a couple of times if they would give it a try, but under the premise that it had been acknowledged that they did not consider themselves great coders. Bonus points for courage and giving it a try, and the opportunity for the some “nice, well done, I knew you could do it” type of feedback to lighten the mood and build some rapport.
To many testers this would all be a unfamiliar way of working. As mentioned, some candidates had a good bit of coding skills, some had very little. Anywhere along this spectrum there are ways a tester can contribute regardless of coding confidence level. From doing the coding, to asking the dev to use you as a rubber duck and explain what they’re doing…at least give something a try! …anything!
At the end of the session the PO would debrief the pair and typically ask if there were any insights had or learnings learned. Any reflections on the quality of estimations (usually a bit optimistic, but not always would be appreciated.
Meet the testers
Getting close to the end of an (hopefully) exiting day we had a more informal chat with the other testers, locally or remotely, with opportunities for questions and answers in any direction.
Behavioural/organisational chat
Finally, last session for the day!
This was typically a chat with the Test Manger/Head of Test and a Dev Manager and would touch on things like team work, what does “agile” mean to you, how do you handle conflict and such topics. I don’t think there was much in that session that was significantly different from what you’d expect, apart from the fact that the successful candidate would be working embedded in the team and not in their own silo. This would often be a little different from how candidates were used to work (it’s remarkable how many job ads I see where companies describe themselves a “truly agile” and cross-functional while still considering writing and maintaining test cases, throwing things back and forth over the wall, and Gatekeeping Quality to be essential skills for a tester/manager).
We thought about moving this upfront to a phone/video call to shorten the day and probably would have experimented with this if we had gone into another round of hiring.
The End
And that was pretty much it.
After shipping the (hopefully still) hopeful candidate on their way home for the evening, or sometimes we would wait until the next day, the people involved would sit down for a chat and a vote. People would typically go Yes, No, or OK (if they weren’t too impressed but didn’t really have a reason to go for No). All OK would probably not have been strong enough to hire, but I don’t think that ever happened. Any No would normally result in a no-hire after a little explanation, questions, and clarifications.
+1 Testing 
3 thoughts on “One Approach to Interviewing Testers: Episode Three”