The other side of the interview table

Introduction

I’ve recently been in the privileged position of being on the other side of the interview table for several interviews over the past year. I’ve decided I’d like to share my experience and get some ideas written down.

Reading CVs

So before an interview, you usually need to review CVs and pick ones that you feel warrant pursuing. Why do we pick out CVs? Because interviewing is a costly process, it takes time and focus away from our daily work, particularly in my case at a mid-sized company where we don’t tend to interview on a regular basis. We simply don’t have the time to interview everyone that we receive a CV for, so we are forced to filter them down.

My general approach for this was the following:

• Read through the CV thoroughly  - everything on the CV is a small clue about the person.
• I looked at first for some sign of personality in the CV, something that told me why this person was looking for work and what motivates them to work.
• I noted any skill that I thought may be relevant, not just programming skills. For example, skills with Business Analysis tools or experience on a Support team. Anything that could be valuable and bring something different to my test team.
• Depending on the role we were looking for, I would review the years of experience.
• I would make a note of any certifications, I personally don’t put a great amount of value on ISTQB certifications, but I considered them just the same as any training a candidate might mention.
• I always looked for some mention that the person attended meetups, conferences, workshops or is somewhat actively engaged with the testing community. While this doesn’t rule people out (as it’s pretty rare that I see it on CVs), when people do mention it, it makes them stand out.
• I would carefully analyse the wording chosen, especially when talking about skills or previous employment. While I wouldn’t necessarily reject a CV because of a typo, it’s pretty embarrassing when people have them in sentences such as “I have a keen eye for qaulity”.

My experience so far didn’t include the initial CV collation and filtering, however, I have done this once or twice with sets of 10 or 12 CVs. Perhaps if I was filtering a stack of 100 CVs, I probably wouldn’t be as thorough reading the CVs and may be more arbitrary about the criteria I reject them on.

My general experience with this part of interviewing is there is not much right and wrong here. Only you can decide what a “good” CV is and what matches your criteria for the role. I have my own personal preferences for people that add a little personality to their CV, with opinions and motivations but other people may value lists of skills or abilities more highly.

I will say though that many, many people seem to have very, very similar CVs, which makes it hard to pick a few to take forward to interview. This is why you may end up using pretty arbitrary rules for filtering and it also biases you towards those CVs that look a bit different. As an interviewee you can use this to your advantage, but as an interviewer I feel you need to be careful not to let this bias lead you too much. Sometimes a dull CV hides a gem of a candidate!

Preparing for the interview

Who is the person? What do I want to find out?

If it’s been quite a while or if I’ve been quite busy with other work between reading the CV the first time and the date of the interview, I will first start with refreshing my memory on the CV. I will try to think about what I like about this person from the CV that I want to see more of, and try to think of questions that will give them opportunity to impress in these areas. Equally, I will also look for areas that I dislike and try to think of questions that explore these.  Some examples of these I’ve had in the past:

• A tester mentioned working closely with developers and managing the relationships with them - I’ve asked them to expand on that, what’s worked well, what hasn’t etc.
• Some CVs have simply listed skills without description of what their level of experience or confidence with them is, or how they’ve used them. So I’ve targeted questions on those skills to try and explore where they really are with them. “I know Java” would usually prompt questions from me about how they’ve used it and how confident they are with it, even specific questions regarding it.
• Some CVs have also described their previous testing experience mainly in terms of “producing Test Cases and Test Plans according to the specifications” which prompts me to probe quite a bit about the candidates feelings on exploratory testing and how they would handle an environment without many written test cases.

Due to the nature of everyone’s CV being different, this means I end up with a different set of questions each time. Currently I feel this is a little inadequate because I feel I end up with inconsistent or biased opinions on the candidates where I’ve asked better questions to some than others.

Interview format

Something that I’ve had not had much chance to experiment with yet is scripting or planning the interview format. But I feel there are several variables that can change and I could experiment with:

• How many people are going to be involved in the interview?
• How long will the interview be?
• Will we include a technical test?
• How many interviews will we conduct with each candidate (e.g. 2nd stage or 3rd stage interviews)?
• Do we ask different questions or the same questions to each candidate? Do we stick to a script?
• Do we ask the candidate to perform homework or a task before the interview?
• Do we ask the candidate to conduct a task (such as a presentation) during the interview?

I’ve been in various interviews with a mix of the above and I’m undecided on what does and doesn’t work. However it’s worth considering and planning these things before the candidate walks through the door! I also feel I can improve how I learn from each interview and compare them. I would like to spend more time in future making sure the experience with each candidate is more consistent and keep better notes on them. In other words I feel I need to plan better how I am going to make a decision on which candidate to choose, rather than leaving it to gut feeling and all of its biases.

The interview itself

Think about your performance

Regardless of whether you are either the interviewer or the interviewee, my number one rule for interviews is to think of interviews as a two-way conversation. Both parties are interviewing each other to figure out if they like each other. As the interviewer I feel it’s important to respect this even if the candidate doesn’t and give them plenty of opportunities to ask questions. Not only that but I try to keep discussions as honest, informal and friendly as possible. If it can feel more like chatting casually in a cafe or a bar, the better, because both interviewer and interviewee are going to think of better questions and answers.

With this mind, I try to be cautious not to assault the candidate with lots of questions one after another. It’s not easy to describe when it makes sense to hold off and give the candidate space, it depends on several factors:

• The personalities of everyone in the interview.
• The mental state of the candidate.
• How difficult the questions being asked are.
• How the conversation has been going (i.e. sometimes the flow is so natural that we may be chatting fairly casually and rattling through lots of questions and that’s ok).
• How much time we have.

I’ve noticed that people very rarely tend to ask questions after the interview, despite being told they can. While I still encourage this, I’ve taken this to mean it’s very important that the interviewee gets chance to ask as much as they can in the interview. If possible, I try to see if I can learn from their questions rather than from the answers they have for mine.

Multiple interviewers

All of the interviews I’ve conducted have been with other interviewers in the room, asking questions. The worst thing that can happen is where you trip over each other, interrupting or awkwardly looking at each other to ask the next question. This is why preparing the interview format and discussing a script or questions beforehand is important to me. For me you get so little time with candidates that you have to spend every minute, every second very carefully. I absolutely hate when an interviewer pursues a line of questioning which has been covered before or that I don’t consider very useful for this reason.

What would a script look like? Would it be a set of strict questions, one after another that we would follow to the letter? No of course not, as I said earlier it’s important to keep the interview casual and informal, letting it flow with the candidate, adapting all of the time. I would like to try scripts in future where we plan out what kinds of questions and discussions we would like have and assign each interviewer to “lead” each. So someone would conduct the introductions, outro and facilitate the interview, another would ask deeper questions on a topic, etc. I would still allow each interviewer to interrupt or go off script but the key is to try and make sure we get the most out of the interview while keeping it natural.

It’s all about opportunities, not tests

If you are thinking of including some kind of task, examination or test of the candidate to assess their skills, bear this in mind - do not look for failure. What do I mean by this? Interviews are very compromised things, there is a lot of pressure involved and people don’t perform anywhere near like they do when they work normally. It is rarely an accurate representation of what the person is like to work with. With this in mind, I try to view questions and tests as opportunities for the candidate to impress me. If the candidate misses or messes up these opportunities, I try to keep in mind that this may be due to the unusual pressure. I feel if I view it as a series of opportunities to impress, then I avoid placing too much emphasis on particular parts of the interview and look for more well-rounded candidates. It also means people have a chance to recover, where they may mess up the start of an interview, but relax and impress later. Or they may impress in their preparation but fluff up their performance because they are not comfortable with interviews. I’m also open to my own questions being terrible and the candidate impressing me in a way that I didn’t expect, on something I didn’t ask them about.

Life is continuous learning and lessons

Even if you don’t hire them, make sure to always give feedback to the candidate and if there are areas they didn’t know or understand, always take the opportunity to teach them if possible. You may not be hiring them but it can be impossible for the candidate to improve themselves if they never receive feedback. I used to find it very frustrating when no-one ever told me why I didn’t get the job, even if I had done nothing wrong it would been helpful for my confidence to know the reasons.

Interviewing testers

So what about testers? What do we talk about and discuss, what is important for testing? My first reference for this is Dan Ashby’s excellent interview mindmap found here:

https://danashby.co.uk/2015/12/07/how-i-interview-testers/

‘Nuff said! But some additional thoughts for me:

• Discussing definitions of “testing” and why people like testing are important because everyone has different ideas and understanding. This as much about making the candidate feel comfortable with what they are applying for as it is establishing they are the right fit for us.
• Discussing “agile” or “devops” are also opportunities to make clear how we work. I’m not looking for people rattle off dictionary definitions of these words but I want to understand what they think it is and how they adapt to topics that affect testing. Its also for me to explain what I believe it is and how the company has interpreted or implemented those ideas. The discussion and understanding  is the important part, not testing the candidate for definitions.
• In terms of technical tests or exams, I’m very skeptical. While there may be certain contexts where you are looking to hire testers with programming experience, I personally don’t view programming as a key testing skill. However, if I could design a technical test that gives a good picture of how capable of learning technical subjects, I would try that! I value testers with the right attitude and approach and the ability to learn a great deal, already knowing programming is useful but not critical. The critical ability is the capacity to learn. I’ve worked with and hired great testers who know little about programming and have contributed a lot of value, if not more value that those that knew programming.
• I’ve experimented with tests of candidate’s testing abilities and seen different ideas, but I’m again unconvinced how much you can judge. You can try and assess them on bugs they find in an application or ask them to explore their lateral thinking skills with a task such as mind-mapping a pencil. I’ve seen some interesting results from these tasks but I’m concerned that these tasks bias us towards candidates that are great on the spot. I suspect there are great testers who don’t perform very well in these situations but are excellent given more time and less pressure.

Summary

• Its rare that we trained how to interview so it’s worth spending time planning how you are going to learn and improve, because it is an area that has particular skills and considerations like any other.
• I’ve got several areas I’d like to focus on improving or learning more about in future, particularly around planning and facilitating interviews.
• It’s easy to feel interviews are about asking lots of questions and testing the interviewee, based on your experience as an interviewee. But the best interviews are where you make it a more natural and informal chat.
• Opportunities to impress, not testing for failure!
• Make sure to always take the time to give feedback, especially if you don’t hire the candidate. Tell them why you are not hiring them, so they can improve.

The temptation to split dev and test work in sprints - don’t do it!

Introduction

About 3 and a half years ago, I was new to sprints and scrum. Coming from the videogames industry, I was used to a process where I would test code that came from developers and return bug reports. I had heard the words “sprint” and “scrum” before but I had no idea how testing fit into them, so I joined a company where I could figure that out. This is what I figured out.

What’s a sprint?

If you’re not familiar with scrum or agile, then a sprint is effectively a short-term project plan where a team of people decide the work that they can complete within a 1, 2 or 3 week window. Work is “committed” (promised) to be completed in that time frame and the team tracks their progress. After each sprint, reviews and retrospectives are held to help the team find what works well and what helps them complete more work to a higher standard while still maintaining their commitment. The main focus of sprint work is to complete the work and trying to avoid leaving work unfinished.

Where does testing fit?

So normally teams set up a task board with columns titled something similar to “To Do, In Progress, Done”. Sometimes people add more columns or use different names but the usage is similar. Anyone from the same background as me would be tempted to then suggest that an additional column could be added between “In Progress” and “Done”. The logic being that “when you’ve finished your development work, I’ll test it”. In my head, this was trying to work with what I knew already in this new environment. We ended up with columns similar to “To Do, Build/Dev, Testing, Done”.

Bad idea

So at first, I thought things were working ok, I feel one of my strengths is learning and picking up things fast so I got stuck in and kept up with the 5 developers in my team. Most of the time I was fortunate that the work dropped sequentially or wasn’t particularly time consuming to test. This didn’t last long though and eventually we started to fail to complete work on time. This happened either because I was testing it all at the end of a sprint or because the work was highly dependant upon each other and the problems with integration weren’t found until very late.

This meant we had to continue some work in future sprints. Now I no longer had plenty of time to write my test plans at the start, but I was busy testing last sprint’s work and then testing this sprint’s work! I no longer had time to spend learning more automation or exploring newer areas to me like performance testing. All of my time was consumed trying to test all of this work and I couldn’t do it. What went wrong?

A change in approach

I would love to say I quickly realised the problem and fixed it but it took me a long time to realise the issue. I think part of this I will put down to not knowing any better and partly working with developers who didn’t know any better. Either way, a while later I realised that the problem was that I was trying to test everything and the developers started to rely on me for that. I’ve since realised that there is a fair bit of psychology involved in software development and this was one of my biggest lessons.

We eventually decided to stop splitting up work between roles, mainly because we found that developers tended to treat work that was in “test” as “done” to them, freeing themselves up to work on even more development work. This created a bottleneck, as the only tester as I was testing work from yesterday while they were busy with today. Instead, I came to the realisation that there is little benefit to splitting the work up in this way, at least not through process. We should be working together to complete the work, not trying to focus on our own personal queue. I shifted from testing after development was thought complete, to trying to test earlier, even trying to “test” code as developers were writing it, pairing with them to analyse the solution.

Understanding what a “role” means

I think for me this lesson has been more about realising that playing the role of “tester” does not necessarily mean I carry out all of the “testing” in a team. It does mean I am responsible for guiding, improving and facilitating good testing, but I do not necessarily have to complete it all personally. An additional part of this lesson is that I cannot rely on other people to define my role for me - as a relative newbie to testing I relied on the developers to help me figure out where I should be. While I’ve learnt from it, I also know that I may need to explain this learning again in future because it is not immediately obvious.

So where does testing really fit?

Everywhere, in parallel and in collaboration to development. Testing is a supportive function of the team’s work, it now doesn’t make sense to me to define it as another column of things to do. It has no set time frame where it’s best to perform, and it doesn’t always have a great deal of repetition in execution. It is extremely contextual.

In addition, that’s not to say you shouldn’t test alone or separately to ongoing teamwork. You absolutely must test alone as well, to allow you to focus and to process information. It’s just that you must choose to do this - where it is appropriate.

Definition of “Done”

One of my recent approaches was to define the definition of “Done” as:

“Code deployed live, with appropriate monitoring or logging and feedback has been gathered from the end user”

Others may have different definitions, but I liked to focus the team on getting our work in a position where we could learn from it and take actions in the following sprint. For me, it meant we could actually pivot based on end user feedback or our monitoring and measure our success. Instead of finishing a sprint with no idea whether our work was useful or not, planning a new sprint not knowing whether we would need to change it.

Summary

• Avoid using columns like “Dev” and “Test” in sprint boards. It seems to lead to a separation of work where work is considered “Done” before it is tested.
• Instead, try to test in parallel as much as possible (but not all of the time), try to test earlier and lower down a technology stack (such as testing API endpoints before a GUI is completed that uses them).
• Encourage developers to test still and instead try to carefully pick when and where to personally carry out the bulk of the testing. Try to coach the team on becoming better at testing, share your skills and knowledge and let them help you.
• Altering the definition of “Done” seemed to help for me, it was useful to focus the team on an objective that meant we really didn’t have to keep returning to work we had considered completed. In other words, make sure “done” means “done”.

The words “Testing” and “QA”

Introduction

I’m not one to harp on about semantics, I mean, I care that we all understand each other, but I’m very conscious of how irritating it can be to constantly point out “you’re using that word wrong”. However, I do find it frustrating when people assume that language used in the community or in documentation is commonly agreed, understood and given the same meaning by everyone in the same way. Very frequently, this is not the case and the words “testing” and “QA” seem to be an example of this. As a tester, keeping in mind these semantics can be very useful in resolving misunderstandings that may have been missed.

Why do I find it frustrating?

Ever since I was a newbie to testing and the tech industry, I’ve been trying to understand. I hear phrases or words and try to find out what they mean. Someone may teach me the word, I may read it in documentation or hear it explained in talks. Most of the time I learn words through the context they are used, which is probably the most natural way we all learn language.

So I learn words or phrases and believe I understand what they mean and hence believe that I would be understood when I reuse them the same way. However, because we are creating new words and phrases all of the time, some of them don’t have a “standard” meaning. An obvious example of this I can think of is regional dialects or slang, and in the UK we have many, many words to describe a bread roll.

So if I asked for a “Bacon Stotty” in London I would get a puzzled look and if I asked for a “Muffin” I might get one of these:

I personally see this as simply a natural development of language, we invent new words all of the time to describe things that are new to us, especially if we don’t have an appropriate word to use already. However, due to distance and culture as humans we may come up with different words to describe the same thing or the same word describing two different things.

The frustration for me is when I encounter people who don’t seem to accept this. The phrase “QA” is widely used in the tech world in many different contexts and doesn’t appear to have much common agreement on its definition. For this reason, I don’t like using the phrase because I don’t believe I would be well understood most of the time. However, I haven’t felt the need to talk about this until now because I haven’t come across an example that justified my feeling on it.

Enough accuracy for enough understanding

So I recently interviewed someone who was looking to start as a tester, they didn’t have any experience of testing and were keen to understand as much as they could about the role. I will add they were impressive for someone with no experience! During the interview one of my colleagues explained the gist of how the development teams were structured and what they worked on. They described a tester on one team as performing “performance testing” and then another one on another team as performing “QA”. I wryly smiled to myself about the use of the word “QA” but I didn’t say anything to it because it was a reasonable, high-level description of how we might work in a very generalised sense. While I knew some of the words were misleading, it wasn’t the time or place to pick it apart because:

1. I didn’t want to embarrass my colleague and I didn’t want to give the candidate a bad impression of our relationship.
2. I didn’t want to spend the limited time we had in the interview explaining why those words weren’t right.
3. The gist that was given felt enough to me for the candidate to understand how we worked, at least for now.

Recognising misunderstanding and addressing it

The candidate was happy with this explanation and I was happy that it was enough to at least give them some context to ask any further questions. The interview continued and at the end the candidate expressed they were really happy we had an “Internal QA” role because they knew of a similar role in their current company which involved checking products were meeting guidelines and standards set out by governing bodies. They felt that this was a role they would like to start with because it would give them an easier step into testing and more technical roles (they openly admitted they didn’t have much technical knowledge of experience but wanted to learn).

Now it had become apparent the gist we had given was clearly not appropriate because the candidate had understood the phrase “QA” differently to how my colleague had meant. The candidate now clearly expressed a desire for such a role, particularly because they were keen to have some guidance. They liked the sound of QA because it sounded more scripted, more guided and therefore an easier leap into the tech world. I immediately explained to the candidate that in the context of this business, we don’t have a role like that and really the roles are far more exploratory in nature. I then had to go into explaining what I defined as “QA” and “testing” what the difference was and why my colleague had interchangeably used them.

This real example hit home for me a justification that the phrase “QA” is misused and misunderstood. It can have multiple interpretations and in this case it could have led an interview candidate into accepting a job completely different to what they had envisioned.

What are my definitions of “QA” and “Testing”?

My definitions are:

• “QA” or “Quality Assurance” involves checking that a piece of software conforms to a predetermined set of qualities. This can come from legal requirements, industry standards or certification. QA is generally scripted in nature.
• “Testing” involves exploring a piece of software to discover information about it. While it may include the use of checking predetermined sets of qualities it focuses on the unknown rather than the known. Testing is generally exploratory in nature.

I do not dare suggest these are commonly agreed definitions and I definitely do not go around correcting people on this. I have learned it’s pretty irritating and counterproductive to do this. However, I use these definitions to help me identify when I have misunderstood or someone else misunderstood these phrases. In other words, I’m aware that these words aren’t commonly understood the same way and I choose to clarify my understanding this way when the discussion comes up.

No one is at fault here

I want to emphasise that in the interview situation I refer to, no one was at fault. I do not expect my colleague to understand all of the semantics of testing. I do not expect a newbie tester to be aware of them either. I’m simply observing that it is useful to keep in mind these semantics and how people may be silently misunderstanding each other and not realising it.

I don’t think we can really do much about preventing this divergence of meanings of words and phrases, I personally feel they are a natural flow of language. The English language is full of words that have many contextual meanings as it is. Nor can we “know” all of these meanings. All we can do is share our different meanings of words and raise each others awareness to these different meanings and perhaps come to some greater consensus on definitions.

Games Testers should be proud of their work, not ashamed

Introduction

A few months ago I interviewed someone for a testing role and the question of past experience came up, to which the interviewee said something along the lines of ‘well this might be a bit sad, but I’ve done a bit of video game beta testing’. Then a week later I noticed a discussion on the Ministry of Testing’s slack team where people were discussing how they had got into testing. A couple of people mentioned how they found it useful to start in games testing but felt games testers get a lot of stick for what they do. This prompted me to jump in and talk about my story as I’m quite passionate about this subject! In fact this very sentiment is where my career pretty much started after university!

Pizza, pop and candy

I will never forget this line. I had just finished university and was applying for every programming job I could find. There was a wild week (they seem to happen every year or two) where I had accrued three interviews to take place in one week, just prior to one of my brothers getting married. I had two interviews for programming roles and one for a games testing job.

The first interview I didn’t pass, the second one I seemingly impressed and got offered the job to test games and the third one? Well, bearing in mind I wanted to get a programming job to make the most of my degree, I went into it feeling glad to have the games testing job in my pocket but wanting the programming job. However, the third interview went something like this:

“So we see from your CV that you’re interested in working in the games industry?”

“That’s right, but I’m keen to learn whatever I can from any programming role I can find!”

“Pizza, pop and candy”

“Excuse me?”

“That’s what you ‘gamers’ like isn’t it?”

“Erm…”

“Yes, we hired a programmer from that games company nearby, what was the name?”

“Traveller’s Tales?”

“Yes, that’s it, but they didn’t last long. Great programmer, but not a good fit for our organisation...”

Needless to say, that was the most awkward, aggressive and stressful interview I’ve ever experienced. I honestly don’t know why they decided to waste their and my time in such an interview. But it made me suddenly feel very, very comfortable to be accepting a job in games testing and definitely removed any second thoughts! My testing career effectively started from a statement of disgust about ‘gamers’.

Learning to test

So here I was in a job testing, having barely heard the word ‘testing’ before (it seems crazy to think now that you can complete programming degrees without ever hearing the phrase ‘unit test’!). I’ve loved video games all of my life and I had seen my fair share of bugs with them, so I knew pretty well what ‘good’ looked like and what ‘bad’ looked like. I also play a lot of different types of game at varying difficulties, so I had a lot of knowledge to draw on - not to mention a degree in games design & programming. I understood what people wanted from games, how they are made and what could go wrong. I hadn’t worked a proper full time job before but I was pretty confident I could do a good job.

I learnt a huge deal games testing, especially regarding the sheer variety of forms a ‘bug’ or ‘problem’ could take. Not only were there the obvious bugs that a lot of people might recognise like crashes, hangs and freezes (yes, they had technical differences) and graphical glitches, but also problems you don’t necessarily see visually. The bulk of the testing was exploratory in nature, usually with a task being to focus on a particular area - be it a collision detection test or a playthrough of the game just to make sure it could be completed.

To give you some idea of the variety of problems we would look out for:

• Graphical - the most obvious and visual kinds of bugs that affected the look and feel of the game.
• Functional - quite simply, bugs to do with what the game should ‘do’. This was very contextual as every game is designed differently, but there are some fairly typical consistencies such as being able to steer a car in a racing game..
• Difficulty - this was pretty subjective, but we had to judge whether ‘easy’ was consistent and didn’t suddenly jump in difficulty on a particular level.
• Audio - maybe the voices in the game don’t synch up with the animations of the characters or the quality of the audio was poor (such as once hearing the ‘beeps’ of the recordings being started and ended for the voice actors!).
• Legal - sometimes games would include trademarked or copyrighted material which they hadn’t credited. Or sometimes they could accidentally feature close similarities to real world products.
• Health - we were trained on photosensitivity and how to identify bugs that would trigger epilepsy.
• Technology - we were sometimes testing for bugs with particular technologies such as 3D, motion control, force feedback devices, augmented reality, virtual reality headsets and more! All of these technologies feature their own particular bugs and problems.
• Compliance - we checked that games were compliant with the standards set out by the corporation. Games that didn’t comply with these standards would be rejected so we tested to make sure they were compliant.
• Localisation - although we didn’t translate games nor could necessarily read all languages, we did check for bugs related to the subtle differences in localisation. An innocent missing language file would be enough to make a game unplayable or crash. You don’t need to be able to read German to notice the text doesn’t fit inside buttons!
• Multiplayer/Performance - we tested for load, stress and also for exploits. Some games need to be able to handle large amounts of players and exploits can ruin people’s enjoyment of their online gaming experience.
• Compatibility - there were some cases where games were ported to new hardware or developed to communicate with older systems. We tested to find problems with how these games were compatible.
• Transactions - we tested to make sure real world payments could be completed and that people couldn’t exploit these systems.

There are probably a few I’ve forgotten, but hopefully that gives you some idea of the wide range of issues we would learn about and find!

Along with learning about the wide variety of problems, I also learned about testing techniques and strategies. Unfortunately the company was heavily using test cases at the time. There were differing views on these and I learned to find and report problems from my exploratory sessions more than fill out the boring and repetitive test cases. There were also the beginnings of automation testing which was being used to help carry out load and stress tests in multiplayer games along with experimented use in running regression test cases.

I learned two major lessons about testing in this job:

1. I hated test cases and found them fairly useless. Occasionally they were useful to find test ideas or guide exploring a product, but they felt awkward to write and use and just generally wasteful of time. I found most of my bugs when I was performing exploratory testing.
2. Automating tests on a product that was constantly changing, especially in a graphical sense was a waste of time. By the time we had written automated tests of the product, it had changed again and we had to re-write them. Quite a lot of game projects didn’t require testing after release, so the value of the automation tests was rarely seen. It only seemed to return some value when running large scale performance tests.

What did I pick up that could apply to other testing jobs?

There were things that I picked up in my first job that I hadn’t formed thoughts around or hadn’t found the language to describe effectively but were important to becoming a better tester. While I gained better understanding and language later in my career, my first job gave me awareness of these issues:

• Managing relations with developers - I quickly learned that diplomacy was important when raising bugs with developers. I picked up an analogy that I re-used in future testing interviews - “imagine an artist has created a magnificent piece of art and you come along and say ‘you missed a bit there’ - you can imagine that’s quite irritating so I look for ways to soften that blow”.
• Costs of automation - I didn’t have the language to describe it, but I learned the hidden costs of automation and the temptation to try and find ways to use it. I was already wary about the value and how easily people could misunderstand that automation is some kind of replacement for ‘all’ testing.
• Knowing when to stop testing - I had started to get experience on when and where I might want to stop testing. While we were generally time-boxed at in my first role, I definitely saw the diminishing returns and learned which risky areas I wanted to focus my testing on.
• Knowing when to repeat tests - I also started to form ideas on how to judge regression testing although I didn’t know at the time how else it could be done other than re-running test cases.
• Justifying my testing - I had started to build up confidence through my experience of being able to justify my testing and write effective bug reports. While I didn’t have the best language to explain myself, I at least had experience to refer to that gave me a determination to learn better language!
• Determination to understand more about the product, earlier - Typically in my first job, we didn’t have very good documentation to refer to (sometimes none at all!). When we did have documentation, it was much easier to quickly provide feedback on the product. So in future roles, I had a strong determination to get involved earlier in projects so I could understand more about them, much more quickly. I knew documentation was expensive but I felt that trying to access sources of information was crucial to getting my testing really started.
• Thinking ‘outside of the box’ - I quickly learned that the ‘best bugs’ were the ones that people hadn’t even thought of. Sometimes these would be fairly expensive bugs to fix and it was valuable to find them as soon as possible.
• Being organised, fast and clear - I took pride in my work being organised but fast and effective. I quickly understood the need to setup my tests quickly and maximise the amount of time I had available. I created my own techniques for being able to do this, while still providing clear feedback in my reports, which have served me well to this very day!
• Being adaptable and willing to learn - My programming background helped me pick up technical or complex products very quickly and I learned that this can be very valuable. Again, it was useful being able to maximise the time I had available to test and it meant I could provide quick and clear feedback. I then looked for opportunities to learn more skills that could complement my testing efforts.
• Understanding and empathising with the end user - If you’re in a games job, you’ve probably played a lot of video games and so you probably have a pretty good idea of what end users do, what they want and what they like. So it was easier to think like an end-user and therefore I really appreciated the value this brings when coming up with test ideas, justifying your testing and justifying your bug reports. This experience still motivates my desire to always understand more about the psychology of the end users as I highly value it in my testing.

Summary

I was a games tester for 3 years in two different companies and I will never talk about it as a negative experience. That’s not to say that everything was perfect, the games industry has a lot of problems that still affect it today. I left the industry partly because testers are treated as an end-of-cycle role and because I was keen to build my career in testing. However, I still managed to leave and secure a job I would describe as being ‘professional testing’ - I wouldn’t have been able to do this without the experience I gained from games testing. If you are currently testing games and you feel like you’re not a professional tester and have to prove something - you don’t! You’re already a professional tester!