Games Testers should be proud of their work, not ashamed

Introduction

A few months ago I interviewed someone for a testing role and the question of past experience came up, to which the interviewee said something along the lines of ‘well this might be a bit sad, but I’ve done a bit of video game beta testing’. Then a week later I noticed a discussion on the Ministry of Testing’s slack team where people were discussing how they had got into testing. A couple of people mentioned how they found it useful to start in games testing but felt games testers get a lot of stick for what they do. This prompted me to jump in and talk about my story as I’m quite passionate about this subject! In fact this very sentiment is where my career pretty much started after university!

Pizza, pop and candy

I will never forget this line. I had just finished university and was applying for every programming job I could find. There was a wild week (they seem to happen every year or two) where I had accrued three interviews to take place in one week, just prior to one of my brothers getting married. I had two interviews for programming roles and one for a games testing job.

The first interview I didn’t pass, the second one I seemingly impressed and got offered the job to test games and the third one? Well, bearing in mind I wanted to get a programming job to make the most of my degree, I went into it feeling glad to have the games testing job in my pocket but wanting the programming job. However, the third interview went something like this:

“So we see from your CV that you’re interested in working in the games industry?”

“That’s right, but I’m keen to learn whatever I can from any programming role I can find!”

“Pizza, pop and candy”

“Excuse me?”

“That’s what you ‘gamers’ like isn’t it?”

“Erm…”

“Yes, we hired a programmer from that games company nearby, what was the name?”

“Traveller’s Tales?”

“Yes, that’s it, but they didn’t last long. Great programmer, but not a good fit for our organisation...”

Needless to say, that was the most awkward, aggressive and stressful interview I’ve ever experienced. I honestly don’t know why they decided to waste their and my time in such an interview. But it made me suddenly feel very, very comfortable to be accepting a job in games testing and definitely removed any second thoughts! My testing career effectively started from a statement of disgust about ‘gamers’.

Learning to test

So here I was in a job testing, having barely heard the word ‘testing’ before (it seems crazy to think now that you can complete programming degrees without ever hearing the phrase ‘unit test’!). I’ve loved video games all of my life and I had seen my fair share of bugs with them, so I knew pretty well what ‘good’ looked like and what ‘bad’ looked like. I also play a lot of different types of game at varying difficulties, so I had a lot of knowledge to draw on - not to mention a degree in games design & programming. I understood what people wanted from games, how they are made and what could go wrong. I hadn’t worked a proper full time job before but I was pretty confident I could do a good job.

I learnt a huge deal games testing, especially regarding the sheer variety of forms a ‘bug’ or ‘problem’ could take. Not only were there the obvious bugs that a lot of people might recognise like crashes, hangs and freezes (yes, they had technical differences) and graphical glitches, but also problems you don’t necessarily see visually. The bulk of the testing was exploratory in nature, usually with a task being to focus on a particular area - be it a collision detection test or a playthrough of the game just to make sure it could be completed.

To give you some idea of the variety of problems we would look out for:

• Graphical - the most obvious and visual kinds of bugs that affected the look and feel of the game.
• Functional - quite simply, bugs to do with what the game should ‘do’. This was very contextual as every game is designed differently, but there are some fairly typical consistencies such as being able to steer a car in a racing game..
• Difficulty - this was pretty subjective, but we had to judge whether ‘easy’ was consistent and didn’t suddenly jump in difficulty on a particular level.
• Audio - maybe the voices in the game don’t synch up with the animations of the characters or the quality of the audio was poor (such as once hearing the ‘beeps’ of the recordings being started and ended for the voice actors!).
• Legal - sometimes games would include trademarked or copyrighted material which they hadn’t credited. Or sometimes they could accidentally feature close similarities to real world products.
• Health - we were trained on photosensitivity and how to identify bugs that would trigger epilepsy.
• Technology - we were sometimes testing for bugs with particular technologies such as 3D, motion control, force feedback devices, augmented reality, virtual reality headsets and more! All of these technologies feature their own particular bugs and problems.
• Compliance - we checked that games were compliant with the standards set out by the corporation. Games that didn’t comply with these standards would be rejected so we tested to make sure they were compliant.
• Localisation - although we didn’t translate games nor could necessarily read all languages, we did check for bugs related to the subtle differences in localisation. An innocent missing language file would be enough to make a game unplayable or crash. You don’t need to be able to read German to notice the text doesn’t fit inside buttons!
• Multiplayer/Performance - we tested for load, stress and also for exploits. Some games need to be able to handle large amounts of players and exploits can ruin people’s enjoyment of their online gaming experience.
• Compatibility - there were some cases where games were ported to new hardware or developed to communicate with older systems. We tested to find problems with how these games were compatible.
• Transactions - we tested to make sure real world payments could be completed and that people couldn’t exploit these systems.

There are probably a few I’ve forgotten, but hopefully that gives you some idea of the wide range of issues we would learn about and find!

Along with learning about the wide variety of problems, I also learned about testing techniques and strategies. Unfortunately the company was heavily using test cases at the time. There were differing views on these and I learned to find and report problems from my exploratory sessions more than fill out the boring and repetitive test cases. There were also the beginnings of automation testing which was being used to help carry out load and stress tests in multiplayer games along with experimented use in running regression test cases.

I learned two major lessons about testing in this job:

1. I hated test cases and found them fairly useless. Occasionally they were useful to find test ideas or guide exploring a product, but they felt awkward to write and use and just generally wasteful of time. I found most of my bugs when I was performing exploratory testing.
2. Automating tests on a product that was constantly changing, especially in a graphical sense was a waste of time. By the time we had written automated tests of the product, it had changed again and we had to re-write them. Quite a lot of game projects didn’t require testing after release, so the value of the automation tests was rarely seen. It only seemed to return some value when running large scale performance tests.

What did I pick up that could apply to other testing jobs?

There were things that I picked up in my first job that I hadn’t formed thoughts around or hadn’t found the language to describe effectively but were important to becoming a better tester. While I gained better understanding and language later in my career, my first job gave me awareness of these issues:

• Managing relations with developers - I quickly learned that diplomacy was important when raising bugs with developers. I picked up an analogy that I re-used in future testing interviews - “imagine an artist has created a magnificent piece of art and you come along and say ‘you missed a bit there’ - you can imagine that’s quite irritating so I look for ways to soften that blow”.
• Costs of automation - I didn’t have the language to describe it, but I learned the hidden costs of automation and the temptation to try and find ways to use it. I was already wary about the value and how easily people could misunderstand that automation is some kind of replacement for ‘all’ testing.
• Knowing when to stop testing - I had started to get experience on when and where I might want to stop testing. While we were generally time-boxed at in my first role, I definitely saw the diminishing returns and learned which risky areas I wanted to focus my testing on.
• Knowing when to repeat tests - I also started to form ideas on how to judge regression testing although I didn’t know at the time how else it could be done other than re-running test cases.
• Justifying my testing - I had started to build up confidence through my experience of being able to justify my testing and write effective bug reports. While I didn’t have the best language to explain myself, I at least had experience to refer to that gave me a determination to learn better language!
• Determination to understand more about the product, earlier - Typically in my first job, we didn’t have very good documentation to refer to (sometimes none at all!). When we did have documentation, it was much easier to quickly provide feedback on the product. So in future roles, I had a strong determination to get involved earlier in projects so I could understand more about them, much more quickly. I knew documentation was expensive but I felt that trying to access sources of information was crucial to getting my testing really started.
• Thinking ‘outside of the box’ - I quickly learned that the ‘best bugs’ were the ones that people hadn’t even thought of. Sometimes these would be fairly expensive bugs to fix and it was valuable to find them as soon as possible.
• Being organised, fast and clear - I took pride in my work being organised but fast and effective. I quickly understood the need to setup my tests quickly and maximise the amount of time I had available. I created my own techniques for being able to do this, while still providing clear feedback in my reports, which have served me well to this very day!
• Being adaptable and willing to learn - My programming background helped me pick up technical or complex products very quickly and I learned that this can be very valuable. Again, it was useful being able to maximise the time I had available to test and it meant I could provide quick and clear feedback. I then looked for opportunities to learn more skills that could complement my testing efforts.
• Understanding and empathising with the end user - If you’re in a games job, you’ve probably played a lot of video games and so you probably have a pretty good idea of what end users do, what they want and what they like. So it was easier to think like an end-user and therefore I really appreciated the value this brings when coming up with test ideas, justifying your testing and justifying your bug reports. This experience still motivates my desire to always understand more about the psychology of the end users as I highly value it in my testing.

Summary

I was a games tester for 3 years in two different companies and I will never talk about it as a negative experience. That’s not to say that everything was perfect, the games industry has a lot of problems that still affect it today. I left the industry partly because testers are treated as an end-of-cycle role and because I was keen to build my career in testing. However, I still managed to leave and secure a job I would describe as being ‘professional testing’ - I wouldn’t have been able to do this without the experience I gained from games testing. If you are currently testing games and you feel like you’re not a professional tester and have to prove something - you don’t! You’re already a professional tester!