Testing in DevOps

I've just spent the past year embedded in a "devops" team (quotation marks explained later) and I've got a few different points to make, so bear with me, this is going to be a long post. Also a bit of a brain dump so it might not be my best writing ever as I want to write this while its relatively fresh in my head
This post is also going to be a little technical and assume some knowledge of DevOps, if you're new to the phrase, I highly recommend Katrina Clokie's book "A Practical Guide to Testing in DevOps" found here - https://leanpub.com/testingindevops

That word "Devops"

In my experience, there are two different understandings of the word/phrase "devops". Basically it boils down to:

• "Devops" is not a role, it's a set of practices, which makes it a bit woolly and vague but in general is about bringing two traditionally separate roles together so that a "team" can both deliver (for example) a software application and its hardware but also maintain, operate and support it in production/live/whatever you want to call it. This can be achieved by training the team in operations or by embedding ops engineers.
• "Devops" is where developers write infrastructure-as-code, typically Ops engineers interested in programming. But sometimes also software programmers interested in getting their hands dirty with more Opsy work. In this definition, you tend to see this become a role called "Devops engineer" and they tend to write re-usable chunks of code that builds infrastructure for software teams to use. For example, creating a generic set of code that provides a MySQL cluster in AWS.

I highlight this because I've realised people aren't aware that there is this difference and personally I prefer to encourage the former rather than the latter. The latter is a bit like SDETs/Developers in Test where you're creating a whole new communication/distance from the end goal role, writing tests that are broken by the dev team because they have no idea about them. The first definition I like because its about teamwork and delivery, encouraging the team to take a more holistic view of software delivery or rather product delivery as a whole. After all, who cares if your code is shit hot if we've put it on hardware too small to run it?

Cloud and infrastructure-as-code

Regardless of those definitions, if you're going to work with cloud-based infrastructure (as opposed to on-premise, where your company owns or rents the physical servers), you're going to be writing infrastructure-as-code. Why? Because in the cloud you are sharing your hardware with other companies and people, which means you have less control over how it operates and this changes the risk profile. The cloud is cheaper than owning the physical servers but this comes at the cost of reliability.
Therefore you may want to automate many aspects of operating your product, such as recoverability and scalability which is where infrastructure-as-code comes in.
There are two main areas that can be expressed in code:

• Code for provisioning the infrastructure you need e.g. the hardware, the network routing, firewall rules and so on.
• Code for configuring an individual server, e.g. setting up users and file permissions, installing software, configuring software (such as setting up Java and then setting up a Java based application to run).

Why consider these two separate areas? In my opinion there is so much to consider and think about in both that its worthwhile considering them in their own discussions, even though you will want to develop and test them together.

Testing in DevOps

So what to test? How do you test? What tools do you use? Is there anything to really test?
Hell yes there’s loads to test, you’ve now got code that builds the foundations of your product, and not just that, but its code that defines how your product will scale and recover and also determine its reliability. Suddenly a developer can potentially make a small change and open all of your servers to the public to access.
These are some ideas of what you can test in an Ops world:

• You can manually run and try out the code as an obvious starting point. Does it build the servers correctly? Can you use the application after destroying it and building it again?
• Destroying the servers leads to OAT (Operational Acceptance Testing) or general operability. Testing what happens in disaster and failure scenarios. Will your product recover if the servers suddenly disappear and new ones are built? Do you test your backups regularly? This also neatly leads to a ideas such as chaos engineering.
• The code itself can be sort-of unit tested. For example Ansible has a framework called Molecule which allows you to run your Ansible scripts against a Docker container and assert what state the scripts will leave a server in. There are also more broader integration test tools such as Test Kitchen which have slightly more capabilities.
• Using tools such as AWS Trusted Advisor or Well-Architected to analyse your infrastructure for common mistakes (such as setting up firewall rules completely open to the public) or under-utilised hardware that could be run more cheaply.
• Given cloud infrastructure is inherently prone to failure, can you monitor and alert those failures? Do you know if your servers fell over overnight? How many errors are happening in your environments? Usually cloud providers don’t have access to your servers to know what is going on, so you need to setup your own access to software logs (e.g. like Java app errors), have you centralised these logs for easy access?
• Tools like Sensu allow you write custom automated checks to monitor your servers, this is very useful for more granular checks like specific software health checks (e.g. your server never failed but the software application has crashed, can you tell from your monitoring?). I think there is a lot of value here for testers to help design, write and create new simple but smart checks and improve how observable systems are not just in production but also in all environments!

Some other things I’ve worked on that were very context specific, and not very DevOpsy or Testing, but could be useful ideas:

• Creating Jenkins pipelines to test out rebuilds of infrastructure-as-code on a daily basis - this was to cover a specific risk we had that our code was very tightly coupled and we were breaking a lot of projects with our changes. This was an interim solution until we un-coupled our architecture but it was useful to be able to do it.
• Off the back of Jenkins pipelines for rebuilding infrastructure-as-code, I also created jobs that would manage downtime periods to help save a good two-thirds of our monthly costs. In general we only needed our test environments to be up 8 hours a day, 5 days a week, not 24-7. I used Jenkins so I could manage dependencies and run custom checks but this could also be achieved with the right auto-scaling policies in AWS too.
• Fixing and writing my own Sensu checks, this was useful although I would be careful as its easy to write a check that produces false positives or negatives. Its very hard to think of all scenarios the check could encounter so avoid writing scenario-based checks where possible. It’s not helpful to have monitoring checks that have bugs themselves and are difficult to debug when they fail, keep them simple.
• Hooking together Dev team Selenium tests, this was because my context was an Ops team changing infrastructure for Dev teams. I wanted a way to test our changes before we potentially broke the Dev teams’ dev environments. This isn’t recommended if you can avoid it as obviously its not very DevOps. But in general finding a way for infrastructure-as-code to be eventually end-to-end tested in an automated way is useful because its hard to really test things like firewall and network routing configuration or file permissions until you actually try to perform certain actions from the server. The hard part is knowing when to run the tests, you to know when your infrastructure-as-code is finished and when the servers and various components have actually completed their setup and app is running. I achieved this with a Jenkins pipeline which polled a Sensu check that might look at a health endpoint from the app, when this went green I knew to proceed with the test and it would timeout if it took longer than usual.
• Writing simple scripts for monitoring or analysing our AWS account. In our context we needed to tag the hardware we were using for our own internal billing purposes so we could appropriately budget for certain projects. As this relied on humans remembering to include the tagging in their infrastructure-as-code, it was useful to regularly audit the account for servers that were missing tags and therefore wouldn’t be billed appropriately. This also made it easier to investigate under-utilised servers and talk to the owners about saving costs.

The temptation to split dev and test work in sprints - don’t do it!

Introduction

About 3 and a half years ago, I was new to sprints and scrum. Coming from the videogames industry, I was used to a process where I would test code that came from developers and return bug reports. I had heard the words “sprint” and “scrum” before but I had no idea how testing fit into them, so I joined a company where I could figure that out. This is what I figured out.

What’s a sprint?

If you’re not familiar with scrum or agile, then a sprint is effectively a short-term project plan where a team of people decide the work that they can complete within a 1, 2 or 3 week window. Work is “committed” (promised) to be completed in that time frame and the team tracks their progress. After each sprint, reviews and retrospectives are held to help the team find what works well and what helps them complete more work to a higher standard while still maintaining their commitment. The main focus of sprint work is to complete the work and trying to avoid leaving work unfinished.

Where does testing fit?

So normally teams set up a task board with columns titled something similar to “To Do, In Progress, Done”. Sometimes people add more columns or use different names but the usage is similar. Anyone from the same background as me would be tempted to then suggest that an additional column could be added between “In Progress” and “Done”. The logic being that “when you’ve finished your development work, I’ll test it”. In my head, this was trying to work with what I knew already in this new environment. We ended up with columns similar to “To Do, Build/Dev, Testing, Done”.

Bad idea

So at first, I thought things were working ok, I feel one of my strengths is learning and picking up things fast so I got stuck in and kept up with the 5 developers in my team. Most of the time I was fortunate that the work dropped sequentially or wasn’t particularly time consuming to test. This didn’t last long though and eventually we started to fail to complete work on time. This happened either because I was testing it all at the end of a sprint or because the work was highly dependant upon each other and the problems with integration weren’t found until very late.

This meant we had to continue some work in future sprints. Now I no longer had plenty of time to write my test plans at the start, but I was busy testing last sprint’s work and then testing this sprint’s work! I no longer had time to spend learning more automation or exploring newer areas to me like performance testing. All of my time was consumed trying to test all of this work and I couldn’t do it. What went wrong?

A change in approach

I would love to say I quickly realised the problem and fixed it but it took me a long time to realise the issue. I think part of this I will put down to not knowing any better and partly working with developers who didn’t know any better. Either way, a while later I realised that the problem was that I was trying to test everything and the developers started to rely on me for that. I’ve since realised that there is a fair bit of psychology involved in software development and this was one of my biggest lessons.

We eventually decided to stop splitting up work between roles, mainly because we found that developers tended to treat work that was in “test” as “done” to them, freeing themselves up to work on even more development work. This created a bottleneck, as the only tester as I was testing work from yesterday while they were busy with today. Instead, I came to the realisation that there is little benefit to splitting the work up in this way, at least not through process. We should be working together to complete the work, not trying to focus on our own personal queue. I shifted from testing after development was thought complete, to trying to test earlier, even trying to “test” code as developers were writing it, pairing with them to analyse the solution.

Understanding what a “role” means

I think for me this lesson has been more about realising that playing the role of “tester” does not necessarily mean I carry out all of the “testing” in a team. It does mean I am responsible for guiding, improving and facilitating good testing, but I do not necessarily have to complete it all personally. An additional part of this lesson is that I cannot rely on other people to define my role for me - as a relative newbie to testing I relied on the developers to help me figure out where I should be. While I’ve learnt from it, I also know that I may need to explain this learning again in future because it is not immediately obvious.

So where does testing really fit?

Everywhere, in parallel and in collaboration to development. Testing is a supportive function of the team’s work, it now doesn’t make sense to me to define it as another column of things to do. It has no set time frame where it’s best to perform, and it doesn’t always have a great deal of repetition in execution. It is extremely contextual.

In addition, that’s not to say you shouldn’t test alone or separately to ongoing teamwork. You absolutely must test alone as well, to allow you to focus and to process information. It’s just that you must choose to do this - where it is appropriate.

Definition of “Done”

One of my recent approaches was to define the definition of “Done” as:

“Code deployed live, with appropriate monitoring or logging and feedback has been gathered from the end user”

Others may have different definitions, but I liked to focus the team on getting our work in a position where we could learn from it and take actions in the following sprint. For me, it meant we could actually pivot based on end user feedback or our monitoring and measure our success. Instead of finishing a sprint with no idea whether our work was useful or not, planning a new sprint not knowing whether we would need to change it.

Summary

• Avoid using columns like “Dev” and “Test” in sprint boards. It seems to lead to a separation of work where work is considered “Done” before it is tested.
• Instead, try to test in parallel as much as possible (but not all of the time), try to test earlier and lower down a technology stack (such as testing API endpoints before a GUI is completed that uses them).
• Encourage developers to test still and instead try to carefully pick when and where to personally carry out the bulk of the testing. Try to coach the team on becoming better at testing, share your skills and knowledge and let them help you.
• Altering the definition of “Done” seemed to help for me, it was useful to focus the team on an objective that meant we really didn’t have to keep returning to work we had considered completed. In other words, make sure “done” means “done”.

The words “Testing” and “QA”

Introduction

I’m not one to harp on about semantics, I mean, I care that we all understand each other, but I’m very conscious of how irritating it can be to constantly point out “you’re using that word wrong”. However, I do find it frustrating when people assume that language used in the community or in documentation is commonly agreed, understood and given the same meaning by everyone in the same way. Very frequently, this is not the case and the words “testing” and “QA” seem to be an example of this. As a tester, keeping in mind these semantics can be very useful in resolving misunderstandings that may have been missed.

Why do I find it frustrating?

Ever since I was a newbie to testing and the tech industry, I’ve been trying to understand. I hear phrases or words and try to find out what they mean. Someone may teach me the word, I may read it in documentation or hear it explained in talks. Most of the time I learn words through the context they are used, which is probably the most natural way we all learn language.

So I learn words or phrases and believe I understand what they mean and hence believe that I would be understood when I reuse them the same way. However, because we are creating new words and phrases all of the time, some of them don’t have a “standard” meaning. An obvious example of this I can think of is regional dialects or slang, and in the UK we have many, many words to describe a bread roll.

So if I asked for a “Bacon Stotty” in London I would get a puzzled look and if I asked for a “Muffin” I might get one of these:

I personally see this as simply a natural development of language, we invent new words all of the time to describe things that are new to us, especially if we don’t have an appropriate word to use already. However, due to distance and culture as humans we may come up with different words to describe the same thing or the same word describing two different things.

The frustration for me is when I encounter people who don’t seem to accept this. The phrase “QA” is widely used in the tech world in many different contexts and doesn’t appear to have much common agreement on its definition. For this reason, I don’t like using the phrase because I don’t believe I would be well understood most of the time. However, I haven’t felt the need to talk about this until now because I haven’t come across an example that justified my feeling on it.

Enough accuracy for enough understanding

So I recently interviewed someone who was looking to start as a tester, they didn’t have any experience of testing and were keen to understand as much as they could about the role. I will add they were impressive for someone with no experience! During the interview one of my colleagues explained the gist of how the development teams were structured and what they worked on. They described a tester on one team as performing “performance testing” and then another one on another team as performing “QA”. I wryly smiled to myself about the use of the word “QA” but I didn’t say anything to it because it was a reasonable, high-level description of how we might work in a very generalised sense. While I knew some of the words were misleading, it wasn’t the time or place to pick it apart because:

1. I didn’t want to embarrass my colleague and I didn’t want to give the candidate a bad impression of our relationship.
2. I didn’t want to spend the limited time we had in the interview explaining why those words weren’t right.
3. The gist that was given felt enough to me for the candidate to understand how we worked, at least for now.

Recognising misunderstanding and addressing it

The candidate was happy with this explanation and I was happy that it was enough to at least give them some context to ask any further questions. The interview continued and at the end the candidate expressed they were really happy we had an “Internal QA” role because they knew of a similar role in their current company which involved checking products were meeting guidelines and standards set out by governing bodies. They felt that this was a role they would like to start with because it would give them an easier step into testing and more technical roles (they openly admitted they didn’t have much technical knowledge of experience but wanted to learn).

Now it had become apparent the gist we had given was clearly not appropriate because the candidate had understood the phrase “QA” differently to how my colleague had meant. The candidate now clearly expressed a desire for such a role, particularly because they were keen to have some guidance. They liked the sound of QA because it sounded more scripted, more guided and therefore an easier leap into the tech world. I immediately explained to the candidate that in the context of this business, we don’t have a role like that and really the roles are far more exploratory in nature. I then had to go into explaining what I defined as “QA” and “testing” what the difference was and why my colleague had interchangeably used them.

This real example hit home for me a justification that the phrase “QA” is misused and misunderstood. It can have multiple interpretations and in this case it could have led an interview candidate into accepting a job completely different to what they had envisioned.

What are my definitions of “QA” and “Testing”?

My definitions are:

• “QA” or “Quality Assurance” involves checking that a piece of software conforms to a predetermined set of qualities. This can come from legal requirements, industry standards or certification. QA is generally scripted in nature.
• “Testing” involves exploring a piece of software to discover information about it. While it may include the use of checking predetermined sets of qualities it focuses on the unknown rather than the known. Testing is generally exploratory in nature.

I do not dare suggest these are commonly agreed definitions and I definitely do not go around correcting people on this. I have learned it’s pretty irritating and counterproductive to do this. However, I use these definitions to help me identify when I have misunderstood or someone else misunderstood these phrases. In other words, I’m aware that these words aren’t commonly understood the same way and I choose to clarify my understanding this way when the discussion comes up.

No one is at fault here

I want to emphasise that in the interview situation I refer to, no one was at fault. I do not expect my colleague to understand all of the semantics of testing. I do not expect a newbie tester to be aware of them either. I’m simply observing that it is useful to keep in mind these semantics and how people may be silently misunderstanding each other and not realising it.

I don’t think we can really do much about preventing this divergence of meanings of words and phrases, I personally feel they are a natural flow of language. The English language is full of words that have many contextual meanings as it is. Nor can we “know” all of these meanings. All we can do is share our different meanings of words and raise each others awareness to these different meanings and perhaps come to some greater consensus on definitions.

Providing value beyond bugs

Introduction

I’ve had several experiences where I’ve worked on projects where a manager has proclaimed “we don’t need testers for this project, we’re not bothered about bugs”. When I started testing 6 years ago, I would have felt that this was wrong simply because “bugs” can take many forms. However nowadays I’ve come to realise that testers provide much more than “bugs”. I still find it incredibly difficult to explain this to people (particularly managers) though and it has only been through bad experiences that I’ve felt justified in arguing the case.

Why people might think they don’t need testing

I have now worked with several projects where I’ve been told that testing wasn’t required, the reasons have varied:

• “This is an internal project and we’re not bothered about embarrassing bugs”
• “This is a quick prototype and we want you to focus on more important projects”
• “This isn’t the software you need to test, it’s just a library/tool/software we’ve bought so we don’t need you to look at it”
• “This project is in its early stages so there is nothing for you to test yet”

It seems that people still see testing as simply being the bugs we report, and not only that, but sometimes people are only thinking of bugs in the popular sense - visually obvious and embarrassing bugs. They also seem to be making decisions regarding risk and priority with the information they have at the time.

Why did I think these projects would benefit from a tester?

Well I think the main reason is because I see testing differently, I don’t see my job as simply reporting bugs, but about telling the truth about software, about observing the people and processes that produce the software and trying to help those people to produce better software. In other words, I see testers as people dedicated to creative and critical thinking on a project.

• Internal projects still need to ‘work’ right? Not to mention, do we really understand all of the kinds of bugs or risks up front about the project? Maybe the project is for internal use, but does it interact with external systems? What about the people and processes, do we not want to help them? Do we not want to track the progress of the project? Even internal projects have costs and implications if they are late or don’t fit the requirements.
• Frequently quick prototypes prove to be quite useful, this is generally their purpose - to rapidly learn what is useful or valuable. Testers are great at rapid learning and especially learning what customers or end users find valuable or useful, so such information is invaluable when moving on to design the eventual fully-fledged product. Why not boost the success of your prototype by involving a tester to help focus the product on the learning as well as bring their skills in analysing the truth of what makes the prototype a success. Not to mention that prototypes tend to very quickly become the “finished product” without any re-design or re-development, it’s easy for people to assume it’s ready to change the usage from “concept” to “ready for mass use”. Involving a tester might help avoid this easy slide and highlight the risks.
• Also, what if the projects we are testing affect your prototype? We won’t know that if we aren’t aware of your prototype, which means there is a risk that we could break your prototype if it depends on other projects.
• Assuming other people’s software is well tested and perfect is an all too easy assumption. Then there is the question about whether it’s even compatible with your own software and the assumption that you understand what it does or how it exactly works.
• Many bugs are caused by assumptions made at the design stage either because of ambiguous language, cognitive bias’ towards information or simply because we cannot think of everything. Does it not make sense to more cheaply catch and resolve these bugs at the design stage rather than finding them after we have spent time building a product?

I believe in all these cases and in any project, you can benefit greatly from involving a human being who dedicates their focus to critically analysing each piece of information and offering feedback. It’s not just about banging on keys or finding “bugs” in the language of computers, you can discover plenty of “bugs” in the language of humans.

Changing the tone from an argument to an invitation

Its draining to constantly have to argue to be included in meetings and projects and this can heavily affect my motivation at times. It’s much easier when I’m invited and do not need to convince people of my value. How to do this then? I think I’m still learning how to become better at this, but one obvious way is simply by becoming extremely knowledgeable about a project, its technology and the end users. By simply being able to answer many questions and provide this knowledge, you naturally become an oracle people refer to, which means you become invited to more meetings.

Take advantage of your opportunities to learn

What do I mean by opportunities? Well firstly, opportunities to become very knowledgeable can take many forms - for example, investigating bugs typically shows you the guts of a system and also provides information of why a system was built a particular way (otherwise how do you know its a “bug”?). Try to view everything as an opportunity to learn more and you may pick up and remember a lot more.

Now you’re in a design meeting, how do you prove your worth?

I also mean opportunities in terms of demonstrating your value in providing critical and creative feedback. It can be very easy to squander these, I’ve repeatedly been too critical, asking the wrong question at the wrong time and earning the ire of my colleagues for wasting time or dragging out discussions. This can lead to being left out of discussions for being disruptive and not constructive.

Another area I’m trying to learn to improve is catching myself when I react emotionally to something, I tend to blurt out a critical question when I see something wrong. Sometimes I should consider how to word my question better to come across as more inquisitive rather than criticising. Sometimes I should really ask the question later and not put people on the spot. However, as ever, such feelings can be heuristics and sometimes I am right to ask the question there and then, sometimes the emotion draws attention to important details. The main point here though is that you shouldn’t always let your emotions guide you.

Trust

Asking too many questions tends to make people feel they are being interrogated, and perhaps not feel trusted to do their jobs. For example, several times I’ve found myself invited to meetings about new projects that I’m not completely up to speed with and a discussion start with the participants assuming I know some things already. It’s too easy in these situations to criticise and say that they’re making a lot of assumptions, when really they established details in another meeting and you’re simply not aware. While this is something that needs to be resolved, by aggressively criticising them for this, they will get defensive and you potentially derail the meeting. It’s being aware of these situations and being critical at the right times that allows you to provide your value, being critical all the time feels counter-productive to me.

Asking questions rather than making declarations

I’ve caught myself declaring things in the past, which comes off as criticism of an individual or a statement of objective fact to argue about. I try to instead ask questions rather than make emotional statements, emphasising that I’m not criticising an individual or think something is bad. I guess in a way, this is a kind of “safety language”, to use the RST (Rapid Software Testing) phrase. For example;

Instead of:

“This isn’t a good idea, I don’t know why you’re suggesting that?”

I might try:

“I don’t mean to criticise you, but are we sure that X is a good idea?”

Instead of:

“These requirements are really ambiguous and unclear”

I might try:

“Apologies if this has already been covered, but could we clarify the requirements here? X and Y could be misinterpreted?”

Conclusion

I think to change the view on whether testers contribute more than just bugs, we have to show value in other ways. One way to do this is becoming an oracle of knowledge about a product, business or end users in general. This can then lead to further opportunities where we can provide value through our critical analysis skills. However, if we are not careful, we can deal more damage to this image than good if we are too disruptive.

I care a lot that my contribution to projects is valued and that I’m of service. I don’t want to waste people’s time and I want people to invite me because they appreciate my skills and knowledge. However, the nature of testing tends to be viewed as negative and sometimes I myself contribute to this negative view when I disrupt meetings with heavy handed criticism. Life is all about carefully balancing my critical feedback with self-criticism and also recognising opportunities and carefully not squandering them.

I love testing and this is why!

Introduction

Over the last year I’ve been on a little bit of a journey of discovery, for 5 years I had been working as a tester but never really seeing it as a career I wanted to keep. In this 6th year of testing, I finally realised I loved it after discovering the friendly and passionate testing community. I’ve learned to appreciate my own work, justify it and learn how to become better and better at it. Suddenly what felt like a job that didn’t have any further growth exploded into a fascinating and immersive world where I could maximise my skills and help others to realise theirs too. I’m taking this moment to write down why I love it so much as a little preparation so that I’ve thought about what I want to say whenever I talk about this again in future.

Testing is endless learning

I’ve spent most of my life (up to this point) in education, I went to nursery at the age of 3 and graduated with my degree at the age of 21. On reflection I feel I’ve always liked having that next target, be it grades for next year, progressing to university or securing full time work. I’ve come to realise one of the things I relish about testing is always having something to learn. I now effectively describe my testing as learning, especially thanks to attending and studying Rapid Software Testing with Michael Bolton which really hit home this point. As an example, before I started my current job, I knew very little about performance, security, API and automation testing. I can now confidently talk about each of these subjects and know where I need to improve.

When someone presents me something to test, I’m trying to learn as much as possible as soon as possible and I love doing this, I love building on each bit of knowledge and using it to ask more and more questions. Let’s say for example I’m presented with an accounting service that integrates with other systems, I start reading the documentation or asking questions and learn that one of these integrations accesses through an externally-facing API. So then I’m asking questions about what is this integration? What does it do? Who is it for? Why have we built it? Can I test it? What does it expect as a response? Is the API secure? What happens when I try to access it without authentication? What happens when an error occurs? What is the desired performance end-to-end with this integration?

Testing is about facts, though opinions are valued too

It’s sometimes easy for people to perceive testers as picky or pedantic people, always criticising or bringing bad news. If we are not careful with our expression, we can over-state a personal opinion or fail to justify why something is a serious problem. I personally enjoy trying to constantly improve my communication, maintain a high standard and keep learning how to interact better with so many different kinds of people. All the while trying to keep in check my own emotions while at the same time knowing when to listen to them.

Testing is about being involved everywhere

As a tester I find myself working at every point from start to finish of a software project. From trying to learn as much about our customers and the business to learning all about the behaviour of a particular line of code. I get to work with so many other roles such as development, business analysis, project management, systems administration, technical support, sales and marketing and all the wonderful and horrible ways people might like to use the software. Testers have moved on to become business analysts or project managers and I think it’s a very natural career step for some.

Testing is about being persistent

Like the world’s best detective, as a tester I try to leave no stone un-turned and I try to look beyond the obvious explanations. I actually find this one of the harder parts of testing, it’s easy to accept an explanation that you don’t fully understand, because sometimes you think you do but don’t really. Sometimes it’s also difficult to keep persisting, especially when a problem resolves itself. Maybe you don’t always have time to keep searching. However, the satisfaction in finally discovering the root of a challenging problem is so high that it makes it worth it. I try to remind myself of the detective analogy because it helps re-energise me (it feels more exciting!) when dealing with tricky problems.

Testing is about directly and indirectly helping people

In providing quality information, I’m helping people all of the time. Helping developers learn more about what code actually does, helping product owners track the progress of a project or helping everyone become more careful with their language and aware of assumptions. I’m also indirectly helping end users of software, trying to understand their needs and understand how these match against the software. I feel caring about people is so very, very central to testing.

Testing is about balance

I like to think of myself as pragmatic and realistic, I care about being critical of my own views with respect to considering someone else’s point of view. So I relish the challenge of balance, I see so many things in life, even life itself being one of balance. There are always pros and cons, always another perspective, another time when things might make more or less sense. The most common challenge that crops up in testing is balancing the desire to learn as much as possible with being quick. Why is time important? Because to learn everything you need all of the time in the world, which you never really have. So you’re always prioritising and trying to identify where the most important pieces of information may be hiding so you can find them as soon as possible. Everything costs time and so we are always asking ourselves whether it’s worth spending the time or if we are spending it in the right places.

Testing is a field ripe for discovery

While many people have discussed, experimented and written about testing for decades, it still feels like many of us are only just beginning to really understand it better as a field. The spread of knowledge and encouragement to discuss how to become better at it seems to be getting closer to other fields like programming. To some extent, it feels like testing is still lagging behind those other fields, we are only just starting to see more conferences and meetups. As a relatively new tester, I can’t speak for what it was like before, but I went a whole 5 years without even knowing there was a community who talked about it, let alone had ideas to improve it. I like this because it feels like an exciting place to be, with so many topics to explore, so much to possibly contribute to. There feels like a lot of work still left to be done to reach out to more people, both new testers and those who may become a tester in future. Not to mention that as we educate ourselves about what testing really means, we also become better at encouraging those other fields to understand it better too and interact with them better.

Testing is great for drawing on diversity

I make use of several different skills and bits of knowledge all of the time in my testing. An obvious example would be making use of my programming background to build useful tools or understand and read code. But I’ve also made use of my education to help me write reports, produce presentations and I’d even credit my studies of history (such as understanding how to analyse sources) with how I approach analysis. I’ve heard of examples of testers using their background in journalism to ask better questions and gather better answers or testers with a keen ear for audio being able to more accurately describe an audio problem. I love that testing so visibly benefits from diversity.

Testing is very psychological

I love psychology, it’s fascinating to me and so much of my work is affected by my psychological state. Topics such as confirmation bias and social interaction come to mind as the most obvious examples. Understanding these areas helps to stay alert to some of the most easily missed issues. For example, if you consider Usability Testing and how you analyse the information you gather from it. You don’t just ask users a list of questions and take away their answers. You are looking out for so many psychological factors too - how are they interpreting the question? Why did they give that answer? What were they thinking when they performed that action? When they say that they want X do they really need Y?

Testing is creative

In our pursuit of learning, it helps to be creative, because it allows us to explore much more than if we stick to the path laid out for us. In testing, being creative is highly rewarding as you are better able to not only discover issues that no one had thought of but also workaround issues or make an innocent-looking problem worse.

Testing is about being a team player

Testing cannot be separated from the work done to produce software. I’m not a rock star, walking onto projects and “assuring quality” and telling people what they’ve done wrong. I’m here to support others in producing software, to work together and learn together. By working more closely with people and learning to become better at working with people, my testing becomes more effective, efficient and supportive. I personally hate working alone and love working in a team. I love “team spirit” and pushing each other to become better.

Summary

This all I can think of right now on why I love testing, I’m sure I have more that will crop up another time. Many thanks to specifically James Bach, Michael Bolton, Jerry Weinberg and Karen Johnson for unlocking this love and appreciation for testing and the ideas and analogies I use in this article. Particular thanks to Michael for this blog post which inspired me to write this and also Andy Tinkham on this podcast for reminding me (through the idea of writing down what I think testing is) to write this.

Games Testers should be proud of their work, not ashamed

Introduction

A few months ago I interviewed someone for a testing role and the question of past experience came up, to which the interviewee said something along the lines of ‘well this might be a bit sad, but I’ve done a bit of video game beta testing’. Then a week later I noticed a discussion on the Ministry of Testing’s slack team where people were discussing how they had got into testing. A couple of people mentioned how they found it useful to start in games testing but felt games testers get a lot of stick for what they do. This prompted me to jump in and talk about my story as I’m quite passionate about this subject! In fact this very sentiment is where my career pretty much started after university!

Pizza, pop and candy

I will never forget this line. I had just finished university and was applying for every programming job I could find. There was a wild week (they seem to happen every year or two) where I had accrued three interviews to take place in one week, just prior to one of my brothers getting married. I had two interviews for programming roles and one for a games testing job.

The first interview I didn’t pass, the second one I seemingly impressed and got offered the job to test games and the third one? Well, bearing in mind I wanted to get a programming job to make the most of my degree, I went into it feeling glad to have the games testing job in my pocket but wanting the programming job. However, the third interview went something like this:

“So we see from your CV that you’re interested in working in the games industry?”

“That’s right, but I’m keen to learn whatever I can from any programming role I can find!”

“Pizza, pop and candy”

“Excuse me?”

“That’s what you ‘gamers’ like isn’t it?”

“Erm…”

“Yes, we hired a programmer from that games company nearby, what was the name?”

“Traveller’s Tales?”

“Yes, that’s it, but they didn’t last long. Great programmer, but not a good fit for our organisation...”

Needless to say, that was the most awkward, aggressive and stressful interview I’ve ever experienced. I honestly don’t know why they decided to waste their and my time in such an interview. But it made me suddenly feel very, very comfortable to be accepting a job in games testing and definitely removed any second thoughts! My testing career effectively started from a statement of disgust about ‘gamers’.

Learning to test

So here I was in a job testing, having barely heard the word ‘testing’ before (it seems crazy to think now that you can complete programming degrees without ever hearing the phrase ‘unit test’!). I’ve loved video games all of my life and I had seen my fair share of bugs with them, so I knew pretty well what ‘good’ looked like and what ‘bad’ looked like. I also play a lot of different types of game at varying difficulties, so I had a lot of knowledge to draw on - not to mention a degree in games design & programming. I understood what people wanted from games, how they are made and what could go wrong. I hadn’t worked a proper full time job before but I was pretty confident I could do a good job.

I learnt a huge deal games testing, especially regarding the sheer variety of forms a ‘bug’ or ‘problem’ could take. Not only were there the obvious bugs that a lot of people might recognise like crashes, hangs and freezes (yes, they had technical differences) and graphical glitches, but also problems you don’t necessarily see visually. The bulk of the testing was exploratory in nature, usually with a task being to focus on a particular area - be it a collision detection test or a playthrough of the game just to make sure it could be completed.

To give you some idea of the variety of problems we would look out for:

• Graphical - the most obvious and visual kinds of bugs that affected the look and feel of the game.
• Functional - quite simply, bugs to do with what the game should ‘do’. This was very contextual as every game is designed differently, but there are some fairly typical consistencies such as being able to steer a car in a racing game..
• Difficulty - this was pretty subjective, but we had to judge whether ‘easy’ was consistent and didn’t suddenly jump in difficulty on a particular level.
• Audio - maybe the voices in the game don’t synch up with the animations of the characters or the quality of the audio was poor (such as once hearing the ‘beeps’ of the recordings being started and ended for the voice actors!).
• Legal - sometimes games would include trademarked or copyrighted material which they hadn’t credited. Or sometimes they could accidentally feature close similarities to real world products.
• Health - we were trained on photosensitivity and how to identify bugs that would trigger epilepsy.
• Technology - we were sometimes testing for bugs with particular technologies such as 3D, motion control, force feedback devices, augmented reality, virtual reality headsets and more! All of these technologies feature their own particular bugs and problems.
• Compliance - we checked that games were compliant with the standards set out by the corporation. Games that didn’t comply with these standards would be rejected so we tested to make sure they were compliant.
• Localisation - although we didn’t translate games nor could necessarily read all languages, we did check for bugs related to the subtle differences in localisation. An innocent missing language file would be enough to make a game unplayable or crash. You don’t need to be able to read German to notice the text doesn’t fit inside buttons!
• Multiplayer/Performance - we tested for load, stress and also for exploits. Some games need to be able to handle large amounts of players and exploits can ruin people’s enjoyment of their online gaming experience.
• Compatibility - there were some cases where games were ported to new hardware or developed to communicate with older systems. We tested to find problems with how these games were compatible.
• Transactions - we tested to make sure real world payments could be completed and that people couldn’t exploit these systems.

There are probably a few I’ve forgotten, but hopefully that gives you some idea of the wide range of issues we would learn about and find!

Along with learning about the wide variety of problems, I also learned about testing techniques and strategies. Unfortunately the company was heavily using test cases at the time. There were differing views on these and I learned to find and report problems from my exploratory sessions more than fill out the boring and repetitive test cases. There were also the beginnings of automation testing which was being used to help carry out load and stress tests in multiplayer games along with experimented use in running regression test cases.

I learned two major lessons about testing in this job:

1. I hated test cases and found them fairly useless. Occasionally they were useful to find test ideas or guide exploring a product, but they felt awkward to write and use and just generally wasteful of time. I found most of my bugs when I was performing exploratory testing.
2. Automating tests on a product that was constantly changing, especially in a graphical sense was a waste of time. By the time we had written automated tests of the product, it had changed again and we had to re-write them. Quite a lot of game projects didn’t require testing after release, so the value of the automation tests was rarely seen. It only seemed to return some value when running large scale performance tests.

What did I pick up that could apply to other testing jobs?

There were things that I picked up in my first job that I hadn’t formed thoughts around or hadn’t found the language to describe effectively but were important to becoming a better tester. While I gained better understanding and language later in my career, my first job gave me awareness of these issues:

• Managing relations with developers - I quickly learned that diplomacy was important when raising bugs with developers. I picked up an analogy that I re-used in future testing interviews - “imagine an artist has created a magnificent piece of art and you come along and say ‘you missed a bit there’ - you can imagine that’s quite irritating so I look for ways to soften that blow”.
• Costs of automation - I didn’t have the language to describe it, but I learned the hidden costs of automation and the temptation to try and find ways to use it. I was already wary about the value and how easily people could misunderstand that automation is some kind of replacement for ‘all’ testing.
• Knowing when to stop testing - I had started to get experience on when and where I might want to stop testing. While we were generally time-boxed at in my first role, I definitely saw the diminishing returns and learned which risky areas I wanted to focus my testing on.
• Knowing when to repeat tests - I also started to form ideas on how to judge regression testing although I didn’t know at the time how else it could be done other than re-running test cases.
• Justifying my testing - I had started to build up confidence through my experience of being able to justify my testing and write effective bug reports. While I didn’t have the best language to explain myself, I at least had experience to refer to that gave me a determination to learn better language!
• Determination to understand more about the product, earlier - Typically in my first job, we didn’t have very good documentation to refer to (sometimes none at all!). When we did have documentation, it was much easier to quickly provide feedback on the product. So in future roles, I had a strong determination to get involved earlier in projects so I could understand more about them, much more quickly. I knew documentation was expensive but I felt that trying to access sources of information was crucial to getting my testing really started.
• Thinking ‘outside of the box’ - I quickly learned that the ‘best bugs’ were the ones that people hadn’t even thought of. Sometimes these would be fairly expensive bugs to fix and it was valuable to find them as soon as possible.
• Being organised, fast and clear - I took pride in my work being organised but fast and effective. I quickly understood the need to setup my tests quickly and maximise the amount of time I had available. I created my own techniques for being able to do this, while still providing clear feedback in my reports, which have served me well to this very day!
• Being adaptable and willing to learn - My programming background helped me pick up technical or complex products very quickly and I learned that this can be very valuable. Again, it was useful being able to maximise the time I had available to test and it meant I could provide quick and clear feedback. I then looked for opportunities to learn more skills that could complement my testing efforts.
• Understanding and empathising with the end user - If you’re in a games job, you’ve probably played a lot of video games and so you probably have a pretty good idea of what end users do, what they want and what they like. So it was easier to think like an end-user and therefore I really appreciated the value this brings when coming up with test ideas, justifying your testing and justifying your bug reports. This experience still motivates my desire to always understand more about the psychology of the end users as I highly value it in my testing.

Summary

I was a games tester for 3 years in two different companies and I will never talk about it as a negative experience. That’s not to say that everything was perfect, the games industry has a lot of problems that still affect it today. I left the industry partly because testers are treated as an end-of-cycle role and because I was keen to build my career in testing. However, I still managed to leave and secure a job I would describe as being ‘professional testing’ - I wouldn’t have been able to do this without the experience I gained from games testing. If you are currently testing games and you feel like you’re not a professional tester and have to prove something - you don’t! You’re already a professional tester!