Which languages are best to learn for testing?

Introduction

I’ve seen this question raised quite a lot in testing circles regarding which programming language is best to learn. Like it or not, the current trend in the industry seems to be asking much more of testers, with a view to creating more automation and having a much greater understanding of the technology they are testing.

Why learn a programming language?

What is your motivation for wanting to learn a programming language? In order to test well, you don’t need to know any programming language. There are very particular situations or contexts where programming may be useful to me as a tester, such as wanting to write some automated checks, learn more about what the product I’m testing is actually doing under the surface or simply wanting to save time by creating tools to help myself. However, these situations don’t arise all of the time.

It’s also worth highlighting that to write programs, I need to understand a lot about the domain I’m working with. If I want to write an automated check, I need to test the product first to understand what is worth checking. If I want to read some code, I need to understand the context that code is used for, what its purpose is.

So even if I did have something that is worth programming, I still need to “test” to identify it, understand it and consider whether it is worth it. Simply learning to program is not enough, which is why as a tester you can bring a lot to the design of automated checks and why developers cannot easily test the product themselves.

Automated checks

So it seems the usual reason testers are looking to learn a programming language is to create automated regression suites for speed and reliability. Typically I find the advice tends to be that you should learn and use the same language as your back-end developers (so if they use Java to build the product you test and Java to write unit tests, then you should learn Java too). The argument being that by using the same language, you can access their support and help more easily when you need it. However, this depends upon your current relationship with your developers and their location. Perhaps you may not be very close to your developers and won’t benefit from their support - this may not be something you can easily change.

You are going to have judge for yourself which language, but the biggest factors that affect my choice would be:

• How comfortable am I writing code in this language?
• What support can I get from the developers I work with?
• What support can I get from other testers?
• How well supported is the language in terms of libraries or capabilities? (for example, if you want to write Selenium checks, is there documentation on how to use Selenium in that language?).
• Can I write programs in this language in a way that makes them easily understood by other people?

There is no easy answer to these questions so I wouldn’t recommend any particular language. However, to help narrow your research, I would suggest focusing on these languages to consider:

• Java
• C#
• Python
• Ruby
• Javascript

At the time of writing, these are some of the more popular languages to learn with regards to automated checks.

Toolsmithing

Maybe you’re interested in simply being able to make use of a programming language to create your own tools. A “tool” in this context can be something small like a script that rapidly repeats a certain action over and over. For example, I once created a script that compared two sets of 100 invoices with each other. It looked at each invoice total and compared the old total with a new one and saved the difference in a text file. This meant I could rapidly compare and identify obvious errors, saving my own time and helping me perform this particular action more accurately. However, it didn’t replace the testing I performed, it simply augmented it, allowing me to focus on more interesting tests.

I created tools like this in a programming language called Python. I personally love using this language because it’s very easy to read, has a lot of support in terms of libraries and documentation and can allow you to experiment with ideas very rapidly. I very much recommend Python as a starting point for building simple tools and it can be used to write automated checks if you so wish.

There’s a great tutorial on getting started with Python here.

Alternatives to programming

Do you want to become a more technically capable tester? Not really keen on learning a programming language but feel like you need to learn? Well perhaps you can find value in learning other technologies and concepts. While programming is a very powerful tool, it’s not the only one that a tester can learn in order to become more technically capable.

• Test lower - maybe you could understand more about the technologies powering the product you’re testing and test lower and earlier? For example, many web services are built upon APIs (Application Programming Interfaces), perhaps you could learn how to test these? A place to start interacting with APIs is trying out Postman.
• A similar approach to testing lower is learning about databases and how to operate them using languages such as MySQL or Postgres.
• Research tools that can help you test or provide more information. For example, Google Chrome DevTools have lots of very useful tools for interacting with websites and debugging problems or emulating mobile devices.
• Talk to developers! Ask them to draw diagrams explaining how the product works technically and ask them explain things you don’t understand. It can be difficult at first knowing what is important to remember and what can be forgotten but simply taking an interest in their work and asking questions can even help them understand their own work better. I find there is no better test of my own understanding than having to explain myself!

Summary

• You don’t need to learn a programming language to be a great tester.
• There is no one particular language that is “the best”, but there are some popular ones that are good to get started with.
• There are other ways to become a more technical tester that don’t involve learning programming.

NWEWT #1 Regression Testing

Introduction

Last weekend I attended my first ever peer conference, which was the North West Exporatory Workshop on Testing (NWEWT). This conference was organised by Duncan Nisbet with help from the Association for Software Testing (AST). I’d met Duncan at the Liverpool meetup for the North West Tester Gathering after he had presented a great talk on ‘Testers! Be more salmon!’. Shortly after that meetup, he contacted me asking if I was interested in attending this brand new peer conference. After mulling it over, I accepted it for two reasons:

1. Why not? (pretty much the main reason I’ve done a lot of things!)
2. It sounded like an opportunity to learn by deep, critical thinking - I liked the challenge of presenting my ideas and having people really critically analyse how I think about testing. If nothing else I was going to get a lot out of it by forcing myself to think about a subject deeply in preparation!

Attendees

The attendees were as follows, the content of this blog post should be attributed to their input as much as mine, the thoughts I have here were brought together through collaboration:

Ash Winter

Callum Hough

Christina Ohanian

Dan Ashby

Duncan Nisbet

Emma Preston

Gwen Diagram

Joep Schuurkes

Jean-Paul Varwijk

Liam Gough

Lim Sim

Richard Bradshaw

Simon Peter Schriver

Toby Sinclair

Tom Heald

Theme

The main theme for this conference was ‘Regression Testing’, specifically what we loved or loathed about it, whether we even do it, whether we should automate it and just generally what our experience and thoughts were.

What the hell is a ‘peer conference’?

I had no idea before I went! I researched around and I knew it was following a format whereby deep discussion and debate were encouraged between professionals, but as with many things, I didn’t really understand it until I started doing it.

Basically, there were 15 or so of us gathered, each bringing our own ‘experience report’ on the theme or topic for the conference. We each took turns in presenting our experience report through slides, flipchart or just simply talking. We then held a Q&A session where the real action started.

The Q&A session featured green, yellow and red cards. People were only allowed to talk when indicated to do so by the facilitator. If people wished to ask a question or contribute to the discussion, they held up one of the cards which had the following uses:

• The green card indicated to the facilitator that you would like to ask a new question or start a new thread based on the current discussion. So at the start, everyone would show green cards because there was no thread yet.
• The yellow card indicated to the facilitator that you would like to ask a further question or talk about the current thread. This is how the discussions got deeper and deeper into particular threads.
• The red card indicated to the facilitator that you felt the current discussion needed to stop or that you think a ‘fact’ being stated by another person was wrong. We didn’t see a lot of use for this card, only really once or twice for when particular threads went too long or something needed clarifying. Red cards can only be used for situations that the facilitator feels are genuine, so they can be taken off people if they are abused.

Typically, the discussions were mainly between the presenter of the experience report and the person asking the question. However, they could shift to a discussion between two other people - when you showed a yellow card you could directly challenge the person who had caused you to raise the card.

On a personal note

I actually think one of my biggest takeaways was cementing the feeling that meetups and conferences are not as scary as they might seem. What I mean by that is it's easy to feel like members of the community who are quite out-spoken or actively involved are not approachable. I’ve thought about this a lot recently and I think for me it comes from the assumption that because people are experienced, they know everything I know and have already come to the same conclusions. At the start of this year, I felt like I didn’t have anything new to add and more experienced people have taken their thoughts to a more advanced level. I guess I also felt that well-known people must get a lot of questions from their public position, so I naturally feel like leaving them alone, especially if I think my questions or thoughts are less developed than theirs.

So if you’re reading this and feel the same way about meeting testers and asking people questions, then fight those thoughts! My experience at all of the testing events I’ve attended so far is that everyone is more than happy to take the time to listen to you and help you! The key here is that you’re open to suggestions and ideas from other people and this is one element of my “oh god I don’t know anything” thought process I’d like to keep - I’d like to remain humble. But don’t be afraid to ask the questions and approach people!

Takeaways

Enough about my personal development, what about the content? Well I think the biggest takeaway that I think everyone agreed upon was that ‘regression testing’ is a phrase that is poorly defined and definitely not consistently used in our industry. I found myself agreeing a lot with Joep’s idea of not even talking about it. He suggested that instead of using ‘regression testing’ we could just talk about whatever we are doing, e.g. “I’m performing these tests to find out if these two systems are functioning as we desire”. However, that doesn’t mean we should never use the phrase, it may be that we feel within a particular circle of people or within a company, there is very clear understanding. The point is more about being aware that phrases such as this may not be as clearly understood as people might think and can even be used to not think about what you are doing. Again, Joep gave a funny example of it being a ‘jedi mind trick’ whereby managers are told “we’re regression testing!” to which they respond - “great” and walk away.

Several people also shared their different approaches to regression testing. Richard shared his F.A.R.T. Model which I had seen before and Ash also shared his own, similar model for exploring the large unknowns of systems. Toby took a different approach and discussed the idea of ‘regression in testing’ - the idea that your skills and knowledge regress and what we might do to try and combat that.  

One of the best parts of attending events like this is learning how exactly people conduct testing within their companies and the different situations and problems they are having to deal with. Christina, Simon and Tom all shared different situations that I think generated a lot of useful discussion and debate, they definitely gave me plenty to think about in terms of considering how I would approach those situations. Richard gave a particularly useful piece of advice that I really love - which I can only paraphrase as ‘don’t focus on the politics, make sure you’re still doing a good job first and foremost at all times’. This really struck a chord for me personally as I have experienced some very political situations that I haven’t agreed with, but I personally value my own professionalism to still deliver good work despite this.

Another idea that stuck in my head (unfortunately a lot of our notes were binned by the hotel staff on the second day so I’m stuck with just a few notes and my memories!) was Jean-Paul’s idea of using the phrase ‘continuous testing’ to help highlight the need to still perform manual testing throughout ‘continuous delivery’ pipelines - in other words combat the feeling that continuous delivery leads to people forgetting about testing. However, we did also discuss that potentially this could have the opposite effect where people treat testing as a separate concern because we are using a separate phrase for it.

In summary, there was a nice mix of ideas and approaches that I felt I could apply to my work or in future as well as a lot of food for thought. I feel like there are some threads that left me with even more questions - I’m guessing this is normal for these things! Unfortunately, I think a lot of the attendees had a lot of similar points of view, so we ended up agreeing on a lot of topics without much debate. However, I think I still learnt a great deal and it was useful to find a lot of validation in my current line of thought on this topic.

My experience report

Not everyone got a chance to share their experience report but I was lucky enough to be one of the chosen! I’ve written up my views on regression testing here:

What do we mean by regression testing?

Summary

I really enjoyed my first experience of a peer conference a lot and it left me wanting more. I really liked the chance to start digging deep into a topic. It was also nice to find a lot of validation of my own ideas on regression testing and to learn new ideas and approaches from other people.

What do we mean by regression testing?

Introduction

I recently attended a peer conference where I presented my experience report on regression testing. I’m going to go into the detail of what I talked about in this post - if you’d like to read more about the peer conference and what on earth a ‘peer conference’ even is, read here:
NWEWT #1 Regression Testing

Disclaimer

I have no data to back this up, before I go any further I’d like to highlight that. What I state here is my opinion based on observations and my interpretation of people’s words and actions. I’ve not really thought about how I might go about measuring and collecting data to back up my words here, but I’m going to bear it in mind for future topics!

Feelings of mis-use

For the last 6 years of my testing career I’ve never been formally trained on testing. I don’t have an ISTQB qualification and a lot of what I know about testing I have either been informally trained or I have learnt myself from observing others. I therefore have a definition of ‘regression testing’ that I have learnt from how others use it.

Before I go into what I personally define this phrase to mean, I’ve noticed ‘regression testing’ referred to as:

“The testing we perform at the end of projects, where we re-run all of our tests and make sure the code hasn’t broken anything.”

This kind of understanding also seemed to crop up and become twisted with statements like:

“I’d like to press a button and run all of the regression tests automatically.”

“Why didn’t your regression tests find this?”

After being invited to the peer conference, I decided to look up the wikipedia definition of ‘regression testing’. I went with wikipedia simply because it’s likely to be a commonly used source. I actually found that wikipedia’s definition fits my own definition fairly well:

“Regression testing is a type of software testing that verifies that software that was previously developed and tested still performs correctly after it was changed or interfaced with other software.”

Combining this definition with my recent understanding that ‘testing’ is about learning, I think this fits with my own take on ‘regression testing’ which would be:

“Regression testing is considering the risk of change”.

Comparing my own definition with how people seem to use the phrase, I realised people seemed to define regression testing with the attributes of:

• Repeating tests and therefore being highly automated.
• Being performed late in a project, at the end of a sprint or a waterfall. Usually it seemed like it was considered an activity to be performed after development activities.

This seemed totally wrong to me, and I started thinking about why I felt that.

Why does this matter?

Well the first question I felt I needed to answer is - why does it matter people have these differing definitions? Well, this is what I could think of:

• If regression testing is only performed at ‘the end’, does that mean we only consider the risk of change at the end?
• If regression testing is an activity to be carried out after development activities, how invested are non-testers in the results?
• If we are only performing regression testing at the end, and therefore only considering it later, how do we effectively identify risks?
• Assumptions are being made about how valuable it is to repeat tests and the cost of executing them.

In Practice

In my opinion, everyone involved with software development is considering the risk of change all of the time even if only subconsciously. Typically, we’re nearly always changing the software and hence discussing what the desired behaviour is. However, for me the danger of the phrase ‘regression testing’ and how it seems to be commonly used is that we’re leaving the bulk of this kind of critical thinking to the end of projects. Not to mention over-estimating the value of simply repeating tests over and over.

Could we be finding problems with change earlier? When it’s cheaper to both identify and act upon?

What if we are developing a component that is part of a larger system, are we not considering the risk of change as we develop it in isolation? As we start integrating it? Is it wise to only to consider the greater implications of a change later on?

What if we’re developing a change to the red component, are we only going to consider its integration with the blue component later? What about the effects of this change to the components that are not directly connected like the green one?

With git branching, where do you perform ‘regression testing’ here? Who performs the testing? How do they identify when and what to test? When is ‘at the end’? What tests would you run ‘at the end’? Are you only repeating tests? Or running new tests?

These questions are hard to answer when you don’t have all of the information available to you, with these git branches you may not know what was changed as part of each branch - you may not even know how many branches there are and when they were merged. The developer who is writing each change and merging it knows this, but do they identify the risks? Or do they ‘leave it for regression testing’?

My conclusions

I’m concerned that ‘regression testing’ is being referred to as a standard testing phase, different to the testing carried out as part of development activities. This seems wrong to me because it encourages the ‘over the fence’ mentality of ‘coders code, testers test’. If we are to effectively identify the risks of change, we need to work together as a team, not as separate roles.

I believe regression testing is a continuous activity that should be performed as soon as possible. The reason it may have to be performed ‘at the end’ is not because it should be, but because that is the earliest possible point. I currently work in an environment which desires a ‘lean agile’ approach (speed) at the same time as developing a microservices architecture (decoupled). Both of these require us to become smarter with our testing, we don’t have time to run large numbers of repeated tests just before releasing only to find out a flaw in our work that could have been realised with some greater collaboration earlier on.
Finally, regression testing is not an automatable activity - it is the process of consideration and analysis, not repetition. Repeating a test may be useful to help us evaluate risk, but sometimes we may want to try a totally new test. Regression testing isn’t just repeating all of the tests that were run before - as the risk changes, so do your tests. Your testing needs to be continuously adapted to this changing risk.

Test Cases - do we need them?

Introduction

I’ve worked in three different companies now as a tester and I’ve read, written and executed a lot of different types and styles of test case. My time especially at a large corporation working with large numbers of test cases written by a large number of different people really gave me some varied experience with them.

Not only that, but these three companies had different approaches to their processes and testing reflected that. I’ve worked with gigantic test suites of thousands of test cases, projects where the test cases were a single spreadsheet and projects where I tested with no test cases at all.

Which then begs the question, do we need test cases? Is there such a thing as too many? Or are there not enough test cases?

The realisation

I once asked my testing team this question - what do you find test cases useful for? Some of the answers I got back were something like this:

“To make sure we check everything”

“To work out what to test”

“To help us learn other areas of the system we haven’t tested before”

“To have confidence we have tested everything and not forgotten anything”

There’s a common theme here, the realisation that test cases are just a form of documentation. Documentation of what you are testing and the kinds of tests you want to run. Not only that, but as testers we use test cases to assist us in learning the system and designing our tests. In other words, we write out test cases in order to figure out what tests we want to run.

So if we’re not even designing our tests before we write them, then how can we hope to write them to a good standard? Are we even thinking about writing them to a standard? Can all tests fit any particular standard?

By having this documentation and putting a tick next to each test, testers also find confidence that they have thoroughly tested the system. So if test cases are a form of documentation….

Do we need documentation?

I think any tester would answer this with a yes. Without documentation, you are wholly reliant on memory and what people tell you. Documentation almost always exists somewhere - even if it’s not “formal” documentation (e.g. a written document, diagram or perhaps a wiki), it might be just an email, a set of requirements or your notes observing the behaviour of the system. Technically, the code of a program is the ultimate form of documentation - it’s just it might not be very easy to read! Documentation is a way of articulating information in a more easily understandable way, and as testers we want to understand as much as possible about the system we are testing. So having easily understandable documentation is very valuable to us.

So, we’ve established we do need documentation, and we’ve established that test cases are only one form of documentation. Maybe then the question to ask is…

Are test cases always the right kind of documentation?

Documentation is a way of articulating information, so the way we produce documentation influences how well people understand it. There is also a cost to documentation, it can never be done in zero time and it takes skill to create documentation that is well written and understandable to a given audience.

Just as there are good authors and bad authors, documentation is a creative art.

My experience at the large corporation and my experience of the wider testing industry is that there appears to be a general understanding that test cases can be written by anybody and therefore read and executed by anybody. Some note that there is a skill to this, but the focus is on the production of test cases, rather than the quality of said test cases. In other words, the idea that you could produce good testing without test cases seemed like a massive risk.

However, this desire for test cases and fear of a lack of them drives people to create test cases where it may not be useful. Most test cases I’ve seen follow the following format:

Summary

Preconditions

Steps

Expected Result

Now, given a very simple piece of software that takes in value X, determines if X > 5 and then does Y or Z depending on this, we might write some test cases for this that look like this:

This is a very simplistic example, but take note of how much I’ve had to write here to fit this format and how long it takes you to read it. Maybe it takes me a few minutes to write these sections, fix any spelling mistakes and review it to make sure it makes sense.

Now, consider if I documented the same example as a diagram:

How long did it take me to draw this? About 30 seconds in a decent flowchart program (such as yEd). I had fewer errors to correct and this particular example mapped very easily into a flowchart. How long did it take you to understand this diagram? How long did it take you to read the test cases? Which was most effective at helping you understand the behaviour? How long would it take you to think of some test ideas?

Hang on, where can I document my tests?

So we’ve not satisfied the desire for some documentation of our tests here. This diagram only satisfies the need for an explanation of the system. Test cases outline specific tests we may want to run. So, maybe we still need test cases and we just need a few diagrams to help explain more complex areas? Is there any other solution?

Enter...mind maps

So I did a little digging and looking around, I felt that surely there are better ways to design tests. Test cases can feel very slow to write, slow to read and hard to keep written in a particular format. This is when I came across mind maps described in this blog by Darren McMillan.  A mind map is not too different from a brainstorm - simply start with a topic and begin branching ideas from it. Taking my example above, I might end up with this mind map (created in a great tool called XMind found here):

Here I’ve explored some test ideas and started to design my tests. Exploring these ideas may raise questions that I don’t know the answer to right now - is storage a concern? If it is, how does it get stored, maybe security is a concern if the underlying system is using a MySQL database, maybe I need to test for SQL injection? Maybe performance needs to be considered? I can also show this to other people and they can quickly understand the scope of my testing without having to read individual test cases - and I can quickly observe the scope of my own testing and keep adding to it. It’s much easier to consider the overall picture of my test plan using this diagram.

But isn’t it difficult to draw diagrams and fit things in?

Yes, I don’t think mind maps are a replacement for test cases. Instead, I think this is a tool that can be used in conjunction with test cases to help you design and document tests in a more readable format, quicker. However, there are still cases where it can be difficult or time-consuming to create a mind map or diagram. I envisage that you may start with a flowchart and formal documentation of a system first, to understand the system you are testing. You would then create a mind map to explore what you want to test around this system. Finally, you would perhaps write test cases to formally document your tests, especially where they have complex steps.

So are there any quick, easy solutions to design and document all tests?

No, I don’t think so. I think by the nature of having to design, write or draw tests, they can never be created in zero time. Some systems or tests will be complex and you cannot run away from the complexity - drawing a diagram or writing test cases may be a way of reducing or making the complexity easier to understand, but sometimes it’s not possible to simplify it.

In the process of thinking about and writing this blog post, I think I’ve come to the conclusion that the problem here is not that we don’t need test cases. The problem is that we are not always using the right tool for the job and sometimes as testers we aren’t thinking carefully about the format we want to write or convey our tests and documentation. Using flowcharts and mind maps allow us more tools for this purpose, and they definitely are not the only forms of diagram or documentation we can use!

But what if I want to collate my tests and re-visit them for regression?

I think this is the crux of why test cases become relied upon so much. Why is it useful to re-visit test cases? Is it because you don’t want to miss an important test that you might have forgotten? I would argue that if we are repeating an important test regularly, we’re unlikely to forget it and simply because it is in a test case format doesn’t mean its importance is always highlighted - which means you’re either regularly running a lot of test cases “just in case” or potentially missing these tests anyway.

Wouldn’t it be better to document a system in a way that conveys information - highlighting important areas, rather than trying to fit all such information in a test case format?

That nagging feeling…

I still feel dissatisfied with my conclusions, I still have concerns that test cases and diagrams still require a lot of skill to write or draw in a way that is easily understandable by others. Every one of us will write or draw things differently and this makes it difficult to be consistent. The need to set standards, train people and conduct reviews still feels like it’s necessary.

However, I definitely do feel that in my career so far as a tester and in my conversations with other testers, there has definitely been too large a reliance on writing test cases over other forms and styles of documentation. I do believe that as testers we can definitely save some time and improve the quality of our testing by considering other techniques and not relying on test cases.

But I still feel there could be a better way!

Summary

• Test cases are not always the most appropriate format to document tests.
• Diagrams like flowcharts can be used to document and explain systems better than a test case. Rather than using test cases to learn a system, we could use diagrams or more formal documentation of the system - if this documentation doesn’t exist already, it’s worth creating it!
• Mind maps can be used to assist in designing and documenting tests, rather than designing the tests as you write test cases. Mind maps provide better visibility of your overall test plan.
• Writing test cases or drawing diagrams still requires skill - diagrams are not necessarily better than test cases all of the time and not necessarily easier to create to a consistent standard.