Introduction
A consistently hot topic in testing is “automation” and how it can be used to improve testing. In this blog post I’m going to talk about how you can really get the most out of “automation” very quickly.
Automation? So you’re going to replace me with a machine?
Nope, I’m going to suggest you augment your abilities as a tester like a cyborg! Part human, part machine. Frequently people assume automation is referring to having a machine perform all of the tests you might perform manually. But actually automation can be used to help you manually test faster. Are you spending a lot of time tearing down databases and re-creating data? Are you spending a lot of time repetitively reading and comparing files? Automation can help you focus on the fun part of testing!
So where do I start?
Before I get onto some of the ideas for what you can automate, you need tools in order to create the automation! First of all, do you have any programming knowledge? If you don’t, don’t fret! (if you do, please ignore me as I’m writing this assuming you don’t). There are things called “scripting languages” - these are a form of programming language that is usually interpreted rather than compiled. What does that mean? Well it means they are much easier to work with and some of them even have more natural english language-like syntax. These scripting languages are a great way to quickly and easily put together programs that automate tasks for you and hopefully you will find them much more accessible than compiled programming languages such as Java or C.
So which language is best?
The one you are most comfortable with really, if you already know a language, then there is nothing wrong sticking with that language. If you don’t know any languages, then some I would recommend are Python and Ruby. I personally prefer Python and I will write the rest of this blog on Python but Ruby is equally good place to start.
Check them both out and decide for yourself which you like the most:
The reason I recommend Python and Ruby is because they are very commonly used and can be run on any machine be it Windows, Mac or Linux. There are lots of examples of free code or libraries (collections of code) available online for you to use to tackle almost any problem too. I personally prefer Python simply because it is more human-readable than Ruby and many other languages.
Ok, so I’ve done a few tutorials but what do I start automating?
Think about tests you’re running and where you spend the most time, can any of it be handled by a script? Is there something about the setup of the test that can be automated?
I don’t know?
That’s ok, it is hard at first to really know what you can and can’t do until you’ve attempted something or seen it done before. Naturally I would recommend trying some ideas as this is always a good way to learn. But to give you some ideas of what is possible, I’ll cover some of the tools that I’ve created myself.
Data Setup
My first port of call whenever I’m considering automation-assisted testing is to look at the data setup for the test. Sometimes there can be a lot of tests that require a lot of setup before hand but you don’t want to vary that setup too much, you simply want to have an environment where you can focus on a particular area. Perhaps you’re attempting to recreate a bug 20 pages into a website or an application. Or perhaps you’re testing an invoicing system and you simply want some base data to repeatedly test against. Sometimes these kinds of tests involve a lot setting up and then restarting to try something else - you want a “clean” set of data that isn’t spoiled too much by testing you’ve done before.
Automation can help a lot in quickly setting the exact same “clean” data up again. You may ask “but why not just have a saved state of an environment?” - sure, that is a valid alternative strategy to deal with this problem. However, it’s a different kind of maintenance - with that you are maintaining a database that you need to keep updating. With a script you may also need to keep updating with changes to the system you are testing. Personally I prefer maintaining scripts, I feel you can more easily create scripts that are less affected by change than keeping a database in a clean state.
Some examples of tools I’ve created to help speed my testing up by creating data are:
- A script that utilises the Selenium Webdriver library to open a browser window and create data in a web application for me. I wouldn’t normally recommend using Selenium for data setup because it is designed for checking UI elements and is therefore quite slow. But in this particular case this was a legacy system with no alternative than a UI for creating the data. I felt it was worth mentioning because Selenium is a useful library if you need to script around a UI. This script became a time-saver simply because it could be left running while I worked on other things.
- A script that controlled two SIP telephones and had one call the other using the sipcmd program and later, the PJSIP library. This was used to quickly and easily create call traffic, especially in large amounts (e.g. making 100 calls). During some of my testing I’ve had instances where I’ve had to simulate telephone calls and it was useful to automate it to create volume. It also had the benefit of being able to then log and provide details about the phone call that would have been more difficult to see manually.
- A script that uses the requests library to interact with a REST API. This allowed very rapid data setup within seconds and the requests library is extremely straightforward to use. I fully recommend this approach to data setup because of its speed.
Speeding up tests
The second area I focus on is to look at where I spend the most time during a test. Am I manually performing something that is just checking? Can a machine do this checking instead? I once had to test a change to a billing system that produced invoices for customers. I wanted to run the invoices on the old system and the new system and compare them to see if I could quickly identify any obvious problems between the two. Manually, this was a lot of “checking” that didn’t really require much intellectual thought - I would be literally comparing one value to another. This is a candidate for automation, so I can focus on more intellectual tests such as focusing on edge cases or perhaps scenarios where we don’t currently have a customer (so I can’t test through this method).
Some examples of tools I’ve created for speeding up tests are:
- A script that compares groups of CSV (comma-separated values) files. I used this to very quickly compare the results of running invoices in one version of a billing system with another. It very simply compared the totals of the invoices - so it wasn’t “testing” very much, but it allowed me to very quickly identify any easy or obvious failures in the billing if the totals didn’t match. This is a good example of a script that augmented my manual testing and allowed me to provide information much faster.
- A script that uses the requests library to quickly check every customer-facing API endpoint returns a 200 OK response. This was useful for very quickly performing a very basic check that the API appeared to be functioning - which would quickly catch any easy bugs.
When not to automate
So this all sounds great right? Remove all the boring, time-consuming testing and leave a machine to do it? Hold your horses! It’s not quite as straight-forward as that. There are times when automation really isn’t a good idea. Here are some common pitfalls for automation:
- The desire for invention. Beware of becoming more interested in engineering automation than providing valuable testing. Make sure that any automation you are writing is going to deliver some value to your testing. Don’t simply automate parts of your testing simply because you can.
- Avoid creating a mirror image of the system you’re testing. This is very easy to do, say you are testing a calculator - it’s easy to end up writing a script that enters 2+2 and then calculates the answer itself. So both the calculator and the script calculate the answer is 4. Why is this bad? Because in a more complex example where a failure occurs, how do you know which is wrong? Have you written a script that calculates the answer wrong or is the calculator failing? Instead you should be writing a script that already knows the answer. You shouldn’t be calculating the answer during the test.
- Once-only tests with little to no repetitive nature to them. When I say repetitive nature, I mean there isn’t anything in this one-off test that requires you to repeat something like data setup or checks - these are obviously good to automate potentially. But otherwise one-off tests are nearly always quicker to perform manually and the cost of creating your automation script won’t be recovered in value because it may never be used again.
One more thing…
This post has been all about creating your own tools via scripting, but sometimes a tool may already exist! It's always worth searching around the Internet for existing tools that may help your testing, but learning scripting is useful in situations where you don’t have time to search and try different options and just want to quickly script something small.
Existing tools can take many forms such as Python libraries, professional software products or perhaps plugins or extensions to another piece of software your company already uses.
Summary
- Automation is useful to augment your testing, improving your speed and efficiency.
- Scripting languages are a good place to start learning how to create your own automation tools.
- It’s hard to know what you can script at first, so never stop asking the question “can I script this?”.
- However, beware of creating automation simply because you can, make sure it’s valuable.
- Even if you can’t create your own tool, have a look around and see if anyone else has.
