Using character recognition for browser automation

Introduction

I was venting recently to my colleague Chris Johnson about the frustration of working with yet another horrendous website that didn’t have any usable locators and he put to me the idea of “what if we had visual driven automation, where the automation treats the website as a complete black box, like a human user?”. It got me thinking, surely this could be possible already in some respect. As I am already very familiar with AWS (Amazon Web Services), I had a look if they had any services that could be useful for this and found their brand new Textract service, which lets you send image documents (.jpg, .png or .pdf) and analyses them for text. This then gave me the idea to try and use this service with Selenium to find elements on a page and click them without any need to interact or understand the HTML. This post is a report on my findings.

What is Textract and how does it work?

Textract is a type of OCR (Optical Character Recognition) service that detects text and data in image documents. It works by supplying it an image file and it responds with the results of its analysis, a list of words, sentences and objects (like forms and tables) that it has identified. Each word or sentence has a set of data including the location in terms a rectangular box and a percentage confidence of the accuracy of its findings. Behind the scenes it manages this via a pre-trained machine learning algorithm.

 
As with other AWS services, you can access Textract via the AWS CLI so for my research I used the boto3 library for Python3 which allows you to interact with the AWS CLI within Python very easily.

When you are using Textract, you receive JSON responses that look like this:

{
        "Blocks": [
            {
                "Geometry": {
                    "BoundingBox": {"Width": 1.0,"Top": 0.0,"Left": 0.0,"Height": 1.0},
                    "Polygon": [
                        {"Y": 0.0, "X": 0.0},
                        {"Y": 0.0, "X": 1.0},
                        {"Y": 1.0, "X": 1.0},
                        {"Y": 1.0,"X": 0.0}
                    ]
                },
            "Text": "Store",
            "Confidence": "99.1231233333",
            "BlockType": "WORD"
        }
        ]
}

My test

So the idea for my test was to create a simple script that would:

1.     Use Selenium to load a website.
2.     Take a screenshot.
3.     Send the screenshot to Textract.
4.     Find the location of some text on the website that would take us to a new page.
5.     Click the location on the website.
6.     Check that we had gone to the correct new page.

Pre-requisites

In order to get this test working, this is what I needed:

• A computer with Python3 installed and the relevant libraries (boto3, Selenium).
• Chromedriver and Chrome installed.
• An AWS account with access to Textract. At the time of writing Textract is in preview and you need to ask Amazon nicely for access. It took about a week for me to get access.
• An AWS IAM (Identity and Access Management) user on my AWS account that has the permissions to interact with Textract.
• AWS CLI installed on my machine and configured with the IAM user and an AWS region that Textract is available in (its currently only available in a few regions).
  

I chose to use Python because I’m very familiar with it, and it allows me to write a simple script like this very rapidly with little setup needed. You can use any programming language, you just need to be able to interact with the AWS CLI.

My findings

Did it work? Yes, effectively. I was able to create a script that did all of the above and successfully navigate between pages...but with a very big caveat - only if the text appeared in the top left corner of the page in any resolution. Why? When you request a screenshot from web browsers, they do not take a screenshot of what you can see, but instead they take a full page screenshot. This means you cannot easily map the pixel locations from the screenshots to the browser window, particularly when the real browser window is a smaller resolution and the website dynamically changes visible location of elements. Even at larger resolutions there is a small difference in resolution sizes because you almost never display the full page in a window.

This means that my idea has limited usefulness until I could find a better way of generating screenshots that are just the visible area rendered by the browser.

However, in principle my idea did work if you can provide an image that maps 1:1 with the browser window, but there are still other issues. I tried testing against different websites just to see what other issues would crop up:

• Naturally you need relatively clear text in order for Textract to confidently find it. However, this could be argued that unclear text would be an accessibility issue anyway and in the below screenshot you can see Textract does a reasonable job of even reading unusual fonts or text that isn’t perfectly straight.

• Obviously if you want to find elements that are not text, Textract isn’t useful at all. Potentially other pattern recognition services could be useful though if you could provide it an image pattern to find rather than just a text string.
• It could be tricky to figure out which is the right element if there is more than one example of some text on the page.
• While Textract is quite fast (roughly 2 seconds or so for it to return a response), some dynamic elements of the page could change in that time. Some website have scrolling elements or pop-ups that only appear after a short time. This means you might not be able to totally run a black box test as you need to check for these elements before proceeding.
  

Due to these issues, I don’t think you could totally remove the need for HTML locators yet. But there is certainly some potential here to expand the abilities of automation tests and maybe make parts of them more powerful. Even if you don’t use this tech to replace locators for Selenium, there are other applications that might not have been easy before - such as testing PDF files generated by the systems you test.

Other considerations

If you were to consider using this tech in your automation, there are some other factors to consider:

• Costs, AWS charge for the use of Textract. Depending on your context and how you use it, this may be a limiting factor. The pricing can be found here: https://aws.amazon.com/textract/pricing/
• Data privacy, AWS state they may store and use the documents you upload to maintain and improve the machine learning algorithm behind Textract. You may work in a workplace where the screenshots contain sensitive information that AWS employees are not authorised to look at. You can request AWS delete documents you upload, but this is a manual process of contacting their support team.
• While I did find Textract responded quite quickly (around 2 seconds), I don’t know its capabilities under significant load nor did I hit the API request limits. I was only running very low numbers of requests.
• Textract is currently only available in certain AWS regions, so if you require data be held in particular geographic locations then this is may be a limiting factor.
  

Of course, other services similar to AWS Textract may be available which overcome these factors.

Summary

This was a fun little exploration into technology I’ve not used before and it was nice to find it so easy to use. I think there is some potential in using this service for automation tooling, particularly if you need to analyse image documents.

If you’re looking to totally remove the need for HTML locators, I think there is more work required though, if you know of a good solution to the browser screenshot resolution problem let me know!

My code

Here’s my code if you’d like to try it yourself:

https://github.com/Ardius/python-selenium-textract/blob/master/textract_test.py

Which languages are best to learn for testing?

Introduction

I’ve seen this question raised quite a lot in testing circles regarding which programming language is best to learn. Like it or not, the current trend in the industry seems to be asking much more of testers, with a view to creating more automation and having a much greater understanding of the technology they are testing.

Why learn a programming language?

What is your motivation for wanting to learn a programming language? In order to test well, you don’t need to know any programming language. There are very particular situations or contexts where programming may be useful to me as a tester, such as wanting to write some automated checks, learn more about what the product I’m testing is actually doing under the surface or simply wanting to save time by creating tools to help myself. However, these situations don’t arise all of the time.

It’s also worth highlighting that to write programs, I need to understand a lot about the domain I’m working with. If I want to write an automated check, I need to test the product first to understand what is worth checking. If I want to read some code, I need to understand the context that code is used for, what its purpose is.

So even if I did have something that is worth programming, I still need to “test” to identify it, understand it and consider whether it is worth it. Simply learning to program is not enough, which is why as a tester you can bring a lot to the design of automated checks and why developers cannot easily test the product themselves.

Automated checks

So it seems the usual reason testers are looking to learn a programming language is to create automated regression suites for speed and reliability. Typically I find the advice tends to be that you should learn and use the same language as your back-end developers (so if they use Java to build the product you test and Java to write unit tests, then you should learn Java too). The argument being that by using the same language, you can access their support and help more easily when you need it. However, this depends upon your current relationship with your developers and their location. Perhaps you may not be very close to your developers and won’t benefit from their support - this may not be something you can easily change.

You are going to have judge for yourself which language, but the biggest factors that affect my choice would be:

• How comfortable am I writing code in this language?
• What support can I get from the developers I work with?
• What support can I get from other testers?
• How well supported is the language in terms of libraries or capabilities? (for example, if you want to write Selenium checks, is there documentation on how to use Selenium in that language?).
• Can I write programs in this language in a way that makes them easily understood by other people?

There is no easy answer to these questions so I wouldn’t recommend any particular language. However, to help narrow your research, I would suggest focusing on these languages to consider:

• Java
• C#
• Python
• Ruby
• Javascript

At the time of writing, these are some of the more popular languages to learn with regards to automated checks.

Toolsmithing

Maybe you’re interested in simply being able to make use of a programming language to create your own tools. A “tool” in this context can be something small like a script that rapidly repeats a certain action over and over. For example, I once created a script that compared two sets of 100 invoices with each other. It looked at each invoice total and compared the old total with a new one and saved the difference in a text file. This meant I could rapidly compare and identify obvious errors, saving my own time and helping me perform this particular action more accurately. However, it didn’t replace the testing I performed, it simply augmented it, allowing me to focus on more interesting tests.

I created tools like this in a programming language called Python. I personally love using this language because it’s very easy to read, has a lot of support in terms of libraries and documentation and can allow you to experiment with ideas very rapidly. I very much recommend Python as a starting point for building simple tools and it can be used to write automated checks if you so wish.

There’s a great tutorial on getting started with Python here.

Alternatives to programming

Do you want to become a more technically capable tester? Not really keen on learning a programming language but feel like you need to learn? Well perhaps you can find value in learning other technologies and concepts. While programming is a very powerful tool, it’s not the only one that a tester can learn in order to become more technically capable.

• Test lower - maybe you could understand more about the technologies powering the product you’re testing and test lower and earlier? For example, many web services are built upon APIs (Application Programming Interfaces), perhaps you could learn how to test these? A place to start interacting with APIs is trying out Postman.
• A similar approach to testing lower is learning about databases and how to operate them using languages such as MySQL or Postgres.
• Research tools that can help you test or provide more information. For example, Google Chrome DevTools have lots of very useful tools for interacting with websites and debugging problems or emulating mobile devices.
• Talk to developers! Ask them to draw diagrams explaining how the product works technically and ask them explain things you don’t understand. It can be difficult at first knowing what is important to remember and what can be forgotten but simply taking an interest in their work and asking questions can even help them understand their own work better. I find there is no better test of my own understanding than having to explain myself!

Summary

• You don’t need to learn a programming language to be a great tester.
• There is no one particular language that is “the best”, but there are some popular ones that are good to get started with.
• There are other ways to become a more technical tester that don’t involve learning programming.

Automation in testing - writing your own tools

Introduction

A consistently hot topic in testing is “automation” and how it can be used to improve testing. In this blog post I’m going to talk about how you can really get the most out of “automation” very quickly.

Automation? So you’re going to replace me with a machine?

Nope, I’m going to suggest you augment your abilities as a tester like a cyborg! Part human, part machine. Frequently people assume automation is referring to having a machine perform all of the tests you might perform manually. But actually automation can be used to help you manually test faster. Are you spending a lot of time tearing down databases and re-creating data? Are you spending a lot of time repetitively reading and comparing files? Automation can help you focus on the fun part of testing!

So where do I start?

Before I get onto some of the ideas for what you can automate, you need tools in order to create the automation! First of all, do you have any programming knowledge? If you don’t, don’t fret! (if you do, please ignore me as I’m writing this assuming you don’t). There are things called “scripting languages” - these are a form of programming language that is usually interpreted rather than compiled. What does that mean? Well it means they are much easier to work with and some of them even have more natural english language-like syntax. These scripting languages are a great way to quickly and easily put together programs that automate tasks for you and hopefully you will find them much more accessible than compiled programming languages such as Java or C.

So which language is best?

The one you are most comfortable with really, if you already know a language, then there is nothing wrong sticking with that language. If you don’t know any languages, then some I would recommend are Python and Ruby. I personally prefer Python and I will write the rest of this blog on Python but Ruby is equally good place to start.

Check them both out and decide for yourself which you like the most:

• Learning Python the hard way
• Learning Ruby the hard way

The reason I recommend Python and Ruby is because they are very commonly used and can be run on any machine be it Windows, Mac or Linux. There are lots of examples of free code or libraries (collections of code) available online for you to use to tackle almost any problem too. I personally prefer Python simply because it is more human-readable than Ruby and many other languages.

Ok, so I’ve done a few tutorials but what do I start automating?

Think about tests you’re running and where you spend the most time, can any of it be handled by a script? Is there something about the setup of the test that can be automated?

I don’t know?

That’s ok, it is hard at first to really know what you can and can’t do until you’ve attempted something or seen it done before. Naturally I would recommend trying some ideas as this is always a good way to learn. But to give you some ideas of what is possible, I’ll cover some of the tools that I’ve created myself.

Data Setup

My first port of call whenever I’m considering automation-assisted testing is to look at the data setup for the test. Sometimes there can be a lot of tests that require a lot of setup before hand but you don’t want to vary that setup too much, you simply want to have an environment where you can focus on a particular area. Perhaps you’re attempting to recreate a bug 20 pages into a website or an application. Or perhaps you’re testing an invoicing system and you simply want some base data to repeatedly test against. Sometimes these kinds of tests involve a lot setting up and then restarting to try something else - you want a “clean” set of data that isn’t spoiled too much by testing you’ve done before.

Automation can help a lot in quickly setting the exact same “clean” data up again. You may ask “but why not just have a saved state of an environment?” - sure, that is a valid alternative strategy to deal with this problem. However, it’s a different kind of maintenance - with that you are maintaining a database that you need to keep updating. With a script you may also need to keep updating with changes to the system you are testing. Personally I prefer maintaining scripts, I feel you can more easily create scripts that are less affected by change than keeping a database in a clean state.

Some examples of tools I’ve created to help speed my testing up by creating data are:

• A script that utilises the Selenium Webdriver library to open a browser window and create data in a web application for me. I wouldn’t normally recommend using Selenium for data setup because it is designed for checking UI elements and is therefore quite slow. But in this particular case this was a legacy system with no alternative than a UI for creating the data. I felt it was worth mentioning because Selenium is a useful library if you need to script around a UI. This script became a time-saver simply because it could be left running while I worked on other things.
• A script that controlled two SIP telephones and had one call the other using the sipcmd program and later, the PJSIP library. This was used to quickly and easily create call traffic, especially in large amounts (e.g. making 100 calls). During some of my testing I’ve had instances where I’ve had to simulate telephone calls and it was useful to automate it to create volume. It also had the benefit of being able to then log and provide details about the phone call that would have been more difficult to see manually.
• A script that uses the requests library to interact with a REST API. This allowed very rapid data setup within seconds and the requests library is extremely straightforward to use. I fully recommend this approach to data setup because of its speed.

Speeding up tests

The second area I focus on is to look at where I spend the most time during a test. Am I manually performing something that is just checking? Can a machine do this checking instead? I once had to test a change to a billing system that produced invoices for customers. I wanted to run the invoices on the old system and the new system and compare them to see if I could quickly identify any obvious problems between the two. Manually, this was a lot of “checking” that didn’t really require much intellectual thought - I would be literally comparing one value to another. This is a candidate for automation, so I can focus on more intellectual tests such as focusing on edge cases or perhaps scenarios where we don’t currently have a customer (so I can’t test through this method).

Some examples of tools I’ve created for speeding up tests are:

• A script that compares groups of CSV (comma-separated values) files. I used this to very quickly compare the results of running invoices in one version of a billing system with another. It very simply compared the totals of the invoices - so it wasn’t “testing” very much, but it allowed me to very quickly identify any easy or obvious failures in the billing if the totals didn’t match. This is a good example of a script that augmented my manual testing and allowed me to provide information much faster.
• A script that uses the requests library to quickly check every customer-facing API endpoint returns a 200 OK response. This was useful for very quickly performing a very basic check that the API appeared to be functioning - which would quickly catch any easy bugs.

When not to automate

So this all sounds great right? Remove all the boring, time-consuming testing and leave a machine to do it? Hold your horses! It’s not quite as straight-forward as that. There are times when automation really isn’t a good idea. Here are some common pitfalls for automation:

• The desire for invention. Beware of becoming more interested in engineering automation than providing valuable testing. Make sure that any automation you are writing is going to deliver some value to your testing. Don’t simply automate parts of your testing simply because you can.
• Avoid creating a mirror image of the system you’re testing. This is very easy to do, say you are testing a calculator - it’s easy to end up writing a script that enters 2+2 and then calculates the answer itself. So both the calculator and the script calculate the answer is 4. Why is this bad? Because in a more complex example where a failure occurs, how do you know which is wrong? Have you written a script that calculates the answer wrong or is the calculator failing? Instead you should be writing a script that already knows the answer. You shouldn’t be calculating the answer during the test.
• Once-only tests with little to no repetitive nature to them. When I say repetitive nature, I mean there isn’t anything in this one-off test that requires you to repeat something like data setup or checks - these are obviously good to automate potentially. But otherwise one-off tests are nearly always quicker to perform manually and the cost of creating your automation script won’t be recovered in value because it may never be used again.

One more thing…

This post has been all about creating your own tools via scripting, but sometimes a tool may already exist! It's always worth searching around the Internet for existing tools that may help your testing, but learning scripting is useful in situations where you don’t have time to search and try different options and just want to quickly script something small.

Existing tools can take many forms such as Python libraries, professional software products or perhaps plugins or extensions to another piece of software your company already uses.

Summary

• Automation is useful to augment your testing, improving your speed and efficiency.
• Scripting languages are a good place to start learning how to create your own automation tools.
• It’s hard to know what you can script at first, so never stop asking the question “can I script this?”.
• However, beware of creating automation simply because you can, make sure it’s valuable.
• Even if you can’t create your own tool, have a look around and see if anyone else has.