The temptation to split dev and test work in sprints - don’t do it!

Introduction

About 3 and a half years ago, I was new to sprints and scrum. Coming from the videogames industry, I was used to a process where I would test code that came from developers and return bug reports. I had heard the words “sprint” and “scrum” before but I had no idea how testing fit into them, so I joined a company where I could figure that out. This is what I figured out.

What’s a sprint?

If you’re not familiar with scrum or agile, then a sprint is effectively a short-term project plan where a team of people decide the work that they can complete within a 1, 2 or 3 week window. Work is “committed” (promised) to be completed in that time frame and the team tracks their progress. After each sprint, reviews and retrospectives are held to help the team find what works well and what helps them complete more work to a higher standard while still maintaining their commitment. The main focus of sprint work is to complete the work and trying to avoid leaving work unfinished.

Where does testing fit?

So normally teams set up a task board with columns titled something similar to “To Do, In Progress, Done”. Sometimes people add more columns or use different names but the usage is similar. Anyone from the same background as me would be tempted to then suggest that an additional column could be added between “In Progress” and “Done”. The logic being that “when you’ve finished your development work, I’ll test it”. In my head, this was trying to work with what I knew already in this new environment. We ended up with columns similar to “To Do, Build/Dev, Testing, Done”.

Bad idea

So at first, I thought things were working ok, I feel one of my strengths is learning and picking up things fast so I got stuck in and kept up with the 5 developers in my team. Most of the time I was fortunate that the work dropped sequentially or wasn’t particularly time consuming to test. This didn’t last long though and eventually we started to fail to complete work on time. This happened either because I was testing it all at the end of a sprint or because the work was highly dependant upon each other and the problems with integration weren’t found until very late.

This meant we had to continue some work in future sprints. Now I no longer had plenty of time to write my test plans at the start, but I was busy testing last sprint’s work and then testing this sprint’s work! I no longer had time to spend learning more automation or exploring newer areas to me like performance testing. All of my time was consumed trying to test all of this work and I couldn’t do it. What went wrong?

A change in approach

I would love to say I quickly realised the problem and fixed it but it took me a long time to realise the issue. I think part of this I will put down to not knowing any better and partly working with developers who didn’t know any better. Either way, a while later I realised that the problem was that I was trying to test everything and the developers started to rely on me for that. I’ve since realised that there is a fair bit of psychology involved in software development and this was one of my biggest lessons.

We eventually decided to stop splitting up work between roles, mainly because we found that developers tended to treat work that was in “test” as “done” to them, freeing themselves up to work on even more development work. This created a bottleneck, as the only tester as I was testing work from yesterday while they were busy with today. Instead, I came to the realisation that there is little benefit to splitting the work up in this way, at least not through process. We should be working together to complete the work, not trying to focus on our own personal queue. I shifted from testing after development was thought complete, to trying to test earlier, even trying to “test” code as developers were writing it, pairing with them to analyse the solution.

Understanding what a “role” means

I think for me this lesson has been more about realising that playing the role of “tester” does not necessarily mean I carry out all of the “testing” in a team. It does mean I am responsible for guiding, improving and facilitating good testing, but I do not necessarily have to complete it all personally. An additional part of this lesson is that I cannot rely on other people to define my role for me - as a relative newbie to testing I relied on the developers to help me figure out where I should be. While I’ve learnt from it, I also know that I may need to explain this learning again in future because it is not immediately obvious.

So where does testing really fit?

Everywhere, in parallel and in collaboration to development. Testing is a supportive function of the team’s work, it now doesn’t make sense to me to define it as another column of things to do. It has no set time frame where it’s best to perform, and it doesn’t always have a great deal of repetition in execution. It is extremely contextual.

In addition, that’s not to say you shouldn’t test alone or separately to ongoing teamwork. You absolutely must test alone as well, to allow you to focus and to process information. It’s just that you must choose to do this - where it is appropriate.

Definition of “Done”

One of my recent approaches was to define the definition of “Done” as:

“Code deployed live, with appropriate monitoring or logging and feedback has been gathered from the end user”

Others may have different definitions, but I liked to focus the team on getting our work in a position where we could learn from it and take actions in the following sprint. For me, it meant we could actually pivot based on end user feedback or our monitoring and measure our success. Instead of finishing a sprint with no idea whether our work was useful or not, planning a new sprint not knowing whether we would need to change it.

Summary

• Avoid using columns like “Dev” and “Test” in sprint boards. It seems to lead to a separation of work where work is considered “Done” before it is tested.
• Instead, try to test in parallel as much as possible (but not all of the time), try to test earlier and lower down a technology stack (such as testing API endpoints before a GUI is completed that uses them).
• Encourage developers to test still and instead try to carefully pick when and where to personally carry out the bulk of the testing. Try to coach the team on becoming better at testing, share your skills and knowledge and let them help you.
• Altering the definition of “Done” seemed to help for me, it was useful to focus the team on an objective that meant we really didn’t have to keep returning to work we had considered completed. In other words, make sure “done” means “done”.

How much testing is enough?

Introduction

Risk analysis is absolutely key to being an effective tester. I have rarely found it effective or possible to “test everything”, and so there is always an element of deciding what and how much I want to test before I have confidence in the quality of a piece of work. Even in cases where I do need vast test coverage, I still need to prioritise what I will test first. The reason I do this is because I want to provide reports on the most important defects as soon as possible, as this is where I deliver the most value.

What to test and how much to test?

So how do I answer these questions? With more questions of course! There are a variety of factors that determine how I will answer, but some general questions that help me decide are:

• What is it? What does it do?
• How complex is it?
• How much do I understand about it?
• Does it have good documentation I can refer to?
• Does it have clear requirements?
• How much time do I have to test? Is there a deadline?
• What resources do I have available to me?
• What tools do I have available to me?
• How critical to the business is it?
• Does it interact with or affect other critical systems?
• Who uses it? How do they use it?
• Is it a modification to an existing system or a brand new system?
• If it's a modification, does the system have a history of instability?
• What is the modification and what does it affect?
• Are there any pre-existing defects I need to know about?
• Are there any performance or security concerns?
• What are the most important parts of the system? Do I have an order of priority?

There are many, many more questions I could ask. Some of these questions I might already know the answers to, but they still influence my decisions on what and how much I test. Its important to realise that some of these questions have impacts on one another, it is only with the full picture that I can effectively identify risks. For example, with limited time and resources, it directly impacts how much I will test and therefore I would prioritise testing the critical areas of the system that are new or have changed.

Asking these questions also allows other members of the team to consider them and helps them gain an insight into my work and what I’m looking for. Over time this can provoke them into providing better information and lead to more collaboration.

But surely you always prioritise critical areas of the system?

Not necessarily, there may be many critical areas and it may not be possible for me to test all of those areas given the time and resources I have available. We may in fact consider some critical areas to be very stable and know that they have not changed. I may decide to accept the risk of not testing these areas in order to focus on areas that I know are more unstable or have been affected by change.

No testing? That’s crazy!

I’m not saying no testing, I’m just suggesting that no testing can be an option - but one of many. Ideally if I had any critical areas that I felt I couldn’t comprehensively test in the time frame I would still look to perform some testing. This can range from very basic smoke tests, time-boxed exploratory tests or some further prioritised order of tests. If this particular area is something that is regularly covered during regression testing, then it may have automation scripts I could run. I may choose to only run automation scripts for that area.

However, ultimately, you will be making a decision to not test something, somewhere. Therefore you must be comfortable with being able to draw this line based on your informed understanding of the risks.

What if there is no deadline?

Then I would ask the business how long is too long. The business will want the work by some ideal time, otherwise they would not ask for the work to be carried out. They will not wait indefinitely and there is always value in delivering work quickly.

Usually a business may give you no deadline simply because they do not understand enough about testing but want you to do a good job. They don’t want to give you an arbitrary deadline because they don’t know themselves how much testing is enough. It is important to start a dialogue at this point to really explore what the business wants and collaboratively come to a decision on how much testing you really want to do.

Summary

• In order to decide what to test, you need to gather information regarding time, resources, priorities, etc.
• Not testing specific areas is a valid option.
• Comprehensive testing is never an option in an agile environment.
• There is always a desired deadline even if it is not explicitly stated.

The role of a tester in backlog grooming and planning

Backlog grooming

As an on-going activity, the product owner and the scrum team should be actively reviewing their backlog and ensuring the work is appropriately prioritised and contains clear information. This allows each piece of work to be more easily planned into a sprint as it can be more accurately estimated.

As a tester I actively try to be involved in this process as it is my first opportunity to assess the requirements and the information provided. It also allows me a chance to gather information required for testing, which allows me to provide more reliable estimates.

The objective is to be in a position where for any piece of work presented in planning you know exactly what the work requires. You should then have a good idea of what you will test and therefore provide reliable estimates. If this is not the case, then the work cannot be effectively planned into the sprint.

Planning and estimation

At the start of each sprint, you have a sprint planning meeting. In this meeting the team collectively decide what work they can commit to being done by the end of the sprint. This may or may not have an estimation process. This is your last opportunity as a tester to ensure that you have all of the information you need before work is started. If the appropriate backlog grooming has been done, then this should be a relatively straight forward process, however inevitably there will be pieces of work that need clarification or may have missed something.

Some example typical thoughts I have in a planning meeting for a piece of work are as follows:

• Do I understand the requirements?
• Does everyone else understand the requirements? (and does their understanding match mine?)
• What requirements have not been written down?
• Are there any external factors such as legal requirements or third parties?
• How can I test this work? Is it even testable?
• Do I need any additional tools to test this work?
• Do I have any dependencies in order to test this work, such as needing live data or having to work directly with a customer or third party?
• Does this work conflict with anything else in this sprint?
• Does this work conflict with work being done by other teams?
• Are there any repetitive checks that I could automate for this work?
• Do I need to consider other forms of testing such as security or performance testing?
• Is the balance of development workload to testing workload viable?
  
  

Hopefully I should have asked most of these questions during backlog grooming, and I wouldn’t always ask all of these questions, it very much depends on the context of the work. But hopefully this demonstrates that you can easily think of a lot of questions and it is important to ask these before and during planning.

Once you have the answers to all of these questions, you should have a good idea of what you are going to test. From this you can provide a more reliable estimate of how much testing you would like to do. Not only should you have a good idea of how long it would take, but also you should be better equipped to analyse risk.

Summary

• Planning a sprint is easier with clearly defined work and when the team has prepared for the planning meeting.
• To achieve this, backlog grooming should be used to ensure tickets are prioritised appropriately and contain enough information.
• Backlog grooming should also be used to prepare for planning such as considering if you need testing tools or environments.

Testers in development teams

Introduction

For this blog post I’m going to talk about my experiences as a tester in a development team working to “scrum” which is an agile development methodology. I write this as someone who has worked to more traditional waterfall style processes prior to working in scrum and as someone who has worked for several years within scrum. I’ve also worked to a “kanban” model too but I won’t cover the pros and cons of the other methodologies here. The goal is to hopefully help someone in a similar situation to me have a clearer idea of how they can fit as a tester into scrum.

I’m going to assume you know the basics of how scrum works but if you are new or unsure about it, please have a read of the wikipedia article first:

https://en.wikipedia.org/wiki/Scrum_(software_development)

I would also add that my thoughts on this topic are from the view of an agile project, where speed is absolutely key. Most of these thoughts would possibly not apply to a project where speed is not desired, though I feel the role of a tester and the skills they use are the same no matter the situation.

Should a development team include a tester?

I’m going to start by tackling this first. You may notice in the link above that there is no distinction between team members in a scrum development team. A common view that I have come across before is that testing should be a process external to development. There are some that feel that testers shouldn’t become too close technically with the system they are testing because they may make some conscious or subconscious assumptions, because of their heightened insight. There is a fear that testers would lose their ability to think like an end user and would think too much like a developer.

For me, this is nonsense, I consider it my job to be aware of end users and to think beyond what I know or see. I have found it to be faster and more efficient in general to work as closely as I can with developers and I have built a much better rapport with them. This has meant that the levels of trust and understanding are much higher and developers see the value in testing. As soon as testers and testing are perceived to be a separate process, I have found the two communicate less and become more frustrated with each other.

In scrum this is crucial as you don’t have the time to waste with the traditional methods. This means that a tester cannot sit outside of development and only test pieces of work when they are complete. This would lead to development work being completed in a sprint, defects being found after a sprint and then the development work having to be revisited in a new sprint. This is obviously not an efficient way of working.

So for me, a tester must sit within a sprint team and be considered part of the development team. While their skills and role is different to a developer, they are an important contributor to considering a piece of work “done”. It is also important to realise that developers are capable of contributing to the test effort. While automation tests do not replace manual exploratory testing, they are very useful for taking the heavy load of more focused tests. Developers can be an alternative resource for constructing automation scripts, expanding upon their unit and integration tests with more advanced Selenium UI tests for example.  Testers should view themselves as being responsible for improving the quality of the work and not as an executor of tests. If tests can be easily automated by developers, then it doesn’t always have to be the tester who creates and executes the automation.

Developers can also provide assistance and help with setting up more intricate or complex tests, reproducing problems and also providing more technical explanations about how the system functions. This can be an invaluable source of information if you have little documentation or wider business knowledge about the system (though it should not be relied upon on its own).

At the same time, a tester can do more than simply test work as it is completed. In some cases, it is possible and very useful to test pieces of work while they are still being developed. Working in close collaboration with the developer, a tester can provide quick and effective feedback on the work and really focus the developer on producing quality work. The tester can also continue to perform preparation activities such as gathering any additional information, ensuring they have the appropriate tools and environments or planning their testing.

I personally feel a tester doesn’t just bring an ability to exploratory test, but also to gather knowledge, critically challenge designs and re-focus a team on what is important to deliver. These skills are best put to practice before the sprint even begins through to finishing the work, rather than after the work has been completed - where it is too late to really reap the benefits of these skills.

Summary

• In Scrum, testers are most effective when they are part of a team of developers.
• A collaborating team is far more productive than separated focused individuals in short time periods.
• Developers can contribute to developing and supporting automation testing.
• Testers should actively ensure they are not becoming too close to a system and are aware of external areas to the team - such as end users, other systems, business concerns, legal requirements, etc.