From finding defects to preventing defects

In this article, Krishna Sowjanya Chandrappagari explains why Access Worldpay adopted the Shift-left process, how it went from finding defects to preventing them, and shares the experience of the journey.

Technical

Written by Krishna Sowjanya Chandrappagari
24 March 2021

Quality Assurance (QA) is traditionally considered a role in which the QA engineer performs the quality control process towards the end of the software development lifecycle. In recent times, most development teams have adopted agile ways of working, but many still see quality assurance as the sole responsibility of the QA engineer whose primary task is to find as many defects as possible post-development. Since finding and fixing defects this late in the process is more expensive and time-consuming, this practice causes delays which negatively impact predictability of delivery.

When building our Access Payments service, we wanted to ship features fast in a predictable and cost-efficient manner while maintaining high quality standards. These imperatives made it essential that we find and prevent defects early in the delivery process. Therefore, starting QA from the requirements analysis phase became a top priority.

Identifying the primary source of defects

Our decision to start QA from the requirements analysis phase was based on independent research, which has shown that most defects emerge from poor requirements. A study by James Martin in 1984 revealed that 56% of all software defects emerge during the requirements analysis phase. Almost a quarter century later, Keith Ellis wrote an analysis on a report from 2008 by IAG Consulting in which he found that "68% of companies suffer from poor requirements specifications practices".

These reports suggest that little has changed over time, both in the nature of software defects and in the approach to dealing with them. They point to a long-standing problem with incorrect or inadequate requirements that go on to cause a large proportion of software defects.

Adopting the Shift-left process: Quality Assurance as a mindset

In 2001, Larry Smith introduced Shift-left Testing, a better way of integrating testing early in software development. By including the QA engineer from the initial phases of the software development lifecycle, Shift-left Testing allows the QA engineer to work in collaboration with the software engineers and the product owner. This creates synergy between development and quality assurance.

In Access Payments, we have adopted a Shift-left process which incorporates Shift-left Testing as well as assuring quality right from the requirements analysis phase. The Shift-left process encourages embedding quality as stage gates throughout the software development lifecycle. Instead of a single role, the Quality Assurance thus becomes a broader mindset which everyone in the team applies, thereby ensuring the team takes collective responsibility of the overall quality.

Improving the quality of requirements

Ensuring that user stories have complete and correct requirements is paramount. Therefore, before they are brought into Sprint, we have mandated that all the user stories meet our Definition of Ready. This means that each user story must satisfy the following quality standards:

The requirements are written using the Connextra template (As a <role> I want <capability>, so that <receive benefit>).
The user story adheres to the INVEST principles.
The acceptance criteria are clearly defined and are expressed in the Gherkin language.

In order to achieve this, we conduct Envisioning sessions, Three Amigos sessions and then Refinement sessions.

Envisioning sessions

In these sessions, we meet with the stakeholder who presents the requirements, explains the context and states the value it brings to the customer. Since this is the first session where we congregate as a team to learn the requirements, our objective is to understand different user journeys, including the business rules. We then capture the requirements in the backlog using the Connextra template which will help us gain clarity on who the user is, what capability is needed and the value the user gets when the feature is delivered.

Three Amigos sessions

After capturing the requirements in the backlog as part of the Envisioning session, we organise Three Amigos sessions. Typically, the participants are the product owner, the QA engineer and a software engineer (hence the name Three Amigos). In practice, any team members are welcome to join, depending on expertise and interest.

Organising a small group session provides flexibility and facilitates multi-directional conversations. The product owner elaborates on the requirements, provides more in-depth details on the user journeys and outlines various use cases. Based on that, the QA engineer comes up with the test scenarios along with the edge cases, confirms the testability and identifies the internal and external dependencies. The software engineer considers the technical feasibility and implications, any design constraints, and then proposes possible solutions. Collaboratively, the Three Amigos work in tandem to come to an agreement on the expected deliverables.

These sessions provide us with an opportunity to understand what problems we are trying to solve and then to bridge the gaps in our understanding of the problem domain. If necessary, Spikes are created to answer any open questions, to prove any assumptions and to verify proposed solution domains. Once questions are answered, concerns addressed and any further assumptions stated, we define the scope and break down the requirements into user stories. Each user story adheres to the INVEST principles and contains a set of acceptance criteria (in the Gherkin language) which create a form of agreement on when the user story is complete.

Next, the stories are taken to the refinement session where we review them as a team.

While the Three Amigos session shapes the requirements into user stories, the refinement session provides an opportunity for the whole team to sculpt the stories themselves. We go through each user story to understand its objectives, the assumptions that have been made so far, and also its scope, so as to be able to size the story. This session encourages the team to ask questions about the stories and challenge the acceptance criteria before sizing them. More eyes often expose 'gotchas'. If there are edge cases that have not been considered, or missing scenarios, then we enrich the acceptance criteria. If it transpires that the stories do not meet the INVEST principles, they are split and resized accordingly.

This process enables the team to become comfortable to work on the user story and to understand the value it will deliver. If the team is not comfortable—either because of unanswered questions or potentially cascading impact—then we diverge again and take the user stories back to another Three Amigos session to refine them further.

The participation of all the team members increases the team's efficiency due to the knowledge being shared within the context of the user stories. It also promotes team alignment.

By the end of this session, all the user stories that have passed the review and meet the Definition of Ready are brought to Sprint planning, after which they go through a two-week Sprint of development and testing.

How are we doing with Shift-left?

To give an idea of our progress on the Shift-left process, we assessed all eight Sprints of a Program Increment. In total, these contained 58 user stories, each of which typically contains a minimum of three acceptance criteria. Of all these user stories and acceptance criteria, we found only two defects: one resulted from a miscommunication of the requirements prior to an envisioning session; the other was due to a scenario that was not thought through when analysing the requirements.

Of course, not everything is perfect. Our adoption of Shift-left prevents defects earlier in the process but it does not mean there are no issues. In particular, we sometimes face two specific challenges that can still result in our discovering defects late in the process.

Knowledge and expertise of the team: During the requirements validation phase, the team members are expected to ask questions and challenge the assumptions. They can only do this if they have the necessary knowledge and expertise of the problem domain. In order to overcome this, whenever we need to fill gaps in our knowledge to deliver new product features or work with new technologies, we attend training, invite a subject matter expert to our team or pair with those who have more experience.
Shifting further to the left: Shifting left as a team is not enough, since we still need to collaborate with other internal and external teams that are further to the left of Product Engineering, whose work we need as an input to our workflow. If they do not apply the same Shift-left mindset to their own ways of working, we end up getting incorrect requirements which can eventually lead to delivering a product that is not valuable to the customer. To address this, we try to influence other teams to apply the Shift-left mindset—for example, by inviting them to our Sprint ceremonies and engaging them on a regular basis—so that we as an organisation become more efficient, not only in terms of delivery but also in getting high quality requirements from inception.

Our overall experience with adopting the Shift-left process

We can see the benefits of Shifting-left in fewer defects being discovered after the development is complete, which can be attributed to the team applying the Shift-left mindset and influencing the rest of the organisation to be quality-focused.

The Shift-left process has also helped our team to share knowledge and have equal understanding of the features we deliver, preventing working in silos. Since all the decisions are collectively taken as a team, the team alignment is strong.

It would appear that applying Shift-left requires a good amount of investment in time, but this investment is all upfront. Furthermore, the investment saves us the potentially high costs involved in fixing defects at later stages and going back to the drawing board. The Shift-left process has made us more predictable in delivering a high quality product suite. In conclusion, the benefits that we get from adopting the Shift-left process are well worth the time invested.

Krishna Sowjanya Chandrappagari is a Senior QA Engineer at Access Worldpay.