How post-incident reviews happen at Access Worldpay

Written by Andy Brodie
15 July 2020

When something goes wrong, it's important to find out why - and how to avoid similar incidents in the future. This is the premise behind Post-Incident Reviews (PIRs). In an accompanying article, Andy Brodie explains why Access Worldpay values PIRs; here, he describes how we conduct them to maximise their benefits.

To summarise my first article, Access Worldpay holds PIRs to maximise the learning opportunity afforded by the service failures that inevitably happen from time to time. Successful PIRs do this in way that avoids blame, which damages our culture and service quality. They also let us examine how we can collectively improve in future, whether by dealing with future incidents faster or more efficiently, or by avoiding them altogether.

How do they work?

The PIR itself centres on a weekly meeting. How long this takes depends on:

1. How many incidents are ready for review - each one usually takes 10-15 minutes.
2. The complexity of each incident.
3. How much pre-review information has been gathered - this can include documentation of the incident's timeline and impact, as well as a problem record.

To maximise productivity, we try to discuss only those incidents that meet certain criteria:

1. All the relevant people can attend - those who managed and/or resolved the incident, and engineering and product representatives.
2. We've documented the incident's timeline and impact.
3. We've created a problem record which is up to date with the current investigation / resolution.
4. We'll discuss proactive problems - where something bad could happen in future - if this can help the wider team.

Describing the incident

PIR meetings are led by a moderator, who introduces each incident and invites someone to manage the discussion - usually the on-call engineer who led the investigation and any subsequent fix.

The discussion begins with a thorough description of the incident, often by walking through the timeline. Timelines can only be completed once the incident has been closed, so it's helpful to keep a simple audit trail - e.g. messages in a chat channel - that can be used to add new events or information to the timeline.

Specifically, the review tries to identify four key stages in the incident lifecycle:

1. Incident Started - the earliest point of the lifecycle.
2. Incident Detected - when the incident is first reported by a person or automated monitoring.
3. Squad Response - the point when the first squad member began addressing the incident.
4. Customer Behaviour - whether the incident prompted customer behaviour to change, e.g. by switching to a backup provider or by retrying failing operations.

A large gap between Incident Started and Incident Detected could suggest a need to improve monitoring, perhaps by making alerts more sensitive or changing where they're directed. A young service often records 'false' incidents because its alert thresholds are too sensitive.

Equally, large gaps between Incident Detected and Squad Response could suggest that we need to examine how we notify squads; that an incident was new enough to miss the alert system; or that a squad member needs a louder alarm!

Assessing the fix

During the incident, the team will investigate its cause and conduct tests, after which it proposes a fix. The process of fixing the incident involves three separate stages:

1. Fix Applied - when a change was made to resolve the incident.
2. Fix Confirmed - when it was confirmed that the fix worked.
3. Incident Resolved - when it was confirmed that the impact of the incident had ended, particularly if an incident required several fixes.

PIRs examine three specific metrics:

1. The time elapsed between Incident Started and Fix Confirmed is important because an active incident is still impacting something or somebody. If this is a customer, we need to stop the incident as quickly and safely as possible.
2. The gap between Fix Applied and Fix Confirmed - a long gap could indicate that we need to improve our monitoring.
3. Fix Applied does not lead to Fix Confirmed - in this scenario, the fix may have no effect or make the situation worse. This is where PIRs can offer enormous value.

When assessing this third metric, we examine the original rationale for the fix. In a PIR, we need to differentiate between what we knew at the time and what we know now, in hindsight. The aim of the PIR is to learn as much as possible so that we can apply this valuable, 20/20 post-event knowledge to future incidents while they are still active.

It's not unknown for PIRs to contain surprising conclusions - "like the big reveal of a whodunnit," according to one regular. Such revelations often identify problem tasks, which often apply across multiple services. This is why bringing disparate people together in a PIR can add genuine value.

What causes incidents?

Broadly, PIRs uncover two types of problem, each of which prompts its own questions.

1. Functional issues When the code doesn't behave in line with customer requirements or expectations, we need to know:

   1. Do we need a new test case to fail if this problem happens again?
   2. Do we have an existing test case that picked this up but the problem wasn't detected?
   3. Were there missing acceptance criteria in the feature or story?
   4. Were any omissions deliberate or accidental? In other words, did something happen that should never happen, or did we miss something?

2. Non-functional issues These cover scenarios where the service is affected by an external influence; where there's a partial failure; or where the incident impacts security, availability, latency, resilience and/or recovery.

   In such situations, the universe of potential questions is enormous and choosing the right ones often comes from combining experience with analysis of the incident's symptoms and impact. Often, we ask questions like these:

3. Could a different action have allowed the service to continue, at least partially? a. Would tweaked configuration of load shedding or circuit breakers reduce the impact? b. If a service instance failed, could we redirect to working instances more quickly? c. Were there any warning signs before it broke; for example, if memory or disk were running out, could we have flagged this earlier in our alerts?

4. How easy it was it to recover? a. Do we know the current state of the customer and downstream third parties or do we need to reconcile? b. Did we need to restore any data from backup and, if so, did we hit our Recovery Point Objective (amount of data lost)?

5. Is this problem specific to the service or could it apply elsewhere? For example, if we tripped over a problem in a third-party library or service, could it affect another of our services later?

Such questions demonstrate why it helps to have representatives from each squad attending PIRs. Experience shows that most incidents and problems affect more than one service. What's more, even if a single fix would resolve the underlying problem across the board, wide attendance reduces the likelihood of another squad finding themselves in the same situation in future.

The human factor

The description of PIRs contained here may give the impression that incidents are discussed entirely dispassionately. Despite everybody's best efforts, this doesn't always happen. To ensure that the response is appropriate and helpful, therefore, moderators listen out for certain types of comments, such as these four below.

Improving, investing, discovering

The success of PIRs rests largely on participants' transparency, honesty and desire to improve. And as a tribe, Access Worldpay appreciates their broader benefits: "We're not just trying to improve the quality of our products, we're trying to improve the quality in our entire way of working, too," explains Jonathan, a Quality Assurance Engineer. "Within our operating model, it's down to us to react to and address every incident that our products encounter," he continues: "To this end, PIRs can be an underrated investment in future time saving."

Speaking from my own point of view, as we follow a Test Driven Development (TDD) approach, I know we do a lot of testing before code is allowed into production so, whatever the incident, it's likely to be a product of multiple factors that we thought were independent, but turned out not to be. As a result, almost all incidents have an "Ah-ha!" moment when the root cause is discovered, followed by an "Oh wow, we didn't think THAT could happen!" Which just goes to show that PIRs can be not only useful, but fascinating, too!

Andy Brodie is Senior Product Manager for Quality of Service at Access Worldpay.