Engineering leadership

The decisions that created Access Worldpay

Written by Oludayo Fafiolu
03 September 2019

Faced with a pile of LEGO, any child knows that building something from scratch involves lots of decisions - about who can help, what to build, and whether it can be adapted later on. Those decisions can transform a simple LEGO plane into a sea-plane, jet-plane, biplane or triplane. Enough good decisions could even, one day, create the prototype Megafast UltraCool StarSurfer 2000™.

That's a complicated vehicle, obviously, but it's nowhere near as knotty as Access Worldpay, which has required decisions of a quantity, complexity and consequence to make LEGO heads spin. Still, the brief was clear - to build a set of payment services and interfaces that could deliver Worldpay products quicker, more safely, more simply and with better service quality than anyone else. It also had to be easily scalable globally, both in capacity and cost. All told, "a reasonably non-trivial request", says Simon Welland, Co-Lead of Access Worldpay.

Stating the obvious

Simon and his colleagues began with some "obvious" decisions, starting with the judgment that Access Worldpay had to be "cloud-first", because the alternatives "simply couldn't deliver the scalability and speed we needed."

They also chose to go open-source. Besides using standard tools, such as GitHub and Nexus, they opted for proven open-source software, "because typically it offers the best engineering and innovation, and also the best quality and good value for money," says Simon. To help ensure a viable supply chain, Access Worldpay decided to invest in key open-source providers and to share any future developments or improvements made by their engineers to open-source projects.

Autonomy and automation

The imperative for high speed, responsiveness and service-quality triggered important decisions. "We felt that only very high levels of automation and rigour in our approach would enable us to meet the required pace of change and level of quality of service," Simon explains: "So we really pushed for investment in automation, in testing and in the use of software agents to have zero defect pipelines."

Another choice involved fully harnessing "the power of the squad", says Pat Bateman, Co-Lead (Build Chapter): "The squads are full of incredibly capable people who are at the coal face, living the problems on a daily basis, so we push as much autonomy in their direction as we can." Reinforcing this, the team's BRO model (Build, Release, Operate) gives squads responsibility for building and releasing software and then maintaining it too, thereby improving service quality.

First, principles

Since autonomy is about decision-making, success requires clarity - specifically, clear guidelines, principles and leadership. One founding principle was to leverage individual knowledge and experience through extensive collaboration - for example, helping Worldpay colleagues understand wider applications that may sprout from the team's decisions, and also discussing experiences informally with developers working for their customers.

Another guiding principle is test-driven development (TDD). "We test everything," says Simon: "If it's not tested, it's broken." Although TDD is commonplace for software developers, it represents a new frontier in cloud and infrastructure engineering. Consequently, the team's cloud engineers have had to "rewire their brains to think about things a little differently," says Paul Wright, Head of Cloud Engineering. "As head, my job has been to ensure people feel comfortable in this journey, understand the challenges and are clear that this is the agreed direction."

Just-enough, just-in-time

Setting clear guidelines and principles - however challenging - has empowered squads to make the decisions necessary to fulfil the task at hand. "We ask the squads just to try and meet the requirements they've been asked to meet and avoid a whole bucket list of hypothetical things that could happen in the future," Pat says. This helps create a "just-enough, just-in-time, every time" architecture that enables squads to start producing results very quickly.

Effective autonomy also requires clear limits and, therefore, leadership. "You need a strong feedback loop," Pat says, because some decisions matter more than others. "If they come up with any significant decisions they can bubble them up to the leadership team," he continues: "If it's a really important decision we'll try to make it for the entire team because it can help an awful lot of the other squads who'll also hit these problems at some point."

Four-wheel decisions

The leadership team consists of Simon and three others, including Paul and Pat. "It means you'll have a difference of opinion, which is good," says Pat. "With a group of three plus Simon, who's the constant check and decision-maker, you end up making a lot more sensible decisions based on everyone's collective knowledge and experience and opinion. It feels like a very safe environment as a result."

Simon adds: "We bring over 100 years of systems leadership experience into the group and we do use that in our decision support." It also helps them develop the knowledge and skills necessary for squads to use their autonomy to best effect. "We've worked hard on supporting our people to grow to the full-stack - which, today, extends from front-end to back-end to infrastructure," says Simon: "We're developing those skills in individuals so that, ultimately, we can take our architecture decisions anywhere within our team." Paul agrees: "That's what we decided from the start - to build strong, cross-functional teams that know what they're operating and owning." Just imagine what they'd make with some LEGO.