How does Monzo Bank in the UK manage 1600 microservices with K8s?

Posted Jun 15, 20205 min read

Matt Heath and Suhail Patel, two senior engineers of British digital bank Monzo, shared their experience of managing 1,600 back-end microservices in a seminar in London. This British bank, established more than 5 years ago, has more than 4 million financial users. It entered the US market last September and is currently developing digital banking services for businesses.

All financial services of Monzo are provided through mobile apps. Therefore, they decided to establish a distributed architecture at the beginning, instead of building a large bank core system. Initially, Mesos was used to build a container cluster. In 2016, it was completely replaced with K8s to create a platform for executing various financial microservices. Monzo is one of the few companies that started to build K8s very early, and also migrated the infrastructure to the AWS platform to reduce operation and maintenance manpower.

[ PIC1.jpg ]( .jpg)

Monzo uses Cassandra, a NoSQL database that is easy to scale horizontally, and the main development language of the back-end system is simple Go. Their reason is that this language is guaranteed to be backward compatible, so when there is a language revision, for example, a new version adds garbage. The collection function, the original code does not need to be modified, it can be executed after recompiling directly with the new version of Go, and the new function can be used to improve the efficiency of memory management.

This is a company that likes to develop its own tools. If there is a need to connect to a third-party system or payment platform, Monzo makes its own development and integration mechanism as much as possible to improve efficiency and avoid the use of third-party integration tools. Code, for example, they developed an interactive tool to connect the AWS environment and the K8s environment, allowing developers to quickly deploy or restore the old version of the deployment through a pull reques command.

In the design of microservices, Monzo runs every microservice in a Docker container. The code of containerized microservices is also divided into two layers. One layer is the shared core code library layer(Shared Core Library Layer) that all microservices must have built-in, including RPC, Cassandra, locking mechanism, log recording mechanism, monitoring Matrics , Queuing and other six types of code bases, and the other layer is the business layer, that is, the code that this microservice will enter.

[ PIC2.jpg ]( .jpg)

The principle of Monzo splitting the granularity of microservices is that the smaller the possible disassembly, they explain that the finer the disassembly, the lower the risk of changes, such as updating a single function, which can reduce other microservices. Impact. However, the smaller the granularity of microservices, the cost is that a large number of microservices will be generated. According to Monzo's statistics, the first year was only hundreds of microservices, but by the beginning of November 2019, it had reached 1,500 microservices, and even increased to 1,600 in December last year. There are more than 9,300 unique calls between these microservices.

[ PIC3.jpg ]( .jpg)

Because all services are online, Monzo hopes to implement a zero-trust security system as much as possible, so a whitelist is used to control the list of other microservices that each microservice can call. At first, when the number of microservices was not large, this whitelist used manual maintenance, but when the number reached thousands, or even nearly 10,000, the maintenance work was very complicated. Therefore, Monzo decided to develop automated maintenance tools.

Monzo, who is accustomed to developing his own tools, first picked a microservice service.ledger with the strictest security control to test, using K8s network policy resources to establish a call whitelist in the configuration file. This is a microservice responsible for moving money across accounts.

Then, Monzo developed a micro service communication analysis rpcmap, which can automatically analyze each Go language program and find all the call sources for service.ledger micro services to establish a white list.

With the list, Monzo's next step is to use the NetworkPolicy resource of K8s to perform filtering, establish a network policy in the ledger service configuration file to which service.ledger belongs, and only the network traffic source with the allowable label can be released, and the ledger In the code directory, put a whitelist file of authorized sources. Once other development teams want to obtain call permissions, they must update the whitelist file and re-create the ledger service(because the files in the code have changed) to take effect. The ledger development team can review this new addition during the build phase. External calls.

However, several problems were found after the test, and the burden of manually reviewing the call was increased, which also increased the risk of the microservice replying to the old version, plus the development team used to manually edit the configuration file, everyone can modify the call white List, but difficult to control. Later, Monzo decided to import Calico, an open source K8s network security control project, to establish a micro firewall function on each micro service to manage access. In addition, Monzo also vigorously improves the information transparency of microservice management, such as the self-made microservice analysis tool, which facilitates the development team to query the list and status information of APIs used by microservices after each code checkout, and uses a large number of visual monitoring tools to track traffic And dosage.

In addition to the management mechanism on the tool side, Monzo has also produced a 101 guide for back-end engineers, which requires the development team to start following the first time the back-end program is written. The content is established from a new service, RPC handler import, information query, such as and through Firehose release and usage information, how to write unit tests, and how to deploy, have detailed instructions and regulations, and provide a Slack channel to discuss the set of back-end application specifications. Monzo requires that every developer member must follow this specification to develop back-end microservices.

Monzo explained that the finer the granularity of microservices, although it helps to improve flexibility, it needs to be matched with a consistent program architecture and tools. Through standardization, engineers can focus on business problems and continue to improve tools and functions, so that a series of iterative modifications can be quickly performed. , To break the complexity of large-scale financial applications and reduce risk.(Source:K8S Chinese Community)