It has been a while since Zuora launched the transformation from a monolithic architecture to microservices. We can see there are a lot of deployments going on every day, in different environments, for example, development, staging and production, and so on.
It’s really a challenge to engineers and Program Managers to track deployment status, since it involves X services in Y environments where X is already larger than 100. To deliver high quality software and manage the dependencies among the services, we need a way to get the real-time status of each service.
The list below represents some of the requirements we have collected internally for this new application.
Ability to get the real-time deployment status of a service, including its version and configurations.
Covering all the environments, which are across multiple AWS accounts and regions.
Easy to scale up with the number of services and environments growing.
Showing the release history of a service which AWS ECS does not provide.
Should be a low cost solution.
With the above requirements in mind, we decided to adopt Serverless Framework, which looks more attractive after Amazon released AWS API Gateway Private Endpoints in June 2018.
Design The following diagram illustrates the high-level architecture:
Figure 1 - Periscope Architecture Diagram
Here’s how this works.
A Node.js application is developed to route the requests and interact with RDS.
All the HTTP requests sent to API Gateway need to include AWS account ID and region as required parameters, which is then used in Lambda functions to assume role. With the IAM policies and STS, the same functions can be executed targeting different environments.
Configure access control to this private API using a Resource Policy which specifies a source VPC endpoint.
Lambda functions are written in Python, and they make use of the APIs provided by Amazon services like S3 and ECS, to build the logical object relationship we need.
Besides the real-time data returned by Lambda functions, we keep the history in RDS, we have the ability to show the data from different dimensions, including displaying the dependencies between services and providing release history for each service.
As an internal service, Periscope has been up and running for several months, and it currently supports both an Environment-Oriented View as well as a Service-Oriented View. This has helped engineers to get the running status of their services in various environments. It also helps program and project managers to be informed about releases without having to check with engineers.
In the future we plan to integrate with our deployment tools to allow for even more visibility for our engineers and program managers.
... View more