Proliferation of micro-services
As part of Zuora’s Reinvent initiative, we see a sharp growth in the number of microservices; there is a new service popping up every few weeks. One of the problems we are trying to solve is how to evangelize and make new services visible as they come into existence.
One possible way to solve this problem is the old school way: Encourage service owners to constantly and actively publicize their services verbally, by email, and so on. The approach we decided on, however, is to create a portal to which everyone has access. This portal - a container if you will - consumes published metadata for each service and is the one-stop-shop for service discovery at Zuora.
The Software Engineering Approach
As an engineering company, we like building things. Early this year, a group of engineers met and started building this hub for fetching information about our microservices. We call it the Service-Registry service, or Registry service for short. Our design of the software is guided by following criteria:
Require minimum effort for service owners to register and maintain metadata in the Registry service
Information must be as up-to-date and as complete as possible
In talking to various stakeholders, we understood that some pieces of metadata are more desirable than others. As such, we classify groups of metadata as follows:
This information is what everyone wishes to know about a service. This includes a description, team contact (email, slack channel), names of service team members, and source code repo links. We consider this information to be the starting point for everyone trying to discover a service.
Documentation and Monitoring
Zuora engineering teams typically use Google Docs, Confluence and Jira to propose/discuss software designs, collaborate and track development progress. Moreover, services also have Swagger specs that describe how to build integrations . Last but not least, service owners monitor their service health very closely. The registry service does not attempt to replicate the existing tooling that provides all of these functions. Instead, service owners only need to provide links to their existing documentation and monitoring dashboards.
Since these first two categories of metadata are static in nature, the Registry Service provides a file template (registry file) for service owners to populate this static content. Service owners simply fill in the content, and once the registry file is put in the root of their source code repository, the service is automatically “registered.” Service metadata is then live and publicly accessible to anyone at Zuora.
Below, you’ll find a glimpse of the registry file template:
The third, and final piece of metadata that the Registry Service exposes is release information. This information answers two questions: Where is the service deployed? Which version is deployed?
The trick with this type of metadata is that we want to display the most up-to-date release information about a service on all deployed environments, while at the same reducing any ongoing dependency on engineers. That is, we don’t want engineers to have to input release information into the registry file after each deployment.
Since most of our microservices are running on ECS (AWS), as long as we have a pointer to the cluster information (such as the name and environment) we can discover everything about the service, including the state change.
As depicted in Figure 1, below, once a new version of the service is deployed to an ECS cluster, CloudWatch receives a notification (we’ve configured this). Through a CloudWatch rule we implemented, CloudWatch passes cluster state change information via a Lambda. We filter only on new ec2-instance-up messages, dramatically reducing the number of messages the Registry Service needs to process. The ec2-instance-up messages are posted to Kafka and subsequently consumed by the Registry Service as release information.
Figure 1 - The Workflow
Since the cluster is unlikely to change once deployed to within any given operating environment, service owners can just register and walk away. Up-to-date release information keeps streaming in from AWS to the Registry Service.
Conclusion and Future work
Since the birth of the Registry Service, all of the existing microservices running in our customer facing production environments are registered. This has helped engineers to better understand what’s built, and where to seek information. It also helps program and project managers to be informed about releases without having to go to Technical Operations or DevOps teams. This enables engineers to spend more time focusing on feature development.
The Registry Service is also integrated into the Earlybird project, making the registration of a new service automatic.
However, our work is not done. The Registry Service keeps on receiving feature requests; a better UI, decreased manual work for initial setups, and notification of service state change are just a few of the feature requests we’ve received.
The Registry Service was a large undertaking, and in that light we’d like to thank members of Zuora’s Core, Insights, TechOps and Platform teams for helping us with guidance and suggestions along the way.
... View more