As Brent Cromley, our SVP of Engineering, outlined in the “Reinventing the Wheel” blog post, we are reinventing large pieces of our core platform and teasing it out into a microservices based architecture. As part of this re-architecture work, we are building smaller, finer-grained and loosely coupled services. This architecture improves the modularity and scalability of the Zuora system. It also enables parallelized development and continuous delivery by smaller, more agile teams. During this transition, we are not only evolving our technology stack; we are delivering services that are easier to understand, develop, test, deploy, monitor and operate. The EarlyBird Gets the Worm At the onset of this journey of reinvention, a project code-named EarlyBird was launched. The deliverables of this project would serve as a reference implementation for a Zuora microservice standard, and also as a template service allowing engineers to build on top of a scaffold microservice written in Java.
To be sure, we were not trying to compete with microservice frameworks like Spring boot or Dropwizard; we were reinventing the wheel by leveraging these best of breed frameworks and building on top of them. The purpose of EarlyBird was to define and implement best practice items that Zuora microservices need to adhere to, as described below.
EarlyBird: Towards a Cookie Cutter World
Zuora has its own development guidelines and design principles. In principle this is a good thing, but in practice they are text documents, and it is sometimes difficult for engineers to follow best practices and all the principles. To solve this problem, EarlyBird implemented these guidelines and principles as built-in functions. By inheriting from the EarlyBird template, most of these guidelines/principals are enabled by default, and the developers also get enforcement of guidelines for free.
API First Design: We are using an API-first design strategy and utilize Swagger as the formal API design specification format. The specification file is tracked and versioned in the microservice codebase. In Earlybird, we designed commonly used API endpoints in Swagger format and also implemented those APIs following Zuora’s internal API Cookbook. For more on Swagger, please check out Chunlei Wang’s blog post on Zuora’s API Reference, built completely using Swagger.
Company Code Style, Mavenized, Static Code Analysis: All projects are built as a maven project. Earlybird uses a company-specific project template inherited from the base pom project, which enforces the Zuora Java code standard and leverages the checkstyle, PMD and Findbugs tools.
Dockerized: Each service runs in a docker container. We have a basic scaffold image Dockerfile for services to build their own docker images and docker compose files for integration tests.
Unit Tests, Component Tests and Integration Tests: Earlybird defines what a unit test is, what a component test is, and what an integration test is. It provides some reference test cases implemented following our testing best practices.
Enable code coverage report: Engineers need insights into their code coverage for each commit, pull request or service version. Earlybird uses JaCoCo enabled for each service and integrates with our CI tools for easy code coverage reporting.
Application Metrics and Monitoring: Metrics are an important mechanism to understand service behavior and monitor overall system status. Earlybird incorporates metrics instrumentation libraries and integrates with MachZ - a Zuora Metrics framework. Engineers can build dashboards and configure alerts based on the application metrics data. For more information on MachZ, check out Jeff Feldstein’s blog post.
Service Logging: Logs are a developer’s best friend. We define standard log attributes for service log entries and integration points with Sumologic. The logs are aggregated and are searchable in a centralized logging console for troubleshooting distributed systems easily.
CICD Integration: Zuora uses a set of infrastructure tools for CICD pipelines. Earlybird provides scaffold YAML files for easy integration with our CICD stack, which includes Jenkins, Terraform and Ansible.
Operations, management APIs: Services need to provide APIs for health checks, tenant provisioning, data migration, and other service operations. EarlyBird provides a default API for these operational services.
Configuration: Each service should build once per release and be runnable in multiple environments without modification. EarlyBird defines a way of externalizing the application configurations in files and can load different environment variables outside of the application. Not everything is cookie cutter
We realize that dev teams and services may have very unique and distinct business and functional requirements, which may imply different project/code organization strategies, or different integrations with Zuora’s infrastructure. We fully recognize there is no one stop shop solution. So for the below items, EarlyBird only provides a basic reference implementation.
Zuora API Gateway Integration
Zuora Reporting pipeline Integration
Zuora Custom Field/Object Service Integration
Zuora Business Event Stream Integration
Project organizing and code organization
Dynamic log level adjustment
At present, we are using an internal command line-based tool to enable dev teams to quickly create scaffolding project code, but our long term goal is to use maven arch-type to generate skeleton microservice code.
Within Zuora, many microservice teams are using EarlyBird as their service template and have built services for different business domains. Many engineers from different teams are actively involved in the effort, either providing feedback or contributing code through internal pull requests or code reviews for EarlyBird. EarlyBird itself is evolving to adopt new methodologies and changing with Zuora’s Reinvent journey. The EarlyBird truly gets the worm!
... View more