Happy Business Starts Here

Towards A Low Cost Agile Development Environment

Zuora Staff

Context

 

Zuora developers need to be able to test the microservices they are building against the Zuora Central Platform and other dependent Zuora microservices.

 

Currently, developers write Docker Compose files for their microservice and Zuora Central Platform; this allows them to test their code quickly against a local stack on their laptop. Although this gives developers some level of flexibility and speed, as the number of microservices continues to increase, running the stack on a commodity laptop is no longer feasible as the laptop does not have the resources to run all the microservices.

 

To head off this problem, we are trying to provide a framework for developers to launch quick, dynamic and customizable development environments in the cloud.  In particular, we are testing a solution that reuses the existing Docker Compose files on a Docker Swarm cluster in AWS, with connectivity from the local laptop’s microservices.

 

Requirements

 

The list below represents some of the requirements that we have collected from developer teams with respect to developer environments. In no particular order:

 

  • Minimize changes developers need to make to adopt new environments
  • Ability to deploy all or subset of microservices and components into an isolated environment for development and testing
  • Dependency management (i.e developer pulls in microservice A, automatically pull in the Zuora Central platform and any dependent microservices)
  • Ability to specify a desired version of the microservice for testing
  • Run services on their laptop that can interact with the Docker Swarm cluster in the cloud
  • Focus on a low cost solution
  • Easy to stand up/tear down
  • Use existing or create a standard for describing a microservice environment setup and dependencies

 

Proof of Concept & Results

 

We implemented a proof of concept (POC) of the Docker Swarm solution, by running the Zuora Central Platform with some of our production ready microservices. One particular goal of the POC was to validate whether we can reuse existing microservice Docker Compose files to run in a Docker Swarm cluster in the cloud.

 

One of the goals was to be able to stand the cluster up and minimize any changes that teams need to make to their configuration files. Another major goal of the POC was to vet out the needs and understand scope for the development environments in order to derive a strategy to auto scale or automate the environments.

 

Figure 1 - High Level model

Figure 1- Integrating local and cloud based developer environments

 

Based on the POC, as displayed in Figure 1, we were able to address most of the initial requirements including avoiding changes that are required from the microservice teams to adopt the new framework.

 

Furthermore, the POC vetted out that developers would be able to run their in-development containerized microservices locally while standing up any dependencies in the Swarm cluster quickly. Developers were also able to run the entire stack on the shared Swarm cluster if they wanted to - Figure 2 represents one such configuration:

 

Figure 2 - Swarm Cluster Integration

 

Future Work

 

With the POC completed, we continue automating the Swarm based framework. As part of our future work, we have identified two cost-based improvements that we think will make the adoption of the framework more economical:

 

  • Ephemeral clusters - transient clusters that will expire after a certain amount of time
  • Spot fleets - leveraging spare AWS capacity to run clusters on the cheap like we do for our CI environments

 

Zuora Infrastructure Engineering Team

Rajesh Dharmalingam, Gary Forrest, Steven Kane