Happy Business Starts Here

Infrastructure Management Tooling in a Micro-services World: The Zuora Connect Journey

Zuora Staff

Preface: Monolith to Micro-Service Evolution


In the last couple of years, there’s been a design paradigm shift from monolithic architectures to micro-service models for web applications. As a monolithic application’s complexity increases, release cycles commonly increase in duration and decrease in efficiency; regression testing, compilation and deployment time are all impacted, which in turn reduces engineering velocity for releases.


Young startups are quick and nimble because they don’t have to deal with thousands - sometimes millions - of lines of code, so a Monolithic architecture approach makes sense for a new product launch. There’s a large amount of risk in spending months over-engineering an application stack when the product itself may never gain traction. But what happens when your product does gain traction?

 

A lot of people, including ourselves, are talking about the benefits of moving to micro-services, but few are talking about best practices for development velocity and maintenance of micro-services. With the advent of micro-services, there is significantly more overhead all around, physically on the infrastructure side and performance-wise due to latency and a desire to eliminate point to point communication between servers.

 

What kind of security, communication and data access frameworks need to be in place to ensure your engineering velocity speeds up instead of down as the number of micro-services your organization supports increases?

 

How do you best enable your team to quickly deliver a micro-service to production without everyone being a full stack engineer with production access?

 

We don’t claim to have all the answers, but we will share our story going through this evolutionary process and the decisions we’ve had to make thus far. We are hoping some of our lessons learned can be takeaways for other engineering teams trying to solve similar problems.

 

The Connect Marketplace Story

 

The Connect Marketplace started about two to three years ago as a monolithic Ruby on Rails application. The Marketplace is a platform that allows “applications” to be developed and integrated against your Zuora environments.

 

Zuora Connect Marketplace

Figure 1 - Connect Markplace Apps

 

Functionally, we are trying to achieve something similar to all of the popular appstores out there: Apple iTunes, Google Play, Salesforce AppExchange are a few that come to mind. By allowing external developers to extend our core Zuora offering, the value Zuora provides can scale exponentially independent of the rate at which Zuora proper delivers platform and product features.

 

Connect - The Monolith

 

The Marketplace minimum viable product (MVP) was centered around our first “app”. Namely, a set of views, models and controllers that were baked into the rest of the Connect codebase that was designed primarily to control user authentication, application listings, provisioning of “installed” applications in Zuora environments, and infrastructure management at the admin level.


We received positive feedback almost immediately and were soon on the horn for releasing two to three additional “applications” using the monolith approach.

 

Our First Micro-service

 

When planning our fourth application, our technical requirements were different than in the past. Whereas past applications focused more on analytics and bulk processing of data, the new app required high responsiveness, high API throughput and would need to support spiky transactional behavior. Based on these requirements, we needed an isolated, standalone application stack.
We tapped our team’s residential AWS guru, Matthew Ingle, to launch a new stack; ELB, a set of EC2 servers for the frontend and the backend and an RDS DB. Matt also wrote a quick deployment script that would pull code from the master branch of the new applications’ git repos and perform rolling restarts…. Success!


Scaling Issues

 

Our engineering team has since grown and Matt ended up owning, launching, configuring and maintaining our separate application stacks:

 

Each time a deployment was required, someone would swivel their chair over and tap him on the shoulder.

 

Each time there was some kind of exception thrown in the production servers Matt would SSH in and pull down the application logs.

 

Server crashed and didn’t self-heal? Matt had to manually launch a new server. 

 

Server crash in the middle of the night? Someone named Matt was getting a phone call. 

 

Need a new server stack? Get in line, put in a ticket, it’ll be available sometime next week.

 

Does the above sound familiar to any of you?

 

Very quickly, Matt was consuming most of his cycles supporting the rest of the team and became a scalability bottleneck. We knew it was time for a programmatic solution.

 

The Next Five Micro-services...

 

We need an easy way for our teams and users to provision their own infrastructure and preferably in a way that abstracts the need for technical knowledge of the underlying components (Servers, DBs, Redis, etc), to deploy their own code, to access application logs, to be able to monitor and benchmark their application performance. The key requirement for all of this is to enable folks without relying on key members on the team, and do it all without direct AWS access or production SSH access.


In the next part of the Zuora Connect Journey series, we’ll share details surrounding the software stack we ended up choosing to solve this scaling issue, and how we ultimately bridged the gap between our components.


Stay tuned!