I’m Scott Ellis, and I’m a technical lead on Zuora’s API Proxy project. In this blog post, I'll describe our API Proxy, which quickly allowed us to create API parity between SOAP and REST for our customers.
For a long time, Zuora customers have had the choice of interacting with our product using both REST and SOAP APIs. Unfortunately, neither protocol exposes the full breadth of functionality available on Zuora’s platform. Some operations can only be performed through our REST interface, while others can only be done over SOAP. Many Zuora customers have had to use a combination of both APIs to meet their needs.
In late 2016 we decided to remedy this problem. We built a system called API Proxy that translates incoming JSON requests into SOAP requests, forwards the translated request to the Core Zuora platform, and then translates the SOAP responses back into JSON. While the exact content of requests and responses are still somewhat constrained by the underlying structure of the SOAP messages, this enables Zuora platform integrators to write JSON clients that can access all of Zuora’s APIs.
To make things more RESTful, we added one more layer: the ‘object’ interface. Adding this abstraction layer allows system integrators to perform RESTful interactions like creating an account by calling POST on:
or to retrieve account data by calling GET on:
Underneath the covers, API Proxy translates these API calls into the appropriate SOAP requests.
As part of the API Proxy project, our documentation team built a fancy new website that describes all the old and new REST APIs from a technical perspective with example input, output, and code. It’s really cool and an invaluable asset if you are building integrations. The site itself will be featured in an upcoming blog post, but I encourage you to click through and check it out.
The rest of this blog provides a high level architectural description of the API Proxy, and describes some of the technical challenges that we overcame as we built a new way to access the Core Zuora platform.
The API Proxy doesn’t just expose a new set of REST APIs for our customers. It also serves as a gateway to existing Zuora REST APIs. Using the API Proxy, Zuora customers can access all of Zuora’s API functionality through a single address: rest.zuora.com. This means that it is a critical piece of our infrastructure, so we designed it with speed and robustness in mind.
The API Proxy system runs as a collection of homogeneous nodes in a cluster, behind a load balancer. Each node runs four Docker containers:
Figure 1: API Proxy Cluster
Requests and responses are handled by Kong, an Openresty application written in Lua and running on Nginx. Kong is extremely fast and can be customized by writing plugins. We have created several plugins to perform the JSON to XML translations we need, to simplify authorization, as well as for logging and metric collection.
For caching data like WSDL files, we use a combination of Twemproxy and Redis. Twemproxy allows us to spread the cache load across our API Proxy cluster using consistent hashing with failure detection, connection pooling, and a few other nice features. This provides us with a high degree of robustness on the caching layer.
Lastly, we use Cassandra to store the small amount of configuration data required for the proxy to operate, such as the mappings between incoming requests and the downstream Core API endpoints. Cassandra’s fault tolerance, scalability, and performance characteristics make it an ideal choice for this use case.
With this architecture there is no single point of failure. The system is fully distributed with no leader nodes or other privileged components; it will continue to operate correctly even when a majority of the nodes fail. In fact API Proxy has been tested under chaos monkey like conditions with no degradation in performance. Scaling the system out can be done easily by adding more identical nodes to the cluster.
The idea sounds simple: convert JSON to XML, and XML back to JSON. There are a couple of interesting elements of the Zuora SOAP APIs that need to be taken into account during this process.
Translating XML Values into Correct JSON Types
The first hurdle is translating XML values into the correct JSON types. Everything in XML is represented as strings of text, but the WSDL document that prescribes the format and content of messages defines types for all of the objects and fields. When SOAP content is translated to JSON, the types need to be translated correctly. For example, boolean fields should contain true and false boolean values, not quoted text like “true” or “false”. Numbers, arrays, and nested objects require the same sort of reinterpretation.
To perform correct data type conversions, API Proxy loads a copy of the Zuora WSDL file and parses it into a data structure that represents all of the different model objects and their fields, including the field data types. When a JSON request is received, this data structure is used to interpret incoming field values. Similarly for outgoing responses, the response object fields are converted to the correct data types depending on the contents of the WSDL file.
Zuora Custom Fields
Another challenge that was solved for in the API Proxy project relates to Zuora’s custom fields feature, which allows almost all objects in Zuora’s object model to have additional custom attributes; these fields are optional, but can be configured separately by each Zuora tenant (customer). Each tenant has the ability to define their own custom fields with different names and types. Thankfully though, these fields are mirrored in the Zuora WSDL file, so the API Proxy uses each tenant’s customized WSDL file to translate messages, and refreshes its cached interpretation of the data structures periodically. In this way the API Proxy stays up-to-date with modifications to the custom fields attached to model objects, adapting as customers make changes to their configurations.
The Object Interface
Implementing the object interface presented yet another challenge. In order to fetch an object with all of its fields in response to a GET call, the API Proxy builds a ZOQL query that selects all of the object’s fields (discovered via the WSDL file). However, some fields can not be queried using ZOQL, and the WSDL file does not tell us which ones can be used in the query.
To solve this problem, API Proxy leverages the ‘datasource describe’ API calls which provide more information about the different object data sources, the fields that they provide, and the ones that can be used in ZOQL select and where clauses. Using this information we build a list of excluded fields, allowing the API Proxy to ask for all available information for an object without causing errors by asking for too much.
Today and Tomorrow
Since the release of API Proxy in November 2016, we have monitored our metric dashboards and seen transaction counts steadily climbing. Today, the system handles thousands of requests per minute.
We’re not stopping there though, and hope you continue to watch this space. There are many more API related developments in our pipeline, some of which will be landing in Production very soon.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.