Maintenance Notifications

EMERGENCY MAINTENANCE for {www,rest,static}.zuora.com @ 6pm PT- 8pm PT Thursday 1/16/20

Please be advised that we are updating following endpoints between 9:00pm - 11:59pm PT Friday, Dec 27th, 2019.

apisandbox-api.zuora.com.

apisandbox.zuora.com.

apisandboxstatic.zuora.com.

pt1-api.zuora.com.

pt1.zuora.com.

pt1static.zuora.com.

These new certificate for these endpoints will be signed with the same root(s)/intermediate(s) as the current certificate. As such, no action should be required.

We will be performing a scheduled maintenance downtime of the Performance Test (PT1) environment starting on Thur, Dec 19, 2019, at 9am PT and ending on Fri, Dec 20, at 9pm. During this maintenance window the PT1 environment will be unavailable to accommodate a physical data center move.

In order to increase network capacity, security, and reliability we will be conducting a network maintenance in our US Production and Sandbox datacenter on Saturday, November 23 2019 from 9:00PM PST to Sunday, November 24 1:00AM PST.

In order to increase network capacity, security, and reliability we will be conducting a network maintenance in our US Production and Sandbox datacenter on Sunday, November 17, 2019 from 11:00AM PST 3:00PM PST.

Please be advised that we are taking a scheduled maintenance window for the following endpoints between 11:55 pm PT Friday, Oct 4th, 2019 - 10 am PT Saturday, Oct 5th, 2019.

Please be advised that we are taking a scheduled maintenance window for rest.zuora.com between 6pm PT and 10pm PT on Thursday Oct 3rd, 2019. We will be performing a routine rotation of an internal certificate which handles traffic for rest.zuora.com.

We are updating the SSL certificate used for the endpoints listed above from Comodo to Sectigo issued certificates.

Which endpoints does this impact?

1. *.zuora.com - (<anything>*zuora.com) not already covered by existing certificates
   
   This means only the following endpoints will be impacted:
   

• services*.zuora.com (Service Environments)
  
• restservices*.zuora.com (Rest for Service Environments)
  
• zconnect.zuora.com and zconnectsandbox.zuora.com (Zuora Reporting APIs)
  

We are updating the SSL certificate used for the following endpoints:

1. na.zuora.com - All NA Production Endpoints (na.zuora.com, rest.na.zuora.com, routing.na.zuora.com, static.na.zuora.com)

We are updating the SSL certificate used for the endpoints listed above from Comodo to AWS issued certificates.

We have an upgrade maintenance scheduled for September 4th, 2019 from 5:00PM - 8:00PM PDT

NO IMPACT EXPECTED 

Impacted Region:

US API-Sandbox Environment

We have an Upgrade maintenance scheduled for Monday August 26TH, 2019 FROM 4:00PM - 10:00PM PDT.
NO IMPACT EXPECTED

Impacted Region:

US Performance Test(PT1) Environment

US API-Sandbox Environment

US SLV Production Environment

We will be performing an preventative upgrade maintenance of our Storage system. We are reserving the 6 hour window for the maintenance where most time is in validation, actual work time is approximately 60 minutes without any impact. 

IMPACT:

A small number of US customers who utilize Zuora’s REST API endpoints may be experiencing the following error when attempting to process POST requests:

Reason{code='50000000', message='Could not read JSON’}

SUMMARY:

Due to a change rolled out by our CDN service provider between Monday, August 12 and continuing through Friday, August 16, some customers may be experiencing failures in posting requests to our REST API endpoints. The change was implemented by the CDN to address security vulnerabilities exposed when clients send both a “content-Length” and “transfer-encoding:chunked” header. To address this issue, the CDN now drops the content of the body of the POST request when both are included.

SOLUTION:

The change necessary to address this problem is to only send the “content-length” header and not utilize the “transfer-encoding” header. 

While we don’t have specific guidance for all possible clients used for API integration, an example of how to make a change for MuleSoft clients is here:

https://help.mulesoft.com/s/question/0D52T00004mXV2JSAW/how-to-remove-http-header-transferencoding-o...

For additional information and a deep dive into the issue, including methods to test and exploit the issue, please see this blog posting:

https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn

Information only, no action required

In order to increase reliability and security, we will be conducting a network maintenance in our Performance Test (PT1) environment Wednesday, July 10 2019 at 8:00PM. API and UI functionality will be unavailable for a brief period during the maintenance window. We anticipate the actual impact time to be less than 2 minutes.

We will be conducting a brief network maintenance at our US Production site on Tuesday, July 16, 2019 at 9:00PM PDT. Actual impact is expected to be less than 2 minutes.

We will be moving traffic to new firewall infrastructure which has been pre-configured and prepared to take network traffic. When the transition is made, existing sessions may be closed. New sessions will establish on the new firewall cluster when the udpated configurations are applied. This maintenance is critical to maintain secure and performant performance.

API and UI availability will be briefly affected. RevPro services will not be affected.

06/21/19 UPDATE: This maintenance will proceed as scheduled for the Sandbox environment only. We will not be moving Production traffic at this time.

During final implementation of the equivalent change in our standby environments on Thursday, June 20, we discovered a service interruption of 30-60 seconds. Due to this, we will schedule the Production maintenance for a date approximately 3 weeks from now. The new date and time will be decided and announced by Wednesday, June 26, with a minimum of 2-3 weeks of advanced notice prior to the implementation.

In order to increase network capacity, security, and reliability we will be conducting a network maintenance in our US Production and Sandbox datacenter on Friday, June 21, 2019 from 9:00PM PDT to 11:00PM PDT. Actual impact is expected to be less than 2 minutes.

We will be moving traffic to new firewall infrastructure, which has been pre-configured and prepared to take network traffic. When the transition is made, existing sessions will be closed. New sessions can be re-established within 2 minutes.

API and UI availability will be briefly affected. RevPro services will not be affected.

A maintenance status message will be posted upon completion of the activity.

SYSTEM MAINTENANCE IN PERFORMANCE TEST (PT1) ENVIRONMENTS ON TUESDAY 6/11 2019 20:00 TO 22:00 PDT

UPDATE: The maintenance window start time has been updated to 7:00PM PDT to 9:00PM PDT

In order to increase network capacity, security, and reliability we will be conducting a network maintenance in our US Production and Sandbox datacenter on Saturday, April 20, 2019 from 9:00PM PDT to 11:00PM PDT 7:00PM PDT to 9:00PM PDT.

API and UI functionality will be unavailable intermittently throughout the maintenance window as we transition in new capacity and network paths optimize. Pre-configuration and placement work has been completed in order to minimize traffic interruptions during the maintenance window. RevPro services will not be impacted.

A maintenance status message will be posted upon completion of the activity.

Update on 3/20: This maintenance will be rescheduled to a later date TBD.

This maintenance has been completed as of 9:47PM PDT.

In order to increase capacity and reliability, we will be conducting a network maintenance in our Performance Test (PT1) infrastructure from Wednesday, March 13 2019 10:00PM PDT to Thursday, March 14, 12:00 AM PDT. API and UI functionality will be unavailable for brief periods during the maintenance window. A maintenance status message will be posted upon completion of the activity.

Update: The scheduled maintenance completed successfully at 11:32 PM PDT on March 13th.

We are updating the SSL certificate used for the endpoint listed above due to its upcoming expiration. This change should be non-impactful as the root and intermediate certificates will remain the same (DigiCert).

We will be conducting a network maintenance in our Performance Test (PT1) infrastructure on Sunday, March 10 2019 from 8:00PM PST to 10:00PM PST.

In order to increase capacity and reliability, we will be conducting a network maintenance in our Performance Test (PT1) infrastructure on Wednesday, February 20 2019 from 8:00PM PST to 10:00PM PST. API and UI functionality will be unavailable for brief periods during the maintenance window. A maintenance status message will be posted upon completion of the activity.

Update: The maintenance has completed.