Maintenance Notifications

We are updating the SSL certificate used for the following endpoints:

1. eu.zuora.com - All EU Production Endpoints (soap.eu.zuora.com,dataloader.eu.zuora.com,zconnect.eu.zuora.com,static.eu.zuora.com,rest.eu.zuora.com,aqua.eu.zuora.com)
2. *.zuora.com - US Production Rest and API, Sandbox, and PT1 (<anything>*zuora.com)
3. rest.pt1.zuora.com - US Production PT1 Rest Endpoint

We are updating the SSL certificate used for the endpoints listed above from Symantec to Comodo, Digicert, or AWS issued certificates.

NOTE: no change required for sandbox.eu.zuora.com as it already has Comodo certificate

Action will be required on your part prior to October 10th, 2018 if your integration certificate store does not trust the appropriate new root and intermediate certificate chain (mentioned below). Please work with your technology teams to determine what actions you must take to ensure you do not experience any disruption in Zuora services.

When will these changes take effect on the Zuora side?

• The change will occur on the schedule outlined below.

• These changes will occur on October 10th, starting at 1AM PT and continuing until 11:00AM PT.

How will this change impact me?

Your integration will stop functioning if your systems do not trust the correct root and intermediate certificate.
* Important Note: Some applications require a restart even if the trusted root store is in place in order to use the new certificate for SSL connections.

What action must I take?

If the Root and Intermediate Certificates are not trusted by your applications or libraries, you must complete the following actions before the scheduled maintenance to avoid any potential service disruption. Please work with your technology teams to determine what actions you must take to trust this CA.

Download and install the Appropriate Root Certificate Bundle
If your integration does not trust the Comodo Root Certificates, then the certificate must be imported into your applications trusted CA store.

Follow these steps to download the Comodo Root Certificates:

1. The CA Certificates can be downloaded from the links below:

https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/Policies/Full_Certification_Chain

.eu.zuora.com - requires "eu-zuora-com-root.cer" and "eu-zuora-com-intermediate.cer"
*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) - requires "zuora-com-root.cer" and "zuora-com-intermediate.cer"
rest.pt.zuora.com - requires "rest-pt1-zuora-com-fullrootchain.ca-bundle"

2. Import these certificates into your trusted CA store based on your system parameters.
3. Restart services if applicable.

We have provided basic instructions to load the the root and intermediate certificates for Java and .NET. For other applications, please follow up with your technology teams to determine what actions must be taken.

For Java:

Run the following command from your application server(s) that make connections to Zuora systems to import root and intermediate certificates into your keystore. Note, text in blue must be replaced based on your system specifics.

• Root Certificate: keytool -import -trustcacerts -alias Zuora-AddTrustExternalCARoot.crt -file AddTrustExternalCARoot.crt -keystore <Name and path to you java keystore file, typically named keystore.jks>
• Intermediate Certificate 1: keytool -import -trustcacerts -alias Zuora-COMODORSAExtendedValidationSecureServerCA.crt -file COMODORSAExtendedValidationSecureServerCA.crt -keystore <Name and path to you java keystore file, typically named keystore.jks>

• Intermediate Certificate 2: keytool -import -trustcacerts -alias Zuora-COMODORSACa.crt -file COMODORSACa.crt -keystore <Name and path to you java keystore file, typically named keystore.jks>

For .NET on Windows 2008/2012 R2 & Windows 2016:

Click here for instructions on adding certificates to Trusted Certification Authorities store for local computer

What happens if I take no action?

If the Root Certificate is not trusted by your integration, and you take no action, your systems will not be able to connect to the Zuora Production endpoint  after this change is implemented. Please discuss this change with your technology teams to ensure you take the appropriate actions.


You are encouraged to register to the Zuora Community in order to receive the latest update on this topic.

Thank you for your support as it allows us to maintain the highest security standards at Zuora ensuring the safety of your data.

Best Regards,

Zuora Support Services & Community

If our services use the URLs you specified should we use the Production Certs or the Services Certs? The Production Certs still seem to reference Symantec, so we're not sure if those are the right ones to use.

You didn't mention the rollout for the EU sandbox? 

How are we supposed to see if this is working or not prior to the EU production roll out?

Hi, 

As per this, Zuora is planning to update symantec to Comodo certs for all US Data centers [Prod/Sandbox/Services Environment]

but the link which you mentioned clearly indicate only for Services Environment. 

Please update the link article with latest certs or clarify here. 

Thanks. 

This is some confusing information, and I have a few questions:

We're pointing to the NA sandbox environment, but not using the "sandbox.na.zuora.com" endpoint. If we don't point specifically to those hosts, (i.e. we use the "apisandbox-api.zuora.com" endpoint) will our integration be affected by the change on September 3rd, September 5th, or September 12th?

The "production" date is listed as September 12, but it includes the api sandbox ("rest.apisandbox.zuora.com") AND the production ("rest.zuora.com") endpoints. But there is also a "production date" which lists the same "rest.zuora.com" endpoint for September 5th. If I'm reading this correctly, the sandbox environment is going to get altered either after, or at the same time as, the production environment? It's impossible to tell which date applies to which environment using the schedule as posted. 

Also, September 3rd is Labor Day, and there will be massive problems with scheduling changes of this nature on that day as most of the workforce will, I imagine, be taking the day off.

Please, give some clarity to this scheule as soon as possible, as we will need to make some emergency changes to the system before next week's scheduled changes, and we need to start working on them RIGHT NOW.

Also in the future please give more time in advance of an important change like this. Posting after COB (EDT) on Friday less than a week before the first scheduled changes is not enough time in which to operate.

Thanks,

Ben

Can this be moved out a week?  We're going into a US holiday weekend and this will not give us enough time for this to bake in QA and allow us time to fully test. 

First and foremost, we need the correct certificate chain attached to this post! Second, it would good to have atleast 2 weeks gaps in between annoucement and actual implementation especially when an long weekend is involved.

So can we have the new comodo certificate chain for all URLs including Sandbox & Production????????

Hi folks

We are reviewing this issue with our Engineering team and expect to have a response shortly. 

Scott

We are using apisandbox.zuora.com URLs. Do we need the SSL certificate changes to be uptaken ?

Thanks !

September 3rd is scheduled for Sandboxes is that also includes the Services(Production copy) environments?. So we can test the SSL cert changes in our services environments. 

Hi Team,

Could you please provide the End time of these changes?

• Wednesday, August 29th 2018 7:00AM PDT: All Regions Connect - connect.zuora.com (connect.eu.zuora.com, connect.na.zuora.com), *.apps.zuora.com, *.apps.eu.zuora.com>End Time of change?
• Monday, September 3rd 2018 7:00AM PDT: US Sandbox - sandbox.na.zuora.com (static.sandbox.na.zuora.com, rest.sandbox.na.zuora.com)>End Time of change?
• Wednesday, September 5th 2018 7:00AM PDT: US Production - www.zuora.com (static.na.zuora.com, static.zuora.com, zuora.com, rest.na.zuora.com, na.zuora.com, rest.zuora.com, api.zuora.com) >End Time of change?
• Wednesday, September 12th 2018 7:00AM PDT: US Production, PT1, Sandbox- *.zuora.com, rest.pt1.zuora.com, rest.zuora.com, rest.apisandbox.zuora.com, eu.zuora.com >End Time of change?(soap.eu.zuora.com,dataloader.eu.zuora.com,zconnect.eu.zuora.com,static.eu.zuora.com,rest.eu.zuora.com,aqua.eu.zuora.com), origin-rest.zuora.com

Please share as soon as possible.

So the plan is to update both rest.apisandbox.zuora.com and rest.zuora.com at the same time? So you're giving your customers no opportunity to test the cert change against your test environment prior to applying the change to your production?

@scottb- Do you have an update on this? You were going to get back to us after checking with Zuora Engineering? Please confirm on priority so that we can plan activities.

Asking again for this to be moved to a week out due to the holiday.  Can this be moved out a week?  We're going into a US holiday weekend and this will not give us enough time for this to bake in QA and allow us time to fully test. 

Let us know if Zuora is seriosuly performing this activity and clarifying all concerns well in advance. 

With lack of information and no response on comment section clearly indicate differently. 

• I don't see the new Cert for the Performance Type Tenants - listed in the KB per the link given above is EU and US - APISandbox and Production along with US Services.
• I am also with everyone else that there is no way for us to test if Production and APISandbox are all done within 2 days of each other
• Our API Sandbox is actually APISANDBOX.Zuora.com but that's not listed above unless it's included in *.zuora.com and then it's the same day as our Production?
• Why would the change be made in Production before the PT and Sandbox environments? We use PT environments for our Day 2 and Project
• Monday 9/3 as previously mentioned is a US Holiday
• We also need a better hourly timeline for implementation; we are a global corporation and things are running 24/7 so we won't know when to update the certificate until it starts failing.

The Connect change was performed successfully. We will postpone production updates until a later date. More information will be provided shortly, and we apologize for the delay.

So you're postponing the production updates, what about the lower environment updates scheduled over the holiday weekend? Are those postponed as well or not?

@scottb@Zuora-CommunityCan we have the full certificate chain for comodo for all URLs? https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/Policies/Full_Certification_Chain does not have comodo certificats for all URLs subject to SSL cert replacement, so request to clear this asap.

Hi Everyone,

Ill follow up with the team to get all of your questions and concerns addressed ASAP. 

Lana Lee

Senior Community Manager and Strategist

Hi Team,

Could you please provide the End time of these changes?

• Wednesday, August 29th 2018 7:00AM PDT: All Regions Connect - connect.zuora.com (connect.eu.zuora.com, connect.na.zuora.com), *.apps.zuora.com, *.apps.eu.zuora.com>End Time of change?
• Monday, September 3rd 2018 7:00AM PDT: US Sandbox - sandbox.na.zuora.com (static.sandbox.na.zuora.com, rest.sandbox.na.zuora.com)>End Time of change?
• Wednesday, September 5th 2018 7:00AM PDT: US Production - www.zuora.com (static.na.zuora.com, static.zuora.com, zuora.com, rest.na.zuora.com, na.zuora.com, rest.zuora.com, api.zuora.com) >End Time of change?
• Wednesday, September 12th 2018 7:00AM PDT: US Production, PT1, Sandbox- *.zuora.com, rest.pt1.zuora.com, rest.zuora.com, rest.apisandbox.zuora.com, eu.zuora.com >End Time of change?(soap.eu.zuora.com,dataloader.eu.zuora.com,zconnect.eu.zuora.com,static.eu.zuora.com,rest.eu.zuora.com,aqua.eu.zuora.com), origin-rest.zuora.com

Please share as soon as possible.

Regards,

Bala.

Hi All,

We'll be delaying deployment with the new dates to occur, tentatively, in early October.  We'll address everyone's questions when we have more information from our Engineering team and will ensure that the new dates will be provided in a more timely fashion.

Thanks for patience and understanding.

Hi Team

Just to add to the list of clarifications needed -

Does this only impact the API integrations, or does it impact any kind of access, i.e. even end users will need to update certificates on their machines.

Thanks

Abhishek

Hi folks

We have updated the original post with the new deployment schedule and clarified a few issues with the original posts endpoints and included the proper links to the certificates in our Knowledge Center. Additionally here are some answers to several of your prior questions.

Question: What is the new date for the deployment
Answer: October 10, 2018 starting at 1am PDT

Question: What is the propagation time of deployment?
Answer: 6-8 hours propagation time (approximate) as it deploys through our CDN

Question: Can you clarify this impacts both UI and API endpoints?
Answer: Technically both although UI (browser) based access to Zuora endpoints impacted isn't likely to be an issue.  

Question: Can you clarify what endpoints are included with *.zuora.com

Answer: Basically anything that ends with zuora.com (www.zuora.com, apisandbox.zuora.com, rest.zuora.com etc)

Question: Why cant Zuora support tell me if Im impacted by this change?

Answer:  As Zuora cannot and does not have access or knowledge of our customers systems, it is important that the customer assess whether their systems are impacted or not by this change. Customer integration and truststore policy along with API integration standard practice is the responsibility of the customer and their security & technology teams to maintain.

Thanks!  Please let us know if there's any further questions

Thanks for the udpate. 

As mentioned in updated article above, 

*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) - requires DigicertRoot.cer and digicert-star-int.cer. 

But attached link for US production data centers has symantec chain [production-zuora-com-fullrootchain-bundle.crt]. 

Kindly clarify at earliest. 

Is that both the sandbox and production environments is going to get altered on the same day?. If this is the case how can we validate the changes in test environments before applying changes in Production?.

Hi Sapatel and PV

Sapatel,  we have updated the filenames in the original article.

.eu.zuora.com - requires "eu-zuora-com-root.cer" and "eu-zuora-com-intermediate.cer"
*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) - requires "zuora-com-root.cer" and "zuora-com-intermediate.cer"
rest.pt.zuora.com - requires "rest-pt1-zuora-com-fullrootchain.ca-bundle"

PV, due to the way our *.zuora.com endpoints are constructed and in alignment with our CDN configuration, unfortunately there's no way to stagger the deployments as a result.

If the deployments can't be staggered, when can we deploy this in our sandbox to test before it hits prod?

We have integrations through Dell Boomi Atom Cloud and Salesforce directly to Zuora. Do we need to reach out to them and find out if they have the latest Zuora certificates, or are you already in touch with these players?

Hi,

Can Zuora please just spin up an instance, ANY instance, of a server using the new certificates so that we can do a keystore check aginst the new certificate chain? It seems like this wouldn't be difficult to do, the server could literally just be an echo service, all we need to do is have something to validate our keystores against so we aren't testing this against live production systems. It's simply not a tolerable situation that we're going to have to ride herd on a production system for 6-8 hours in order to know if it's going to blow up or not.

Thoughts?

Ben

Hi Guys

Can you please confirm if Zuora is in touch with Dell Boomi and Salesforce for this SSL certificate update on their end as well?

They are your partners, so you should be following up with them for this change.

Hi Everyone

After review with our technical team, we believe you can test against the following Zuora API endpoint since the new certificates are currently present:

rest.apisandbox.zuora.com 

A successful test from your integration that does not result in a SSL handshake error confirms your integration is capable of making the connection without updates on your side, or if it connects when you have updated your keystore.  

In addition, we believe that there will be no impact to managed Zuora partner integrations such as Salesforce and Netsuite/Boomi which use Zuora API endpoints 

Regards, 

Scott

@scottb with the phrasing:

[...] we believe that there will be no impact to Zuora partner managed integrations such as Salesforce [...]

are you sure or not? Do we need to test something? A bit to week phrasing in my opinion. 

We have tested and found no impact to API integrations such as Salesforce and other systems where Zuora specifically maintains the API call integration.  

The truth is that historacally, the majority of SSL Certificate updates go unnoticed by most integrations.  Where it gets tricky is when a customer elects to use a trust store in their API integration, of if the code being used requires it.  

Is there a SOAP endpoint that we can use for testing the new certificates?

Additional information: A properly configured server will provide all certificates necessary to verify its server certificate, minus the root, during the TLS handshake.

You can verify that the server provides a proper chain using https://www.ssllabs.com/ssltest/?leadsource=3787244

For example, if you take a look at www.zuora.com through the Qualys SSL Server Test. You will see that the server provides all of its intermediate CA certificates. For reference: https://www.ssllabs.com/ssltest/analyze.html?d=www.zuora.com&latest

Hi Guys, can we only test it against the rest end point - rest.apisandbox.zuora.com.

Or is there a SOAP endpoint available to test as well, to make sure all is good on our end, before the cutover tonight.

Thanks

Abhishek

Hi @abhishek_grover

Unfortunatly there is no SOAP endpoint available, however as stated above you can test the conenction of your SOAP service against the cited endpoint (rest.apisandbox.zuora.com).  Any connection that does not result in a SSL handshake error represents a successful test (example, HTTP 401 Authentication error is confirmation you've passed the SSL Cerficate check and reached out app)

Hi Team,

May i know the status of this maintenance please?Let me know is it complete or still in progress?

Regards,

Bala.

@baluanisetti

The maintenace has been completed. 

https://trust.zuora.com/incidents/sp6v9ws7hmgs

Those of you who may not subscribed to our trust notification site, I would like to take the opportunity to suggest everyone subscribe to both Community notifications in "Release Notifications" (here) and on https://trust.zuora.com to ensure wide notification coverage of potential Zuora issues and maintenance.

Regards

Scott 

Hi,

We're still not seeing the changes for `rest.zuora.com` and `api.zuora.com` - openssl is showing the same certificate chains at this point. Is this just taking a long time to propagate?

Ben

When are these changes getting propogatednto production ? We are seeing an impact after SSL certificate change in sandbox instances. We need to seek the downtime if this change is going in production as well. Please publish the timeline for production changes and give us ahead notice.

@ysireesha - The changes were promoted to production yesterday (10/10) as per our original schedule
@btemko - I'm not sure we see the issue, the chain is updated as expected.

I JUST (10:37 AM EDT, October 12) ran openssl on this, here's what it's saying:

=====

>openssl s_client -connect rest.zuora.com:443 -showcerts
CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global CA G2
verify return:1
depth=0 C = US, ST = California, L = San Mateo, O = Zuora Inc., CN = WWW.ZUORA.COM
verify return:1

=====

CA CERT:

=====

>openssl x509 -text -in zs1
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
18:da:d1:9e:26:7d:e8:bb:4a:21:58:cd:cc:6b:3b:4a
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
Validity
Not Before: Nov 8 00:00:00 2006 GMT
Not After : Jul 16 23:59:59 2036 GMT
Subject: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5

=======

SERVER CERT:

=======

>openssl x509 -text -in zs2
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:12:e3:19:35:0d:d4:16:2a:21:18:1a:56:7c:a2:5e
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, CN=DigiCert Global CA G2
Validity
Not Before: Dec 20 00:00:00 2017 GMT
Not After : Dec 21 12:00:00 2018 GMT
Subject: C=US, ST=California, L=San Mateo, O=Zuora Inc., CN=WWW.ZUORA.COM

=====

That's not a Comodo CA cert at the root.

That server certificate still expires in December.

I dunno what else to do here except to say that "rest.zuora.com" doesn't have the new cert chain yet. It's the same situation for "api.zuora.com".

Am I maybe just misunderstanding the scope of the environments affected in this certificate update?

@btemko

Thanks for clarifying.  Our Technical team is looking into this.  

After review, we understand that the previous posting aluded to "anything".zuora.com, which we recognize could cause some confusion for our customers and we apologize for the ambiguity in that language. We consider all maintenances related to this point that were scheduled for this time period to be complete and that these additional endpoints will be scheduled for a follow-up update in December and we will like those details here shortly.

The certificates for the following endpoints will be replaced in a subsequent maintenance to address the certificate expiration in the December timeframe:

api.zuora.com

rest.zuora.com

static.zuora.com

www.zuora.com

zuora.com

Hi,

The notification specifically calls out "US Production Rest and API, Sandbox, and PT1" endpoints by name in the instructions ("*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) ", so it seems unambiguous that we should expect the above endpoints to be updated. The amount of angst on this thread was specifically related to the fact that all of the API endpoints in all the environments were changing at once as stated in the announcement, giving no chance to test the certificates before going to production, when in fact it was apparently never the intention to update the production REST or API endpoints at all. We have all been very clear about the issues we've had with this procedure from the start. I don't see how we were the ones who were confused here. If it was known that the production API integration endpoints weren't going to be updated it could have saved a lot of grief to simply say so. Please improve the specificty of communication on upgrades like this going forward?

Thanks,

Ben

Agree with @btemko. It goes for all release information and configurations that you do. Both timings and efftected environments and efftects are in general unclear. 

Regards

Per

Hi,

I also feel it necessary to point out that Zuora Moderator Viktor states above that "A properly configured server will provide all certificates necessary to verify its server certificate, minus the root, during the TLS handshake." and provides a tool for us to make this check against servers.

This is the result using that tool for "rest.pt1.zuora.com":

You will notice that it says "This server's certificate chain is incomplete."

We actually had an outage on our UAT system when the certificate upgrade went in because we expected the full chain to be presented as Viktor asserted above and so configured our keystore with the root certificate, expecting that to suffice. Currently the Zuora PT1 environment does not adhere to the Zuora-recommended standard.

Is this something Zuora is going to fix, or should we expect to have to adhere to this, and continue to load the entire certificate chain into our keystores, and to therefore need to update them whenever any certificate in the chain changes? We need to get this clarified before the production certificate updates in December.

Thanks,

Ben

Hi folks

I wanted to x-reference a new certificate deployment for December to address a coming cert expiry.  Please see https://community.zuora.com/t5/Release-Notifications/ACTION-REQUIRED-Maintenance-to-Update-SSL-Certificate-for-zuora/ba-p/25830

Hi @btemko

We have reviewed internally and can relay the following:

Regarding PT1 not sending a full chain, this is an oversight on our part we will correct in the near future with a separate PT1 deployment, likely early next year.

For production certificates, I can confirm we send the full chain. You can see this running SSL Labs against one of our production-staging endpoints in Akamai. URL: https://www.ssllabs.com/ssltest/analyze.html?d=e6853.ce.akamaiedge-staging.net.  Please recognize that this test is not testing the correct SNI in this test, subsequently resulting in a certificate mismatch error.  Ignoring the mismatch, you can see we are presenting the full chain in staging and the same will apply once it's deployed to produciton on the 12th.