We are updating the SSL certificate used for the following endpoints:
We are updating the SSL certificate used for the endpoints listed above from Symantec to Comodo, Digicert, or AWS issued certificates.NOTE: no change required for sandbox.eu.zuora.com as it already has Comodo certificate
Action will be required on your part prior to October 10th, 2018 if your integration certificate store does not trust the appropriate new root and intermediate certificate chain (mentioned below). Please work with your technology teams to determine what actions you must take to ensure you do not experience any disruption in Zuora services.
When will these changes take effect on the Zuora side?
How will this change impact me?
Your integration will stop functioning if your systems do not trust the correct root and intermediate certificate.* Important Note: Some applications require a restart even if the trusted root store is in place in order to use the new certificate for SSL connections.
What action must I take?
If the Root and Intermediate Certificates are not trusted by your applications or libraries, you must complete the following actions before the scheduled maintenance to avoid any potential service disruption. Please work with your technology teams to determine what actions you must take to trust this CA.
Download and install the Appropriate Root Certificate Bundle If your integration does not trust the Comodo Root Certificates, then the certificate must be imported into your applications trusted CA store. Follow these steps to download the Comodo Root Certificates:
.eu.zuora.com - requires "eu-zuora-com-root.cer" and "eu-zuora-com-intermediate.cer"*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) - requires "zuora-com-root.cer" and "zuora-com-intermediate.cer"rest.pt.zuora.com - requires "rest-pt1-zuora-com-fullrootchain.ca-bundle"
We have provided basic instructions to load the the root and intermediate certificates for Java and .NET. For other applications, please follow up with your technology teams to determine what actions must be taken. For Java:
Run the following command from your application server(s) that make connections to Zuora systems to import root and intermediate certificates into your keystore. Note, text in blue must be replaced based on your system specifics.
For .NET on Windows 2008/2012 R2 & Windows 2016:
Click here for instructions on adding certificates to Trusted Certification Authorities store for local computer What happens if I take no action? If the Root Certificate is not trusted by your integration, and you take no action, your systems will not be able to connect to the Zuora Production endpoint after this change is implemented. Please discuss this change with your technology teams to ensure you take the appropriate actions. You are encouraged to register to the Zuora Community in order to receive the latest update on this topic. Thank you for your support as it allows us to maintain the highest security standards at Zuora ensuring the safety of your data.
Best Regards, Zuora Support Services & Community
I wanted to x-reference a new certificate deployment for December to address a coming cert expiry. Please see https://community.zuora.com/t5/Release-Notifications/ACTION-REQUIRED-Maintenance-to-Update-SSL-Certificate-for-zuora/ba-p/25830
If the deployments can't be staggered, when can we deploy this in our sandbox to test before it hits prod?
September 3rd is scheduled for Sandboxes is that also includes the Services(Production copy) environments?. So we can test the SSL cert changes in our services environments.
Hi Guys, can we only test it against the rest end point - rest.apisandbox.zuora.com.
Or is there a SOAP endpoint available to test as well, to make sure all is good on our end, before the cutover tonight.
Hi @abhishek_groverUnfortunatly there is no SOAP endpoint available, however as stated above you can test the conenction of your SOAP service against the cited endpoint (rest.apisandbox.zuora.com). Any connection that does not result in a SSL handshake error represents a successful test (example, HTTP 401 Authentication error is confirmation you've passed the SSL Cerficate check and reached out app)
Is that both the sandbox and production environments is going to get altered on the same day?. If this is the case how can we validate the changes in test environments before applying changes in Production?.
Could you please provide the End time of these changes?
Please share as soon as possible.
@ysireesha - The changes were promoted to production yesterday (10/10) as per our original schedule@btemko - I'm not sure we see the issue, the chain is updated as expected.
We have integrations through Dell Boomi Atom Cloud and Salesforce directly to Zuora. Do we need to reach out to them and find out if they have the latest Zuora certificates, or are you already in touch with these players?
Ill follow up with the team to get all of your questions and concerns addressed ASAP.
Senior Community Manager and Strategist
Agree with @btemko. It goes for all release information and configurations that you do. Both timings and efftected environments and efftects are in general unclear.
Hi Sapatel and PV
Sapatel, we have updated the filenames in the original article.
PV, due to the way our *.zuora.com endpoints are constructed and in alignment with our CDN configuration, unfortunately there's no way to stagger the deployments as a result.
If our services use the URLs you specified should we use the Production Certs or the Services Certs? The Production Certs still seem to reference Symantec, so we're not sure if those are the right ones to use.
I JUST (10:37 AM EDT, October 12) ran openssl on this, here's what it's saying:
>openssl s_client -connect rest.zuora.com:443 -showcertsCONNECTED(00000003)depth=3 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5verify return:1depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2verify return:1depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global CA G2verify return:1depth=0 C = US, ST = California, L = San Mateo, O = Zuora Inc., CN = WWW.ZUORA.COMverify return:1
>openssl x509 -text -in zs1Certificate:Data:Version: 3 (0x2)Serial Number:18:da:d1:9e:26:7d:e8:bb:4a:21:58:cd:cc:6b:3b:4aSignature Algorithm: sha1WithRSAEncryptionIssuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5ValidityNot Before: Nov 8 00:00:00 2006 GMTNot After : Jul 16 23:59:59 2036 GMTSubject: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. - For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
>openssl x509 -text -in zs2Certificate:Data:Version: 3 (0x2)Serial Number:03:12:e3:19:35:0d:d4:16:2a:21:18:1a:56:7c:a2:5eSignature Algorithm: sha256WithRSAEncryptionIssuer: C=US, O=DigiCert Inc, CN=DigiCert Global CA G2ValidityNot Before: Dec 20 00:00:00 2017 GMTNot After : Dec 21 12:00:00 2018 GMTSubject: C=US, ST=California, L=San Mateo, O=Zuora Inc., CN=WWW.ZUORA.COM
That's not a Comodo CA cert at the root.
That server certificate still expires in December.
I dunno what else to do here except to say that "rest.zuora.com" doesn't have the new cert chain yet. It's the same situation for "api.zuora.com".
Am I maybe just misunderstanding the scope of the environments affected in this certificate update?
The Connect change was performed successfully. We will postpone production updates until a later date. More information will be provided shortly, and we apologize for the delay.
Asking again for this to be moved to a week out due to the holiday. Can this be moved out a week? We're going into a US holiday weekend and this will not give us enough time for this to bake in QA and allow us time to fully test.
Thanks for the udpate.
As mentioned in updated article above,
*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) - requires DigicertRoot.cer and digicert-star-int.cer.
But attached link for US production data centers has symantec chain [production-zuora-com-fullrootchain-bundle.crt].
Kindly clarify at earliest.
We have updated the original post with the new deployment schedule and clarified a few issues with the original posts endpoints and included the proper links to the certificates in our Knowledge Center. Additionally here are some answers to several of your prior questions.
Question: What is the new date for the deploymentAnswer: October 10, 2018 starting at 1am PDT
Question: What is the propagation time of deployment?Answer: 6-8 hours propagation time (approximate) as it deploys through our CDN
Question: Can you clarify this impacts both UI and API endpoints?Answer: Technically both although UI (browser) based access to Zuora endpoints impacted isn't likely to be an issue. Question: Can you clarify what endpoints are included with *.zuora.com
Answer: Basically anything that ends with zuora.com (www.zuora.com, apisandbox.zuora.com, rest.zuora.com etc)
Question: Why cant Zuora support tell me if Im impacted by this change?
Answer: As Zuora cannot and does not have access or knowledge of our customers systems, it is important that the customer assess whether their systems are impacted or not by this change. Customer integration and truststore policy along with API integration standard practice is the responsibility of the customer and their security & technology teams to maintain.Thanks! Please let us know if there's any further questions
After review, we understand that the previous posting aluded to "anything".zuora.com, which we recognize could cause some confusion for our customers and we apologize for the ambiguity in that language. We consider all maintenances related to this point that were scheduled for this time period to be complete and that these additional endpoints will be scheduled for a follow-up update in December and we will like those details here shortly.
The certificates for the following endpoints will be replaced in a subsequent maintenance to address the certificate expiration in the December timeframe:
As per this, Zuora is planning to update symantec to Comodo certs for all US Data centers [Prod/Sandbox/Services Environment]
but the link which you mentioned clearly indicate only for Services Environment.
Please update the link article with latest certs or clarify here.
This is some confusing information, and I have a few questions:
We're pointing to the NA sandbox environment, but not using the "sandbox.na.zuora.com" endpoint. If we don't point specifically to those hosts, (i.e. we use the "apisandbox-api.zuora.com" endpoint) will our integration be affected by the change on September 3rd, September 5th, or September 12th?
The "production" date is listed as September 12, but it includes the api sandbox ("rest.apisandbox.zuora.com") AND the production ("rest.zuora.com") endpoints. But there is also a "production date" which lists the same "rest.zuora.com" endpoint for September 5th. If I'm reading this correctly, the sandbox environment is going to get altered either after, or at the same time as, the production environment? It's impossible to tell which date applies to which environment using the schedule as posted.
Also, September 3rd is Labor Day, and there will be massive problems with scheduling changes of this nature on that day as most of the workforce will, I imagine, be taking the day off.
Please, give some clarity to this scheule as soon as possible, as we will need to make some emergency changes to the system before next week's scheduled changes, and we need to start working on them RIGHT NOW.
Also in the future please give more time in advance of an important change like this. Posting after COB (EDT) on Friday less than a week before the first scheduled changes is not enough time in which to operate.
Can this be moved out a week? We're going into a US holiday weekend and this will not give us enough time for this to bake in QA and allow us time to fully test.
We have reviewed internally and can relay the following:
Regarding PT1 not sending a full chain, this is an oversight on our part we will correct in the near future with a separate PT1 deployment, likely early next year.
For production certificates, I can confirm we send the full chain. You can see this running SSL Labs against one of our production-staging endpoints in Akamai. URL: https://www.ssllabs.com/ssltest/analyze.html?d=e6853.ce.akamaiedge-staging.net. Please recognize that this test is not testing the correct SNI in this test, subsequently resulting in a certificate mismatch error. Ignoring the mismatch, you can see we are presenting the full chain in staging and the same will apply once it's deployed to produciton on the 12th.
We are using apisandbox.zuora.com URLs. Do we need the SSL certificate changes to be uptaken ?
@scottb with the phrasing:
[...] we believe that there will be no impact to Zuora partner managed integrations such as Salesforce [...]
are you sure or not? Do we need to test something? A bit to week phrasing in my opinion.
We have tested and found no impact to API integrations such as Salesforce and other systems where Zuora specifically maintains the API call integration. The truth is that historacally, the majority of SSL Certificate updates go unnoticed by most integrations. Where it gets tricky is when a customer elects to use a trust store in their API integration, of if the code being used requires it.
Hi EveryoneAfter review with our technical team, we believe you can test against the following Zuora API endpoint since the new certificates are currently present:rest.apisandbox.zuora.com A successful test from your integration that does not result in a SSL handshake error confirms your integration is capable of making the connection without updates on your side, or if it connects when you have updated your keystore. In addition, we believe that there will be no impact to managed Zuora partner integrations such as Salesforce and Netsuite/Boomi which use Zuora API endpoints
We are reviewing this issue with our Engineering team and expect to have a response shortly.
The notification specifically calls out "US Production Rest and API, Sandbox, and PT1" endpoints by name in the instructions ("*.zuora.com (apisandbox.zuora.com, rest.zuora.com, and api.zuora.com) ", so it seems unambiguous that we should expect the above endpoints to be updated. The amount of angst on this thread was specifically related to the fact that all of the API endpoints in all the environments were changing at once as stated in the announcement, giving no chance to test the certificates before going to production, when in fact it was apparently never the intention to update the production REST or API endpoints at all. We have all been very clear about the issues we've had with this procedure from the start. I don't see how we were the ones who were confused here. If it was known that the production API integration endpoints weren't going to be updated it could have saved a lot of grief to simply say so. Please improve the specificty of communication on upgrades like this going forward?
Let us know if Zuora is seriosuly performing this activity and clarifying all concerns well in advance.
With lack of information and no response on comment section clearly indicate differently.
@scottb- Do you have an update on this? You were going to get back to us after checking with Zuora Engineering? Please confirm on priority so that we can plan activities.
We'll be delaying deployment with the new dates to occur, tentatively, in early October. We'll address everyone's questions when we have more information from our Engineering team and will ensure that the new dates will be provided in a more timely fashion.
Thanks for patience and understanding.
Additional information: A properly configured server will provide all certificates necessary to verify its server certificate, minus the root, during the TLS handshake.
You can verify that the server provides a proper chain using https://www.ssllabs.com/ssltest/?leadsource=3787244
For example, if you take a look at www.zuora.com through the Qualys SSL Server Test. You will see that the server provides all of its intermediate CA certificates. For reference: https://www.ssllabs.com/ssltest/analyze.html?d=www.zuora.com&latest
So you're postponing the production updates, what about the lower environment updates scheduled over the holiday weekend? Are those postponed as well or not?
Can Zuora please just spin up an instance, ANY instance, of a server using the new certificates so that we can do a keystore check aginst the new certificate chain? It seems like this wouldn't be difficult to do, the server could literally just be an echo service, all we need to do is have something to validate our keystores against so we aren't testing this against live production systems. It's simply not a tolerable situation that we're going to have to ride herd on a production system for 6-8 hours in order to know if it's going to blow up or not.
First and foremost, we need the correct certificate chain attached to this post! Second, it would good to have atleast 2 weeks gaps in between annoucement and actual implementation especially when an long weekend is involved.
So can we have the new comodo certificate chain for all URLs including Sandbox & Production????????
Just to add to the list of clarifications needed -
Does this only impact the API integrations, or does it impact any kind of access, i.e. even end users will need to update certificates on their machines.
@baluanisettiThe maintenace has been completed.
Those of you who may not subscribed to our trust notification site, I would like to take the opportunity to suggest everyone subscribe to both Community notifications in "Release Notifications" (here) and on https://trust.zuora.com to ensure wide notification coverage of potential Zuora issues and maintenance.Regards
Can you please confirm if Zuora is in touch with Dell Boomi and Salesforce for this SSL certificate update on their end as well?
They are your partners, so you should be following up with them for this change.
Is there a SOAP endpoint that we can use for testing the new certificates?
We're still not seeing the changes for `rest.zuora.com` and `api.zuora.com` - openssl is showing the same certificate chains at this point. Is this just taking a long time to propagate?
@btemkoThanks for clarifying. Our Technical team is looking into this.
@scottb@Zuora-CommunityCan we have the full certificate chain for comodo for all URLs? https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/Policies/Full_Certification_Chain does not have comodo certificats for all URLs subject to SSL cert replacement, so request to clear this asap.
You didn't mention the rollout for the EU sandbox?
How are we supposed to see if this is working or not prior to the EU production roll out?
I also feel it necessary to point out that Zuora Moderator Viktor states above that "A properly configured server will provide all certificates necessary to verify its server certificate, minus the root, during the TLS handshake." and provides a tool for us to make this check against servers.
This is the result using that tool for "rest.pt1.zuora.com":
You will notice that it says "This server's certificate chain is incomplete."
We actually had an outage on our UAT system when the certificate upgrade went in because we expected the full chain to be presented as Viktor asserted above and so configured our keystore with the root certificate, expecting that to suffice. Currently the Zuora PT1 environment does not adhere to the Zuora-recommended standard.
Is this something Zuora is going to fix, or should we expect to have to adhere to this, and continue to load the entire certificate chain into our keystores, and to therefore need to update them whenever any certificate in the chain changes? We need to get this clarified before the production certificate updates in December.