Removed SSL certificates causing errors in WP 4.4

[DvanKooten]

In #30434, a bunch of certificates were removed from the bundled root certificate file.

Unfortunately, it seems this is forcing plugin author to disable sslverify again as certain versions of CURL / OpenSSL are not liking the change.

I'm still working out the exact configuration that's causing the issue to pop-up but all I know is that disabling sslverify in WP_HTTP wasn't necessary for WordPress 3.7 up to WordPress 4.3.x.

User-submitted error reports:
​https://wordpress.org/support/topic/error-connecting-to-mailchimp-ssl-certificate-problem?replies=8
​http://myonlinesecurity.co.uk/wordpress-4-4-update-breaks-itself-with-ssl-certificate-problem-unable-to-get-local-issuer-certificate/
​https://github.com/ibericode/mailchimp-for-wordpress/issues/219

[javorszky]

We get a number of support tickets for this. Will post details as they come in.

Happens on customer's site:

Customer 1:

PHP 5.4.45
cURL 7.38.0
OpenSSL 1.0.1e
Library and Header versions: OpenSSL 1.0.1e-fips 11 Feb 2013

Customer 2:

PHP 5.4.40
cURL 7.38.0
OpenSSL 1.0.1e
Library and Header versions: OpenSSL 1.0.1e-fips 11 Feb 2013

I've asked them to update OpenSSL to f+, will report back as soon as that's done.

Works on own server:

PHP  5.5.9-1ubuntu4.14
cURL 7.35.0
OpenSSL 1.0.1f
Library and Header versions: OpenSSL 1.0.1f 6 Jan 2014

Answer from one host about updating:

We are checking but it might be harder than it sounds. That is tied to the OS distribution so we are checking the version we are using now. We have a pretty standard distribution.. Will see what we can do.

Also, unless I'm doing something monumentally wrong, neither of these have helped:

• downloading the ca certs from haxx.se and using those instead (I did clear the caches of WP Rocket)
• setting sslverify to false or removing it from the args

Edit 1: added more customer environment data
Edit 2: added answer from host
Edit 3: added info about certs + sslverify
Edit 4: added php version

[dd32]

I feared this would come up again.. Unfortunately the curl cacert bundles are seemingly sometimes incompatible with older versions of curl for an unknown reason.. It's a bug deep within curl (possibly fixed at some point) which I don't think we can easily work around.

Can everyone please include details of the curl version, openssl version (or whatever ssl library curl is using) AND the php version affected.

[dd32]

And to clarify, I don't think it's the removal, rather the adding or reordering of certs. We had this issue before, and were able to fix it by moving one certificate to the start of the file, it's trial and error to find which one though.

[claudiosanches]

I see many people complaining about it in forums of my plugins... But everyone ends up solving by updating or installing again cURL and OpenSSL.

[javorszky]

Not sure if it notifies on comment edit. I've added PHP versions.

[DvanKooten]

Some of ​my users are posting server details in this support topic on w.org

PHP: 5.5 (cPanel)
cURL: 7.19.7
OpenSSL: 1.0.1e

Apache: 2.4.12 (Win32)
PHP:	5.6.8
MySQL: 5.6.24
cURL: 7.40.0
OpenSSL: 1.0.1l

​PHP: 5.3.8 CentOS release 5.5 (Final)
​CURL: 7.15.5
​OpenSSL:  OpenSSL/0.9.8b

Older versions of OpenSSL seem to be the only consistent thing, as CURL versions are all over the place (from 7.19.x up to 7.40.x) as are PHP versions.

[rmccue]

I think this might be ​https://github.com/openssl/openssl/commit/6c03af135b285429a71ab3dac953ad9a70d8a1ac but not sure yet. Will try and debug ASAP.

[rmccue]

Managed to replicate using OpenSSL only from the command line. Appears to be broken in 1.0.1e and fixed in 1.0.1q, so some git bisect magic lead me to ​this commit. This then lead me on to ​this ticket; ​this comment goes into specifics about it: ​(Use guest/guest as username/password)

Recently, Mozilla has started to cleanup the Mozilla CA trust list and
remove CA certificates that use a weaker 1024-bit RSA key. I'll call
them legacy CAs.

When upgrading the trust store used by openssl to exclude the legacy CA
certificates, by default, openssl (e.g. s_client) can no longer verify
the server certificate of several popular SSL/TLS servers, examples are
www.flickr.com and www.amazon.com.

The cURL page for the certificate bundle also mentions this:

RSA-1024 removed

Around early September 2014, Mozilla removed the trust bits from the certs in their CA bundle that were still using RSA 1024 bit keys. This may lead to TLS libraries having a hard time to verify some sites if the library in question doesn't properly support "path discovery" as per RFC 4158. (That includes OpenSSL and GnuTLS.)

​This file is linked as the last version built from the NSS store before they started removing these certs.

These certificates have been removed for being 1024-bit:

• GTE CyberTrust Global Root
• Thawte Server CA
• Thawte Premium Server CA
• Verisign Class 3 Public Primary Certification Authority
• Verisign Class 3 Public Primary Certification Authority - G2
• ValiCert Class 1 VA
• ValiCert Class 2 VA
• ValiCert Class 3 VA (incorrectly called ​RSA Root Certificate 1)
• Entrust.net Secure Server CA
• Equifax Secure Global eBusiness CA
• Equifax Secure eBusiness CA 1
• NetLock Business (Class B) Root
• NetLock Express (Class C) Root
• Verisign Class 3 Public Primary Certification Authority

(Importantly, note that "Verisign Class 3 Public Primary Certification Authority - G2" is the current root certificate for api.paypal.com.)

After some more checking, the removed ones are all 1024 bit, with the exception of the following (links to reasoning for removal):

• ​RSA Root Certificate 1
• ​America Online Root Certification Authority 1
• ​America Online Root Certification Authority 2
• ​TDC Internet Root CA
• ​AC Raíz Certicámara S.A.
• ​TC TrustCenter Class 3 CA II
• ​E-Guven Kok Elektronik Sertifika Hizmet Saglayicisi

Therefore: I believe we should be able to take our existing bundle, and pull the 1024 bit certificates back in.

Attaching patched version of the CA bundle that adds the 1024 bit certificates back; this fixes resolution for me via OpenSSL on the command line, but needs testing on a site that's broken.

[rmccue]

Certificate bundle with 1024-bit root certificates re-added

[javorszky]

Feedback from one customer who was affected:

Me: Alternatively, you can replace the wp-includes/certificates/ca-bundle.crt file with the one I attached. Please make a copy of the original one first. If you're on 4.4, this file should fix the issue. Let me know whether it does or doesn't.

Customer: OK, so that fixed the SSL error.

Another feedback:

Good news, i have updated to wo 4.4 applied the new certificate and all working now.

Exact steps i followed were:

Deactivated all plugins
Performed the wp automated updated.
Then replaced the ca-bundle.crt with the new one u gave
Next activated woo commerce
Made a dummy purchase -- All worked
Next activated subscriptions
Made a dummy purchase -- All worked
Activated all other plugins
Made a dummy purchase -- All worked
Deactivated plugin- Temporary SSLverify Disable for WooCommerce Subscriptions (sets sslverify to false)
Made a dummy purchase -- All worked

So that is very good news indeed - thanks for your help in solving that

[Kent Brockman]

Hello guys. I'm seeing this issue when trying to receive IPN feedback from PayPal in a WP 4.4 running on PHP 5.5. Do you have any ca-bundle.crt file I can betatest please?

[javorszky]

Replying to Kent Brockman:

Hello guys. I'm seeing this issue when trying to receive IPN feedback from PayPal in a WP 4.4 running on PHP 5.5. Do you have any ca-bundle.crt file I can betatest please?

Hi Kent,

@rmccue's fix worked for a customer for the same issue. You'd need to check out the wordpress development svn branch and apply the patch.

[WilltheWebMechanic]

ca-bundle.crt resolved the issue here:

cURL 7.19.7
OpenSSL 1.0.1e-fips 11 Feb 2013

and here:
cURL 7.24.0
OpenSSL 1.0.1e-fips 11 Feb 2013

[Kent Brockman]

Thanks! For those of us not using git nor svn and that could land here, you can use the following file as ca-bundle.crt and I can confirm my problems were solved after using it:
​https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt

Hope it helps others. Best regards

[toddlahman]

Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release. Using the test results to allow WordPress to detect incompatible versions of cURL and OpenSSL on the server, and warn the WordPress admin to have their host update cURL or OpenSSL, would provide admins actionable feedback, so they can solve their own issue, even if the ca-bundle.crt isn't perfect.

[javorszky]

Replying to toddlahman:

Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release. Using the test results to allow WordPress to detect incompatible versions of cURL and OpenSSL on the server, and warn the WordPress admin to have their host update cURL or OpenSSL, would provide admins actionable feedback, so they can solve their own issue, even if the ca-bundle.crt isn't perfect.

Following feedback from users who had this problem and we asked them to update openssl and / or curl, they came back with that being fairly hard, as they're part of the server distribution they used.

One of them provided feedback from the host: they updated to 1.0.2, but php did not pick that change up. It's a lot easier to bundle a working ca-bundle.crt than asking hosts to update their openssl.

[Kent Brockman]

Absolutelly agree with @javorszky: people using shared hosting will be trapped in a cage because vast majority of hosting providers are not able/willing to upgrade their OS or templates due to just a few shared hosting customer complaints.

A well implemented ca-bundle.crt file demonstrated to do the magic. The tricky part is... keeping that .crt file up to date, given the lots of updates that may arise. All in all, is a foolproof solution. I bet that monitoring changes in a couple of git ca bundles (like this one: ​https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt ) will be enough to keep up with it.

[sneader]

Hosting provider input here: Very standard cPanel shared hosting environment, running CentOS 6.7 OS with yum updates, as well as cPanel updates, every night. OpenSSL reports as version "1.0.1e-fips" but has backported patches (the most recent being June 23, 2015).

We had one customer running WooCommerce get bit by this ca-bundle issue. Specifically, a paid plugin for the Elavon payment gateway (purchased from WooCommerce, now owned by Automattic) generating 500 errors. WooCommerce tech support blamed our server configuration and suggested we "reinstall OpenSSL and Curl". I did not do that... however, after I updated ca-bundle.crt, the problem was immediately corrected.

If Automattic thinks this issue is a shared-hosting provider problem, then they should immediately contact cPanel tech support. There are likely hundreds of thousands of web servers set up exactly like ours, in this very standard shared hosting configuration.

[rmccue]

Replying to toddlahman:

Adding automated testing of different versions of cURL and OpenSSL with WordPress using HTTPS to connect to remote URLs, like an API, would help avoid these issues in the next release.

It is a huge amount of effort to test different cURL and OpenSSL versions (as various versions need to be used); finding and verifying this bug took me ~4 hours, even using git bisect. Thankfully, this issue turned out to be an easy one caused by OpenSSL itself, which meant I only needed to rebuild OpenSSL and use the command line client openssl s_client; if this was caused by OpenSSL and cURL (or worse, PHP cURL) having issues together, it could have taken days just to find the issue.

Testing every version of cURL and OpenSSL that's available isn't something we can do, which is why it's important for people to test release candidates.

Replying to Kent Brockman:

A well implemented ca-bundle.crt file demonstrated to do the magic. The tricky part is... keeping that .crt file up to date, given the lots of updates that may arise. All in all, is a foolproof solution. I bet that monitoring changes in a couple of git ca bundles (like this one: ​https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt ) will be enough to keep up with it.

We do monitor changes in the bundle, which is what caused this issue in the first place. The key of the issue is that Mozilla has decided to remove 1024 bit root certificates from their root (that bundle file is generated from it). Mozilla's software and infrastructure uses NSS (a different SSL library) that can handle alternate chains; OpenSSL 1.0.1l was the first version to introduce this to OpenSSL.

Monitoring the upstream changes isn't enough when we also need to have compatibility with years old versions of OpenSSL.

Replying to sneader:

Hosting provider input here: Very standard cPanel shared hosting environment, running CentOS 6.7 OS with yum updates, as well as cPanel updates, every night. OpenSSL reports as version "1.0.1e-fips" but has backported patches (the most recent being June 23, 2015).

It may be worth reporting this to CentOS, since 1024 bit root certificates are going away eventually. This is going to break certificates that rely on alternate chains. We will endeavour to keep supporting this, but I wouldn't be surprised when other software starts breaking too.

---

Given that this patch is working, recommending for commit in 4.4.1.

[Kent Brockman]

By reading about the OpenSSL versions included for CentOS (​http://rpmfind.net/linux/rpm2html/search.php?query=openssl), I see 1.0.1e is even the last version on CentOS 7.1. So, even cPanel installs using their recent support for CentOS 7 (​https://features.cpanel.net/topic/rhel-7-centos-7-support) will have this issue too, right?

[dd32]

Replying to rmccue:

Therefore: I believe we should be able to take our existing bundle, and pull the 1024 bit certificates back in.

Attaching patched version of the CA bundle that adds the 1024 bit certificates back; this fixes resolution for me via OpenSSL on the command line, but needs testing on a site that's broken.

I agree with this direction. With an explicit mention in the commit (and file if possible) specifically explaining why we're including no-longer-trusted-by-browser roots.

If there's any way of automatically generating the crt file for future maintenance that would also be grand. I'm thinking that even just pulling the latest NSS store + suffixing the 1024bit certs may be enough. Also worth noting is that we now need to monitor the status of these root certs, we've relied upon NSS to do that in the past, but since they no longer trust the certs, if one of them is compromised we'll need to be aware of it somehow.

[dd32]

In 35919:

HTTP: Partially revert [34283] which removed the 1024bit certificates from our trust store.

Most browsers no longer trust 1024bit certificates, or certificates signed by them, instead verifying them by a trusted intermediate or a cross-sign from another trusted certificate.

Unfortunately, as it turns out, OpenSSL prior to 1.0.1g cannot correctly handle certificates chains such as this, even if one of the intermediates is trusted.
The solution is that we need to continue to trust the 1024bit legacy root certificates forthe foreseeable future

This adds the following certificates back into our trust store:

GTE CyberTrust Global Root
Thawte Server CA
Thawte Premium Server CA
Verisign Class 3 Public Primary Certification Authority
Verisign Class 3 Public Primary Certification Authority - G2
ValiCert Class 1 VA
ValiCert Class 2 VA
RSA Root Certificate 1
Entrust.net Secure Server CA
Equifax Secure Global eBusiness CA
Equifax Secure eBusiness CA 1
America Online Root Certification Authority 1
America Online Root Certification Authority 2
NetLock Business (Class B) Root
NetLock Express (Class C) Root
Verisign Class 3 Public Primary Certification Authority

Props rmccue
Fixes #34935 for trunk.

[dd32]

In 35921:

HTTP: Partially revert [34283] which removed the 1024bit certificates from our trust store.

Most browsers no longer trust 1024bit certificates, or certificates signed by them, instead verifying them by a trusted intermediate or a cross-sign from another trusted certificate.

Unfortunately, as it turns out, OpenSSL prior to 1.0.1g cannot correctly handle certificates chains such as this, even if one of the intermediates is trusted.
The solution is that we need to continue to trust the 1024bit legacy root certificates forthe foreseeable future

This adds the following certificates back into our trust store:

GTE CyberTrust Global Root
Thawte Server CA
Thawte Premium Server CA
Verisign Class 3 Public Primary Certification Authority
Verisign Class 3 Public Primary Certification Authority - G2
ValiCert Class 1 VA
ValiCert Class 2 VA
RSA Root Certificate 1
Entrust.net Secure Server CA
Equifax Secure Global eBusiness CA
Equifax Secure eBusiness CA 1
America Online Root Certification Authority 1
America Online Root Certification Authority 2
NetLock Business (Class B) Root
NetLock Express (Class C) Root
Verisign Class 3 Public Primary Certification Authority

Props rmccue.
Merges [35919] to the 4.4 branch.
Fixes #34935.

[garubi]

Got the same errors trying to update some plugins.

I just replaced the ca-bundle.crt with the one provided here and on my server the errors doesn't show up anymore and the plugin updates where all ok.

I can confirm that on my environment this solves the problem.

Here my server's (shared hosting) details:

Server: OS Linux Software Apache Version 32Bit
Name ufficiarredati.it Address 89.31.74.213 Port 80
Type Linux www.sosufficiarredati.it 2.6.24-23-server #1 SMP Wed Apr 1 22:22:14 UTC 2009 i686
System: PHP 5.2.4-2ubuntu5.10 Active Plugins 43 Zend 2.2.0
Database: SQL 5.0.51a Build 5.0.51a-3ubuntu5.4 Charset utf8