Page MenuHomePhabricator

curl https://git.wikimedia.org/ gives invalid certificate
Closed, ResolvedPublic

Description

hashar@integration-selenium-driver:~$ curl https://git.wikimedia.org/
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
hashar@integration-selenium-driver:~$


Version: unspecified
Severity: normal

Details

Reference
bz59910

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:36 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz59910.
bzimport added a subscriber: Unknown Object (MLST).

With curl in verbose mode:

hashar@integration-selenium-driver:~$ curl -v https://git.wikimedia.org/

  • About to connect() to git.wikimedia.org port 443 (#0)
  • Trying 208.80.154.241... connected
  • successfully set certificate verify locations:
  • CAfile: none CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv3, TLS alert, Server hello (2):
  • SSL certificate problem, verify that the CA cert is OK. Details:

error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

  • Closing connection #0

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
hashar@integration-selenium-driver:~$

I guess labs is missing some certificates :(

Change 106771 had a related patch set uploaded by Hashar:
star.wikimedia.org cert chain fix

https://gerrit.wikimedia.org/r/106771

Found out the same issue on production machine lanthanum.eqiad.wmnet. It is lacking the Rapid SSL CA cert:

$ ll /etc/ssl/certs/*apid*
ls: cannot access /etc/ssl/certs/*apid*: No such file or directory
$

No chained either since that machine never had any certificate installed.

According to the history of Jenkins job https://integration.wikimedia.org/ci/job/mwext-browsertests-UniversalLanguageSelector-phantomjs/ (which uses https://git.wikimedia.org/ ). That stopped working between Dec 10 2013 17:15 and Dec 11 2013 13:40UTC.

Damn I was wrong! The job last success was Jan 8th 13:20 , first failure Jan 10th 3:37 UTC.

Trying on lanthanum:

hashar@lanthanum:~$ openssl s_client -connect git.wikimedia.org:443
...
Certificate chain
0 s:/serialNumber=06QcQ9dUSZqu5ru7oQSfeCpXiBccrCyh/C=US/O=*.wikimedia.org/OU=GT11518520/OU=See www.rapidssl.com/resources/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=*.wikimedia.org

i:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA

1 s:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

$

Equifax is wrong, should be Geotrust :(

And on my machine the chain is:

Certificate chain
0 s:/serialNumber=06QcQ9dUSZqu5ru7oQSfeCpXiBccrCyh/C=US/O=*.wikimedia.org/OU=GT11518520/OU=See www.rapidssl.com/resources/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=*.wikimedia.org

i:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA

1 s:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA

i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

(correct)

Change 106785 had a related patch set uploaded by RobH:
fixes star.wikimedia.org intermidite certificate chain

https://gerrit.wikimedia.org/r/106785

Change 106785 merged by RobH:
fixes star.wikimedia.org intermidite certificate chain

https://gerrit.wikimedia.org/r/106785

Change 106842 had a related patch set uploaded by RobH:
install rapidssl_ca_2.pem

https://gerrit.wikimedia.org/r/106842

Change 106842 merged by RobH:
install rapidssl_ca_2.pem

https://gerrit.wikimedia.org/r/106842

Ok, So the two patchsets I submitted for this are linked in ticket here. Turns out the wildcard rapidssl has two intermediate certificates, versus the non-wildcard rapidssl certs that use the single one we've had on cluster and in puppet for awhile.

I've added the second cert to our repo and included it in use for misc-web-lb.eqiad. Now curls return correctly, folks shouldn't be getting certificate errors anymore.