Here's a thing I came across today that confused me. openssl s_clientappeared to be claiming that a certificate was expired, when it wasn't.

(For reference, this is “OpenSSL 1.0.1j 15 Oct 2014″ on OSX Yosemite, as installed by Homebrew).

$ openssl s_client -connect ssl.bbc.co.uk:443
…
Verify return code: 20 (unable to get local issuer certificate)

So far so good. Let's see the certificate chain:

$ openssl s_client -connect ssl.bbc.co.uk:443 -showcerts
…
0 s:/C=GB/ST=London/L=London/O=British Broadcasting Corporation/CN=*.bbc.co.uk
  i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA — SHA256 — G2
1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA — SHA256 — G2
  i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA

This chain leads back to the “GlobalSign Root CA” cert, which I've already got downloaded as it happens:

$ openssl s_client -connect ssl.bbc.co.uk:443 -showcerts -CAfile ~/GlobalSignRootCA.pem
…
Verify return code: 10 (certificate has expired)

Expired? Hmm. Let's dig into that.

depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify error:num=10:certificate has expired
notAfter=Jan 28 12:00:00 2014 GMT

Today is December 10th 2014, so yes that's in the past. So which cert is it claiming is expired?

Let's inspect each cert in turn using openssl x509 -noout -subject -issuer -startdate -enddate — the first two using the output ofs_client, and the last using the cert I already have a local copy of:

subject= /C=GB/ST=London/L=London/O=British Broadcasting Corporation/CN=*.bbc.co.uk
issuer= /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA — SHA256 — G2
notBefore=Jun 2 09:56:02 2014 GMT
notAfter=Aug 19 13:50:57 2015 GMT

subject= /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA — SHA256 — G2
issuer= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
notBefore=Feb 20 10:00:00 2014 GMT
notAfter=Feb 20 10:00:00 2024 GMT

subject= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
issuer= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
notBefore=Sep 1 12:00:00 1998 GMT
notAfter=Jan 28 12:00:00 2028 GMT

None of those are expired. None of those have an expiry date of “Jan 28 12:00:00 2014 GMT”, as claimed by s_client.

Several hours and some digging into the openssl source code later, I think that what's going on is this:

In certain circumstances (but not always), openssl will try to perform certificate verification. For example, when you specify the -CApath and/or -CAfile options to s_client.

When you do this, openssl can also load the certificates given by $SSL_CERT_FILE (default: OPENSSLDIR + “/cert.pem”, which for me means “/usr/local/etc/openssl/cert.pem”); and I think also those given by $SSL_CERT_DIR (default: OPENSSLDIR + “/certs”, which for me means “/usr/local/etc/openssl/certs”).

Therefore, even though I've explicitly told openssl only one extra cert to use, it's also using the certs in the default cert.pem file.

So what's in that file? For me, 229 certs, that's what. Including this one:

subject= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
issuer= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
notBefore=Sep 1 12:00:00 1998 GMT
notAfter=Jan 28 12:00:00 2014 GMT

The GlobalSign Root CA, which expired back in January. Bingo.

As a workaround, we can use the SSL_CERT_FILE environment variable to suppress loading of the default cert bundle:

$ env SSL_CERT_FILE=/dev/null openssl s_client -connect ssl.bbc.co.uk:443 \
  -CAfile ~/GlobalSignRootCA.pem
…
Verify return code: 0 (ok)