SSL Certificate Chain differs; how to verify?

Short version: I'm seeing an SSL certificate chain that's different based on how I access the https server. What's going on, and how do I verify the certificate under these circumstances?

Slightly longer version:

I'm trying to use libcurl to verify the certificate of an SSL connection. The server I'm connecting to is Amazon S3.

When I access Amazon S3 in Firefox, I get this certificate chain:

VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A
- VeriSign Class 3 Secure Server CA - G3
  Serial Number: 6E:CC:7A:A5:A7:03:20:09:B8:CE:BC:F4:E9:52:D4:91
  - *.s3.amazonaws.com
    Serial Number: 43:FB:BA:C2:66:27:E0:97:1E:1C:11:E0:30:C3:6B:66

When I access the same URL via the OpenSSL command line tool, I get this certificate chain:

*.s3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66
VeriSign Class 3 Secure Server CA - G3
Serial number: 6ecc7aa5a7032009b8cebcf4e952d491
VeriSign Class 3 Public Primary Certification Authority - G5
Serial number: 35973187f3873a07327ece580c9b7eda

The "*.s3.amazonaws.com" and "VeriSign Class 3 Secure Server CA - G3" certificates appear to be the same, but the one after that is different! It's named "VeriSign Class 3 Public Primary Certification Authority - G5" in both chains, but the certificate serial number is different. (Other information is different as well; let me know if you want a longer dump.)

I believe that this difference is the reason that I can't get libcurl to verify the SSL certificate. The certificate with serial number "18:DA:D1..." is in my CACERT.PEM file, but the certificate with serial number "35:97:31..." is not.

Obviously, the simple fix would be to add certificate "35:97:31..." to my CACERT.PEM file, but I want to make the right change here, not just a quick fix.

What does this difference in certificate chains signify?
Is it even possible for a SSL server to return different certificate chains based on the client (Firefox vs. OpenSSL/libcurl)?
How should I get libcurl to verify this SSL certificate?

Really long version and background information:

I'm using libcurl with OpenSSL to download from Amazon S3. Libcurl is returning "SSL certificate problem: unable to get local issuer certificate", which I know means that the root certificate isn't listed in my CACERT.PEM file. (I'm using the one downloaded from curl's website, which is converted from Mozilla's certificate store.) I am able to verify certificates on other SSL connections, so I know my libcurl setup is correct.

To see what was going on, and why the certificate didn't verify, I pulled up the same URL in Firefox. Firefox didn't show any SSL certificate warnings. Here's the certification path Firefox shows. The root certificate, "VeriSign Class 3 Public Primary Certification Authority - G5", is listed in my CACERT.PEM file, and the serial number of the certificate matches what's shown in the screenshot.

Here's the serial numbers of all three certificates in the chain:

VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18:DA:D1:9E:26:7D:E8:BB:4A:21:58:CD:CC:6B:3B:4A
- VeriSign Class 3 Secure Server CA - G3
  Serial Number: 6E:CC:7A:A5:A7:03:20:09:B8:CE:BC:F4:E9:52:D4:91
  - *.s3.amazonaws.com
    Serial Number: 43:FB:BA:C2:66:27:E0:97:1E:1C:11:E0:30:C3:6B:66

On a different platform (different OS, different version of OpenSSL, etc.), I tried accessing the same URL using the OpenSSL command line tool, and I got a different certification path!

$ openssl s_client -showcerts -connect stackoverflowtest.s3.amazonaws.com:443
CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority
verify return:1
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = Terms of use at https://www.verisign.com/rpa (c)10, CN = VeriSign Class 3 Secure Server CA - G3
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = Amazon.com Inc., OU = S3-A, CN = *.s3.amazonaws.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
-----BEGIN CERTIFICATE-----
MIIFOTCCBCGgAwIBAgIQQ/u6wmYn4JceHBHgMMNrZjANBgkqhkiG9w0BAQUFADCB
tTELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMm
VmVyaVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwHhcNMTQwNjI1
MDAwMDAwWhcNMTUwNjA1MjM1OTU5WjB6MQswCQYDVQQGEwJVUzETMBEGA1UECBMK
V2FzaGluZ3RvbjEQMA4GA1UEBxQHU2VhdHRsZTEYMBYGA1UEChQPQW1hem9uLmNv
bSBJbmMuMQ0wCwYDVQQLFARTMy1BMRswGQYDVQQDFBIqLnMzLmFtYXpvbmF3cy5j
b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+mjGH4PrN0TDNmTsF
my2fHpHpTuySRwBMe/nuGIkL3cvKIxFKLHvkK9kx1UpO0skdQTdCY55LywhubLNO
fD19IzJdoRGdlqgkDAYC8vz3LRYj8WWsGnROfS/YFtgj25YaHPnsNp6lWrff4/qi
ctbojJpMxm+9Q0A4nTzrZymHEUkRbx6AVVUBVKH3uZi/w0aV+i4cp2bs+CYIK3DL
Qp584DJ9bOImgUhDfz19+Wtv64zIezE0Uz9eOkqgQ1X//XumyZWyD6N6+h/XqTnc
YTvIer/s83T/IngGMbfPRqjpQCay6ySXCNbJ5izMgo+gwN84t7JhaI+EYcxf1dDN
w0mXAgMBAAGjggF9MIIBeTAJBgNVHRMEAjAAMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwZQYDVR0gBF4wXDBaBgpghkgBhvhF
AQc2MEwwIwYIKwYBBQUHAgEWF2h0dHBzOi8vZC5zeW1jYi5jb20vY3BzMCUGCCsG
AQUFBwICMBkaF2h0dHBzOi8vZC5zeW1jYi5jb20vcnBhMB8GA1UdIwQYMBaAFA1E
XBZTRMGCfh0gqyX0AWPYvnmlMCsGA1UdHwQkMCIwIKAeoByGGmh0dHA6Ly9zZC5z
eW1jYi5jb20vc2QuY3JsMFcGCCsGAQUFBwEBBEswSTAfBggrBgEFBQcwAYYTaHR0
cDovL3NkLnN5bWNkLmNvbTAmBggrBgEFBQcwAoYaaHR0cDovL3NkLnN5bWNiLmNv
bS9zZC5jcnQwLwYDVR0RBCgwJoISKi5zMy5hbWF6b25hd3MuY29tghBzMy5hbWF6
b25hd3MuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQBlXrn1FTPjVIFbcuQNbBesrAMI
NV4L7jS1mobEwFrb7UrqZ7kHvuvoR/BpDygATyqLvPihs7nUc2TUHsw/41EAHKoq
QBVfRTOH0yWaTC6SYSx7fiElL+k55Pvrz4+7gLRy5zUlVX3iUw93zr95ka/LPuCL
7PQOFPeQDOgveDjcSNVtLcTfQfvog/rMSu/4XPFHu7zaZwUEurt9CzLeVbdB6O25
bHuHTaZLP0wmjCIbwgXu8bFWqOTnAjG70EtYrbIiQhl/ISJU6HioFzLiy5Ibp07r
RbV4ir5EI2EPKxIy30YDvpCQ0WQvYLWFV0qQOuOXkMC2M2IsBmVn2/9GQ7eP
-----END CERTIFICATE-----
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
-----BEGIN CERTIFICATE-----
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp
U2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW
ZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0
aG9yaXR5IC0gRzUwHhcNMTAwMjA4MDAwMDAwWhcNMjAwMjA3MjM1OTU5WjCBtTEL
MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW
ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQg
aHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMmVmVy
aVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCxh4QfwgxF9byrJZenraI+nLr2wTm4i8rCrFbG
5btljkRPTc5v7QlK1K9OEJxoiy6Ve4mbE8riNDTB81vzSXtig0iBdNGIeGwCU/m8
f0MmV1gzgzszChew0E6RJK2GfWQS3HRKNKEdCuqWHQsV/KNLO85jiND4LQyUhhDK
tpo9yus3nABINYYpUHjoRWPNGUFP9ZXse5jUxHGzUL4os4+guVOc9cosI6n9FAbo
GLSa6Dxugf3kzTU2s1HTaewSulZub5tXxYsU5w7HnO1KVGrJTcW/EbGuHGeBy0RV
M5l/JJs/U0V/hhrzPPptf4H1uErT9YU3HLWm0AnkGHs4TvoPAgMBAAGjggHfMIIB
2zA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnZlcmlz
aWduLmNvbTASBgNVHRMBAf8ECDAGAQH/AgEAMHAGA1UdIARpMGcwZQYLYIZIAYb4
RQEHFwMwVjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL2Nw
czAqBggrBgEFBQcCAjAeGhxodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhMDQG
A1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMtZzUu
Y3JsMA4GA1UdDwEB/wQEAwIBBjBtBggrBgEFBQcBDARhMF+hXaBbMFkwVzBVFglp
bWFnZS9naWYwITAfMAcGBSsOAwIaBBSP5dMahqyNjmvDz4Bq1EgYLHsZLjAlFiNo
dHRwOi8vbG9nby52ZXJpc2lnbi5jb20vdnNsb2dvLmdpZjAoBgNVHREEITAfpB0w
GzEZMBcGA1UEAxMQVmVyaVNpZ25NUEtJLTItNjAdBgNVHQ4EFgQUDURcFlNEwYJ+
HSCrJfQBY9i+eaUwHwYDVR0jBBgwFoAUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwDQYJ
KoZIhvcNAQEFBQADggEBAAyDJO/dwwzZWJz+NrbrioBL0aP3nfPMU++CnqOh5pfB
WJ11bOAdG0z60cEtBcDqbrIicFXZIDNAMwfCZYP6j0M3m+oOmmxw7vacgDvZN/R6
bezQGH1JSsqZxxkoor7YdyT3hSaGbYcFQEFn0Sc67dxIHSLNCwuLvPSxe/20majp
dirhGi2HbnTTiN0eIsbfFrYrghQKlFzyUOyvzv9iNw2tZdMGQVPtAhTItVgooazg
W+yzf5VK+wPIrSbb5mZ4EkrZn0L74ZjmQoObj49nJOhhGbXdzbULJgWOw27EyHW4
Rs/iGAZeqa6ogZpHFt4MKGwlJ7net4RYxh84HqTEy2Y=
-----END CERTIFICATE-----
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
-----BEGIN CERTIFICATE-----
MIIExjCCBC+gAwIBAgIQNZcxh/OHOgcyfs5YDJt+2jANBgkqhkiG9w0BAQUFADBf
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsT
LkNsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkw
HhcNMDYxMTA4MDAwMDAwWhcNMjExMTA3MjM1OTU5WjCByjELMAkGA1UEBhMCVVMx
FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVz
dCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2lnbiwgSW5jLiAtIEZv
ciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJpU2lnbiBDbGFzcyAz
IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzUwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1nmAMqudLO07cfLw8
RRy7K+D+KQL5VwijZIUVJ/XxrcgxiV0i6CqqpkKzj/i5Vbext0uz/o9+B1fs70Pb
ZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6/WhkcIzSdhDY2pSS9KP6HBR
TdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQGBO+QueQA5N06tRn/
Arr0PO7gi+s3i+z016zy9vA9r911kTMZHRxAy3QkGSGT2RT+rCpSx4/VBEnkjWNH
iDxpg8v+R70rfk/Fla4OndTRQ8Bnc+MUCH7lP59zuDMKz10/NIeWiu5T6CUVAgMB
AAGjggGRMIIBjTAPBgNVHRMBAf8EBTADAQH/MDEGA1UdHwQqMCgwJqAkoCKGIGh0
dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMuY3JsMA4GA1UdDwEB/wQEAwIBBjA9
BgNVHSAENjA0MDIGBFUdIAAwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVy
aXNpZ24uY29tL2NwczAdBgNVHQ4EFgQUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwNAYD
VR0lBC0wKwYJYIZIAYb4QgQBBgpghkgBhvhFAQgBBggrBgEFBQcDAQYIKwYBBQUH
AwIwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAHBgUr
DgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVyaXNp
Z24uY29tL3ZzbG9nby5naWYwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhho
dHRwOi8vb2NzcC52ZXJpc2lnbi5jb20wDQYJKoZIhvcNAQEFBQADgYEADyWuSO0b
M4VMDLXC1/5N1oMoTEFlYAALd0hxgv5/21oOIMzS6ke8ZEJhRDR0MIGBJopK90Rd
fjSAqLiD4gnXbSPdie0oCL1jWhFXCMSe2uJoKK/dUDzsgiHYAMJVRFBwQa2DF3m6
CPMr3u00HUSe0gST9MsFFy0JLS1j7/YmC3s=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com
issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
---
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4624 bytes and written 399 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA
    Session-ID: B642ED1E6FE32F7A374B5A62847DEC63C2F37DCA7A18FD669B8F0FCDC98C49BF
    Session-ID-ctx:
    Master-Key: EE2D31F43D341A0895B36E0BCCE7557B221F1469AC1B7B0BA22D843C75F25F949822B0D0E22E967A1F373F034E9624E4
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1430871883
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

When I decode the certificates it lists, I get a different certificate chain than Firefox gave me.

*.s3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66
VeriSign Class 3 Secure Server CA - G3
Serial number: 6ecc7aa5a7032009b8cebcf4e952d491
VeriSign Class 3 Public Primary Certification Authority - G5
Serial number: 35973187f3873a07327ece580c9b7eda

The cert in use, and its immediate parent cert are the same, but the next one up has the same name, but a different serial number.

Here's my version information:

> curld --version
curl 7.40.0 (i386-pc-win32ce) libcurl/7.40.0 OpenSSL/1.0.1e
Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp
Features: NTLM SSL

> curld --version
curl 7.40.0 (i386-pc-win32) libcurl/7.40.0 OpenSSL/1.0.1c
Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp
Features: AsynchDNS Largefile NTLM SSL

$ cat /etc/redhat-release
CentOS release 6.6 (Final)
$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
$ yum list openssl
Installed Packages
openssl.x86_64               1.0.1e-30.el6.8               @updates

Firefox: 37.0.2, running on Windows 7 x64.

I did review this question: SSL Certificate - Certification Path in browser different from Certificate Chain File, but my problem appears to be different: In that case, the certificate chain in the OpenSSL command line tool went 1 - 2 - 3 - 4, and in IE it was 1 - 2 - 3. Because IE considered "3" to be a root certificate, the chain stopped early. In my case, Firefox is reporting 1 - 2 - 3, and OpenSSL is reporting 1 - 2 - 4; The chain is different.

The server sends the same chain certificates to firefox and s_client:

CN=.s3.amazonaws.com SAN=DNS:.s3.amazonaws.com,DNS:s3.amazonaws.com
CN=VeriSign Class 3 Secure Server CA - G3
CN=VeriSign Class 3 Public Primary Certification Authority - G5

But the way the certificates will be verified differs depending on the SSL stack and the trusted root certificates of the client. And in case of curl you run into an old OpenSSL validation problem. Details:

Firefox has a trusted root certificate similar to the certificate#3 send by the client. This means that it is a different certificate but it contains the same public key, so that the signature for certificate#2 is still valid. Since Firefox underlying TLS stack (NSS) thus has found a usable trust anchor it will consider the chain as verified and ignore the certificate#3 sent by the server.
But the version of curl you have uses OpenSSL as the TLS library. OpenSSL tries to get the longest match, that it it will try to find a trusted root certificate which signed certificate#3. If it fails it will not try with a shorter trust chain but simply fail. This is a long-standing bug which is the cause of many many strange problems like this and it looks like the issue is for now only fixed in the latest development branch (not released).

The solution is probably to use a CA store for curl which still includes the old deprecated 1024bit CA's so that it contains the trust anchor "/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority"

For even more details about this problem see

more detailed writeup
OpenSSL bug #2732 from 2012
workarounds with patch
newer Bug reports which indicate a bug fix for current development branch (after 1.0.2)