I'm running into a strange problem with certificates that I can't figure out how to debug. When I run wget inside of a docker container on one specific server it cannot verify certificates. The same wget works fine on the server machine itself (outside docker) and it works inside that same docker container on different servers.
Here's the setup for the docker container:
docker run --rm -ti debian:jessie bash
apt-get update
apt-get install wget
wget https://google.com
The response is:
converted 'https://google.com' (ANSI_X3.4-1968) -> 'https://google.com' (UTF-8)
--2016-06-22 14:22:02-- https://google.com/
Resolving google.com (google.com)... 216.58.217.142, 2607:f8b0:4004:807::200e
Connecting to google.com (google.com)|216.58.217.142|:443... connected.
ERROR: The certificate of 'google.com' is not trusted.
ERROR: The certificate of 'google.com' hasn't got a known issuer.
The certificate's owner does not match hostname 'google.com'
Since this same process works on other servers, it seems like the problem could only be some certificate problem on that server itself. But I must be confused: why should the certificates on the server itself have anything to do with what's happening inside of the docker container?
I would really appreciate any insight into this, in particular any debugging steps I can take to understand the problem better.
Docker uses iptables.
If you have iptable rules set up it's possible to direct EVERY https request to your own running server.
If you are, for example, running jenkins locally and using iptables to redirect 443 to default 8080 port than all your container traffic to port 443 ports will be redirected to that local jenkins server which will be unable to verify your certificate.
We ran into this problem when using Jenkins to build our docker images. our jenkins used iptables to get around running jenkins as root.
It seems that the certificates are out of date inside the jessie image.
try apt-get install ca-certificates before the wget
This worked fine for me, though to be safe, make sure your "ca-certificates" package is up to date. Most likely, you have some kind of security device on the network that is inspecting the traffic, and to do so, decrypting and encrypting with it's own certificate. Here's the certificate I get from my own testing:
bash$ openssl s_client -showcerts -connect www.google.com:443
CONNECTED(00000003)
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = www.google.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
-----BEGIN CERTIFICATE-----
MIIEgDCCA2igAwIBAgIIdnEF+1C/AZowDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTYwNjA4MTIzNzI5WhcNMTYwODMxMTIzMDAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxYx0X
auhoQ4cobo6J3UMUNCbzKmJ/XSzDB5RLtjvbvtDfCMHm8hO91vvlcKRRrwqdYpiw
2zUcPDyjwOrZZsJlQglQw/rRpbfQQ6aKsKQWiT3sAIz5joXXi/622YhhGAAdyGGy
/tzsQkW0IqWAIFLFBbHWMvgDmvMwacps34B80U+p5Iq2xx2xHegl0RMb4HfSEpW/
H4A0MKvYR6uZZEEr39E+4R6IY9HSv1ZDq8csspyWjXpaIxd6ZD6+lGwvgyszQtNa
0aEP9+tNhF7jPRD5TedfvyOz81dUCvE0O2E+nfripG2gNBm4r5N6XWUH/lvoopaR
00eE2fKpk/fvZgZXAgMBAAGjggFLMIIBRzAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
A1UdDgQWBBStL+4j1/n+vGwj3sL861LWCYDUGTAMBgNVHRMBAf8EAjAAMB8GA1Ud
IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMCEGA1UdIAQaMBgwDAYKKwYBBAHW
eQIFATAIBgZngQwBAgIwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL3BraS5nb29n
bGUuY29tL0dJQUcyLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAK1BpCyAbID8gpwWI
bQReJv81H/qvYvaaOFa7PLnqHhaAZmzjV1tkCsVB60IgsBDoNuPdtJ5klpxV+njs
VhDnxneaHL21zwUCuZvNVyYL9VCSYGWV1iNe6PtYYtbWt7of6bEiwZsSuPWaRuRp
YcvJH+mpv/EIDw1shU5UK3FpvnSHEH2jrs2psnC4BYSovT3pH2nxTCpiLiya1UNn
+qtqiCJyDKhEV6f1j+Awg/Fr+tVCZLjHcSmGqL3DNcHbCUalXrq4EwVK3Pg8lghU
Dm3e0J3EcjgMacIg+RP+2pOM5GvIwQ6BrKbmcnTjqsjcG1tQEV0ZSb6hx2bGYhc2
3TG+rQ==
-----END CERTIFICATE-----
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
-----BEGIN CERTIFICATE-----
MIID8DCCAtigAwIBAgIDAjqDMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMwNDA1MTUxNTU2WhcNMTYxMjMxMjM1OTU5WjBJMQswCQYDVQQG
EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
ahqyzFPdFUuLH8gZYR/Nnag+YyuENWllhMgZxUYi+FOVvuOAShDGKuy6lyARxzmZ
EASg8GF6lSWMTlJ14rbtCMoU/M4iarNOz0YDl5cDfsCx3nuvRTPPuj5xt970JSXC
DTWJnZ37DhF5iR43xa+OcmkCAwEAAaOB5zCB5DAfBgNVHSMEGDAWgBTAephojYn7
qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUSt0GFhu89mi1dvWBtrtiGrpagS8wDgYD
VR0PAQH/BAQDAgEGMC4GCCsGAQUFBwEBBCIwIDAeBggrBgEFBQcwAYYSaHR0cDov
L2cuc3ltY2QuY29tMBIGA1UdEwEB/wQIMAYBAf8CAQAwNQYDVR0fBC4wLDAqoCig
JoYkaHR0cDovL2cuc3ltY2IuY29tL2NybHMvZ3RnbG9iYWwuY3JsMBcGA1UdIAQQ
MA4wDAYKKwYBBAHWeQIFATANBgkqhkiG9w0BAQsFAAOCAQEAqvqpIM1qZ4PtXtR+
3h3Ef+AlBgDFJPupyC1tft6dgmUsgWM0Zj7pUsIItMsv91+ZOmqcUHqFBYx90SpI
hNMJbHzCzTWf84LuUt5oX+QAihcglvcpjZpNy6jehsgNb1aHA30DP9z6eX0hGfnI
Oi9RdozHQZJxjyXON/hKTAAj78Q1EK7gI4BzfE00LshukNYQHpmEcxpw8u1VDu4X
Bupn7jLrLN1nBz/2i8Jw3lsA5rsb0zYaImxssDVCbJAJPZPpZAkiDoUGn8JzIdPm
X4DkjYUiOnMDsWCOrmji9D6X52ASCWg23jrW4kOVWzeBkoEfu43XrVJkFleW2V40
fsg12A==
-----END CERTIFICATE-----
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
-----BEGIN CERTIFICATE-----
MIIDfTCCAuagAwIBAgIDErvmMA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDIwNTIxMDQwMDAwWhcNMTgwODIxMDQwMDAw
WjBCMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UE
AxMSR2VvVHJ1c3QgR2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA2swYYzD99BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9m
OSm9BXiLnTjoBbdqfnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIu
T8rxh0PBFpVXLVDviS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6c
JmTM386DGXHKTubU1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmR
Cw7+OC7RHQWa9k0+bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5asz
PeE4uwc2hGKceeoWMPRfwCvocWvk+QIDAQABo4HwMIHtMB8GA1UdIwQYMBaAFEjm
aPkr0rKV10fYIyAQTzOYkJ/UMB0GA1UdDgQWBBTAephojYn7qwVkDBF9qn1luMrM
TjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjA6BgNVHR8EMzAxMC+g
LaArhilodHRwOi8vY3JsLmdlb3RydXN0LmNvbS9jcmxzL3NlY3VyZWNhLmNybDBO
BgNVHSAERzBFMEMGBFUdIAAwOzA5BggrBgEFBQcCARYtaHR0cHM6Ly93d3cuZ2Vv
dHJ1c3QuY29tL3Jlc291cmNlcy9yZXBvc2l0b3J5MA0GCSqGSIb3DQEBBQUAA4GB
AHbhEm5OSxYShjAGsoEIz/AIx8dxfmbuwu3UOx//8PDITtZDOLC5MH0Y0FWDomrL
NhGc6Ehmo21/uBPUR/6LWlxz/K7ZGzIZOKuXNBSqltLroxwUCEm2u+WR74M26x1W
b8ravHNjkOR/ez4iyz0H7V84dJzjA1BOoa+Y7mHyhD8S
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 3727 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 09AF6D01D3E3059EA0E4543E880035C34D74CEFCBB9D20F34F8CC1789D2485B2
Session-ID-ctx:
Master-Key: 575CCE0D8562480D591DE3983B2B6709D1FF5F0FCF219FFF66C30B90A5A906E5A8BD6688DED22EDFE6F7DC9702915E5B
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 100800 (seconds)
TLS session ticket:
0000 - 3e 73 9d 09 9a 16 a9 a2-70 64 76 b4 16 b1 ca d0 >s......pdv.....
0010 - 70 37 62 e2 d3 e6 ac b3-31 31 4d 4b 1c 9b 2b 6c p7b.....11MK..+l
0020 - cc 1c 0d 3d ae dc ce c2-d4 36 41 4c 04 54 f0 e3 ...=.....6AL.T..
0030 - 15 03 04 b5 32 0d 8b c0-5b c0 d6 03 8d df d8 bf ....2...[.......
0040 - 74 7c ae ac da 3b 1a 8d-d7 56 3d 3a ee dd 69 d3 t|...;...V=:..i.
0050 - fb 2d 34 4a c4 51 0c e6-39 18 20 f1 cc 5d ab 66 .-4J.Q..9. ..].f
0060 - 9f f9 47 6f b4 09 6f 4f-42 6c 72 42 fd 92 a3 3b ..Go..oOBlrB...;
0070 - 95 3d a1 14 e5 33 b8 b4-8a de 0f f4 4b b6 08 2b .=...3......K..+
0080 - bb f6 18 3c 51 90 c8 ce-8c 9d 84 37 de be 07 72 ...<Q......7...r
0090 - 5d 5a fa 6a 28 70 95 29-28 5e 0d 26 0f 59 c7 d2 ]Z.j(p.)(^.&.Y..
00a0 - b5 86 1e 99 ....
Start Time: 1466605956
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
To make this work on your own network, you'll need to add the CA from your local security appliance into your container:
sudo cp ca.pem /usr/local/share/ca-certificates/my-ca.crt
sudo update-ca-certificates
