“dotnet restore” fails with “SSL peer certificate

Question

I've just followed the procedure here: https://www.microsoft.com/net/core#ubuntu

and that's the output of dotnet restore

log  : Restoring packages for /home/test/project.json...
error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
error:   An error occurred while sending the request.
error:   SSL peer certificate or SSH remote key was not OK

I've added the relevant certificates to the trusted ones in order to make curl work but the error remains for dotnet restore.

I've tried to dig in the core source to find out how Nuget checks the SSL certificates without luck. Versions I've tried:

• 1.0.0-preview1-002702
• 1.0.0-preview2-003100

I've configured curl using the .curlrc :

cacert=/etc/ssl/certs/ca-certificates.crt

It has fixed curl -I https://api.nuget.org invocation.

However dotnet restore -v Debug still fails:

trace: Running restore with 8 concurrent jobs.
trace: Reading project file /home/user/test/project.json.
log  : Restoring packages for /home/user/test/project.json...
trace: Restoring packages for .NETCoreApp,Version=v1.0...
error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
error:   An error occurred while sending the request.
error:   SSL peer certificate or SSH remote key was not OK
trace: System.AggregateException: One or more errors occurred. (Unable to load the service index for source https://api.nuget.org/v3/index.json.) ---> NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://api.nuget.org/v3/index.json. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: SSL peer certificate or SSH remote key was not OK
trace:    at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)

So dotnet core uses libcurl but it doesn't use the .curlrc obviously.

EDIT: 21/06/2016

Tried to update the certificates database with mozroots also but it failed to have any effect. (Seems to be more related to mono than to dotnetcore even if dotnet core building page mention it).

After digging into corefx code, Curl handler of System.Net.Http doesn't seem to set the right ssl options in all cases (like in Simple Curl SSL sample).

I've tried Tyler solutions:

certmgr -ssl -m https://api.nuget.org

This isn't adding the last certificate even if I type 'y', 'yes', '1', 'true' or whatever.

mozroots --url https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt --sync --import

This does something:

Importing certificates into user store...
194 new root certificates were added to your trust store.
Import process completed.

I'm not convinced that dotnet core used the libcurl nss build (just because their development page tells about openssl version (and they're mutually exclusive). By the way, I've tried to use nss build of libcurl and dotnet restore still fails.

IMHO, the problem is not related to bad certificates registration but more on the fact that curl builtin certificate validation is not properly disabled (because the certificate validation is done in System.Net.Http and that is mandatory to offer to client code the ability to customize this validation).

Why is it happening on my machine and not elsewhere ? It must be related to my version of libcurl.

However all of these are just assumptions for the moment.

EDIT 22/05/2016:

By looking at the code more thoroughly, especially when comparing master branch and the RC2 release, it's clear that SSL handling code is still changing a lot.

So I just grab the RC2 code and modify it to reflect what master branch does:

easy.SetCurlOption(Interop.Http.CURLoption.CURLOPT_SSL_VERIFYHOST, 0);

However, it didn't change anything... But forecasted it was. so here the code I used:

easy.SetCurlOption(Interop.Http.CURLoption.CURLOPT_SSL_VERIFYPEER, 0);

and then replace System.Net.Http.dll with the ssl certificates check disabled. Not safe but unblocking me for the moment.

I didn't add that as an answer because it's more a hack than a fix.

(a real fix would be to disable completely certificates checks done by curl and always handle it in .Net core but in the current code on master, it's still not the case, it's more a kind of mix of both).

For the root cause, I think that I'm on a specific setup :

• libcurl built without any default certificate bundle path. curlconfig --ca return an empty string. And it doesn't read the CURL_CA_BUNDLE environment variable or .curlrc file.
• System.Net.Http (of dotnet-core) does neither setup a ca default nor disable the certificate validation.

Answer 1

Open SSL missing bundle certificates

The root cause is a missing configuration of openssl.
If you run the following command (or a similar one):

openssl verify /usr/share/ca-certificates/nuget.crt

and you receive the following result:

/usr/share/ca-certificates/nuget.crt: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, OU = Microsoft IT, CN = Microsoft IT SSL SHA2
error 2 at 1 depth lookup:unable to get issuer certificate
140075910137504:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
140075910137504:error:0B06F009:x509 certificate routines:X509_load_cert_file:PEM lib:by_file.c:162:

It's because openssl (on which dotnet / libcurl ultimately relies to do the ssl checks) did not know where to find the ca bundle. I didn't seen any related parameter in /etc/ssl/openssl.cnf so even

export OPENSSL_CONF=/etc/ssl/openssl.cnf

won't help on the openssl verification failure.

However, the following has fixed both problems (openssl and nuget)

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

Now the output for openssl:

zsh/2 906 [1] # openssl verify /usr/share/ca-certificates/nuget.crt
/usr/share/ca-certificates/nuget.crt: OK

and for dotnet restore:

zsh 904 # dotnet restore
log  : Restoring packages for /home/user/test/project.json...
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.dotnethostresolver/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.dotnethostresolver/index.json 412ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.dotnethost/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.dotnethost/index.json 409ms
info : Committing restore...
log  : Lock file has not changed. Skipping lock file write. Path: /home/user/test/project.lock.json
log  : /home/user/test/project.json
log  : Restore completed in 4412ms.

NuGet Config files used:
    /home/user/.nuget/NuGet/NuGet.Config

Feeds used:
    https://api.nuget.org/v3/index.json

Special thanks to Tyler who help me to keep the motivation to fix this.

Answer 2

The place where the certificates are imported from currently redirects to an under construction/outage page. Once the maintenance is done, try again. I think this is a tad bit of a blunder on either Mozilla or the mozroots maintainer. All you see in the console output is either a stack trace or Couldn't retrieve the file using the supplied information. depending on your build of mozroots.

The workaround is to somehow use certmgr to import the correct CA certs, and then import the endpoint certs (use certmgr -ssl -m https://api.nuget.org). If the CA for the certs are missing, the certs are considered invalid. If the certs are considered invalid, you can import them, but the restore will still panic, calling out the certs lack of matching issuer CA cert.

I say somehow, because I haven't figured out a good safe way to recommend doing it yet. I'm definitely going to have the proper mozroots certs baked into the next mono+dotnet docker image I build.

Newer builds of mozroots have command line parameters to substitute the certificate data endpoint. Going to use a web.archive.org copy as a substitute for mozroots --url ... for now. Going to see if I can just use this or maybe this instead, or perhaps another certdata.txt from Mozilla's official mercurial repo.