I've put security.provider.1=org.bouncycastle.

Here is the stack trace of one of the connections in my process:

"ServerConnection on port 10000 Thread 27" #521 prio=5 os_prio=0 tid=0x0000000002db4800 nid=0x2d79 runnable [0x00007f0ababb1000] 
java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 at java.net.SocketInputStream.read(SocketInputStream.java:171)
 at java.net.SocketInputStream.read(SocketInputStream.java:141)
 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
 - locked <0x00000006d63c51f0> (a java.lang.Object)
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
 - locked <0x00000006d6405210> (a sun.security.ssl.AppInputStream)
 at org.apache.geode.internal.cache.tier.sockets.Message.fetchHeader(Message.java:691)
 at org.apache.geode.internal.cache.tier.sockets.Message.readHeaderAndPayload(Message.java:709)
 at org.apache.geode.internal.cache.tier.sockets.Message.read(Message.java:657)
 at org.apache.geode.internal.cache.tier.sockets.Message.recv(Message.java:1105)
 - locked <0x00000006d6405288> (a java.nio.HeapByteBuffer)
 at org.apache.geode.internal.cache.tier.sockets.Message.recv(Message.java:1118)
 at org.apache.geode.internal.cache.tier.sockets.BaseCommand.readRequest(BaseCommand.java:869)
 at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:723)
 at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
 at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1171)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
 at java.lang.Thread.run(Thread.java:745)

Here I guess instead of sun.security.ssl I should see something used from bouncy castle library.

A couple of things:

1) Which bouncy castle provider you are adding? Bouncy castle packages the JCE provider and JSSE provider in separate jars and have to use separate provider class. JSSE provider class is org.bouncycastle.jsse.provider.BouncyCastleJsseProvider and the JCE provider is org.bouncycastle.jce.provider.BouncyCastleProvider

2) Yes, the providers are looked up in the order of priority but as mentioned in above responses, which implementation is returned also depends on how the algorithm/protocol is requested in the application code. First a provider should be implementing the algorithm/protocol you are requesting and also it has to register it using the name/alias that you are using while requesting.

For example, if the code is requesting TLS context as javax.net.ssl.SSLContext.getInstance("SSL"), BC won't return any context as it does not register any implementation with that alias. However, SunJSSE will return a context as it add "SSL" as an alias to "TLS"

Yes, you can explicitly ask the implementation from a specific provider. All JCE/JSSE api has an additional overloaded method that takes provider name. For example,

javax.net.ssl.SSLContext.getInstance("TLS", "BCJSSE");

javax.net.ssl.KeyManagerFactory("PKIX", "BCJSSE");