Service Fabric https endpoint with kestrel and rev

I've been trying to setup Https on a stateless API endpoint following the instructions on the microsoft documentations and diverse post/blogs I could find. It works fine locally, but I'm struggling to make it work after deploying it on my dev server getting

Browser : HTTP ERROR 504
Vm event viewer : HandlerAsyncOperation EndProcessReverseProxyRequest failed with FABRIC_E_TIMEOUT
SF event table : Error while processing request: request url = https://mydomain:19081/appname/servicename/api/healthcheck/ping, verb = GET, remote (client) address = xxx, request processing start time = 2018-03-13T14:50:17.1396031Z, forward url = https://0.0.0.0:44338/api/healthcheck/ping, number of successful resolve attempts = 48, error = 2147949567, message = , phase = ResolveServicePartition

in code I have in the instancelistener

    .UseKestrel(options =>
    {
       options.Listen(IPAddress.Any, 44338, listenOptions =>
       {
           listenOptions.UseHttps(GetCertificate());
       });
    })

servicemanifest

<Endpoint Protocol="https" Name="SslServiceEndpoint" Type="Input" Port="44338" />

startup

        services.AddMvc(options =>
        {
            options.SslPort = 44338;
            options.Filters.Add(new RequireHttpsAttribute());
        });

            var options = new RewriteOptions().AddRedirectToHttps(StatusCodes.Status301MovedPermanently, 44338);
        app.UseRewriter(options);

here is what I got in azure (deployed through ARM template)

Health probes
NAME                    PROTOCOL    PORT    USED BY
AppPortProbe            TCP         44338   AppPortLBRule
FabricGatewayProbe      TCP         19000   LBRule
FabricHttpGatewayProbe  TCP         19080   LBHttpRule
SFReverseProxyProbe     TCP         19081   LBSFReverseProxyRule

Load balancing rules
NAME                    LOAD BALANCING RULE                 BACKEND POOL                    HEALTH PROBE
AppPortLBRule           AppPortLBRule (TCP/44338)           LoadBalancerBEAddressPool       AppPortProbe
LBHttpRule              LBHttpRule (TCP/19080)              LoadBalancerBEAddressPool       FabricHttpGatewayProbe
LBRule                  LBRule (TCP/19000)                  LoadBalancerBEAddressPool       FabricGatewayProbe
LBSFReverseProxyRule    LBSFReverseProxyRule (TCP/19081)    LoadBalancerBEAddressPool       SFReverseProxyProbe

I have a Cluster certificate, ReverseProxy Certificate, and auth to the api through azure ad and in ARM

                "fabricSettings": [
                {
                    "parameters": [
                        {
                            "name": "ClusterProtectionLevel",
                            "value": "[parameters('clusterProtectionLevel')]"
                        }
                    ],
                    "name": "Security"
                },
                {
                    "name": "ApplicationGateway/Http",
                    "parameters": [
                      {
                        "name": "ApplicationCertificateValidationPolicy",
                        "value": "None"
                      }
                    ]
                }
            ],

Not sure what else could be relevant, if you have any ideas/suggestions, those are really welcome

Edit : code for GetCertificate()

    private X509Certificate2 GetCertificate()
    {
        var certificateBundle = Task.Run(async () => await GetKeyVaultClient()
            .GetCertificateAsync(Environment.GetEnvironmentVariable("KeyVaultCertifIdentifier")));
        var certificate = new X509Certificate2();
        certificate.Import(certificateBundle.Result.Cer);
        return certificate;
    }

    private KeyVaultClient GetKeyVaultClient() => new KeyVaultClient(async (authority, resource, scope) =>
    {
        var context = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCred = new ClientCredential(Environment.GetEnvironmentVariable("KeyVaultClientId"),
            Environment.GetEnvironmentVariable("KeyVaultSecret"));
        var authResult = await context.AcquireTokenAsync(resource, clientCred);
        return authResult.AccessToken;
    });

Digging into your code I've realized that there is nothing wrong with it except one thing. I mean, as you use Kestrel, you don't need to set up anything extra in the AppManifest as those things are for Http.Sys implementation. You don't even need to have an endpoint in the ServiceManifest(although recommended) as all these things are about URL reservation for the service account and SSL binding configuration, neither of which is required with Kestrel.

What you do need to do is to use IPAddress.IPv6Any while you configure SSL. Aside the fact that it turns out to be the recommended way which allows you to accept both IPv4 and IPV6 connections, it also does a 'correct' endpoint registration in the SF. See, when you use IPAddress.Any, you'll get the SF setting up an endpoint like https://0.0.0.0:44338, and that's how the reverse proxy will try to reach the service which obviously wouldn't work. 0.0.0.0 doesn't correspond to any particular ip, it's just the way to say 'any IPv4 address at all'. While when you use IPAddress.IPv6Any, you'll get a correct endpoint mapped to the vm ip address that could be resolved from within the vnet. You could see that stuff by yourself in the SF Explorer if you go down to the endpoint section in the service instance blade.