ASP.NET Core 2.2 kestrel server's performance

Question

I'm facing problem with kestrel server's performance. I have following scenario :

    TestClient(JMeter) -> DemoAPI-1(Kestrel) -> DemoAPI-2(IIS) 

I'm trying to create a sample application that could get the file content as and when requested. TestClient(100 Threads) requests to DemoAPI-1 which in turn request to DemoAPI-2. DemoAPI-2 reads a fixed XML file(1 MB max) and returns it's content as a response(In production DemoAPI-2 is not going to be exposed to outside world).

When I tested direct access from TestClient -> DemoAPI-2 I got expected result(good) which is following :

1. Average : 368ms
2. Minimum : 40ms
3. Maximum : 1056ms
4. Throughput : 40.1/sec

But when I tried to access it through DemoAPI-1 I got following result :

1. Average : 48232ms
2. Minimum : 21095ms
3. Maximum : 49377ms
4. Throughput : 2.0/sec

As you can see there is a huge difference.I'm not getting even the 10% throughput of DemoAPI-2. I was told has kestrel is more efficient and fast compared to traditional IIS. Also because there is no problem in direct access, I think we can eliminate the possible of problem on DemoAPI-2.

※Code of DemoAPI-1 :

string base64Encoded = null;
            var request = new HttpRequestMessage(HttpMethod.Get, url);
            var response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseContentRead).ConfigureAwait(false);

            if (response.StatusCode.Equals(HttpStatusCode.OK))
            {
                var content = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
                base64Encoded = Convert.ToBase64String(content);
            }
return base64Encoded;

※Code of DemoAPI-2 :

[HttpGet("Demo2")]
    public async Task<IActionResult> Demo2Async(int wait)
    {
        try
        {
            if (wait > 0)
            {
                await Task.Delay(wait);
            }
            var path = Path.Combine(Directory.GetCurrentDirectory(), "test.xml");
            var file = System.IO.File.ReadAllText(path);
            return Content(file);
        }
        catch (System.Exception ex)
        {
            return StatusCode(500, ex.Message);
        }
    }

Some additional information :

1. Both APIs are async.
2. Both APIs are hosted on different EC2 instances(C5.xlarge Windows Server 2016).
3. DemoAPI-1(kestrel) is a self-contained API(without reverse proxy)
4. TestClient(jMeter) is set to 100 thread for this testing.
5. No other configuration is done for kestrel server as of now.
6. There are no action filter, middleware or logging that could effect the performance as of now.
7. Communication is done using SSL on 5001 port.
8. Wait parameter for DemoAPI2 is set to 0 as of now.
9. The CPU usage of DEMOAPI-1 is not over 40%.

Answer 1

The problem was due to HttpClient's port exhaustion issue. I was able to solve this problem by using IHttpClientFactory. Following article might help someone who faces similar problem.

https://www.stevejgordon.co.uk/httpclient-creation-and-disposal-internals-should-i-dispose-of-httpclient

Answer 2

DEMOAPI-1 performs a non-asynchronous read of the streams:

var bytes = stream.Read(read, 0, DataChunkSize);

while (bytes > 0)
{
  buffer += System.Text.Encoding.UTF8.GetString(read, 0, bytes);
  // Replace with ReadAsync
  bytes = stream.Read(read, 0, DataChunkSize);
}

That can be an issue with throughput on a lot of requests.

Also, I'm not fully aware of why are you not testing the same code with IIS and Kestrel, I would assume you need to make only environmental changes and not the code.