Why is my .net core API cancelling requests?

2019-08-20 12:18发布

问题:

I have a an aync method that is looped:

    private Task<HttpResponseMessage> GetResponseMessage(Region region, DateTime startDate, DateTime endDate)
    {
        var longLatString = $"q={region.LongLat.Lat},{region.LongLat.Long}";
        var startDateString = $"{startDateQueryParam}={ConvertDateTimeToApixuQueryString(startDate)}";
        var endDateString = $"{endDateQueryParam}={ConvertDateTimeToApixuQueryString(endDate)}";
        var url = $"http://api?key={Config.Key}&{longLatString}&{startDateString}&{endDateString}";
        return Client.GetAsync(url);
    }

I then take the response and save it to my ef core database, however in some instances I get this Exception message: The Operaiton was canceled

I really dont understand that. This is a TCP handshake issue?

Edit:

For context I am making many of these calls, passing response to the method that writes to db (which is also so slow Its unbelievable):

private async Task<int> WriteResult(Response apiResponse, Region region)
        {
            // since context is not thread safe we ensure we have a new one for each insert
            // since a .net core app can insert data at the same time from different users different instances of context
            // must be thread safe
            using (var context = new DalContext(ContextOptions))
            {
                var batch = new List<HistoricalWeather>();
                foreach (var forecast in apiResponse.Forecast.Forecastday)
                {
                    // avoid inserting duplicates
                    var existingRecord = context.HistoricalWeather
                        .FirstOrDefault(x => x.RegionId == region.Id &&
                            IsOnSameDate(x.Date.UtcDateTime, forecast.Date));
                    if (existingRecord != null)
                    {
                        continue;
                    }

                    var newHistoricalWeather = new HistoricalWeather
                    {
                        RegionId = region.Id,
                        CelsiusMin = forecast.Day.Mintemp_c,
                        CelsiusMax = forecast.Day.Maxtemp_c,
                        CelsiusAverage = forecast.Day.Avgtemp_c,
                        MaxWindMph = forecast.Day.Maxwind_mph,
                        PrecipitationMillimeters = forecast.Day.Totalprecip_mm,
                        AverageHumidity = forecast.Day.Avghumidity,
                        AverageVisibilityMph = forecast.Day.Avgvis_miles,
                        UvIndex = forecast.Day.Uv,
                        Date = new DateTimeOffset(forecast.Date),
                        Condition = forecast.Day.Condition.Text
                    };

                    batch.Add(newHistoricalWeather);
                }

                context.HistoricalWeather.AddRange(batch);
                var inserts = await context.SaveChangesAsync();

                return inserts;
            }

Edit: I am making 150,000 calls. I know this is questionable since It all goes in memory I guess before even doing a save but this is where I got to in trying to make this run faster... only I guess my actual writing code is blocking :/

var dbInserts = await Task.WhenAll(
                getTasks // the list of all api get requests
                .Select(async x => {
                    // parsed can be null if get failed
                    var parsed = await ParseApixuResponse(x.Item1); // readcontentasync and just return the deserialized json
                    return new Tuple<ApiResult, Region>(parsed, x.Item2);
                })
                .Select(async x => {
                    var finishedGet = await x;
                    if(finishedGet.Item1 == null)
                    {
                        return 0;
                    }

                    return await writeResult(finishedGet.Item1, finishedGet.Item2);
                })
            );

回答1:

.net core has a DefaultConnectionLimit setting as answered in comments.

this limits outgoing connections to specific domains to ensure all ports are not taken etc.

i did my parallel work incorrectly causing it to go over the limit - which everything i read says should not be 2 on .net core but it was - and that caused connections to close before receiving responses.

I made it greater, did parallel work correctly, lowered it again.



标签: c# .net-core