HTTPClient Buffer Exceeded 2G; Cannot write more b

Question

The latest in what is becoming the saga of the Walmart API. I'm submitting a call to get a list of clearance items using HttpClient. Things work fine with other requests, but this one is so large it busts the HTTPRequest buffer. Odd, too, that it's a REQUEST buffer error and not a RESPONSE error, since the request is only the URL.

Exception Information:

 Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. (System.Net.Http)

at System.Net.Http.HttpContent.LimitMemoryStream.CheckSize(Int32 countToAdd)\r\n at System.Net.Http.HttpContent.LimitMemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n at System.Net.Http.StreamToStreamCopy.TryStartWriteSync(Int32 bytesRead)\r\n at System.Net.Http.StreamToStreamCopy.StartRead()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Wal_Mart_Crawler.NET_IO.<walMart_Special_Feed_Lookup>d__4.MoveNext() in c:\\users\\user\\documents\\visual studio 2015\\Projects\\Wal-Mart_Inventory_Tracker\\Wal-Mart_Crawler\\NET_IO.cs:line 65\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Wal_Mart_Crawler.Special_Feeds.d__2.MoveNext() in c:\users\user\documents\visual studio 2015\Projects\Wal-Mart_Inventory_Tracker\Wal-Mart_Crawler\Special_Feeds.cs:line 22

(for the curious, Line 22 is):

 API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Wal_Mart_Crawler.Properties.Resources.API_Key_Walmart));
 //which uses HttpClient to call the API

At first, my eyes twinkled a bit when I saw "configured," as that means I can change it, right? I'm running in x64 after all - might be able to bust 2G can't figure out how.

Read up on SO, found that I should disable streaming. Tried:

 var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);

... no Bueno.

Found an AllowReadStreamBuffering setting for WebRequest - can't seem to find one for HttpClient.

I can't control how much data I'm getting -- that's on Walmart. I'm also limited by my shrunken head on what to do with the response stream because it's going straight to deserialize:

   var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
   return JsonConvert.DeserializeObject<API_Json_Special_Feeds.RootObject>(result);

So even if I could break up the response, I'd be stuck because I don't think I can feed partial data to deserialize.

Question: How can I either increase the buffer beyond 2G or otherwise avoid the buffer size exceeded exception?

I'm hoping it's another easy fix my weary, ignorant brain just can't figure out. As always - a sincere THANK YOU for your time and in advance for any help you can provide.

Response Headers:

 StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
 X-Mashery-Responder: prod-j-worker-us-west-1c-63.mashery.com
 transfer-encoding: chunked
 Connection: keep-alive
 Date: Sat, 10 Sep 2016 00:57:38 GMT
 Server: Mashery
 Server: Proxy
 Content-MD5: BvJMDJiZPUvmAxxmwKGSog==
 Content-Type: application/json; charset=utf-8
 Last-Modified: Fri Sep 09 15:31:08 PDT 2016
}

Answer 1

Will something like this work?

using (var http = new HttpClient())
using (var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (StreamReader sr = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
    var serializer = new JsonSerializer();               
    ITraceWriter tw = new MemoryTraceWriter();
    serializer.TraceWriter = tw; 
    var obj = (API_Json_Special_Feeds.RootObject)serializer.Deserialize(sr, typeof(API_Json_Special_Feeds.RootObject));
    // Stop and inspect the tracewriter object here to diagnose 
    return obj;                
}

EDIT: By including the tracewriter, you can try and determine why the deserialization isn't behaving as you would expect. JSON deserialization tends to silently throw away information it doesn't know how to handle and results in an empty or sparsely populated object. The tracewriter will help diagnose these issues. Example