The latest in what is becoming the saga of the Walmart API. I'm submitting a call to get a list of clearance items using HttpClient. Things work fine with other requests, but this one is so large it busts the HTTPRequest buffer. Odd, too, that it's a REQUEST buffer error and not a RESPONSE error, since the request is only the URL.
Exception Information:
Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. (System.Net.Http)
at System.Net.Http.HttpContent.LimitMemoryStream.CheckSize(Int32 countToAdd)\r\n at System.Net.Http.HttpContent.LimitMemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n at System.Net.Http.StreamToStreamCopy.TryStartWriteSync(Int32 bytesRead)\r\n at System.Net.Http.StreamToStreamCopy.StartRead()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter
1.GetResult()\r\n at Wal_Mart_Crawler.NET_IO.<walMart_Special_Feed_Lookup>d__4.MoveNext() in c:\\users\\user\\documents\\visual studio 2015\\Projects\\Wal-Mart_Inventory_Tracker\\Wal-Mart_Crawler\\NET_IO.cs:line 65\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter
1.GetResult()\r\n at Wal_Mart_Crawler.Special_Feeds.d__2.MoveNext() in c:\users\user\documents\visual studio 2015\Projects\Wal-Mart_Inventory_Tracker\Wal-Mart_Crawler\Special_Feeds.cs:line 22
(for the curious, Line 22 is):
API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Wal_Mart_Crawler.Properties.Resources.API_Key_Walmart));
//which uses HttpClient to call the API
At first, my eyes twinkled a bit when I saw "configured," as that means I can change it, right? I'm running in x64 after all - might be able to bust 2G can't figure out how.
Read up on SO, found that I should disable streaming. Tried:
var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
... no Bueno.
Found an AllowReadStreamBuffering setting for WebRequest - can't seem to find one for HttpClient.
I can't control how much data I'm getting -- that's on Walmart. I'm also limited by my shrunken head on what to do with the response stream because it's going straight to deserialize:
var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
return JsonConvert.DeserializeObject<API_Json_Special_Feeds.RootObject>(result);
So even if I could break up the response, I'd be stuck because I don't think I can feed partial data to deserialize.
Question: How can I either increase the buffer beyond 2G or otherwise avoid the buffer size exceeded exception?
I'm hoping it's another easy fix my weary, ignorant brain just can't figure out. As always - a sincere THANK YOU for your time and in advance for any help you can provide.
Response Headers:
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
X-Mashery-Responder: prod-j-worker-us-west-1c-63.mashery.com
transfer-encoding: chunked
Connection: keep-alive
Date: Sat, 10 Sep 2016 00:57:38 GMT
Server: Mashery
Server: Proxy
Content-MD5: BvJMDJiZPUvmAxxmwKGSog==
Content-Type: application/json; charset=utf-8
Last-Modified: Fri Sep 09 15:31:08 PDT 2016
}