OutOfMemoryException when reading a string

Question

I have a C# .NET system that takes a JSON data feed and converts it to an object using the Newtonsoft.Json.JsonConvert.DeserializeObject converter.

This process works perfect as long as the JSON string is below a certain size (a few Mb) but as soon as the returned data is large (almost 100Mb) I get the error OutOfMemoryException

This code works great for small data:

// WebClient ------------------------------------------------------------------
var _client = new System.Net.WebClient();
var _content = _client.DownloadString(_url);

but blows up on the last line (DownloadString)

I tried changing to this which also works for small data but it still blew up on the ReadToEnd line when the data grew in size.

using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            string _responseFromServer = _streamReader.ReadToEnd();
        }
    }
}

Finally I tried this which worked:

StringBuilder _stringBuilder = new StringBuilder();
using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            while (!streamReader.EndOfStream)
            {
                char[] _buffer = new char[4096];
                _streamReader.ReadBlock(_buffer, 0, _buffer.Length);
                var _bufferString = new String(_buffer);
                _stringBuilder.Append(_bufferString);
            }
        }
    }
}

But it blew up with an OutOfMemoryException error when it got to the next line here:

var _results = Newtonsoft.Json.JsonConvert.DeserializeObject<List<MyObject>>(_stringBuilder.ToString());

It didn't like the ToString() method.

It also crashed with a simple line like

string _convertedString = _stringBuilder.ToString();

The full error is:

An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code

The machine is running 64bit windows with 16Gb of memory.

So, what are my options?

All I want is an IQueryable<MyObject> from a (very large) JSON string.

Answer 1

Your code essentially emulates what StreamReader.ReadToEnd does, taking at least 4 times the memory needed to read a large response (the memory of the string response itself, the StringBuilder's internal buffer, the size of all the intermediate temporary strings and the final string).

You can avoid this by deserializing from the stream directly with a JsonTextReader. Copying from the documentation sample:

using (var json= new JsonTextReader(streamReader))
{
    JsonSerializer serializer = new JsonSerializer();
    return (List<MyObject>)serializer.Deserialize(json, typeof(List<MyObject>));
}

O