Dealing with large JSON data returned by Web API

Question

We are building an web API that receives the array of strings as input parameter which queries the oracle database and returns the result as a JSON file.

So the code is like

 namespace PDataController.Controllers
{
  public class ProvantisDataController : ApiController
  {
    public HttpResponseMessage Getdetails([FromUri] string[] id)
    {

       List<OracleParameter> prms = new List<OracleParameter>();
        string connStr = ConfigurationManager.ConnectionStrings["PDataConnection"].ConnectionString;
        using (OracleConnection dbconn = new OracleConnection(connStr))
        {
            var inconditions = id.Distinct().ToArray();
            var srtcon = string.Join(",", inconditions);
            DataSet userDataset = new DataSet();
            var strQuery = @"SELECT 
                           STCD_PRIO_CATEGORY_DESCR.DESCR AS CATEGORY, 
                           STCD_PRIO_CATEGORY_DESCR.SESSION_NUM AS SESSION_NUMBER, 
                           Trunc(STCD_PRIO_CATEGORY_DESCR.START_DATE) AS SESSION_START_DATE, 
                           STCD_PRIO_CATEGORY_DESCR.START_DATE AS SESSION_START_TIME , 
                           Trunc(STCD_PRIO_CATEGORY_DESCR.END_DATE) AS SESSION_END_DATE, 
                             FROM 
                             STCD_PRIO_CATEGORY_DESCR, 
                             WHERE 
                            STCD_PRIO_CATEGORY_DESCR.STD_REF IN(";
            StringBuilder sb = new StringBuilder(strQuery);
             for(int x = 0; x < inconditions.Length; x++)
                 {
                   sb.Append(":p" + x + ",");
                   OracleParameter p = new OracleParameter(":p" + x,OracleDbType.NVarchar2);
                   p.Value = inconditions[x];
                   prms.Add(p);
                 }
            if(sb.Length > 0) sb.Length--;
            strQuery = sb.ToString() + ")"; 
            using (OracleCommand selectCommand = new OracleCommand(strQuery, dbconn))
              {
               selectCommand.Parameters.AddRange(prms.ToArray());
                 using (OracleDataAdapter adapter = new OracleDataAdapter(selectCommand))
                {
                    DataTable selectResults = new DataTable();
                    adapter.Fill(selectResults);
                    var returnObject = new { data = selectResults };
                    var response = Request.CreateResponse(HttpStatusCode.OK, returnObject, MediaTypeHeaderValue.Parse("application/json"));
                    ContentDispositionHeaderValue contentDisposition = null;
                    if (ContentDispositionHeaderValue.TryParse("inline; filename=ProvantisStudyData.json", out contentDisposition))
                    {
                        response.Content.Headers.ContentDisposition = contentDisposition;
                    }
                    return response;
                }
            }

        }
    }
}
}

The data returned for the API is in the below format

{"data":[{"CATEGORY":"Internal Study","SESSION_NUMBER":7,"SESSION_START_DATE":"2015-02-13T00:00:00","SESSION_START_TIME":"2015-02-13T10:33:59.288394","SESSION_END_DATE":"2015-02-13T00:00:00"}]}

We are sometimes having issue in returning the large amount of data it throws the OutOfMemory Exception. It was suggested to use the JSON property, parallel to the “data” property: like “next_data”, with a value of the value you need to pass into the SQL OFFSET (which works in MySQL, I am not sure if this works in oracle),if there no data remaining then set the value of “next_data” to 0.I am not sure how to implement this.Not sure if this can be implemented. Any help with this is greatly appreciated.

Answer 1

Your problem is that you are running an Oracle query that is returning a very large number of results, and then loading that entire result set into memory before serializing it out to the HttpResponseMessage.

To reduce your memory usage, you should find and eliminate all cases where the entire set of results from the query is loaded into a temporary intermediate representation (e.g. a DataTable or JSON string), and instead stream the data out using a DataReader. This avoids pulling everything into memory at once according to this answer.

First, from your traceback, it appears you have Enable Browser Link checked. Since this apparently tries to cache the entire response in a MemoryStream, you will want to disable it as explained in FilePathResult thrown an OutOfMemoryException with large file.

Next, you can stream the contents of an IDataReader directly to JSON using Json.NET with following class and converter:

[JsonConverter(typeof(OracleDataTableJsonResponseConverter))]
public sealed class OracleDataTableJsonResponse
{
    public string ConnectionString { get; private set; }
    public string QueryString { get; private set; }
    public OracleParameter[] Parameters { get; private set; }

    public OracleDataTableJsonResponse(string connStr, string strQuery, OracleParameter[] prms)
    {
        this.ConnectionString = connStr;
        this.QueryString = strQuery;
        this.Parameters = prms;
    }
}

class OracleDataTableJsonResponseConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(OracleDataTableJsonResponse);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException("OracleDataTableJsonResponse is only for writing JSON.  To read, deserialize into a DataTable");
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var response = (OracleDataTableJsonResponse)value;

        using (var dbconn = new OracleConnection(response.ConnectionString))
        {
            dbconn.Open();
            using (var selectCommand = new OracleCommand(response.QueryString, dbconn))
            {
                if (response.Parameters != null)
                    selectCommand.Parameters.AddRange(response.Parameters);
                using (var reader = selectCommand.ExecuteReader())
                {
                    writer.WriteDataTable(reader, serializer);
                }
            }
        }
    }
}

public static class JsonExtensions
{
    public static void WriteDataTable(this JsonWriter writer, IDataReader reader, JsonSerializer serializer)
    {
        if (writer == null || reader == null || serializer == null)
            throw new ArgumentNullException();
        writer.WriteStartArray();
        while (reader.Read())
        {
            writer.WriteStartObject();
            for (int i = 0; i < reader.FieldCount; i++)
            {
                writer.WritePropertyName(reader.GetName(i));
                serializer.Serialize(writer, reader[i]);
            }
            writer.WriteEndObject();
        }
        writer.WriteEndArray();
    }
}

Then modify your code to look something like:

    public HttpResponseMessage Getdetails([FromUri] string[] id)
    {
        var prms = new List<OracleParameter>();
        var connStr = ConfigurationManager.ConnectionStrings["PDataConnection"].ConnectionString;
        var inconditions = id.Distinct().ToArray();
        var strQuery = @"SELECT 
                       STCD_PRIO_CATEGORY_DESCR.DESCR AS CATEGORY, 
                       STCD_PRIO_CATEGORY_DESCR.SESSION_NUM AS SESSION_NUMBER, 
                       Trunc(STCD_PRIO_CATEGORY_DESCR.START_DATE) AS SESSION_START_DATE, 
                       STCD_PRIO_CATEGORY_DESCR.START_DATE AS SESSION_START_TIME , 
                       Trunc(STCD_PRIO_CATEGORY_DESCR.END_DATE) AS SESSION_END_DATE, 
                         FROM 
                         STCD_PRIO_CATEGORY_DESCR, 
                         WHERE 
                        STCD_PRIO_CATEGORY_DESCR.STD_REF IN(";
        var sb = new StringBuilder(strQuery);
        for (int x = 0; x < inconditions.Length; x++)
        {
            sb.Append(":p" + x + ",");
            var p = new OracleParameter(":p" + x, OracleDbType.NVarchar2);
            p.Value = inconditions[x];
            prms.Add(p);
        }
        if (sb.Length > 0)// Should this be inconditions.Length > 0  ?
            sb.Length--;
        strQuery = sb.Append(")").ToString();

        var returnObject = new { data = new OracleDataTableJsonResponse(connStr, strQuery, prms.ToArray()) };
        var response = Request.CreateResponse(HttpStatusCode.OK, returnObject, MediaTypeHeaderValue.Parse("application/json"));
        ContentDispositionHeaderValue contentDisposition = null;
        if (ContentDispositionHeaderValue.TryParse("inline; filename=ProvantisStudyData.json", out contentDisposition))
        {
            response.Content.Headers.ContentDisposition = contentDisposition;
        }
        return response;
    }

This avoids the in-memory DataSet representation of the results.

Incidentally, I reckon the line

        if (sb.Length > 0)
            sb.Length--;

instead should be:

        if (inconditions.Length > 0)
            sb.Length--;

I believe you're trying to peel off the trailing comma in the query, which will be present if and only if inconditions.Length > 0

Please note - I'm not an Oracle developer and I don't have Oracle installed. For testing I mocked up the OracleClient classes using an underlying OleDbConnection and it worked fine.

Answer 2

Are you allowed to change your method to get that data? I mean, if you are using RESTful services its not a good idea to traffic so much data on a single request. Maybe downloading a file for that purpose or maybe getting the data by pagination.

You can also try to change the max request lenght like this answer: Change max request lenght

But again, it's not good for a web application to traffic and/or process so much data at once.