IIS Application Pool Process Using A lot of Memory

Question

I have a very weird problem with one of my IIS application pool processes. Ive been getting a System.OutOfMemoryException error lately and ive been trying to figure out exactly what is going on. Basically I have a script that uses a web service to get a file from our DAM. It then checks the file stores it a byte array, then uses the Response to output the file. The only one ive been having problems with is the PDF's when they are over 20MB now it seems that they cause an error sometimes. If I increase the memory in the app pool it fixes the problem temporarily. I watched the w3wp.exe process and seen that sometimes when I run this script it increase the memory up to 400MB the largest file we have is 45MB what would be causing this type of behavior to happen. The problem seems to go away every night and in the morning it will work for a while and then start doing the same thing again. This application is c# asp.net application. It run inside of sharepoint.

After watching the service for a while I did notice that since these PDF's are rendered in the browser window that until the file downloads completely it doesn't release from memory. which makes sense but I can see that this is somewhat of my problem. If I have several people loading the file, with a average (no file downloading) memory usage at 385,000 kb.It can easily get to 900,000-1,100,000 KB which is the limit of the application pool.

Im not so much looking for an exact answer but more like a direction to head because I am all out of ideas.

Answer 1

When you bring the file data into memory as an array of bytes, you are putting a lot of pressure on the web server.

Instead of storing the entire file data in an array of bytes, you should try writing the file stream into the reponse stream in chunks.

Pseudo example:

context.Response.Buffer = false;

byte[] buffer   = new byte[4096];
int bytesRead   = 0;

using(var stream = new FileStream(path, FileMode.Open, FileAccess.Read))
{
    while ((bytesRead = stream.Read(buffer, 0 , buffer.Length)) > 0)
    {
        context.Response.OutputStream.Write(buffer, 0, buffer.Length);
        context.Response.OutputStream.Flush();
    }
}

The idea here being that you only bring a chunk of the file data into memory on each read of the file stream and then write it to the response. Note that response buffering has been disabled, and that you can substitute using a file stream with another Stream data source (I have used this approach while reading binary data from a SQL database).

Edit: (Response to how to stream data from SQL to HTTP Response)

In order to stream data from a SQL server database table (e.g. a varbinary(max) column), you use sequential access on SqlCommand:

#region WriteResponse(HttpContext context, Guid id)
/// <summary>
/// Writes the content for a media resource with the specified <paramref name="id"/> 
/// to the response stream using the appropriate content type and length.
/// </summary>
/// <param name="context">The <see cref="HttpContext"/> to write content to.</param>
/// <param name="id">The unique identifier assigned to the media resource.</param>
private static void WriteResponse(HttpContext context, Guid id)
{
    using(var connection = ConnectionFactory.Create())
    {
        using (var command = new SqlCommand("[dbo].[GetResponse]", connection))
        {
            command.CommandType = CommandType.StoredProcedure;

            command.Parameters.Add("@Id", SqlDbType.UniqueIdentifier);
            command.Parameters.AddReturnValue();

            command.Parameters["@Id"].Value = id;

            command.Open();

            using(var reader = command.ExecuteReader(CommandBehavior.SequentialAccess))
            {
                if(reader.Read())
                {
                    WriteResponse(context, reader);
                }
            }
        }
    }
}
#endregion

#region WriteResponse(HttpContext context, SqlDataReader reader)
/// <summary>
/// Writes the content for a media resource to the response stream using the supplied <paramref name="reader"/>.
/// </summary>
/// <param name="context">The <see cref="HttpContext"/> to write content to.</param>
/// <param name="reader">The <see cref="SqlDataReader"/> to extract information from.</param>
private static void WriteResponse(HttpContext context, SqlDataReader reader)
{
    if (context == null || reader == null)
    {
        return;
    }

    DateTime expiresOn      = DateTime.UtcNow;
    string contentType      = String.Empty;
    long contentLength      = 0;
    string fileName         = String.Empty;
    string fileExtension    = String.Empty;

    expiresOn               = reader.GetDateTime(0);
    fileName                = reader.GetString(1);
    fileExtension           = reader.GetString(2);
    contentType             = reader.GetString(3);
    contentLength           = reader.GetInt64(4);

    context.Response.AddHeader("Content-Disposition", String.Format(null, "attachment; filename={0}", fileName));

    WriteResponse(context, reader, contentType, contentLength);

    ApplyCachePolicy(context, expiresOn - DateTime.UtcNow);
}
#endregion

#region WriteResponse(HttpContext context, SqlDataReader reader, string contentType, long contentLength)
/// <summary>
/// Writes the content for a media resource to the response stream using the 
/// specified reader, content type and content length.
/// </summary>
/// <param name="context">The <see cref="HttpContext"/> to write content to.</param>
/// <param name="reader">The <see cref="SqlDataReader"/> to extract information from.</param>
/// <param name="contentType">The content type of the media.</param>
/// <param name="contentLength">The content length of the media.</param>
private static void WriteResponse(HttpContext context, SqlDataReader reader, string contentType, long contentLength)
{
    if (context == null || reader == null)
    {
        return;
    }

    int ordinal     = 5;
    int bufferSize  = 4096 * 1024; // 4MB
    byte[] buffer   = new byte[bufferSize];
    long value;
    long dataIndex;

    context.Response.Buffer         = false;
    context.Response.ContentType    = contentType;
    context.Response.AppendHeader("content-length", contentLength.ToString());

    using (var writer = new BinaryWriter(context.Response.OutputStream))
    {
        dataIndex   = 0;
        value       = reader.GetBytes(ordinal, dataIndex, buffer, 0, bufferSize);

        while(value == bufferSize)
        {
            writer.Write(buffer);
            writer.Flush();

            dataIndex   += bufferSize;
            value       = reader.GetBytes(ordinal, dataIndex, buffer, 0, bufferSize);
        }

        writer.Write(buffer, 0, (int)value);
        writer.Flush();
    }
}
#endregion

Answer 2

Oppositional has very good advice about handling the file itself. I would also look at references you may be hanging on to in your web processing. Do store anything in Session State or Application State? If so, trace those carefully to make sure they don't in turn point to the page or somethign else involved with handling your files.

I mention this becuase we had a nasty 'leak' a few years ago caused by placing an object in application state. Turns out that object subscribed to page events, and since the object never died neither did all the pages! oops