ERR_EMPTY_RESPONSE when processing a large number

Question

I created an intranet system to handle TXT files in a directory and process it's data into a SQL Server database. It's recursive, so I have a file structure like this:

C:\RawData\2012_06_01\2012_06_01_0002144493.txt C:\RawData\2012_06_01\2012_06_01_0002954412.txt

...

C:\RawData\2012_06_15\2012_06_15_0012554778.txt

And so on, summing up to about 10.530 files.

It works perfectly when selecting just some folders (like all folders from a month). However, when trying to process, let's say, an year of files (from 2011_06_01 to 2012_06_01), it begins processing, and then I get this ERR_EMPTY_RESPONSE error. It is not connection related, since it happens under VS2010 engine (I just run the code, it opens the browser, I select all folders I want - on the web interface - and click "process" button).

The processing is being made by doing loops under loops, like this sample:

string[] folders = Directory.GetDirectories(parentFolder);
foreach (string folder in folders)
{
    string[] files = f.GetFilesRecursive(@folder);
    foreach (string file in files)
    {
        if (File.Exists(file))
        {
            string buffer = "";
            StreamReader sr = File.OpenText(file);
            while (!sr.EndOfStream)
            {
                string line = sr.ReadLine();
                buffer += line;

                // All the internal file processing and database calls goes here

            } // while
            sr.Close();
        } // if file exists
    } // foreach (files)
} // foreach (folders)

This code is a simplified version of the original code. The internal file processing code is really big, involves classes, database objects, etc. but they're working fine (with few files, up to 300 files or so, it works flawlessly).

Is there a way to avoid this error, or any suggestion about other ways to optimize file processing?

Also (and of minimum importance), I would like to implement an ajax response, so the user could see a percentage bar or something like that... I've heard about ICallbackEventHandler and IPostBackEventHandler, but I have no idea on how implement this under loops like this one I use.

Anything would really be appreciated.

EDIT ON 2012-06-21:

I found the answer. The solution suggested by @BumbleBee gave me a start point for researching, since it returned the message:

Asynchronous operations are not allowed in this context. Page starting an asynchronous operation has to have the Async attribute set to true and an asynchronous operation can only be started on a page prior to PreRenderComplete event.

Found this: ASP.NET true asynchronous operation which has the solution on a link in the answer post ( ThreadPool.QueueUserWorkItem uses ASP.Net )

Now, I just need to find a way to send data using AJAX to 3 Divs on page.

Answer 1

You might want to change the RequestTimeout and see.

<httpRuntime executionTimeout="some big value here"  />

Check this for an example

But that actually isn't the real fix for your problem. You need to fire up a background thread and let it handle the big processing and let the UI thread return with a message to the user. Here is one example how you could achieve that.

Answer 2

Don't do such heavy job in a page code behind. Push the job to do in a separate queue, which will perform the job when possible, in a separate thread.

Basically, the click on the button should "fire and forget" the job.

If you need to have a progress bar, put a [WebMethod] that will return the queue length, and call it with ajax.

You can also add a cancel button if you want.

The UI will be responsive using this technic.