File upload and result pooling kills ASP.NET web a

We have created ASP.NET MVC app which accept file upload(up to 80mb) and has result pooling implemented by AsincController. Hosted on Windows 2008 R2 IIS7.5 .NET 4. Server 2 Cores 2.6GHZ, 2GB Ram, fast HDD.

The web site has many users and Performance Monitor show ASP.NET Requests/Sec ~15 and Request Current ~270

After several minutes ASP.NET starts queuing request and ASP.NET Request Queued counter starts growing and application become extremely slow. I am hunting the problem almost a month, tried to profile code, no performance issues and no memory leaks. Increased maxWorkerThreads to 400 and maxIoThreads to 400. Set maxConcurrentRequestsPerCPU to 5000 and MaxConcurrentThreadsPerCPU to 0 but that either didn't helps.

One thing which seems helps is increasing app pool Maximum Worker Processes to two or three processes and making app pool web garden. After that Request Current jumps to ~350 and no request queuing. But web garden introduce several new issues which we will not mention here.

Please post any suggestions how we could increase application performance without making our app run in IIS pool web garden?

I will say to you to double check the pages that take long time to proceed a work, or the pages that download or upload files, and disable the session on this pages. If this is not possible, then you may need to write a totally custom way to handle the session.

Why, because session is locks everything until the page fully return.

You can read relative to session and to this issue:
call aspx page to return an image randomly slow
ASP.NET Server does not process pages asynchronously
Trying to make Web Method Asynchronous
Web app blocked while processing another web app on sharing same session
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Replacing ASP.Net's session entirely