UPDATE
As Mathias notes below, this exact problem has been reported and resolved here: ASP.NET-MVC (IIS6) Error on high traffic: Specified cast is not valid
ORIGINAL POST
This may be too specific a debugging issue to be posted here, but I'm posting it anyway in the hopes that it produces a solution that others find useful.
I have a web application that operates under moderate load -- maybe 5 requests per second. It has some older code talking to Sql via ADO.NET + DataReaders and has been using this same technique for at least five years without problem. It also has some newer code using LINQ-to-SQL. Both techniques use the same connection string to maximize connection pool reuse.
Recently I'm experiencing a very weird behavior described by these symptoms:
Everything will work perfectly for about a day, then suddenly every call (or nearly every call) to the data layer (both ADO.NET and LINQ) returns data that cannot be parsed by my code -- I'll get exceptions like "Unable to cast object of type 'System.Int32' to type 'System.String'." or "Sequence contains no elements" or "IndexOutOfRangeException" or "Invalid attempt to call Read when reader is closed".
Interestingly, I never get exceptions from SqlCommand.ExecuteReader() or DataReader.Read() -- the exceptions only occur when I try to parse the IDataRecord that is returned.
I am able to fix the problem temporarily by restarting either Sql or IIS. After a few hours it comes back again.
I've tried monitoring the number of connections in the connection pool and it never goes above 3 or so. Certainly never above 100.
I'm not getting anything in the event log that indicates any problem with Sql or IIS.
The drive has 9 GB empty space.
I suspected bad RAM, but the server is using registered ECC DIMMs.
I have other applications using ADO.NET that work fine and never exhibit the problem.
When the problem is occurring I can call the exact same stored procedures via Management Studio and they return the correct, expected results.
Here is my pattern for ADO.NET access:
using (var dbConn = Database.Connection) // gets already-open connection
{
var cmd = new SqlCommand("GetData", dbConn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("@id", id);
SomeDataObject dataObject = null;
var dr = cmd.ExecuteReader(CommandBehavior.CloseConnection | CommandBehavior.SingleRow);
if (dr.Read())
dataObject = new SomeDataObject(dr);
dr.Close();
return dataObject;
}
Theory: Is it possible that the combination of ADO.NET in one part of the code and LINQ in another part of the code, both using the same connections from the connection pool, is having some weird side-effect?
Question: Are there any debugging steps I should be trying? Any events logs or performance metrics that might help?