What is the accepted way to instrument a web-site to record execution statistics?

How long it takes to X

For example, i want to know how long it takes to perform some operation, e.g. validating the user's credentials with the Active Directory server:

authenticated = CheckCredentials(Login1.UserName, Login1.Password);

A lot of people will suggest using Tracing, of various kinds, to output, or log, or record, the interesting performance metrics:

var sw = new System.Diagnostics.Stopwatch();
sw.Start();
authenticated = CheckCredentials(Login1.UserName, Login1.Password);
sw.Stop();

//write a number to a log
WriteToLog("TimeToCheckCredentials", sw.ElapsedTicks);

Not an X; all X

The problem with this is that i'm not interested in how long it took to validate a user's credentials against Active Directory. i'm interested in how long it took to validate thousands of user's credentials in ActiveDirectory:

var sw = new System.Diagnostics.Stopwatch();
sw.Start();
authenticated = CheckCredentials(Login1.UserName, Login1.Password);
sw.Stop();

timeToCheckCredentialsSum = timeToCheckCredentialsSum + sw.ElapsedTicks;
timeToCheckCredentialsCount = timeToCheckCredentialsCount + 1;
if ((sw.ElapsedTicks < timeToCheckCredentialMin) || (timeToCheckCredentialMin == 0))
   timeToCheckCredentialMin = sw.ElapsedTicks;
if ((sw.ElapsedTicks > timeToCheckCredentialMax) || (timeToCheckCredentialMax == 0))
   timeToCheckCredentialMax = sw.ElapsedTicks;
oldMean = timeToCheckCredentialsAverage;
newMean = timeToCheckCredentailsSum / timeToCheckCredentialsCount;
timeToCheckCredentialsAverage = newMean;

if (timeToCheckCredentailsCount > 2)
{
   timeToCheckCredentailsVariance = (
         ((timeToCheckCredentialsCount -2)*timeToCheckCredentailsVariance  + (sw.ElapsedTicks-oldMean)*(sw.ElapsedTicks-newMean))
         / (timeToCheckCredentialsCount -1))
}
else
    timeToCheckCredentailsVariance = 0;

Which is a lot of boilerplate code that can easily be abstracted away into:

var sw = new System.Diagnostics.Stopwatch();
sw.Start();
authenticated = CheckCredentials(Login1.UserName, Login1.Password);
sw.Stop();

//record the sample
Profiler.AddSample("TimeToCheckCredentials", sw.ElapsedTicks);

Which is still a lot of boilerplate code, that can be abstracted into:

Profiler.Start("TimeToCheckCredentials");
authenticated = CheckCredentials(Login1.UserName, Login1.Password);
Profiler.Stop("TimeToCheckCredentials");

Now i have some statistics sitting in memory. i can let the web-site run for a few months, and at any time i can connect to the server and look at the profiling statistics. This is very much the ability of SQL Server to present it's own running history in various reports:

But ASP kills apps without warning

The problem is that this is an ASP.net web-site/application. Randomly throughout the course of a year, the web-server will decide to shut down the application, by recycling the application pool:

perhaps it has been idle for 3 weeks
perhaps it reached the maximum recycle time limit (e.g. 24 hours)
perhaps a date on a file changed, and the web-server has to recompile the application

When the web-server decides to shut down, all my statistics are lost.

Are there any ASP.net performance/instrumentation frameworks that solve this problem?

Try persisting to SQL Server

i thought about storing my statistics in SQL Server. Much like ASP.net session state can be stored in SQL Server after every request is complete, i could store my values in SQL Server every time:

void AddSample(String sampleName, long elapsedTicks)
{
    using (IDbConnection conn = CreateDatabaseConnection())
    {
        ExecuteAddSampleStoredProcedure(conn, sampleName, elapsedTicks);
    }
}

Except now i've introduced a huge latency into my application. This profiling code is called many thousand times a second. When the math is performed only in memory it takes few microseconds. Now it takes few dozen milliseconds. (Factor of 1,000; noticeable delay). That's not going to work.

Save only on application shutdown

i have considered registering my static helper class with the ASP.net hosting environment by implementing IRegisteredObject:

public class ShutdownNotification : IRegisteredObject
{
    public void Stop(Boolean immediate)
    {
        Profiler.SaveStatisticsToDatabase();
    } 
}

But i'm curious what the right way to solve this problem is. Smarter people than me must have added profiling to ASP.net before.

We use Microsoft's Application Performance Monitoring for this. It captures page load times, DB call times, API call times, etc. When a page load is unexpectedly slow, it also alerts us and provides the stack trace along with the timings of various calls that impacted the load time. It's somewhat rudimentary but it does the trick and allowed us to verify that we didn't have any variations that were not performing as expected.

Advance warning: the UI only works in IE.

http://technet.microsoft.com/en-us/library/hh457578.aspx