Controller actions hit multiple times - ASP.NET MV

This might take some time to explain, but here I go :).

I created two Visual Studio 2012 projects:

A project using the (Web -> ASP.NET MVC 4 Web Application) template in Visual Studio's Add Project dialog. I didn't add a unit test project.
A project using the Azure cloud service template (Cloud -> Windows Azure Cloud Service). I added a single role, an ASP.NET MVC 4 Web Role, and again didn't add a unit test project.

I left both of these projects totally untouched from the way that Visual Studio created them for me.

For each project, I then went to the HomeController class and set a breakpoint in each of the About and Contact action methods (boring actions that just return a view). For example, I set a breakpoint on the only line in the method:

public ActionResult About()
{
     return View();
}

I then started debugging the first project (the non-Azure project). Everything was as expected - i.e. when I use the browser to navigate between the About and Contact pages, the breakpoint in the respective action methods would get hit once on each request. Seemed fine to me.

I then started debugging the Azure project. I followed the same procedure of navigating between the About and Contact pages. This time I found interesting non-deterministic behaviour. On some requests, the breakpoint in the action method would be hit multiple times (sometimes more than twice) before the page would render. Sometimes the request would just hang and the page wouldn't render (even after minutes of waiting).

I'd like to know why this is happening, because it is affecting more complicated things in another Azure MVC application that we're working on (e.g. due to this problem the application may attempt to create two or more instances of a model object in my database for the one request).

I'm in the following environment:

Windows 8 Pro (x64)
Visual Studio 2012 Ultimate
Windows Azure SDK for Visual Studio 2012
IIS 8 Express
Firefox browser with Firebug (also confirmed the behavior using IE)
SQL Server 2012 (not really relevant I guess)

The only interesting thing that I noticed is that the following warning message comes up in the Output window in Visual Studio whenever I debug the Azure project:

Remapping private port 80 to 81 in role 'My_Web_Role_Name' to avoid conflict during emulation.

Perhaps this remapping has something to do with it. Then again, after doing a netstat -ano I saw that the Azure Development Fabric process is listening on port 80, so maybe that's why it needs to do the remapping...sounds fair enough.

Anyway, I'm hoping that somebody might have an idea as to what could be causing this behavior. Here are some extra points and approaches I tried:

Reset IIS, restart Azure Compute/Storage emulator, restart entire machine
Strip the views of all script (i.e. the behavior is still there if the pages just have a basic HTML paragraph on them)
I tried to find the relevant IIS Express log files, but it seems that Azure projects don't run through IIS Express like the other project does (is that correct?).
I monitored the Azure Compute Emulator console...nothing very interesting there.

So...finally, some questions:

Can anybody else reproduce this behavior?
Are there any additional logs generated by the Azure Emulator that might help me out?

I'd really appreciate a push in the right direction here :).

Cheers!

EDIT

Here's some more information (I'm updating this question as I continue to debug this problem):

I decided to run the solution without debugging, and I put in some simple debug messages in both the About and Contact action methods, for example:

public ActionResult About()
{
     System.IO.File.AppendAllText(@"c:\Logs\azure.log", DateTime.Now + ": Contact, thread " + System.Threading.Thread.CurrentThread.ManagedThreadId + "\r\n");
     return View();
}

I attached Tail to this file and also started up Fiddler to see what was going on over HTTP. Navigating between the two About and Contact links again, I saw the following output:

From Tail:

...
9/12/2012 12:14:07 PM: About, thread 7
9/12/2012 12:14:08 PM: Contact, thread 7
9/12/2012 12:14:09 PM: About, thread 7
9/12/2012 12:14:10 PM: Contact, thread 7
9/12/2012 12:14:11 PM: About, thread 6
9/12/2012 12:14:12 PM: Contact, thread 8
9/12/2012 12:14:31 PM: Contact, thread 7
9/12/2012 12:14:50 PM: Contact, thread 7
9/12/2012 12:15:03 PM: About, thread 7
9/12/2012 12:15:05 PM: Contact, thread 8
9/12/2012 12:15:23 PM: About, thread 6
9/12/2012 12:15:42 PM: About, thread 14
9/12/2012 12:16:01 PM: About, thread 6
9/12/2012 12:16:31 PM: Contact, thread 7
9/12/2012 12:16:33 PM: About, thread 14
...
9/12/2012 12:17:08 PM: Contact, thread 12
9/12/2012 12:17:09 PM: About, thread 12
9/12/2012 12:17:28 PM: About, thread 5
...

From Fiddler for the same requests (note that I've also been playing around with my role endpoint settings, to no avail):

So...the take-home messages here are:

The HTTP traffic seems normal (Fiddler is not showing any duplicate requests)
The action methods are being called twice or three times sometimes for the same request, seemingly non-deterministically (i.e. the original problem described above)
Interestingly, in my situation, it seems as though exactly 19s goes by before another 'hit' happens on the action method (i.e. when action methods are being hit multiple times, the time difference between the hits is deterministic)
The thread IDs are generally different for each hit whenever the action methods are hit multiple times

I'll keep digging deeper...maybe somebody would be kind enough to offer a suggestion based on this updated info though :).

EDIT 2 I decided to print out the process ID along with the thread ID inside the action method in each case. For every request, the process ID was the one corresponding to the IIS Express Worker Process. So it seems as though the IIS Express Worker Process is sometimes spawning multiple threads to handle the request, each one 19s apart.

...and deeper I go... :)

EDIT 3

I think I'm getting closer to the problem...I had a look at the IIS Express tray and saw the following after I had started the application (without debugging):

I decided to use this URL directly (instead of 127.0.0.2:8888 as shown in Fiddler above), and the problem seemed to go away. This is now starting to make sense...if I invoke the IIS application directly over port 8889, everything works fine as it did with the original non-Azure project. If I invoke the web role (i.e. the Azure-specific part) over port 8888, it sometimes seems to invoke the IIS application multiple times.

So I'm starting to narrow my focus on the web role now. I'll try to see if I can find some relevant web role logs...

EDIT 4

I made sure diagnostics was set up properly on the application and ran it in debug mode. I then went to the Azure Compute Emulator UI, right-clicked on my web-role instance, and selected "Open Local Store". This took me to Explorer, where I was able to find a child folder "temp\temp\RoleTemp\iisexpress" which had a log file in it with contents that looked like this (note that I've changed my endpoint back to port 80, so the IIS Express application is now back on port 81):

Request started: "GET" 127.0.0.1:81/Home/About
Request ended: 127.0.0.1:81/Home/About with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/About
Request ended: 127.0.0.1:81/Home/About with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/About
Request ended: 127.0.0.1:81/Home/About with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/About
Request ended: 127.0.0.1:81/Home/About with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/About
Request ended: 127.0.0.1:81/Home/About with HTTP status 200.0
Request started: "GET" 127.0.0.1:81/Home/Contact
Request ended: 127.0.0.1:81/Home/Contact with HTTP status 200.0

Again I was just toggling between the About and Contact pages, so from the log one can see that a single web-role endpoint request sometimes resulted in the web role calling the IIS Application multiple times. Now to the search for some higher-level web-role logs to figure out why it's doing this :)!

So based on what I wrote in the various edits of the original post:

On my machine at least, the DevFC.exe process (development fabric, Azure compute emulator) is sometimes spawning multiple requests for the requested resource from IIS Express. That is, it'll take a single request on the emulator endpoint, and sometimes spawn multiple requests to the underlying IIS Express application, one after another, even though they all succeed with HTTP status code 200. The time difference between these spawned requests seems to be constant, but aside from that, the behavior seems non-deterministic (i.e. it doesn't happen for every request to the emulator endpoint).
In order to debug this, I'd need access to the source code of the Azure compute emulator, which I'm guessing I can't get.

Now, my theory (and hope) is that this behavior is somehow specific to my development environment and won't be reproducible on the real Azure infrastructure. That said, I have come up with a workaround...

I created a simple proxy file for Firefox to redirect all traffic going to my local Azure emulator port to the IIS Express application port. This solves the problem, however I'm aware that I'm essentially bypassing the Azure emulator now, and I'm guessing I may have issues in the future because I'm doing that (e.g. if I decide to use the AppFabric cache?)!

In the mean-time I'll continue with this workaround for development and periodically deploy to my Azure staging environment for testing there.

Regardless, if anybody knows how I might solve the fundamental problem, let me know! :)

EDIT: I am no longer able to reproduce the issue using version 2.2 of the Azure SDK Tools. If you're still seeing this behaviour, try upgrading!