.net webclient returns 500 error, but url in brows

Question

When I connect to a url via a web client, it returns a 500 internal server error. I can visit that URL in my browser just fine. This was working for months and the problem started yesterday. No changes on my end. I understand that a 500 status code is a server error, but I wanted to see if there was something else I could do at this point as I'm freaking out.

Example url: http://www.myfitnesspal.com/reports/printable_diary/chuckgross?from=2015-07-17&to=2015-07-17

The code:

 string results;
 using (var client = new WebClient())
 {
    results = client.DownloadString(url);
 }

This gives the exception: The remote server returned an error: 500.

Using Fiddler, I'm seeing a page returned that is their "Site Down" page with the message: "Sorry, but a server error occurred processing your request. Our team has been notified of the issue".

Is there anything that I can do given that the URL itself works fine in a browser? Any alternative ways to try to troubleshoot?

Answer 1

It seems they try to detect browser version on server side by querying User-Agent header but do not expect it to be missing.

To fix error on client side you can for instance fill it with one used by IE 9 before sending request:

client.Headers["User-Agent"] = "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)";

Answer 2

It's normal to receive an exception when the server returns a HTTP error while using the WebClient class, it signals that the server couldn't complete the request successfully. That doesn't means that it didn't return a proper html response, however. You should catch the exception, where you can get the response stream object and extract the data from there.