WCF Data Services Error - 302 Moved on SaveChanges

Question

Problem: DataServiceContext.SaveChanges() fails with "302 - moved" response.

Background/Suspected Cause: Load balancer! - We recently changed our infrastructure so that our web servers now sit behind a load balancer which also handles the ssl. Clients address the service as HTTPS but IIS ends up processing HTTP requests since the SSL was done by the load balancer(I am sure most of you are familiar with this type of setup). Anyway, what we end up with is a feed that contains URIs that are using HTTP rather than HTTPS. (see GET request/response below with http in the response instead of httpS). The behavior is pretty strange, as when I call saveChanges() it sends a MERGE(as expected), and I get back a 302:

HTTP/1.1 302 Object moved
Location: https://some.domain.org/CMSProfileService/ProfileDataService.svc/Mails(guid'80fef993-a4b5-4343-a908-28c2c6517a81')
Connection: close

but WCF keeps trying the MERGE over and over using HTTP(about 50 times) then finally throws an exception with the message, ""An error occurred while processing this request.", inner excepion message is "Found". :)

When I point directly at the server(bypassing the loadbalancer and ssl) everything works fine. Everything also works fine when SSL is registered directly in IIS.

There must be some config setting/property that I am not finding. Viteks answer to a similar question scares me a bit.

This is the raw GET request from fiddler

GET https://some.domain.org/CMSProfileService2/ProfileDataService.svc/Mails()?$filter=Status%20eq%20'Queued'&$orderby=Timestamp&$expand=Attachments HTTP/1.1
User-Agent: Microsoft ADO.NET Data Services
DataServiceVersion: 1.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
UserName: 
Accept: application/atom+xml,application/xml
Accept-Charset: UTF-8
Host: some.domain.org
Connection: Keep-Alive

---

and here is the raw response(trimmed):

HTTP/1.1 200 OK
Set-Cookie: ARPT=RPZVOOS192.168.94.118CKOUM; path=/
Cache-Control: no-cache
Content-Length: 45849
Content-Type: application/atom+xml;charset=utf-8
Server: Microsoft-IIS/7.5
DataServiceVersion: 1.0;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 22 Aug 2011 14:56:46 GMT

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="http://some.domain.org/CMSPRofileService2/ProfileDataService.svc/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
  <title type="text">Mails</title>
  <id>http://some.domain.org/CMSProfileService2/ProfileDataService.svc/Mails</id>
  <updated>2011-08-22T14:56:47Z</updated>
  <link rel="self" title="Mails" href="Mails" />
  <entry>
    <id>http://some.domain.org/CMSPRofileService2/ProfileDataService.svc/Mails(guid'c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd')</id>
    <title type="text"></title>
    <updated>2011-08-22T14:56:47Z</updated>
    <author>
      <name />
    </author>
    <link rel="edit" title="Mail" href="Mails(guid'c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd')" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Attachments" type="application/atom+xml;type=feed" title="Attachments" href="Mails(guid'c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd')/Attachments">
      <m:inline>
        <feed>
          <title type="text">Attachments</title>
          <id>http://some.domain.org/CMSPRofileService2/ProfileDataService.svc/Mails(guid'c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd')/Attachments</id>
          <updated>2011-08-22T14:56:47Z</updated>
          <author>
            <name />
          </author>
          <link rel="self" title="Attachments" href="Mails(guid'c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd')/Attachments" />
        </feed>
      </m:inline>
    </link>
    <category term="XXX.YYY.Profile.Repository.Mail" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <content type="application/xml">
      <m:properties>
        <d:MailId m:type="Edm.Guid">c7edb158-7a61-4fca-a40e-7f4a3a0b2bbd</d:MailId>
        <d:Timestamp m:type="Edm.DateTime">2011-07-28T12:51:37.69</d:Timestamp>
        <d:ApplicationCode>EREF</d:ApplicationCode>
        <d:Status>Queued</d:Status
.
.
.

Answer 1

The problem here is likely that the self/links edit links are HTTP, not HTTPS, so the client can't use the links to access the data. Per Vitek's scary answer.

The unfortunate thing is the server doesn't know where it is.

I can think of a couple of options (neither of them particularly nice):

1. Use a custom IDataServiceHost2 that tells Data Services its root is https://... via the AbsoluteServiceUri property. That way Data Services should produce appropriate URLS
2. Implement a WCF Behavior that strips out http:// urls and replaces them with https://, this would involve a lot of buffering etc. Checkout this JSONP example which modifies responses from the server.

-Alex

Answer 2

Finally solved this nasty issue. There were actually two issues.

Issue #1: WCF Data Service hosted in IIS behind a load balancer with https creates feeds with http URIs rather than https. This is understandable and was my first guess into the issue. Through lots of digging I came across this article which provided a pretty straight forward solution.

In the constructor of my service I hook up to the processing request event

ProcessingPipeline.ProcessingRequest += ProcessingPipeline_ProcessingRequest;

In the event handler, I use the article mentioned and made some tweaks that basically take the host, scheme and port from the clientExpectsUri and applies them to the request Uri. I also just look for a custom header that I called "X-Client-Expects-RootUri" that clients outside the firewall need to send if they want valid feeds.

static void ProcessingPipeline_ProcessingRequest(object sender, DataServiceProcessingPipelineEventArgs e)
{
    if (e.OperationContext.RequestHeaders.AllKeys.Contains(ClientExpectsUriKey))
        ProcessUri(new Uri(e.OperationContext.RequestHeaders[ClientExpectsUriKey]));    
}

private static void ProcessUri(Uri clientExpectsRootUri)
{
    if (clientExpectsRootUri != null)
    {
        var requestUri = OperationContext.Current.IncomingMessageProperties.ContainsKey("MicrosoftDataServicesRequestUri") 
                                ? OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] as Uri : HttpContext.Current.Request.Url;

        var serviceUri = clientExpectsRootUri;

        var serviceUriBuilder = new UriBuilder(serviceUri);

        var requestUriBuilder = new UriBuilder(requestUri)
                                    {
                                        Host = serviceUriBuilder.Host,
                                        Scheme = serviceUriBuilder.Scheme,
                                        Port = serviceUriBuilder.Port
                                    };

        if (!serviceUriBuilder.Path.EndsWith("/")) //the base uri should end with a slash... 
            serviceUriBuilder.Path = serviceUriBuilder.Path += "/";

        OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRootUri"] = serviceUriBuilder.Uri;
        OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = requestUriBuilder.Uri;
    }
}

I would actually like some feedback on this design, as I am basically asking the client to tell me how to return the URI via a custom header. I will be looking into modifying the load balancer in the future to add these headers automatically.

Issue #2: Load balancer was not configured for MERGE and other non-RFC http methods. The solution to this issue was simple, just set usePostTunneling to true on the DataServiceContext. If anyone has the scripts to enable non-RFC http methods for a cisco 11503, I'd love to have them. ;-)