REST WCF - stream download is VERY slow with 65535

Question

we have a WCF method that returns a stream - exposed via REST. we compared a regular download (from web site) to the WCF method, and we found out the following for 70MB file:

• in regular site - the download took ~10 seconds - 1MB chunk size
• in WCF method - took ~20 seconds - the chunk size was ALWAYS 65,535 bytes.

we have a custom stream that actually streams into another product, which makes the difference of times ever worse - 1 minute for regular site, while it takes 2 minutes for the WCF.

because we need to support very large files - its getting crucial.

we stopped on debug, and found out that the method "Read" of the Stream that the WCF calls always have a chunk size of 65,535 - which causes the slowness.

we tried several server configurations - like this:

The endpoint:

   <endpoint address="Download" binding="webHttpBinding" bindingConfiguration="webDownloadHttpBindingConfig"  behaviorConfiguration="web" contract="IAPI" />

The binding:

<binding name="webDownloadHttpBindingConfig" maxReceivedMessageSize="20000000" maxBufferSize="20000000" transferMode="Streamed">
                              <readerQuotas maxDepth="32" maxStringContentLength="20000000" maxArrayLength="20000000" maxBytesPerRead="20000000" maxNameTableCharCount="20000000"/>
                              <security mode="Transport">
                                     <transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
                              </security>
                       </binding>

The client which is a REST client (cannot use WCF binding - we don't want to reference it) - is built this way:

System.Net.HttpWebRequest request = (HttpWebRequest)WebRequest.Create(CombineURI(BaseURL, i_RelativeURL));

     request.Proxy = null; // We are not using proxy
     request.Timeout = i_Timeout;
     request.Method = i_MethodType;
     request.ContentType = i_ContentType;

     string actualResult = string.Empty;
     TResult result = default(TResult);
     if (!string.IsNullOrEmpty(m_AuthenticationToken))
     {
        request.Headers.Add(ControllerConsts.AUTH_HEADER_KEY, m_AuthenticationToken);
     }

     using (var response = request.GetResponse())
        {
           using (Stream responseStream = response.GetResponseStream())
           {
              byte[] buffer = new byte[1048576];

              int read;
              while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
              {
                 o_Stream.Write(buffer, 0, read);
              }
           }
        }

basically we're just streaming into a stream.

so, no matter we do - the server ALWAYS receives chunk size of 65,535 (we tried several client / server configurations)

What are we missing?

Thanks!

== EDIT 8/4/15 Microsoft response == Hi, we worked with microsoft about this case, this is their answer:

When the WCF client calls a WCF method that returns a Stream, it actually gets a reference to a MessageBodyStream instance. MessageBodyStream ultimately relies on WebResponseInputStream to actually read data, through this graph of relationships:

• MessageBodyStream has a member, message, that references an InternalByteStreamMessage instance
• InternalByteStreamMessage has a member, bodyWriter, that references a StreamBasedStreamedBodyWriter instance
• StreamBasedStreamedBodyWriter has a member, stream, that references a MaxMessageSizeStream instance
• MaxMessageSizeStream has a member, stream, that references a WebResponseInputStream instance

When you call Read() on the stream, WebResponseInputStream.Read() is ultimately called (you can test this yourself by setting the breakpoint in Visual Studio – one caveat: “Just My Code” option in Visual Studio – Debugging must be disabled, in order for the breakpoint to be hit). The relevant part of WebResponseInputStream.Read() is the following:

                    return BaseStream.Read(buffer, offset, Math.Min(count, maxSocketRead));

where maxSocketRead is defined to be 64KB. The comment above maxSocketRead says “in order to avoid blowing kernel buffers, we throttle our reads. http.sys deals with this fine, but System.Net doesn't do any such throttling.”. This means that if you specify too large a read value, you exceed the kernel’s own buffer size and causes poorer performance as it needs to do more work.

Does this cause a performance bottleneck? No, it should not. Reading too few bytes at a time (say, 256 bytes) will cause a performance degradation. But 64KB should be a value that causes good performance. In these cases, the real bottleneck is typically the network bandwidth, not how fast data is read by the client. In order to maximize performance, it is important that the reading loop is as tight as possible (in other words, there are no significant delays between reads). Let’s also keep in mind that objects larger than 80KB go to the Large Object Heap in .Net, which has a less efficient memory management than the “normal” heap (compaction does not take place under normal conditions, so memory fragmentation can occur).

Answer 1

we worked with microsoft about this case, this is their answer:

When the WCF client calls a WCF method that returns a Stream, it actually gets a reference to a MessageBodyStream instance. MessageBodyStream ultimately relies on WebResponseInputStream to actually read data, through this graph of relationships:

MessageBodyStream has a member, message, that references an InternalByteStreamMessage instance InternalByteStreamMessage has a member, bodyWriter, that references a StreamBasedStreamedBodyWriter instance StreamBasedStreamedBodyWriter has a member, stream, that references a MaxMessageSizeStream instance MaxMessageSizeStream has a member, stream, that references a WebResponseInputStream instance When you call Read() on the stream, WebResponseInputStream.Read() is ultimately called (you can test this yourself by setting the breakpoint in Visual Studio – one caveat: “Just My Code” option in Visual Studio – Debugging must be disabled, in order for the breakpoint to be hit). The relevant part of WebResponseInputStream.Read() is the following:

                return BaseStream.Read(buffer, offset, Math.Min(count, maxSocketRead));

where maxSocketRead is defined to be 64KB. The comment above maxSocketRead says “in order to avoid blowing kernel buffers, we throttle our reads. http.sys deals with this fine, but System.Net doesn't do any such throttling.”. This means that if you specify too large a read value, you exceed the kernel’s own buffer size and causes poorer performance as it needs to do more work.

Does this cause a performance bottleneck? No, it should not. Reading too few bytes at a time (say, 256 bytes) will cause a performance degradation. But 64KB should be a value that causes good performance. In these cases, the real bottleneck is typically the network bandwidth, not how fast data is read by the client. In order to maximize performance, it is important that the reading loop is as tight as possible (in other words, there are no significant delays between reads). Let’s also keep in mind that objects larger than 80KB go to the Large Object Heap in .Net, which has a less efficient memory management than the “normal” heap (compaction does not take place under normal conditions, so memory fragmentation can occur).

Possible solution: is to cache in memory bigger chunks (for example, use MemoryStream and while the WCF Stream calls your custom "Read" - cache inside 1MB or more / less - whatever you want.

then, when 1MB (or other value) exceeds - push it to your actual custom stream, and continue caching bigger chunks

this wasn't checked but i think it should solve the performance issues.