C# HttpWebRequest Server not returning full respon

Question

I am making a HTTP request to an server that returns HTML with data. But sometimes it "stops in the middle" without any clear explanation. For example the end of response:

[...Content length 14336 chars ...]</tbody></table><p /><br /><ul id=

It simply stops in the middle of generating the HTML. My code:

var request = (HttpWebRequest) WebRequest.Create("http://example.com);
var authInfo = string.Format("{0}:{1}", "username", "password");
authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
request.Headers.Add("Authorization", "Basic " + authInfo);
request.Credentials = new NetworkCredential(user.Xname, decryptedPassword);
request.Method = WebRequestMethods.Http.Get;
request.AllowAutoRedirect = true;
request.Proxy = null;
var response = (HttpWebResponse) request.GetResponse();
var stream = response.GetResponseStream();
var streamreader = new StreamReader(stream, Encoding.GetEncoding("ISO-8859-2"));
var s = streamreader.ReadToEnd();

Is it somehow possible that the code is not waiting until the end?

Answer 1

When you are executing a http web request locally (debugging) there almost no network delay and everything is fine, however when you are using the same code on internet the servers does not send all response data in one chunk, server could be busy processing other requests or there could be some network latency, then, you receive the response data in several chunks.

The ReadToEnd methods from StreamReader will read just the data available when you received the first chunk of data into the stream, and won't wait for further data.

This not means the response is complete and you have received all response data, next code found somewhere on internet handle the read process properly... (code is not mine and I cannot get the credit for it)

    public static byte[] ReadFully(Stream stream, int initialLength)
    {
        // If we've been passed an unhelpful initial length, just
        // use 32K.
        if (initialLength < 1)
        {
            initialLength = 32768;
        }


        byte[] buffer = new byte[initialLength];
        int read = 0;


        int chunk;
        while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
        {
            read += chunk;


            // If we've reached the end of our buffer, check to see if there's
            // any more information
            if (read == buffer.Length)
            {
                int nextByte = stream.ReadByte();


                // End of stream? If so, we're done
                if (nextByte == -1)
                {
                    return buffer;
                }


                // Nope. Resize the buffer, put in the byte we've just
                // read, and continue
                byte[] newBuffer = new byte[buffer.Length * 2];
                Array.Copy(buffer, newBuffer, buffer.Length);
                newBuffer[read] = (byte)nextByte;
                buffer = newBuffer;
                read++;
            }
        }
        // Buffer is now too big. Shrink it.
        byte[] ret = new byte[read];
        Array.Copy(buffer, ret, read);
        return ret;
    }