How do I emulate Internet Explorer url encoding in

Question

The objective:

Make a console app that send a simple get request - in exactly the right way.

The problem:

I have a url like this: http://myhost/somepage.do?Search01=コード番号=123456&Search02=改訂番号=2

When I copy and past this into IE 11 or lower, the page I want returns correctly. When I copy and past this into IE Edge, chrome, firefox, it returns an error saying it does not properly recognize the parameters.

I investigated this with Wireshark, and I can see quite clearly that ol' IE is sending the url with some kind of different encoding:

Whereas Chrome does a more expected encoding:

I don't fully understand what is happening here, but it seems that this server I am sending the message to, is somewhat TAILORED towards ol' IE's messed up way of encoding the url - because it is only replying to the messed up requests.

I have checked other things like the useragent etc - it makes no difference. This server is running a service which is very old (maybe using ASP).

So, my objective is to emulate this messed up encoding in a console app. How do I do it?

Answer 1

So, with some help from understanding what might be happening via: This stackoverflow question

I came to realize how my url is being encoded.

My computer is Japanese, so the default codepage is 932. After much messing around with a sample console app, and watching the packets in Wireshark, I realized that no matter what I did, the default HttpClient and WebClient will always UrlEncode my URL correctly regardless of what encoding I used. This is not how ol' IE encodes it's URLs.

I dug deeper and found that in the source for HttpClient (and WebClient) it uses the class Uri, which has a constructor with parameter : DontEscape which I thought, "Eurika!" but it turns out this constructor is Obsolete, and there is no way to not make URL's automatically escape themselves when using HttpClient or WebClient.

So I had to use TcpClient and make my own request instead. Which I stole from here:

    /// <summary>
    /// The initial request to search only works if the url is encoded using Shift-JIS, which means we cannot use any client library and must use a custom TCP message.
    /// </summary>
    /// <param name="serveripaddress"></param>
    /// <param name="restoftheurl"></param>
    /// <returns></returns>
    private async Task<string> HttpRequestAsync(string serveripaddress, string restoftheurl)
    {
        Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
        string result = string.Empty;

        using (var tcp = new TcpClient(serveripaddress, 80))
        using (var stream = tcp.GetStream())
        {
            tcp.SendTimeout = 500;
            tcp.ReceiveTimeout = 1000;
            Console.WriteLine("URL rest:" + restoftheurl);
            // Send request headers
            var builder = new StringBuilder();
            builder.AppendLine("GET " + restoftheurl + " HTTP/1.1");
            builder.AppendLine("Host: " + serveripaddress);
            //builder.AppendLine("Content-Length: " + data.Length);   // only for POST request
            builder.AppendLine("Accept: text/html, application/xhtml+xml, */*");
            builder.AppendLine("Accept-Language: ja-JP");
            builder.AppendLine("User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko");
            builder.AppendLine("Accept-Encoding: gzip, deflate");
            builder.AppendLine("Connection: Close");
            builder.AppendLine();
            Console.WriteLine("Sending message:" + builder.ToString());
            var header = Encoding.GetEncoding(932).GetBytes(builder.ToString());
            await stream.WriteAsync(header, 0, header.Length);

            // Send payload data if you are POST request
            //await stream.WriteAsync(data, 0, data.Length);

            // receive data
            using (var memory = new MemoryStream())
            {
                await stream.CopyToAsync(memory);
                memory.Position = 0;
                var data = memory.ToArray();

                var index = BinaryMatch(data, Encoding.ASCII.GetBytes("\r\n\r\n")) + 4;
                var headers = Encoding.ASCII.GetString(data, 0, index);
                memory.Position = index;

                if (headers.IndexOf("Content-Encoding: gzip") > 0)
                {
                    using (GZipStream decompressionStream = new GZipStream(memory, CompressionMode.Decompress))
                    using (var decompressedMemory = new MemoryStream())
                    {
                        decompressionStream.CopyTo(decompressedMemory);
                        decompressedMemory.Position = 0;
                        result = Encoding.UTF8.GetString(decompressedMemory.ToArray());
                    }
                }
                else
                {
                    result = Encoding.UTF8.GetString(data, index, data.Length - index);
                    //result = Encoding.GetEncoding("gbk").GetString(data, index, data.Length - index);
                }
            }

            //Debug.WriteLine(result);
            return result;
        }
    }

    private int BinaryMatch(byte[] input, byte[] pattern)
    {
        int sLen = input.Length - pattern.Length + 1;
        for (int i = 0; i < sLen; ++i)
        {
            bool match = true;
            for (int j = 0; j < pattern.Length; ++j)
            {
                if (input[i + j] != pattern[j])
                {
                    match = false;
                    break;
                }
            }
            if (match)
            {
                return i;
            }
        }
        return -1;
    }

}

The key part of this code is:

var header = Encoding.GetEncoding(932).GetBytes(builder.ToString());

This forces the string to be encoded in my codepage, which required that the codepage provider was registered, so at the top: Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);

The usage is simple:

await HttpRequestAsync("123.456.789.123", "/somepage.do?Search01=コード番号=123456&Search02=改訂番号=2");