How should I decode a UTF-8 string

Question

I have a string like:

About \xee\x80\x80John F Kennedy\xee\x80\x81\xe2\x80\x99s Assassination . unsolved mystery \xe2\x80\x93 45 years later. Over the last decade, a lot of individuals have speculated on conspiracy theories that ...

I understand that \xe2\x80\x93 is a dash character. But how should I decode the above string in C#?

Answer 1

Scan the input string char-by-char and convert values starting with \x (string to byte[] and back to string using UTF8 decoder), leaving all other characters unchanged:

static string Decode(string input)
{
    var sb = new StringBuilder();
    int position = 0;
    var bytes = new List<byte>();
    while(position < input.Length)
    {
        char c = input[position++];
        if(c == '\\')
        {
            if(position < input.Length)
            {
                c = input[position++];
                if(c == 'x' && position <= input.Length - 2)
                {
                    var b = Convert.ToByte(input.Substring(position, 2), 16);
                    position += 2;
                    bytes.Add(b);
                }
                else
                {
                    AppendBytes(sb, bytes);
                    sb.Append('\\');
                    sb.Append(c);
                }
                continue;
            }
        }
        AppendBytes(sb, bytes);
        sb.Append(c);
    }
    AppendBytes(sb, bytes);
    return sb.ToString();
}

private static void AppendBytes(StringBuilder sb, List<byte> bytes)
{
    if(bytes.Count != 0)
    {
        var str = System.Text.Encoding.UTF8.GetString(bytes.ToArray());
        sb.Append(str);
        bytes.Clear();
    }
}

Output:

About John F Kennedy’s Assassination . unsolved mystery – 45 years later. Over the last decade, a lot of individuals have speculated on conspiracy theories that ...

Answer 2

If you have a string like that, then you have used the wrong encoding when you decoded it in the first place. There is no "UTF-8 string", the UTF-8 data is whent the text is encoded into binary data (bytes). When it's decoded into a string, then it's not UTF-8 any more.

You should use the UTF-8 encoding when you create the string from binary data, once the string is created using the wrong encoding, you can't reliably fix it.

If there is no other alternative, you could try to fix the string by encoding it again using the same wrong encoding that was used to create it, and then decode it using the corrent encoding. There is however no guarantee that this will work for all strings, some characters will simply be lost during the wrong decoding. Example:

// wrong use of encoding, to try to fix wrong decoding
str = Encoding.UTF8.GetString(Encoding.Default.GetBytes(str));

Answer 3

Finally I've used something like this:

public static string UnescapeHex(string data)
{
    return Encoding.UTF8.GetString(Array.ConvertAll(Regex.Unescape(data).ToCharArray(), c => (byte) c));
}