What I tried it to add new header:

request.Headers.GetType().InvokeMember("ChangeInternal",
    BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod,
    Type.DefaultBinder, request.Headers, new object[] { "Connection", "keep-alive" }
);

Actually it adds keep-alive header into Connection but it doesn't replace old one. So I get Connection: Keep-Alive,keep-alive.

I tried experimenting with Reflection but didn't got anything working.

There is other similar questions about this but there was no solution.

Just do the following:

request.Headers.Remove("Connection");
request.Headers.Add("Connection", "keep-alive");

It's not neccessary to set these headers via reflection. In the first place it's important to remove the old entry as an call to Add adds another value if the key already exists (the result you saw with comma separated values).

It'd be even better to use the HttpRequestHeader Enumeration instead of the header name as string:

request.Headers.Remove(HttpRequestHeader.Connection);
request.Headers.Add(HttpRequestHeader.Connection, "keep-alive");

Edit:

My bad. There's an explicit Connection property on the request-object which must be used in that case:

request.Connection = "keep-alive";

FYI: There are some more headers that must be set via their explicit propertries. For a list refer to this page, section remarks: https://msdn.microsoft.com/en-us/library/System.Net.HttpWebRequest%28v=vs.110%29.aspx

Edit2:

Well, looking at the connection property's source code, you can see that it restricts setting these values:

bool fKeepAlive = text.IndexOf("keep-alive") != -1;
bool fClose = text.IndexOf("close") != -1;
if (fKeepAlive || fClose)
{
    throw new ArgumentException(SR.GetString("net_connarg"), "value");
}

So you have 2 options:

Stick with the upper-case value (which I'd prefer) as anyway you have no real reason for it being lower-case ("So I want to have headers exactly as my for example firefox browser."). And as Darin Dimitrov already stated, headers shouldn't be case-sensitive anyway.
Extend your reflection-approach in that way, that you first remove the header an then set it again in lower-case.