In HTTP/1.1 specs I get this when it comes to define headers:
message-header = field-name ":" [ field-value ]
[...]
field-value = *( field-content | LWS )
field-contet = <the OCTETs making up the field-value and consisting of either *TEXT or combinations of token, separators, and quoted-string>
and the definition for OCTET and TEXT is:
OCTET = <any 8-bit sequence of data>
TEXT = <any OCTET except CTLs, but including LWS> ; where CTL refers to control characters from US-ASCII charset.
Question: Now, when it comes to header names (called field-names in definition), the encoding used is US-ASCII (specified in HTTP/1.1 specs), but how would a server application know what encoding to use for header values?
Note: I think it's normal to be US-ASCII encoded, but the definition lets enough room for different situation.