Python special characters in Strings

Question

I am trying to show email within a webpage. The program is written in Python. Unfortunately, I have some character encoding issues. I have quotes and double quotes in the text.

Original mail:

“All is good”
‘it is getting better’

with character set 'windows-1252' I get from the ISP:

=93All is good=94
=91it is getting better=92

with character set 'utf-8' I get from the ISP:

=E2=80=9CAll is good=E2=80=9D
=E2=80=98it is getting better=E2=80=99

I replace the =.. with their corresponding hex characters. The text then looks like:

character set 'windows-1252'
ôAll is goodö
æit is getting betterÆ


character set 'utf-8'
ΓÇ£All is goodΓÇ¥
ΓÇÿit is getting betterΓÇÖ

Subsequent calls to the unicode function fail with

UnicodeEncodeError: 'charmap' codec can't encode character u'\u201d' in position 6: 
character maps to <undefined>

or similar.

The call looks like unicode( message, 'utf-8', 'replace' ). Any idea what I am doing wrong?

Answer 1

Why are you replacing anything with anything?

>>> m = email.message_from_string('''Content-Type: text/plain; utf-8\nContent-Transfer-Encoding: quoted-printable\n\n=E2=80=9CAll is good=E2=80=9D\n=E2=80=98it is getting better=E2=80=99''')
>>> m.get_payload(decode=True).decode(m['Content-Type'].split('; ')[1])u'\u201cAll is good\u201d\n\u2018it is getting better\u2019'

Answer 2

Because did I try this and I ran into problems. Here is another try:

The output looks like:

# lines is already prefilled with a valid HTML message
m = email.message_from_string( lines );
email.iterators._structure( m );
print m.is_multipart();
print m.get_payload( decode=True );
print m.get_payload();

The output looks like:

>>> execfile( 'email2.py' )
multipart/alternative
     text/plain
     text/html
True
None
[<email.message.Message instance at 0x0235FDF0>, <email.message.Message instance at 0x02355F08>]

You see, fails if I use decode='true'. Here is the simplified email:

Content-Type: multipart/alternative;
    boundary="----=_NextPart_000_0130_01CC1E30.41026040"

This is a multi-part message in MIME format.

------=_NextPart_000_0130_01CC1E30.41026040
Content-Type: text/plain;
    charset="utf-8"
Content-Transfer-Encoding: quoted-printable

plain

------=_NextPart_000_0130_01CC1E30.41026040
Content-Type: text/html;
    charset="utf-8"
Content-Transfer-Encoding: quoted-printable

html

------=_NextPart_000_0130_01CC1E30.41026040--