Python: Which encoding is used for processing sys.

Question

In what encoding are the elements of sys.argv, in Python? are they encoded with the sys.getdefaultencoding() encoding?

sys.getdefaultencoding(): Return the name of the current default string encoding used by the Unicode implementation.

PS: As pointed out in some of the answers, sys.stdin.encoding would indeed be a better guess. I would love to see a definitive answer to this question, though, with pointers to solid sources!

PPS: As Wim pointed out, Python 3 solves this issue by putting str objects in sys.argv (if I understand correctly). The question remains open for Python 2.x, though. Under Unix, the LC_CTYPE environment variable seems to be the correct thing to check, no? What should be done with Windows (so that sys.argv elements are correctly interpreted whatever the console)?

Answer 1

"What should be done with Windows (so that sys.argv elements are correctly interpreted whatever the console)?"

For Python 2.x, see this comment on issue2128.

(Note that no encoding is correct for the original sys.argv, because some characters may have been mangled in ways that there is not enough information to undo; for example, if the ANSI codepage cannot represent Greek alpha then it will be mangled to 'a'.)

Answer 2

A few observations:

(1) It's certainly not sys.getdefaultencoding.

(2) sys.stdin.encoding appears to be a much better bet.

(3) On Windows, the actual value of sys.stdin.encoding will vary, depending on what software is providing the stdio. IDLE will use the system "ANSI" code page, e.g. cp1252 in most of Western Europe and America and former colonies thereof. However in the Command Prompt window, which emulates MS-DOS more or less, the corresponding old DOS code page (e.g. cp850) will be used by default. This can be changed by using the CHCP (change code page) command.

(4) The documentation for the subprocess module doesn't provide any suggestions on what encoding to use for args and stdout.

(5) One trusts that assert sys.stdin.encoding == sys.stdout.encoding never fails.

Answer 3

I'm guessing that you are asking this because you ran into issue 2128. Note that this has been fixed in Python 3.0.

Answer 4

I don't know if this helps or not but this is what I get in DOS mode:

C:\Python27>python Lib\codingtest.py нер
['Lib\\codingtest.py', '\xed\xe5\xf0']

C:\Python27>python Lib\codingtest.py hello
['Lib\\codingtest.py', 'hello']

In IDLE:

>>> print "hello"
hello
>>> "hello"
'hello'
>>> "привет"
'\xef\xf0\xe8\xe2\xe5\xf2'
>>> print "привет"
привет
>>> sys.getdefaultencoding()
'ascii'
>>> 

What can we deduce from this? I don't know yet... I'll comment in a little bit.

A little bit later: sys.argv is encoded with sys.stdin.encoding and not sys.getdefaultencoding()

Answer 5

sys.getfilesystemencoding() works for me, at least on Windows. On Windows it is actually 'mbcs', and 'utf-8' on *nix.

Answer 6

On Unix systems, it should be in the user's locale, which is (strangely) not tied to sys.getdefaultencoding. See http://docs.python.org/library/locale.html.

In Windows, it'll be in the system ANSI codepage.

(By the way, those elementary school teachers who told you not to end a sentence with a preposition were lying to you.)