How `[System.Console]::OutputEncoding/InputEncodin

Question

Under Powershell v5, Windows 8.1, Python 3. Why these fails and how to fix?

[system.console]::InputEncoding = [System.Text.Encoding]::UTF8; 
[system.console]::OutputEncoding = [System.Text.Encoding]::UTF8; 
chcp; 
"import sys
print(sys.stdout.encoding)
print(sys.stdin.encoding)
sys.stdout.write(sys.stdin.readline())
" | 
sc test.py -Encoding utf8; 
[char]0x0422+[char]0x0415+[char]0x0421+[char]0x0422+"`n" | py -3 test.py

prints:

Active code page: 65001
cp65001
cp1251
п»ї????

Answer 1

You are piping data into Python; at that point Python's stdin is no longer attached to a TTY (your console) and won't guess at what the encoding might be. Instead, the default system locale is used; on your system that's cp1251 (the Windows Latin-1-based codepage).

Set the PYTHONIOENCODING environment variable to override:

PYTHONIOENCODING
If this is set before running the interpreter, it overrides the encoding used for stdin/stdout/stderr, in the syntax encodingname:errorhandler. Both the encodingname and the :errorhandler parts are optional and have the same meaning as in str.encode().

PowerShell doesn't appear to support per-command-line environment variables the way UNIX shells do; the easiest is to just set the variable first:

Set-Item Env:PYTHONIOENCODING "UTF-8"

or even

Set-Item Env:PYTHONIOENCODING "cp65001"

as the Windows UTF-8 codepage is apparently not quite UTF-8 really, depending on the Windows version and on wether or not pipe redirection is used.

Answer 2

Why not embed CPython in powershell?! CPython is so easy to embed, and powershell is very good REPL to play with .NET and COM objects. Here is a simple introduction to using pythonnet from PowerShell. Note how encoding is automatically propagated from powershell to python.

Windows PowerShell
Copyright (C) 2015 Microsoft Corporation. All rights reserved.

PS C:\Users\denfromufa> [system.console]::InputEncoding = [System.Text.Encoding]::UTF8;
PS C:\Users\denfromufa> [system.console]::OutputEncoding = [System.Text.Encoding]::UTF8;
PS C:\Users\denfromufa> [Reflection.Assembly]::LoadFile("C:\Python\Miniconda3_64b\Lib\site-packages\Python.Runtime.dll")


GAC    Version        Location
---    -------        --------
False  v4.0.30319     C:\Python\Miniconda3_64b\Lib\site-packages\Python.Runtime.dll


PS C:\Users\denfromufa> $gil = [Python.Runtime.Py]::GIL()
PS C:\Users\denfromufa> $sys=[Python.Runtime.Py]::Import("sys")
PS C:\Users\denfromufa> $sys.stdin.encoding.ToString()
cp65001
PS C:\Users\denfromufa> $sys.stdout.encoding.ToString()
cp65001
PS C:\Users\denfromufa> $gil.Dispose()
PS C:\Users\denfromufa> [Python.Runtime.PythonEngine]::Shutdown()
PS C:\Users\denfromufa>

---

[EDIT]

Here is snek package that was released by one of powershell developers for embedding Python in powershell:

https://github.com/adamdriscoll/snek