Encoding issue with JLine

Question

Jline is a module for intercepting user input at a console before the user presses Enter. It uses JNA or similar wizardry.

I'm doing a few experiments with it and I'm getting encoding problems when I input more "exotic" Unicode characters. The OS here is W10 and I'm using Cygwin. Also this is in Groovy but should be obvious to Java people.

def terminal = org.jline.terminal.TerminalBuilder.builder().jna( true ).system( true ).build()
terminal.enterRawMode()
// NB the Terminal I get is class org.jline.terminal.impl.PosixSysTerminal
def reader = terminal.reader()

def bytes = [] // NB class ArrayList
int readInt = -1
while( readInt != 13 && readInt != 10 ) {
    readInt = reader.read()
    byte convertedByte = (byte)readInt
    // see what the binary looks like:
    String binaryString = String.format("%8s", Integer.toBinaryString( convertedByte & 0xFF)).replace(' ', '0')
    println "binary |$binaryString|"
    bytes << (byte)readInt // NB means "append to list"
    println ">>> read |$readInt| byte |$convertedByte|"
}
// strip final byte (13 or 10)
bytes = bytes[0..-2]
println "z bytes $bytes, class ${bytes.class.name}"
def response = new String( (byte[])bytes.toArray(), 'UTF-8' )
// to get proper out encoding for Cygwin I then need to do this (I have no idea why!)
def psOut = new PrintStream(System.out, true, 'UTF-8' )
psOut.print( "using PrintStream: |$response|" )

This works fine with one-byte Unicode, and letters like "é" (2-bytes) get handled fine. But it goes wrong with "ẃ":

ẃ --> Unicode U+1E83 
    UTF-8 HEX: 0xE1 0xBA 0x83 (e1ba83) 
    BINARY: 11100001:10111010:10000011

Actually the binary it puts out when you enter "ẃ" is 11100001:10111010:10010010.

This translates to U+1E92, which is another Polish character, "Ẓ". And that is indeed what gets printed out in the response String.

Unfortunately the JLine package hands you this reader, which is class org.jline.utils.NonBlocking$NonBlockingInputStreamReader... So I don't really know what I can do to investigate its encoding (I presume UTF-8) or somehow modify it... Can anyone explain what the problem is?

Answer 1

As far as I can tell this relates to a Cygwin-specific problem, as asked and then answered by me a year ago.

There is a solution, in my answer to the question I asked directly after this one... which correctly deals with Unicode input, even when outside the Basic Multilingual Plane, using JLine, ... and using a Cygwin console ... hopefully.