Line buffered serial input

Question

I have a serial device that I'm trying to read input from. I sent it a string "ID\r", and it returns "ID XX\r" (where \r is an ASCII carriage return, hex 0x0d).

Since the eol option on serial.readline is no longer supported, I'm using TextIOWrapper to read from the serial port and return a line at a time.

My problem is that instead of returning my string as soon as it sees the carriage return, it's waiting until the twice the timeout I set when I opened the serial port. I'd like it to return the string immediately as soon as it reads an entire line since I may have hundreds of these commands to send to the device and I don't want to wait for the timeout each time. If I set timeout to 0, then I get no output at all (presumably because my script stops waiting before the device has a chance to output anything), and if I set the timeout to None, the script blocks forever.

Here's a simple test script:

import serial
import io
import time

ser = serial.Serial("/dev/ttyUSB0", baudrate=9600,
                    bytesize=8, parity='N', stopbits=1,
                    xonxoff=0, rtscts=1, timeout=5)

sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser),
                       newline=None)


sio.write(unicode("ID\r"))
sio.flush()

print "reading..."

x = sio.readline()

print len(x)
print x

The script always takes 10 seconds from the time it says "reading" until it prints the "ID XX" string that it read from the serial port.

I'm certain that the device is outputting the carriage return, as I've used strace to watch the reads:

select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 991704})
read(3, "I", 8192)                      = 1
select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 999267})
read(3, "D", 8191)                      = 1
select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 999420})
read(3, " ", 8190)                      = 1
select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 999321})
read(3, "X", 8189)                      = 1
select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 999355})
read(3, "X", 8188)                      = 1
select(4, [3], [], [], {5, 0})          = 1 (in [3], left {4, 999171})
read(3, "\r", 8187)                     = 1
select(4, [3], [], [], {5, 0})          = 0 (Timeout)
select(4, [3], [], [], {5, 0})          = 0 (Timeout)

You can see the 2 select() timeouts that give the 10 second delay, but you can also clearly see the carriage return being read. I've tried setting the newline parameter to 'None' and '' (which should automatically allow \r, \n, and \r\n), and to '\r', but with the same result each time.

I've also tried setting the buffer_size in the BufferedRWPair() call to '1' to keep it from buffering input, but that made no difference.

Any idea what I'm doing wrong?

If I can't get this working, my next step will be to use serial.read() to read a character at a time and do my own line buffering, but I wanted to try to do it the "right" way with textiowrapper first.

Answer 1

Caveat: I am using Python 3.4 on a Mac so your mileage may vary, though I believe with TextIOWrapper backported to Python 2.7, the situation in Python 2.7 (and other OSs) will essentially be the same as what I describe below.

The main issue is that io.TextIOWrapper itself uses a buffering mechanism, controlled by the undocumented _CHUNK_SIZE attribute. This is pretty unpleasant. So you have two choices:

1. Use a timeout as you tried out. This is what is hinted on in the documentation of readline on the pyserial documentation page. However, if you use a large value (as you did), when there is not enough data to fill the buffer of TextIOWrapper, your code will block until the timeout is reached. This is what you are experiencing essentially (I did not go after why you have to wait double the timeout value, but I think that this could be sorted out by looking at the implementation of TextIOWrapper and ultimately is irrelevant to your question).

2. The second choice is to change _CHUNK_SIZE to 1. In your case, simply add the line

   sio._CHUNK_SIZE = 1
   

   to your code right after you initialized sio. This has the perhaps unpleasant effect that the buffering within TextIOWrapper itself will be turned off (this is used for the incremental decoding of the input). If performance is not an issue, this is the simplest solution. If performance is an issue, you can set a low value of timeout, not toucing _CHUNK_SIZE. However, in this case be prepared to get an empty string from readline() (if the device sends you an empty line, that will come through as '\n', so it can be distinguished from the empty string that you will get when a read runs out of the alloted time).

There is another problem with your code: When sio will be removed, the close method of ser will be called twice, which will result in an exception when your program will be about to finish (at least this is what happens if I try your code on my computer). You should create (it seems) to instances of serial.Serial and pass those to BufferedRWPair.

I have also created a wrapper class based on TextIOWrapper, which I could also post, if there is interest, just I did not want to litter response with some extra code which, strictly speaking, is not needed.

PS: In the meanwhile, I have experimented with the code on Ubuntu. While on my Mac, I did not see a need for setting the buffer size of io.BufferedRWPair to 1, on Ubuntu I had to do this, too, in addition to setting _CHUNK_SIZE to 1.

Answer 2

This would be hard to debug without actually being there to see it. But see if you can use my tty module.

http://code.google.com/p/pycopia/source/browse/trunk/aid/pycopia/tty.py

Try the SerialPort object in there. I've successfully used this to interact with serial instruments, where "that other serial module" had lots of problems similar to what you describe. This one can also tell you if you have data in the FIFO.

Let me now how it goes.

Answer 3

Wasted a few hours on this today. It turned out that io.BufferedReader reads until it has filled its buffer and then passes the buffer to io.TextIOWrapper. The default buffer size is 8192, so depending on your device this might take a while.

The correct example code should be:

# buffer size is 1 byte, so directly passed to TextIOWrapper
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser, 1), encoding='ascii')
print sio.readline()[:-1]

Answer 4

Thanks for the code Keith, but I wanted to keep this code somewhat portable, so I'd like to stick with the default "serial" package.

Plus, since I'm still learning Python, I wanted to try to learn how to use the TextIOWrapper in the way it was intended.

I gave up trying to make serial.readline() work, so for now I'll just use a simple "readLine" function to read a character at a time and look for a carriage return terminator. Though if I run into more serial quirkyness, I may revisit using your code.

Thanks!

def readLine(ser):
    str = ""
    while 1:
        ch = ser.read()
        if(ch == '\r' or ch == ''):  
            break
        str += ch

    #"print "str = " + str

    return str