#150 io conformity and serials read()

[Erik Bernoth]

Problem

The read() method without parameters, in the sense of Python's io module is expected to read all bytes until an EOF is found, which happens indirectly by calling readall() (which indirectly also calls read(), but with another number. pyserial's read() defaults to reading one byte. And even actively calling read(-1) doesn't result in the expected behavior. This is somewhat frustrating, if you want to write code that is exchangeable for all Python io-like interfaces.

Solution

In the stdlib can be found, that read() is already implemented and either calls readall() if n < 0 or calls a readinto() method, which should be implemented by subclasses of RawIOBase. Therefore from my point of view, serial should implement it's reading capabilities in a readuntil() function and leave the other functions to the standard behaviour of RawIOBase. Otherwise it doesn't make much sense to use RawIOBase at all. Does that sound about right?

Notes

I mostly wrote this ticket, to get to a common ground of understanding with the developers of pyserial. I know that my solution will break current projects based on pyserial. So maybe the end result might be, to change something on a lower level, to allow users of the io interface to write a Plugin for their API instead of changing the API for everybody. We'll see.

And I don't think I will find the time to produce a suitable implementation myself in the next weeks, sorry. Writing my Master's thesis currently. But maybe someone else will, if we can agree on a meaningful solution plan or I might get to it later on. In any case it looks to me like a general problem of pyserial which must be addressed at some point.

(btw. is there a mailing list? couldn't find one in the docs)

[Chris Liechti]

read() defaults to one byte as this is the only sensible value that works in all situations. the default (and currently only option within pySerial) is to handle a read timeout as EOF, so if the port is opened with no timeout, reading to EOF would never return...

that leads to the question what an EOF with a serial connection is anyway. CTRL+D you may say, or CTRL+Z on that other OS, but that is not suitable if you want to handle binary protocols too. someone with a binary protocol might wish to have each frame handled as "file", but such a thing would require that it can be specified what a frame is - there are so many different protocols out there...

i do not see a readuntil function in the io API.

[Erik Bernoth]

Hi Chris,

my mistake. The method is called readinto() not readuntil() (see line 886 for where it is called in the standard implementation of read()).

You really don't have to care about the EOF part, because that is already solved in the default methods. You can also try that it works, when you call readall() from a serial.Serial object. This method is still the default behavior, because pyserial doesn't overwrite it. It's awfully slow but it always returns you the command you sent over the wire, its output, and in the end a new prompt. Because I tried that with pyserial and with normal files I have to assume, that the slowness comes from the currently non standard implementation of read().

I also didn't get that idea myself, but because the /dev/tty* device files are files, you should really read from them as from a file.

And I completely agree that the current implementation of timeouts wouldn't stop reading when the timeout runs out. Now of course you can say, that the current implementation is perfect, but you can also assume, that the implementation might be another bug, correlated with this one here.

A third point might be blocking reading. Can you still remember why pyserial tries to do the blocking itself instead of using a self-blocking filedescriptor?

Last but not least, isn't it interesting to use already inherited standard behavior, when possible? I always like to spend a lot of time in that area, because reusing code from others means I don't have to maintain it myself. And reading from a (special) file is already a solved problem in (at least C-) Python itself. ^^

[Erik Bernoth]

In case you are interested, the current state of investigation for our own framework, which builds on top of pyserial.