#23 pyserial read dropping characters when logging high baud

[Kevin Lillard]

I have been trying to write a simple script to just log serial data from a few different comm ports. The comm ports are running at 115200 baud, I am using a WinXP computer and i am using Python 2.4. I would like to get this to work where it can log up to at least 5 different Comm ports at 115200 baud each. (hyperterminal does this with no problems)
The majority of the data is logged, but there are pretty occasional places where many characters in a row are dropped and are not logged. I am able to use Hyperterminal and log the same exact Comm ports and all of the data is there, so i know the Serial card and drivers are good and are not at fault.

I have attached the script that I created (the latest attempt). It mainly starts up and reads a config file to see how many different comm ports are going to be logged. It then spawns a new task for each port that is to be logged. Each spawned task just calls serial.read() and then adds any characters to a string variable. The original task monitors the string variables and calls a function which will log the strings into a text file.
The sub tasks basically have this code structure:
ser0 = serial.Serial(SerialPort-1, BaudRate, timeout=0)
while True:
s = ser0.read(50000)

if len(s) > 0:
ScpaString = ScpaString + s

time.sleep(1)

I have tried many different combinations of code to see what might work, but have had no luck at all. In the above example I have added a print statement to see if it was the read command that was loosing the data, or if it was the actual writing to the file - but it is the read command that is droping the characters, because the printout had characters lost as well.

Does anyone have any suggestions on what to do? I have looked for about 2 days now for examples or other people who have had the same problem and have found nothing.

The SerialLogging_Config.txt referenced in the script file just needs to contain the following for it to work:
SCPA = 7, 115200
MIRA1 = 4, 115200
MIRA2 = 5, 115200
MIRA3 = 6, 115200
MIRA4 = 8, 115200

[Kevin Lillard]

Serial Logging script

[Chris Liechti]

1. adding long strings is inefficient in python as strings are read-only (it has to create an all new string). use a list instead and use ''.join() when needed.

2. do you have a thread per port? i hope the time.sleep() call was just there to show the problem and it's not in your real code? (the attachment uses it though :(

the problem with a one seconds sleep is, that currently pySerial configures 4000 bytes buffer for the serial port. with 115200kBaud that's less than 0.5 seconds until the buffer overflows (assuming the data is coming in at the full rate)

also note that non-blocking I/O is probably not the best way to go with pySerial on Windows unless you implement waiting for data someway using other windows lowlevel functions.
i'd suggest a reader thread per port instead, if you want to read at least a single char but as many as possible something like this can be used: data = s.read() + s.read(s.inWaiting())
setting the port timeout to any value from 1 to 10 seconds (the timeout is only needed if your thread want to check an "alive" flag from time to time)

you might want to take a look at the examples directory and maybe tcp_serial_redirect.py

[Kevin Lillard]

From your comments I believe I have gotten this to work correctly. It really helps to know that the pySerial has a limit of 4000 bytes buffer (I couldn't find that information anywhere - or didn't know where to look)

The final solution that appears to be working is as follows:
def Scpa_thread_run (SerialPort, BaudRate):
ser0 = None

try:
ser0 = serial.Serial(SerialPort-1, BaudRate, timeout=10)
except serial.serialutil.SerialException:
print "Could not open serial port %s." % SerialPort

if ser0 != None:
while True:
s = ser0.read(1000)
s = s.replace('\r', ' ')

if len(s) > 0:
LogDebug(s, 0) # This function writes the string to the *.txt file

I have multiple threads running virtually the same code as above to log each comm port.

thank you for your help, I was about to give up on pySerial completely