ser2net-devel — General ser2net development

Re: [Ser2net-devel] How to guarantee data delivery?

[Corey M. <mi...@ac...>]

On Thu, Jan 05, 2023 at 02:24:08PM -0300, Marcelo de Campos Franco Leal wrote:
> Hi,
> 
> Is there any sort of documents with the architecture and the design (UML
> diagrams) of ser2net?
> If so, could you please share it with me?
> The docs will help speed up my understanding of the source code.

I don't have anything like that, sorry.  I don't find them very useful.
If you wanted to created some design diagrams, I'd be happy to include
them, though.

I assume you are talking about the newest version of ser2net, too.

> 
> I would like to understand whether the data coming from the serial is
> queued and then removed from the queue only when there is a confirmation of
> receipt by the telnet client.
> 
> In other words, is there a mechanism that guarantees 100% delivery of data
> received from the serial to the telnet client?

It is a serial port, so there aren't any guarantees on that side besides
the standard RTS/CTS.

When connected, the data is flow-controlled, so if the TCP connection
can't keep up the data will be held in queue (until the buffers run out,
of course, and if RTS/CTS is available it will flow-control it there).

Really, nothing can provide 100% delivery guarantees.  It's impossible
from a basic theoretical view.  (See the Byzantine general's problem, if
you are interested.).  But that's not what you are asking, I don't
think.

> 
> The machine that runs ser2net is connected to a wireless network and may
> lose its connection from time to time. At these times, the serial data
> would not be lost if it is queued and then sent as soon as the wireless
> connection is re-established.

That's a different question, you are asking if ser2net queues if nothing
is connected.  If there is nothing connected on the accepter side,
ser2net will disconnect from the connector side.  So at that point any
data would be lost.

You could have ser2net hold the data while disconnected, but that's less
than half the problem.

TCP has no end-to-end delivery check.  The receiver has no way to tell
the other end "I got this data".  So ser2net might send some data, the
connection go down, and then on reconnect ser2net has no way to know
what data was received on the other end.  TCP (and telnet) do not
provide that.  That requires a higher level protocol, which is not as
simple as it seems.

It would be possible to write a gensio to provide this, but it would be
non-trivial and you have to have a protocol layer on both ends to do it.

-corey

> 
> Thanks in advance and regards,
> Marcelo Leal
> 
> -- 
> 
> 
> "Esta mensagem pode conter informações confidenciais ou privilegiadas. 
> Caso não seja o destinatário desta mensagem ou caso ela lhe tenha sido 
> enviada por engano, pedimos a gentileza de eliminar esta mensagem e todos 
> os seus anexos imediatamente. Utilizar, divulgar a quaisquer terceiros ou 
> copiar esta mensagem para qualquer fim é expressamente proibido e punido 
> por lei."
> 
> "This message may contain privileged or confidential 
> information. If you are not the intended recipient of this message or if 
> you have received it in error, please kindly delete this message and all 
> its attachments immediately. Unauthorized use, disclosure to any third 
> parties or copying of this message is expressly prohibited and may be 
> subject to the penalties of law."


> _______________________________________________
> Ser2net-devel mailing list
> Ser...@li...
> https://lists.sourceforge.net/lists/listinfo/ser2net-devel

[Ser2net-devel] How to guarantee data delivery?

[Marcelo de C. F. L. <le...@cp...>]

Hi,

Is there any sort of documents with the architecture and the design (UML
diagrams) of ser2net?
If so, could you please share it with me?
The docs will help speed up my understanding of the source code.

I would like to understand whether the data coming from the serial is
queued and then removed from the queue only when there is a confirmation of
receipt by the telnet client.

In other words, is there a mechanism that guarantees 100% delivery of data
received from the serial to the telnet client?

The machine that runs ser2net is connected to a wireless network and may
lose its connection from time to time. At these times, the serial data
would not be lost if it is queued and then sent as soon as the wireless
connection is re-established.

Thanks in advance and regards,
Marcelo Leal

-- 


"Esta mensagem pode conter informações confidenciais ou privilegiadas. 
Caso não seja o destinatário desta mensagem ou caso ela lhe tenha sido 
enviada por engano, pedimos a gentileza de eliminar esta mensagem e todos 
os seus anexos imediatamente. Utilizar, divulgar a quaisquer terceiros ou 
copiar esta mensagem para qualquer fim é expressamente proibido e punido 
por lei."

"This message may contain privileged or confidential 
information. If you are not the intended recipient of this message or if 
you have received it in error, please kindly delete this message and all 
its attachments immediately. Unauthorized use, disclosure to any third 
parties or copying of this message is expressly prohibited and may be 
subject to the penalties of law."

[Ser2net-devel] ser2net 4.3.11 released

[Corey M. <mi...@ac...>]

A bug that, in very rare circumstances, could cause ser2net to go into
an infinite loop was discovered. I was able to trace it down to an
incorrect usage of gensio (something that changed in gensio at the
beginning and I never fixed in ser2net). This really only showed up
because the user was using rate limiting with multiple threads,
increasing the odds of this showing up, and it still only showed up
occasionally in a busy setup.

Anyway, it's a hard to reproduce bug, but it is a bug, and I suggest
upgrading if possible.

-corey

[Ser2net-devel] ser2net 4.3.10, gensio 2.6.1 released

[Corey M. <mi...@ac...>]

The gensio update fixes some build issues when tcpd is enabled or when
swig is not enabled.

The ser2net update fixes a long-standing issue with startup order. There
have always been issues with ser2net starting before networking and the
startup code not working because gethostbyname() fails in that case, and
you can't open a listen socket in that case. I thought working around
this would be hard, but it turns out that it wasn't hard at all. Now
ser2net, if it fails to create the accepter or connector, will retry
periodically with the accepter-retry-time interval. So hopefully nobody
will have to deal with this in systemd any more.

-corey

[Ser2net-devel] ser2net 4.3.9, gensio2.3.9, 2.5.3, 2.5.6, and 2.6.0 released

[Corey M. <mi...@ac...>]

ser2net is updated to compile with the newest version of gensio, and a
few other small issues.

gensio 2.3, 2.4, and 2.5 are updated to fix an issue with possibly
delivering data out of order in the ssl layer. I've only seen it with
openssl 3.x, but it is a bug that might happen.

This is likely the last 2.3 release, unless something big comes along.

gensio-2.6.0 is a new big release with lots of changes:

 * The gensios have been restructured to be dynamically loadable. This
   removes a bunch of dependencies from the main gensio library, thus
   improving decoupling. It also makes it easy to create your own
   gensios, stick them in the proper directory, and they are ready. You
   can also still statically compile them in, and you can now choose
   gensios individually.

 * os handler and mdns code have been moved to separate libraries. This
   solved a few problems, and makes them more independent. This does
   mean you have to change the way you link against these things.

 * pty and stdio have a change in the way euid and uid setting work.
   There wasn't a security issue, per se, but the way it was before
   didn't work on BSD. This makes the usage more standard.

 * New gensios:

  * sound - You can access sound devices through gensio much the same
    way as any other data.

  * ratelimit - This limits the flow of data, useful for slow devices
    without flow control.

  * afskmdm - A ham radio modem commonly used with AX.25. It uses the
    sound gensio.

-corey

[Ser2net-devel] gensio 2.6.0-rc6 released

[Corey M. <mi...@ac...>]

This release fixes issues with the afskmdm gensio and the sound
interfaces.  Both have been fairly stable for a while now.

It also fixes a long-standing bug with the Windows pty code, primarily
affecting gtlssd logins.  It was causing dropped characters when you
logged in with gtlsshd then started msys64 .  An update of msys2 caused
it to completely lock up, making it easier to trace down the problem.
It had to do with a subtlety of the extremely complex Windows security
model.  I had to create an intermediary process that sits between the
pty code and the application to get all the security stuff right.

I expect the next release will not be an RC unless something else comes
up.

-corey

[Ser2net-devel] gensio 2.3.8, 2.5.5, 2.6.0-rc4 released

[Corey M. <mi...@ac...>]

gensio 2.3.8 and 2.5.5 fix some crashes in the serial code that people reported.

gensio 2.6.0-rc4 has a number of changes:

  * The internal structure has been reworked so that gensios are
    dynamically loaded by default and can be individually selected at
    build time. The library itself was getting too many dependencies, so
    these dependencies are pushed to the individally loaded gensios.
    There should be no difference from the user's point of view; gensios
    go into libexec and get loaded on demand. And you can write your own
    and stick them in this directory and it will pick them up.
    
  * The os interface and mdns code has been put into their own
    libraries. There were circular dependencies, and mdns brings a
    boatload of other libraries. They really aren't gensios, they are
    gensio support, so they needed to be in their own library anyway.
    This make the gensio library just about gensios. But it will change
    the way you have to link against it if you are using mdns.

  * The sound gensio is now no longer experimental, I'd say it's beta
    quality now.

  * An AFSK modem gensio has been added. It sits on top of the sound
    gensio and implements the 1200 baud modem used for AX.25 in ham
    radio.

  * pty and stdio have a change in the way euid and uid setting work.
    There wasn't a security issue, per se, but the way it was before
    didn't work on BSD. This makes the usage more standard.

I'd expect that 2.6.0 will be coming out pretty soon now.

-corey

[Ser2net-devel] gensio 2.5.4, 2.6.0-rc2 released

[Corey M. <mi...@ac...>]

Well, sorry for the quick update, but for 2.6.0-rc1 a bug slipped by
because the pkgconfig setup on my new system was broken and there was a
bug in libgensiomdns.pc. That will break ser2net if your pkgconfig is
working. So there's a new update for that.

I missed a backport of a patch for BSD, so the compile was broken on BSD
for 2.5.3. Thus a 2.5.4 release.

-corey

[Ser2net-devel] gensio 2.6.3, 2.6.0-rc1 and ser2net 4.3.8 released

[Corey M. <mi...@ac...>]

gensio-2.5.3 fixes a bunch of little bugs, mostly found working on
gensio 2.6.0. It also adds two gensios:

ratelimit - limit the data throughput on the write side of the gensio
  stack. This is useful if you have a slow device on a serial port that
  doesn't have flow control.

sound - Yes, you can access sound on Linux and Windows now. It's a heck
  of a lot easier than using the standard interfaces. This still somewhat
  experimental and the interface may change some, but it works basically.
  It's fairly flexible. It's based on ALSA on Linux and the native Windows
  interface there. A raw file interface also is available on all
  platforms. Nothing for BSD (except the file interface), but I'm happy
  for someone else to work on that.

gensio-2.6.0-rc1 reworks the initialization infrastructure for gensios
and makes them dynamically loadable. You can also individually select or
deselect gensios at configure time and decide if they are built into the
library or dynamically loaded. This has all kinds of good things about
it. The gensio library was getting a lot of dependencies; dynamic
loading decouples the library itself from the individual gensios and the
libraries they require. If you only need certain gensios, you only have
to compile those. And you can create and add your own gensios and throw
them in the gensio libexec directory and they will be automatically
picked up. There are some major changes:

* The individual gensio allocators are removed and replaced with a
  generic mechanism to do the same thing. Individual gensio allocators
  was a bad idea from the beginning and was never going to work with
  dynamically loaded gensios.

* The mdns code was moved to it's own libgensiomdns library. It's not a
  gensio, so having it in the main library caused some dependency
  issues.

* cmake support is removed. It was just too hard to support two build
  systems. cmake was always broken. It would have taken me a week to
  firure how to do all the new stuff under cmake. The main reason I
  added it was to build under Visual Studio, and that hasn't turned out
  to be that useful of a thing. mingw64 has worked well for me.

There are lots of little reworks, mostly removing hacks, required to
make this work, but they should be transparent to users.

ser2net 4.3.8 adjusts for having a separate mdns library. and fixes some
reported typos. No functional changes.

-corey

[Ser2net-devel] gensio 2.5.2 released

[Corey M. <mi...@ac...>]

2.5.1 was a quick release for one purpose and not general.

2.5.2 adds a "script" gensio. This runs an external program an basically
stacks it's stdio on top of the child of the script gensio. Once the
external program exits, the script gensio will report itself as open and
pass through all data. This is useful for running startup scripts, thus
the name.

There are also some small bug fixes.

-corey

[Ser2net-devel] gensio 2.5.0, ser2net 4.3.7 released

[Corey M. <mi...@ac...>]

The new gensio release adds a bunch of new stuff:

 *  ed25519 key support for gtlssh and certauth
 *  Windows pty support
 *  Windows gtlsshd support
 *  A bunch of little fixes all over the place, especially Windows

If you are running on Windows you will want to upgrade. If you are not,
it's probably still worth upgrading.

This is one unfortunate non-backwards compatible change I had to make. I
turns out that having a struct gensio inside of a gensio namespace in
C++ was a really bad idea. gcc was able to handle it, but clang didn't
like it because it couldn't resolve the name. Anyway, the C++ namespace
is changed to gensios. If you are not using C++, this won't affect you.

There is still a bug lurking in the Windows pty code. When working on
it, I found it worked fine on Windows consoles, but on msys2 consoles it
randomly dropped characters. The bug appears to be in msys2, but I'm not
sure. I found a workaround; adding a short delay before writing to the
Windows pty seems to work around the issue, but it's obviously not
ideal. Hopefully I can track this down at some point.

The ser2net release just fixes some build issues around pam from the
previous release.

-corey

[Ser2net-devel] gensio 2.4.2 released

[Corey M. <mi...@ac...>]

This release adds a workaround for non-compliant AX.25 stacks.

It also fixes a long-standing bug with blocking I/O; in certain
situations it can crash. This will not affect ser2net, but if you are
using blocking I/O, it could affect you.

If you are using AX.25 or blocking I/O, you should upgrade.

-corey

[Ser2net-devel] gensio 2.4.1 and ser2net 4.3.6 released

[Corey M. <mi...@ac...>]

ser2net 4.3.6 adds the ability to use PAM passwords to authenticate
ser2net connections, in addition to passwords in the ser2net admin
structure and certificates. It fixes some test failures due to Linux
changes.

gensio 2.4.1 adds a "keepopen" filter gensio that attempts to keep the
gensio below it open by retrying an open on it periodically if it closes
or returns an error. This was for certain serial ports that disappear
under certain conditions, and then come back later, but it can be used
for anything, like TCP connections. It also adds a lot of little fixes
for lots of things. If you are using 2.4.0, you should upgrade.

-corey

[Ser2net-devel] gensio 2.4.0 release

[Corey M. <mi...@ac...>]

It's been a long time in the works, but it has stabilized and hasn't
changed much recently.

The base C library hasn't changed much since the 2.3 series. Three new
filters have been added, the kiss and ax25 filters for ham radio
protocols, and a filter for translating characters. Some small semantic
change has been made: * The return value for GENSIO_EVENT_READ and
GENSIO_EVENT_WRITE_READY events now matter. Before they were ignored,
but that led to issues in certain situations. * The return value for
GENSIO_EVENT_NEW_CHANNEL events now matter. It was ignored before, but
that would leave new channels just lying around if the upper layer
didn't handle the message. * The way out of bound data is handled has
changed. It is no longer delivered by default, you have to enable it
using a gensio_control. Since it is always requires special handling,
providing it to the user when the user wasn't expecting it could cause
issues.

There are numerous bug fixes; these should all have been backported to
2.3 if relevant.

The big changes are in C++ and other ;anguages. That interface has been
heavily reworked for two reasons: * It wasn't a great C++ interface. *
It didn't map well into swig.

So the C++ code has been pretty much rewritten. The same basic concepts
are there, but lots of details are tweaked.

But as part of that, a Go interface was done and is completely
functional. And there is a new Python interface named pygensio. The old
python interface had hand-written callbacks into Python, which was
really messy and unmaintainable. The new ones are based on swig
directors and are much more easily maintained.

I expect the old Python interface will eventually go away, but will be
maintained for the near future.

-corey

[Ser2net-devel] gensio 2.4.0-rc7, 2.3.7 released

[Corey M. <mi...@ac...>]

Release 2.3.7 to fix a regression from 2.2 dealing with setting the
serial break on startup. A few other bugs are fix. You should upgrade if
you can.

Release 2.4.0-rc7 features the same fixes as 2.3.7, and lots of fjxes
for AX.25. AX.25 is very stable now. It also adds an xlt gensio, which
will let you translate characters in this stream. This was for dealing
with "\r" (0x0d) meaning "end-of-line" in AX25 chats, as opposed to "\n"
(0x0a) like all C programs expect.

-corey

[Ser2net-devel] gensio 2.4.0-rc2, 2.3.6 released

[Corey M. <mi...@ac...>]

gensio 2.3.6 is a bugfix release, lots of little bugs. You should
upgrade if you are using 2.3.5. There is one security bug if you are
using gtlssh, but there probably aren't many people using that.

gensio 2.4.0-rc2 is a major rework of the C++ interface; it's a lot
cleaner and more C++-like that it was before. This was done because I've
done some language bindings based upon the C++ interface, and the
cleaner on there now make language bindings a lot easier. There is a new
python language binding named pygensio. The old python binding is still
there and working, but I expect to phase it out over time. And there is
a new Go binding. These bindings use swig directors for the callbacks,
so they are cleaner and more maintainable than the hand-coded python
interfaces in the old python binding.

The biggest change in the C++ interface is that the gensio/accepter is
no longer passed back in the callbacks. With language bindings, this
caused all sorts of issues and complexities. Having it made some things
easier for the user, but it was just too hard to make it work right.

-corey

[Ser2net-devel] gensio-2.4.0-rc1, ser2net-4.3.5 released

[Corey M. <mi...@ac...>]

ser2net-4.3.5 had some little fixes for handling some sysfs attributes,
some improved error reporting, fixed handling of xon/xoff, and a few
other small things.

gensio-2.4.0-rc1 is a pre-release with added support for the AX25
Amateur Radio packet protocol. This required some rework. There are also
a lot of bug fixes that the new protocol shook out of other places. AX25
is not well interop tested.

If you are using gensio_os_funcs directly, there are now functions to do
the accessing for you for anything that is public. You should use those
instead of directly calling functions in gensio_os_funcs for better
binary compatibility.

Also, most of the address handling was moved out of gensio_os_funcs and
into a separate function call structure that is now in the address. This
was required to add a different address type for AX25.

There is a semantic change to the way new incoming channels are handled.
Before, if you returned an error from the new channel callback event, it
was ignore. I realized that if you had an event handler that didn't
handle the event, it would return GE_NOTSUP, but nothing would be done
with the new channel and it would leak. So now if an error is returned
from the new channel callback, it will shut down and free that channel.

This is a pre-release so semantic or API changes can be easily made to
AX25. The rest of the library is stable, it should be a pretty good
improvement from the previous release. You shouldn't be afraid to try it
out.

-corey

[Ser2net-devel] gensio 2.3.5 released

[Corey M. <mi...@ac...>]

Yet another quick release. gcc-11 and the new sctp library threw some
curve balls, so those needed to be handled. No functional changes, just
compile and bug fixes related to the new compiler and library.

-corey

[Ser2net-devel] gensio 2.3.4 released

[Corey M. <mi...@ac...>]

This is quick after the previous release, but it fixes some significant
issues that some people might want. If you are using:

    gtlssh/gtlsshd
    ssl
    mux
    relpkt

Then you should upgrade. These have significant performance increases
and a rather massive wasted CPU usage problem was fixed.

Otherwise, this isn't so important.

-corey

[Ser2net-devel] gensio 2.3.3 released

[Corey M. <mi...@ac...>]

The major feature part of this is gensiot is rewritten to support a
server mode, which allows it to continue to accept connections after a
connection comes in. As part of this, it supports multiple threads,
pidfiles, and output to syslog. This is useful as a stripped down
ser2net, and also for creating a forwarding application, letting you
convert a non-ssl connection to an ssl on, or receive ssl connections
and forward them to a local application that doesn't support ssl. It's
also useful for testing. Adding threads to gensiot found a couple of
bugs in the library.

This release also fixes a number of Windows issues. The oomtests are
getting better. UDP tests still have issues, but it still have issues on
BSD systems due to buffer size settings that I haven't worked on. But
Windows is much better than it has been.

You should upgrade if you can.

-corey

[Ser2net-devel] ser2net 3.5.4 released

[Corey M. <mi...@ac...>]

This is the old version of ser2net, but it had a nasty bug dealing with
telnet connections crashing in some situations. So a release was in
order. If you are using ser2net 3.x, you should upgrade.

-corey

[Ser2net-devel] gensio 2.3.2 released

[Corey M. <mi...@ac...>]

This release contains lots of little fixes and enhancements. Nothing
huge, but enough that if you are using the 2.3 series you should
definitely upgrade.

-corey

[Ser2net-devel] gensio 2.3.1 released

[Corey M. <mi...@ac...>]

I know I released 2.3.0 yesterday, and I thought "I'll do the coverage
testing later, it shouldn't matter." Big mistake, it turns out. Coverage
testing uncovered some a couple of bugs. I've automated a little more of
the testing, too. so this doesn't happen again.

Anyway, you shouldn't use 2.3.1, use this one instead.

-corey

[Ser2net-devel] gensio 2.3.0 released

[Corey M. <mi...@ac...>]

I've been fiddling with it long enough, it's time to get it out there
for general use.  I've heavily tested it and done all sorts of code
analysis on it.

This adds a bunch of new functions, including:
* Windows support.  It compiles under MinGW and Visual Studio.  OpenSSL
  is required for SSL and certauth support.
* C++ support.  The interface was already OO, this is a mapping into
  C++.  It's light, but not much was required.
* Built-in TCL and glib OS handlers were added.
* 2-factor authentication support is built-in to the certauth protocol.
* Support for elliptical keys was added to certauth.

The main user interface is unchanged.  If you have written your own OS
handler, it will need to be redone.  But beyond that, it should just
work.

-corey

[Ser2net-devel] ser2net 4.3.4 released

[Corey M. <mi...@ac...>]

Nothing huge, just fixing some minor annoyances. headers from git are:

591361e Add some build docs, clean it up a little
1c9d4d6 Fix month output on tracing
ed88798 Don't report an mdns error if it's not supported by gensio
946cb57 Add an option to allow one-way transfers with ser2net
61630c9 Fix a shutdown crash
f6144dc tests: Fix a typo in a test print
5cabe48 tests: Add a delay after ipmi_sim terminates
6795df8 Fix setportcontrol dtr configuration