tcl-core

[TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[David G. <dav...@po...>]

Just a happy announcement that tip-162-branch in CVS is building and
working on windows:

E:\tcl_ws\tcl_tip162\win>release_vc8\tclsh86
% set s [socket -type tcp6 www.kame.net 80]
sock672
% fconfigure $s -sockname
2001:470:1f04:ed::2 davygrvy-pt.tunnel.tserv3.fmt2.ipv6.he.net 3182
% fconfigure $s -peername
2001:200:0:8002:203:47ff:fea5:3085 orange.kame.net 80

The old Win3.11 WSAAsyncSelect based I/O notification method is now
gone.  In it's place is a more modern and scalable use of overlapped
sockets using completion port notification.  The code base it came from
has a good history of being stable.

http://technet.microsoft.com/en-us/sysinternals/bb963891.aspx

A non-NT based implementation using WSAEventSelect has yet to be
invented.  In the meantime, I'm thinking of using the old code as a
runtime fallback if needed.

Hopefully by Christmas I'll have it all polished.

I've started skeletal implementations of IrDA and Bluetooth additions
mainly to force me to consider what the best API changes are needed so
that we don't have to revisit API changes again.  Whether the new
protocols make it to 8.6 or not is not my short term goal.

I've also started a UDP section of the code in the hopes that one day we
might fill it and turn it on.

An asynchronous and protocol agnostic resolver is also in the works.

That's about it for my dream networked Tcl..  Is it your's too?

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Sun, Dec 7, 2008 at 11:54 AM, David Gravereaux <dav...@po...> wrote:
> Just a happy announcement that tip-162-branch in CVS is building and
> working on windows:

Congrats Dave ! That's the kind of xmas gift I appreciate.

> I've started skeletal implementations of IrDA and Bluetooth additions
> mainly to force me to consider what the best API changes are needed so
> that we don't have to revisit API changes again.  Whether the new
> protocols make it to 8.6 or not is not my short term goal.
> I've also started a UDP section of the code in the hopes that one day we
> might fill it and turn it on.

Wow. UDP is a killer feature. Imagine, starting to be able to write
userland drivers for various devices in pure Tcl !
I never understood why there wasn't a "UDP netcat". This will bring it for free.

What about portability ? Is there a Win-specific catch to UDP or can
we hope to have it for unix with little work ?

> An asynchronous and protocol agnostic resolver is also in the works.
> That's about it for my dream networked Tcl..  Is it your's too?

You bet it is :)

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Sun, Dec 7, 2008 at 12:23 PM, Alexandre Ferrieux
<ale...@gm...> wrote:
>
> Wow. UDP is a killer feature. Imagine, starting to be able to write
> userland drivers for various devices in pure Tcl !
> I never understood why there wasn't a "UDP netcat". This will bring it for free.
>
> What about portability ? Is there a Win-specific catch to UDP or can
> we hope to have it for unix with little work ?

Oops, I have written "UDP" where I meant "USB". Sorry.
The mistake was triggered by the analogy with Bluetooth.

But by the way, is USB dreamable of in this context ?

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[David G. <dav...@po...>]

Alexandre Ferrieux wrote:

> But by the way, is USB dreamable of in this context ?

Not for sockets.  Talking direct USB is like picking the lock on the
backdoor instead of using the unlocked frontdoor.

Even raw sockets is far off until a viable way to do message-based I/O
comes about.  I hope Pat Thoyts has some thoughts on merging his tcludp
into the core.

I'm excited about ipv6 multicast because it extends to internet routers.
 It's rather a moving target, which makes it even more exciting.

I've been away from core hacking for a while, and just found
Tcl_SetChannelError() yesterday.  An awesome addition to help bypass the
huge POSIX translation problem.  All thumbs up!

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Sun, Dec 7, 2008 at 9:25 PM, David Gravereaux <dav...@po...> wrote:
> Alexandre Ferrieux wrote:
>
>> But by the way, is USB dreamable of in this context ?
>
> Not for sockets.  Talking direct USB is like picking the lock on the
> backdoor instead of using the unlocked frontdoor.

Can you elaborate ?
Is there a fundamental difference between sending a data packet in
point-to-point mode over USB, Bluetooth, or IP(UDP) for that matter ?

By this I mean, I acknowledge the slight mismatch between the
stream-like Tcl channel model and packetized transmissions, but it is
doable anyway, as is shown by TclUDP: [puts -nonewline] (below the
MTU) and short-[read]s are just a peculiar packet interface, but a
working one...

Now is there something intrinsically harder with USB ?

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[David G. <dav...@po...>]

Alexandre Ferrieux wrote:

> Now is there something intrinsically harder with USB ?

There is no sockets interface for it, mainly.  So it extends to a
different interface.  Usermode is seen as the interface the USB end
device driver decides it wants to give you.  Some chose a comport,
others a C API.  What the stream details over the wire are not for
public consumption.  You could debug the wire and reverse engineer the
packet exchange.  CreateFile() might have a way to get below the drivers
and open the port itself, but you are on your own for that.

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Reinhard M. <ma...@tc...>]

Hi,

On Sun, 7 Dec 2008 at 23:18, David Gravereaux wrote:

> CreateFile() might have a way to get below the drivers and open the 
> port itself, but you are on your own for that.

I think libusb would be the way to go for talking directly to USB 
devices in a cross-platform way.

http://libusb.sf.net/

cu
 	Reinhard

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Sun, Dec 7, 2008 at 11:18 PM, David Gravereaux <dav...@po...> wrote:
> Alexandre Ferrieux wrote:
>
>> Now is there something intrinsically harder with USB ?
>
> There is no sockets interface for it, mainly.

Of course :-)

What I was dreaming of was taking whatever API exists for USB and
disguising it into a Tcl channel type.
(From what you wrote I believed you were about to do the same for
Bluetooth and IrDA; knowing nothing about them I assumed the three
were one the same level, exposing via specific APIs a point-to-point
packet transfer.)

Needless to say, such an USB channel would say nothing about the
actual bytes to send; that part would be the job of the "userland
driver" of each individual device.

Do you think this "transport-level-only" abstraction is doable ?

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[David G. <dav...@po...>]

Alexandre Ferrieux wrote:

> What I was dreaming of was taking whatever API exists for USB and
> disguising it into a Tcl channel type.
> (From what you wrote I believed you were about to do the same for
> Bluetooth and IrDA; knowing nothing about them I assumed the three
> were one the same level, exposing via specific APIs a point-to-point
> packet transfer.)

Oh, what's really neet is that IrDA and Bluetooth *do* have a sockets
interface.

What you want is another basis of driving I/O for a new channel driver
type.  Probably file based.

The list of available protocols for sockets is sizable actually.  Here
goes the list of what the socket() command _might_ understand.  This is
of course dependent on OS and what software drivers to support hardware
are installed.  On win, they're called layered service providers.

a general name | address family  |    type       |  protocol
-------------------------------------------------------------
tcp4            AF_INET           SOCK_STREAM     IPPROTO_TCP
tcp6            AF_INET6          SOCK_STREAM     IPPROTO_TCP
udp4            AF_INET           SOCK_DGRAM      IPPROTO_UDP
udp6            AF_INET6          SOCK_DGRAM      IPPROTO_UDP
irda            AF_IRDA           SOCK_STREAM     0
bluetooth       AF_BTH            SOCK_STREAM     BTHPROTO_RFCOMM
appletalk       AF_APPLETALK      SOCK_STREAM     ATPROTO_ATP
atm             AF_ATM            SOCK_STREAM     ATMPROTO_AAL5
decnet          AF_DECnet         SOCK_SEQPACKET  DNPROTO_NSP
ipx             AF_IPX            SOCK_DGRAM      NSPROTO_IPX
spx_seq         AF_IPX            SOCK_SEQPACKET  NSPROTO_SPX
spx             AF_IPX            SOCK_STREAM     NSPROTO_SPX
spx2_seq        AF_IPX            SOCK_SEQPACKET  NSPROTO_SPXII
spx2            AF_IPX            SOCK_STREAM     NSPROTO_SPXII
isotp4          AF_ISO            SOCK_STREAM     ISOPROTO_TP0
netbios         AF_NETBIOS        SOCK_DGRAM      IPPROTO_UDP
vines           AF_BAN            SOCK_STREAM     0

This isn't an exhaustive list either, but is all the easily available
info I have.  The non-stream protocols are the tough ones to use with
Tcl for the stream-only limitation of channels.  But I was just gabbing
with Pat Thoyts and I have an idea worth try to use channels for their
associate data, but bypass the buffer space altogether and use a
different [read] procedure such as [recvfrom] which unloads "packets"
from a linkedlist I'd keep in the associate data.

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Donal K. F. <don...@ma...>]

David Gravereaux wrote:
> The list of available protocols for sockets is sizable actually.  Here
> goes the list of what the socket() command _might_ understand.  This is
> of course dependent on OS and what software drivers to support hardware
> are installed.  On win, they're called layered service providers.
[...]
> This isn't an exhaustive list either, but is all the easily available
> info I have.

If you're working on Unix, consider AF_UNIX as well.

Donal.

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Reinhard M. <ma...@tc...>]

Hi,

On Sun, 7 Dec 2008 at 12:23, Alexandre Ferrieux wrote:

> I never understood why there wasn't a "UDP netcat".

the original netcat has a -u switch and socat supports UDP in various 
ways. What else would be needed to make them a "UDP netcat"?

cu
 	Reinhard

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Donal K. F. <don...@ma...>]

Alexandre Ferrieux wrote:
> Can you elaborate ?
> Is there a fundamental difference between sending a data packet in
> point-to-point mode over USB, Bluetooth, or IP(UDP) for that matter ?
> 
> By this I mean, I acknowledge the slight mismatch between the
> stream-like Tcl channel model and packetized transmissions, but it is
> doable anyway, as is shown by TclUDP: [puts -nonewline] (below the
> MTU) and short-[read]s are just a peculiar packet interface, but a
> working one...

That's the really big issue. With channels, it doesn't really matter if
you break a message up into pieces: the other side will get them in
order and can reassemble. With message-oriented connections such as
UDP[*] that assumption doesn't hold, so TclUDP can break "interestingly"
if used naïvely with large messages. The way to fix this is to use a
better fundamental model of message-oriented communications for those
types of connections, but nobody's done one for us yet. (In short,
TclUDP does it "the wrong way" for the _excellent_ reason that "the
right way" isn't defined yet.) We ought to fix that, but that's a pretty
big project.

Apart from that, there's not much fundamental difference between the
different types of sockets and, of course, to date Tcl's been good at
hiding those differences which exist. We'd be able to reuse important
things like the Notifier, so in fact most of the key bits already exist.

Oh, and the other key thing: if we're doing the introduction of this
whole new API right, we should remember to allow Tcl-implemented message
layers. (Someone might be odd enough to layer email/SMTP or XMPP or SOAP
over it. Crazy, but who am I to say no? :-D)

Donal.
[* Anything that's SOCK_DGRAM in David's table. And I don't understand
    SOCK_SEQPACKET. ]

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Mon, Dec 8, 2008 at 4:32 AM, Donal K. Fellows
<don...@ma...> wrote:
>
>> By this I mean, I acknowledge the slight mismatch between the
>> stream-like Tcl channel model and packetized transmissions, but it is
>> doable anyway, as is shown by TclUDP: [puts -nonewline] (below the
>> MTU) and short-[read]s are just a peculiar packet interface, but a
>> working one...
>
> That's the really big issue. With channels, it doesn't really matter if
> you break a message up into pieces: the other side will get them in
> order and can reassemble. With message-oriented connections such as
> UDP[*] that assumption doesn't hold, so TclUDP can break "interestingly"
> if used naïvely with large messages.

Of course. But programming naïvely (thanks for the non-ASCII
character) can lead to problems in many other circumstances ;-)
When you're doing UDP, on the write side you have to be aware of the
MTU anyway . On the receive side you can use TclUDP's guaranteed short
read semantics, so the only danger is if you pass a maximum length
that is too small...
So it seems the concern is more of an aesthetic than practical nature.
What I'm driving at is the following dichotomy. Should we:

 (a) stick to the TclUDP tradition for all the new packet-based socket
types in David's Pandora box.

 (b) first define a clean API extension before proceeding.

You seem to advocate (a)  because (b) is "a big project".
I can see another reason: (a) will probably have to be supported
anyway, eg  for backwards compatibility with TclUDP, or for the
uniformity of the channel API (no [read] nor [puts] sounds odd).
So, when (b) comes,  there will be two ways of sending/receiving
packets. I find this odd too :-(

On the other hand, being courageous might help for once. Compatibility
with TclUDP is not a hard constraint since it is not in the core. And
the "right way" might not be that complex. What about the following:

 # Example built with UDP in mind; please generalize.

 # sendto()
 chan configure/fconfigure $channel -dest $host:$port
 chan putpacket $channel $data

 # recvfrom()
 set data [chan getpacket $channel]
 set from [chan configure $channel -src]

 # connect()+recv() => filter
 chan configure $channel -src $host:$port
 set data [chan getpacket $channel]

 # ioctl() => immediate actions unfit for [fconfigure]
 chan control $channel FOO bar

Reactions ?

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[Alexandre F. <ale...@gm...>]

On Mon, Dec 8, 2008 at 4:32 AM, Donal K. Fellows
<don...@ma...> wrote:
>
> [ ... And I don't understand SOCK_SEQPACKET. ]

That's reliable-ordered-datagrams (or stream-with-boundaries).
The only place I ever used it is on an emulated Bluetooth layer
exposing sound IO on a braindead home VOIP gateway made by my company
:-}

   socket(PF_BLUETOOTH, SOCK_SEQPACKET, BTPROTO_SCO)

Now whatever the semantics (possibly barely useful) or support
(possibly patchy), exposing it in Dave's work should not be a problem
(just map a string to an integer  constant).

-Alex

Re: [TCLCORE] six lane highway just paved through the swamp. (tip162 - IPv6)

[David G. <dav...@po...>]

Alexandre Ferrieux wrote:

>    socket(PF_BLUETOOTH, SOCK_SEQPACKET, BTPROTO_SCO)
> 
> Now whatever the semantics (possibly barely useful) or support
> (possibly patchy), exposing it in Dave's work should not be a problem
> (just map a string to an integer  constant).

On Win, just BTPROTO_RFCOMM and BTPROTO_L2CAP are defined.  Looks SOL
for BTPROTO_SCO and BTPROTO_HCI here.  The string mapping project is
going well.  Thanks for the above.  I'm up to a total of 45 different
socket types that might be possible in my dream list.  Whether I can
realize them is a different issue.

> On the other hand, being courageous might help for once. Compatibility
> with TclUDP is not a hard constraint since it is not in the core. And
> the "right way" might not be that complex. What about the following:
> 
>  # Example built with UDP in mind; please generalize.
> 
>  # sendto()
>  chan configure/fconfigure $channel -dest $host:$port
>  chan putpacket $channel $data
> 
>  # recvfrom()
>  set data [chan getpacket $channel]
>  set from [chan configure $channel -src]
> 
>  # connect()+recv() => filter
>  chan configure $channel -src $host:$port
>  set data [chan getpacket $channel]
> 
>  # ioctl() => immediate actions unfit for [fconfigure]
>  chan control $channel FOO bar


I like it.  I was thinking a more simple path just to get going, but
that certainly is my direction minus some details.  The
Tcl_Driver(Input|Output)Proc could both return EOPNOTSUPP for those
types when one accidentally uses [puts],[read] or Tcl_Gets, etc..

You're not kidding mentioning Pandora's box.  I'm getting dizzy seeing
all the possible network support we could have.