Thread: [6bed4-devel] PtP tunnel forking instead of a tunnel spanning the ipv4 internet
zeroconfig IPv6 tunnel
Status: Beta
Brought to you by:
vanrein
From: <ebi...@xm...> - 2012-07-04 01:34:02
|
There are a lot of scenarios other that Ipv6 transitions that involve Point to Point tunnels in a mesh over a Non-Broadcast Multi-Access network. VPN and the like. In those tunnels you can work very hard to achieve multicast and broadcast parity with existing ipv6 links or you can take the line that you simply have a lot of point to point connections that give you connectivity, and you have the ability to create a new point to point connection at need. In the startup configuration of these kinds of networks you bootstrap with a tunnel to some well known central point so you have connectivity. Then you create tunnels to other endpoints on demand. I have started playing the idea of encoding the ICE offer/answer exchange as an STUN message and then I can let ICE/STUN take care of trying different paths and finding one that works, and agreeing on a common route. It looks like a general approach that is going to be simple and easy to implement. I don't know if it is going to be really useful in the 6bed4 design scenario. Having to throttle hole punching to 20ms or slower looks like it is going to be a real drag on tunnel creation speed and my hunch is that most transitive internet connections will be over before a tunnel can be established. However I do think this is a fun way to experiment with ICE and a very interesting way of coping with implementing NAT traversal outside end applications, so I am going to keep playing with this as time permits. So far except for tunnel setup speeds this looks very promising, and the least NIH that I have found. Eric |
From: Rick v. R. <ri...@op...> - 2012-07-12 08:50:11
|
Hello Eric, Thanks again for your feedback. You've been so productive that I had to allocate some serious time, causing the delay in the response. You picked up that the name is not bed64 but 6bed4, right? > - There should be a recommendation of what Reachable Time should > be set to in router advertisements and on bed64 links to allow > the neighbour reachability logic to both keep ports open > and to detect when the udp port has timed out. The possible values are 0 for unspecified, or another value to set a valid time for a route. Such a time depends on assumptions about the NAT router, which is not generally available at the location where the 6bed4 client is run. It would be awkward to specify that "some value" should be set, without specifying what value. > Hmm you do mention tha the default neighbour cache times > are sufficient in the current draft. A little more detail > and a description of how the values work to keep NAT ports open > might be useful. Agreed. I can mention the times that are common in practice, that makes sense to me. > - bed64 creates has NBMA non-broadcast multi-access links. > As such RFC2491 should be looked at and referred to.- Thanks, I hadn't seen this RFC before. I will read it and integrate its suggestions as well as I can -- and then refer to it. > - Using ipv4 address and udp port as the link layer address > looks like a good solution. Thanks. > - Avoiding problems with spoofed ipv4 packets. > Most commonly spoofed ipv4 packets result from a lack of source > address validation at some point in the network. As such > verifying that there is two way connectivity will frequently > fix this issue. This is what is done with Neighbor Solicitation + Advertisement. > Fpr neighbour solicitations and router solicitations I recommend > requiring the nonce option from RFC3971 Secure Neighbor Discovery. Interesting suggestion. Copying it would be a MUST when responding, and sending it initially a MAY or SHOULD. I had looked at Secure ND but found it was patented and therefore useless -- but that does not seem to apply to the nonce. I would also include the Timestamp. > With a decent random number generator that should make replies > impossible to spoof. > For router solicitations this closes a pretty significant hole. For > neighbour solictitations this prevents DOS attacks where you think a > neighbour is reachable and they are not. Some perspective on this: We never send to an untrusted address, but only to a router address determined out-of-band and hopefully securely, the target address itself or a LAN peer. As for the router address, we have a /24 into which we allocate it, so it is a fixed address. What you are worried about is the reply; keep in mind that guessing both the timing and UDP port and IP address for a client take quite some effort; but I do agree that Nonce/Timestamp options improve on that. As you seem to have grasped, it is never possible to divert peer traffic to an arbitrary IPv4 address by bombarding a victim with Neighbor Advertisements. The influence of adding Secure ND to those would be to thwart DoS attacks. > - Stateless operation of 6bed4 routers: > Ultimately this appears to be a critical property for 6bed4. > Without state the routers can be simpler and can scale to larger > numbers of packets processed per second. Yes, I think routing protocols should be stateless, to avoid drowning in administration. > - Interface identifiers: > I suggest you simply drop the concept of illegal port values > Only going through an interface that strongly assumes ethernet > will they even that even be a problem. And making some port > numbers illegal looks like a good way to break in surprising ways > on weird NAT implementations. Note that odd *external* UDP ports are said to be illegal. This is at the outside of NAT. The reason to ban odd port numbers on all nodes is to be able to show one's address to a node that assumes ethernet semantics for the 6bed4 interface. It seems a waste to disable interoperability with existing stacks; this is why odd external UDP ports are banned. This is not a conceptual beauty, but it is the sort of pragmatics that I'm willing to incorporate because 6bed4 is a transitional technique. > - Well known vs providered assigned addresses: > > Automatic tunnel like 6to4 and Teredo without a well defined place of > where do you look to find the problems and who do you blame have been > undebuggable. 6rd on the other hand works well enough it is being > rolled out by several vendors in production deployment, because they > can isolate all of the bits within their own networks. > > Therefore I suggest 6bed4 learn from history and be designed to be > used only with the gateway providers ivp6 addresses and to be > primarily used with unicast ipv4 addresses going to those gateways. 6bed4 Cannot assume that it runs entirely within one AS, but an ISP is still able to route 6bed4 exactly as they like; beyond the ISP-internal scope, it will run into the complexity of the broad Internet. The advantage of an anycast address is the advantage of zero config operation, which IMHO is a vital part of 6bed4. It is difficult to debug 6to4, but according to research by the ISP Column the differences between forward and return routes are not the primary cause of problems; for 6to4 it was mainly that users run it behind NAT for which it hasn't been designed. I have been thinking about this, and it may make sense to setup some infra that is explicitly intended for debugging; not aiming to control it all, but at least to find a problem if it exists. For reasons of routability we have a /24 for anycast, and use only the .1 in it for 6bed4. It is easy enough to define behaviour for a .2 as well, possibly based on a /64 prefix that is local to the network. This would make it possible to (1) detect the node being used, (2) detect route changes, (3) send back to the original node. (It would even be possible for end users to prefer this address over an anycast address in the IPv6 space, at the expense of a change of their external address whenever anycast routing changes.) > - Bootstrapping. I suggest a slightly modified form of the existing > bootstrap mechanism. I'm not completely confident that I understand you here. Are you proposing to split the current Router Discovery into a phase of Router Discovery (providing the /64 prefix) and Neighbor Discovery (provoding our externally observed address/port as an LLaddr)? > . Avoidance of all vageries of global anycast routing. Please explain? > . Moving our link layer address determination into a the part of the > process that is most likely to notice when our link local address > changes I think I will get this if I understand what you mean exactly. > - On link determination: > > 6bed4 should honor the on-link prefix bit in the prefix option in the > router advertisement. That way a gateway with plenty of bandwidth can > direct 6bed4 clients to always route through it. There is currently no statement on this, indeed. Note that "real IPv6 traffic" always travels through the 6bed4 gateway; only 6bed4 to 6bed4 can be optimised with direct routes. > To make NAT traversal work most of the time through NAT requires a > process where both nodes coordinate so that they are both poking holes > in the NAT firewalls simultaneously, If an attempt fails on one end, it will send through the router and mostly that would provide a response which is then permitted directly due to the hole opened by the initially failed attempt. At that time, the direct connection should be open in both directions. This will only fail on what is usually classified as symmetric NAT. > so in the try hard scenario they > have poked simultaneous holes. If we can map ICE into the core ipv6 > primitives I think it might be worth trying hard. Otherwise trying > harder than neighbour discovery to the public ipv4 address and port of > our neighbour feels silly. I went through the same thoughts; ICE is not designed for this sort of thing and ND seemed to be just the right fit. > My gut feel is that the 6bed4 effort should be split into two parts. > 6bed4-simple that does not worry about the challenges of talking to > anything except the router, and 6bed4-bypass that figures out how to > build on 6bed4-simple to reduce the latency of common conections. Hmmm, a more complex spec to derive at potentially simpler software. This could be done by rearranging MUST (for direct route attempts) to SHOULD. The disadvantage would be that direct routes would only work if both endpoints supported it, which causes more attempts than warranted, and also retries. This is why it is currently a MUST. Furthermore, I'd like to learn what node operators think; I do not expect them to be as open to IPv6 traffic without bypasses. It is true that no work has currently been done to exploit direct links between 6bed4 nodes if at all possible, but trying to contact a 6bed4 peer over a 6bed4 interface if at all possible has been my angle. Setting up an interface with the proper /64 for 6bed4 as a low-priority route that looks like a locally attached interface would help a lot though. Normal routing rules would then do the right thing. And given the simplicity (zero config) of 6bed4 I would not be surprised if it got popular in distributions. Think alone of what it could do for peer-to-peer exchange of files in solutions (such as SIP, or chat, or YaCy) that are now fighting NAT traversal issues. > Or to put it another way your bypass route that invokes ipv4 multicast > and has all kinds of magic conditions I find very scary. This is the LAN flavour, right? You're being subjective when you call it scary ;-) but do note that this is optional in v01 because it might scare more people than just you. If your node decides not to use it, then you have nothing to fear. > At the same time if there can be made an abstraction layer that does > as well STUNbis + ICE for sip and xmpp so I could have the advantages > of ipv6 sooner rather than later I figure I love that notion. Not > that I actually believe it for a second, but a guy can dream. :) At least for SIP, I am working very hard to realise it. The whole 6bed4 tunnel was designed to enable SIP over IPv6. I'm also doing other transitioning work in that field, including the necessary translation between IPv4- and IPv6-based SIP/RTP. http://devel.0cpm.org/sitemap.html > - Ports: > > It is probably obvious but clients should randomize as much as > possible their listening ports. This creates the greatest chance of > clients preserving their port number when traversing the NAT. As NATs > like iptables preserve the source port when there is no conflict. I'm inclined to leave that as an implementation issue, not to be mentioned in a specification which basically declares the boundaries of proper conduct. On Fri, Jun 15, 2012 at 10:20:27AM -0700, Eric W. Biederman wrote: > > Sending Router solcitis: > > The 6bed4 router being stateless will never send unsolicited > router advertisements. Therefore the 6bed4 client must > have timers setup so that it requests the router to send > new solicitations. Those timers are not standard ipv6 > behavior so the spec needs to be extended. What is currently done, is assume that the routes live forever. Then, if outgoing traffic gets mapped to a different external UDP port, it will lead to a response with a corrective Rtr Adv. The client for v01 incorporates a keepalive timer to keep NAT traversal open, and indeed make the route's time infinite. > I can see the practicality in infinite address timeouts, but since the > addresses may not be used for ever that seems unfortunately naive. As soon as we've standardised an RFC and have a final prefix assigned, the addresses are actually static. Just like 2002::/16 for 6to4 or 2001::/32 for Teredo. The addresses can be used as long as the 6bed4 interface is up, and sends keepalives as part of being up. The routes are tied to the interface, and will drop if the interface goes down. Before the interface goes down, the route, prefix, address are basically forever-lasting. > Router redirects: > > If it is desired to trust have a model where redirects may be trusted in > 6bed4 I recommend Secure ND is a good suggestion, indeed. Redirects are treated as suggestions, and lead to a normal ND process; Secure ND was discussed above. On Fri, Jun 15, 2012 at 02:19:58PM -0700, Eric W. Biederman wrote: > > I would really appreciate being able to tell a 6bed4 address when I see > one. Let me recommend using the format: > > ::64bd:PP:II:II We'll get an experimental IPv6 anycast range assigned on Sep 3, 2012. This is a /48 range that we can recognise as use of 6bed4. On Sun, Jun 17, 2012 at 04:45:51PM -0700, Eric W. Biederman wrote: On Sun, Jun 17, 2012 at 05:12:18PM -0700, Eric W. Biederman wrote: I will process your patches separately. Probably after my holiday, which lasts until July 30. Thanks! On Sun, Jun 17, 2012 at 04:52:25PM -0700, Eric W. Biederman wrote: > > I am concerned that the problems of the non-debugability of > asymetric paths through unicast addresses will repeat. But I don't > see any reason why we should not try to get this to work. Agreed -- see above. > My request will be that we specify the protocol in such a way that the > same client can be used with anycast addresses and without. With just a > change in the initial address to contact. Yes, that can be done. I've already run into an admin who preferred to run 6bed4 on his own IPv4 address instead of locally diverting the anycast address. I'm not sure if this should be part of the spec, or just of the software. On Sun, Jun 17, 2012 at 05:10:01PM -0700, Eric W. Biederman wrote: > > I have been playing with this and I really don't like the current > solution in 6bed4. As it currently stands the solution is enormously > complex in it's implementation and it does not actually work to pierce > any connection tracking firewall (like iptables) that tracks the entire > <srcip,srcport,destip,destport> tuple. Symmetric NAT is not going to work, that is a known problem. If it is *just* a firewall, then the fact that a node sends out UDP between then known internal and external nodes should suffice. > To get through firewalls that track the entire source and destination > address we must attempt simultaneous connections from both sides. This > is easy to arrange if we send an icmp packet through the router from > one machine to another to get it started. This is currently done, under the assumption that a packet send will usually lead to a response in a short time. Explained above. > The idea is we start off with a 4 way handshake to exchange possible > ip:port addresses and to estabilish the existence of bidirectional > routing connectivity (to foil most ipv4 source address spoofing). This is the ICE idea, right? On Tue, Jun 19, 2012 at 06:40:31PM -0700, Eric W. Biederman wrote: > > The general problem. Stateful firewalls incoporated into many NAT > devices only allow traffic to flow based on the 4 tuple > <srcip,srcport,destip,destport>. Symmetric NAT is a problem, symmetric firewalls are not. Please note that port changes constitute a potential change of machine and/or user, and could therefore lead to security problems. > I have attached the code and the setup for my test cases (based on > network namespaces below). Thanks -- I will look into it along with the patches. On Tue, Jul 03, 2012 at 05:27:04PM -0700, Eric W. Biederman wrote: > > There are two issues that using anycast addresses can trigger that both > impact debugging, and until I had those two issue conflated. > > - Anycast addresses are a good way to trigger assymetric routing. > - Anycast addresses hide which machine is the router in your path. > > Assymetric routes can happen with ipv4 and they are a pain, because > getting a reverse traceroute requires tracing from the other side. Yes, and this can be solved by using the .2 in the /24 and being setup with a locally available IPv6 address in the 6bed4 router's AS. Another return route could even be selected by communicating over IPv6, using the locally availabe IPv6 address of whatever target 6bed4 router's AS you'd like. > Having multiple routers at very different places in the network > operated by different people report the same address in traceroute > makes figuring out who to blame for slow traffic a real challenge. Agreed, hence the idea towards a tracing solution. > So I suggest adding: "ICMP time exceeded errors MUST NOT be reported > with a 6bed4 address." to make it possible to figure out whose 6bed4 > router is malfunctioning. Interesting. Especially when combined with the IPv6 address that does point to the target to blame. On Tue, Jul 03, 2012 at 05:30:20PM -0700, Eric W. Biederman wrote: > > Reading through RFC 5245 Interactive Connectivity Establishment I came > across a nasty observation. > > In some NATs you can not create a new hole faster than once ever 20ms. > > Which makes NAT holepunching something that has to be done a lot more > carefully than I would have guessed. I am *so* done with NAT and IPv4... On Tue, Jul 03, 2012 at 05:45:46PM -0700, Eric W. Biederman wrote: > > I took a quick look and the complexity of stateless nat64 looks to > be about 1000 lines of code or about the same as your original 6bed4 > tunnel that did not directly attempt to connect peer endpoints. > See http://www.litech.org/tayga/ NAT64 is typically used on a LAN or by an ISP; it is not generally considered an option for the Internet as a whole, so it does not function to provide "IPv6 for the masses" which is the intention of 6bed4. > This does indicate that you would need to implement ICE aka RFC5245 in > your VOIP clients but RFC5245 is required for proper IPv6 support > anyway, and the implementation if not the specification is pretty > simple. You're clearly a fan of ICE ;-) and the way of publishing multiple LLaddr in a single ND message is intruiging. On Tue, Jul 03, 2012 at 06:11:42PM -0700, Eric W. Biederman wrote: > > You have to add support for multiple source and destination addresses. > You have to require support for nonces, and use them to prevent > problems with source address spoofing. The nonces may indeed solve the matter with symmetric NAT, to ensure talking to the same process as before. > You have to change the semantics so that you don't trust source and > destination addresses that have been provided until you have verified > them. This is basically what happens with a Redirect; and it is done through Neighbor Discovery. > And there are a whole host of slight differences in semantics that > you have to handle very carefully. This is what I seriously dislike... I've tried all along to stay within the existing protocols as much as possible, with the only diversion being the source LLaddr suggested in the Rtr Adv. > And ultimately you have what is effectively a whole new protocol. This might be a bit over the edge, I fear ;-) and you say you agree. On Tue, Jul 03, 2012 at 06:33:49PM -0700, Eric W. Biederman wrote: > > I have started playing the idea of encoding the ICE offer/answer > exchange as an STUN message and then I can let ICE/STUN take care > of trying different paths and finding one that works, and agreeing > on a common route. Interesting! > However I do think this is a fun way to experiment with ICE and a very > interesting way of coping with implementing NAT traversal outside > end applications, so I am going to keep playing with this as time > permits. So far except for tunnel setup speeds this looks very > promising, and the least NIH that I have found. Meanwhile, my line will be to provide a working tunnel based on what I already have. Delivering is more important to me, so I will be a bit more conservative. As stated above, several of your suggestions are directly usable, but not all are. I want to thank you for all of them, and made quite a few notes for further processing; I am interested in hearing what you will cook up! Once again, my apologies for the slow response. I'm afraid 6bed4 is only one of my activities... no surprise there, I suppose ;-) Cheers, -Rick |
From: <ebi...@xm...> - 2012-07-12 11:01:05
|
Rick van Rein <ri...@op...> writes: > Hello Eric, > > Thanks again for your feedback. You've been so productive that I had > to allocate some serious time, causing the delay in the response. > > You picked up that the name is not bed64 but 6bed4, right? Yes. I'm not certain why I wrote bed64 in the initial subject line. >> However I do think this is a fun way to experiment with ICE and a very >> interesting way of coping with implementing NAT traversal outside >> end applications, so I am going to keep playing with this as time >> permits. So far except for tunnel setup speeds this looks very >> promising, and the least NIH that I have found. > > Meanwhile, my line will be to provide a working tunnel based on what > I already have. Delivering is more important to me, so I will be > a bit more conservative. As stated above, several of your > suggestions are directly usable, but not all are. I want to > thank you for all of them, and made quite a few notes for further > processing; I am interested in hearing what you will cook up! Sounds like a reasonable plan. > Once again, my apologies for the slow response. I'm afraid 6bed4 is > only one of my activities... no surprise there, I suppose ;-) No surprise there. Let me recommend to you rfc5128 "State of peer to peer communication across NATs", for it's vocabularly in talking about NATs. The vocabulary is much more specific and much more relevant then the vocabulary from RFC3489 STUN. Not that you want to fight NATs issues. But some at least when talking about bypass solutions it comes up. What is your vision for the set of problems you see 6bed4 solving? I think I have seen part of your vision when you said "Think alone of what it could do for peer-to-peer exchange of files in solutions (such as SIP, or chat, or YaCy) that are now fighting NAT traversal issues." You also mentioned not wanting to include both a full ipv4 stack and a full ipv6 stack, in thin embedded clients. The problem that is near and dear to my heart, is that when I talk to someone at a distance (say 2000 Kilometers) over a VOIP phone it works well if the RTP packets flow direction from my phone to the other parties. If the packets go through a relay connections are unreliable, and essentially undebugable. So I am very interested in figuring out which technologies will make those direct connections possible and easy to achieve. IPv6 is one of those technologies and ICE looks like it may be another. Eric |
From: <ebi...@xm...> - 2012-07-23 07:03:32
|
Rick van Rein <ri...@op...> writes: >> Reading through RFC 5245 Interactive Connectivity Establishment I came >> across a nasty observation. >> >> In some NATs you can not create a new hole faster than once ever 20ms. >> >> Which makes NAT holepunching something that has to be done a lot more >> carefully than I would have guessed. > > I am *so* done with NAT and IPv4... I can understand the feeling. I want to point out some specific things to take into consideration for future desgins. RFC6296 IPv6 to IPv6 Network Prefix Translation. All that changes is one IPv6 address in the IPv6 header. RFC6204 Basic Requirements for IPv6 Customer Edge Routers. This recommends following RFC6092 for packet filtering and it recommends deploying RFC4193 Unique Local addresses. The birthday paradox suggests that once there are above 1 million (10^6) users using ULAs there will be duplicate ULA prefixes. RFC6092 IPv6 CPE Simple Security. This recommends using at least address indepedent filtering, and possibly address dependent filter for IPv6 flows. Which means that many of the same challenges that existed in IPv4 will exist in a fully deployed IPv6. Where IPv6 improves upon the current situation is that there is no need for address sharing, and as such there is no excuse for implementing endpoint dependent translation. Without endpoint dependent translation (the EVIL of symmetric NATs) peer to peer communication will always be possible. Eric |
From: <ebi...@xm...> - 2012-07-23 13:46:47
|
Rick van Rein <ri...@op...> writes: >> And ultimately you have what is effectively a whole new protocol. > > This might be a bit over the edge, I fear ;-) and you say you agree. > On Tue, Jul 03, 2012 at 06:33:49PM -0700, Eric W. Biederman wrote: >> >> I have started playing the idea of encoding the ICE offer/answer >> exchange as an STUN message and then I can let ICE/STUN take care >> of trying different paths and finding one that works, and agreeing >> on a common route. > > Interesting! I have something working at this point, although not quite finished, I haven't coded up the message-integrity check attribute of the verification attribute but that is essentially orthogonal from the rest of the work. It takes about a 3rd more code than your version of peer.c but the code is absolutely bored when it comes to peircing my NAT test cases. The packet flow: A B ipv6 port 3478 <----------ice offer----------- ipv6 port 3478 ipv6 port 3478 ----------ice answer----------> ipv6 port 3478 ipv4 6bed4 port <-----stun binding request----> ipv4 6bed4 port ipv4 6bed4 port <-----stun binding reply------> ipv4 6bed4 port ipv4 6bed4 port <== shortcut tunnel traffic ==> ipv4 6bed4 port The ipv6 address does not need to be a 6bed4 address. This allows any host to bypass the 6bed4 router if it has the ability to receive traffic on a ipv4 udp port. Only the ice offer/answer packet exchange go must go through the 6bed4 router. Currenlty I am running I am not buffering traffic and running traffic to it's destination over the 6bed4 tunnel, until a shortcut tunnel forms. The awesome and terrible feature of the shortcut tunnels is that traffic is not sent over them until round trip reachability is confirmed, which looks like it means just in time for a small flow to be completely routed over the 6bed4 router while the tunnel is forming. Ugh. So I expect I may need to delay the initial traffic to a host a little, to get the tunnel to be useful to short bursts of traffic. On the side of debugability I have coded up a ICMPv4 to ICMPv6 converter and started storing sync the hop_count and the traffic_class between v4 and v6. And I have explicitly not configured an address on the 6bed4 router interface. Which seems to give me parity on debugability with traceroute over ipv4 and ipv6. I can traceroute from an ipv6 host and see all of the ipv4 hops past the 6bed4 router. Likewise I can traceroute on the 6bed4 client side and see all of the hops on the way to the 6bed4 router. My traceroutes through the 6bed4 router give me a router name other than the generic 6bed4 name so I can tell which router I am dealing with. It is kind of fun seeing a traceroute before and after the shortcut tunnel forms. Eric |
From: <ebi...@xm...> - 2012-07-23 14:20:41
|
Rick van Rein <ri...@op...> writes: >> - Interface identifiers: >> I suggest you simply drop the concept of illegal port values >> Only going through an interface that strongly assumes ethernet >> will they even that even be a problem. And making some port >> numbers illegal looks like a good way to break in surprising ways >> on weird NAT implementations. > > Note that odd *external* UDP ports are said to be illegal. This is at > the outside of NAT. > > The reason to ban odd port numbers on all nodes is to be able to show > one's address to a node that assumes ethernet semantics for the 6bed4 > interface. It seems a waste to disable interoperability with existing > stacks; this is why odd external UDP ports are banned. > > This is not a conceptual beauty, but it is the sort of pragmatics that > I'm willing to incorporate because 6bed4 is a transitional technique. The trade off is breaking some clients in myserious ways, instead of implementing an indirection table mapping mac addresses to ipv4 address and ports. Banning odd ports is a BAD idea. It means hosts will not be able to assume they have a working 6bed4 tunnel. Your reference implementation does not even handle the case when the NAT takes your even port and gives you an odd port. It is a bug you won't find in testing, it is a bug that will make people decide your protocol is flaky and deprecate it. So I disagree with the restriction to only use even ports in the strongest possible terms. On the flip side you can interoperate as ethernet with existing stacks with one simple layer of indirection. You can go farther and do a clean implemenation that doesn't claim to be eithernet and not even need that layer of indirection. But for the sake of a bit I don't want to see a protocol lost. Eric |