From: Xin L. <luc...@gm...> - 2021-07-06 18:22:29
|
This patchset is to implement PLPMTUD and GSO for TIPC, Patch 1-5 are for PLPMTUD while 6-8 are for GSO. It gets some ideas from SCTP as their similarities like both are reliable datagram packets and possible to run over IP(v6)/UDP. But also it does some adjustments for TIPC. Xin Long (8): tipc: set the mtu for bearer properly for udp media tipc: add the constants and variables for plpmtud tipc: build probe and its reply in tipc_link_build_proto_msg tipc: add probe send and state transition tipc: add probe recv and state transition tipc: add offload base tipc: add software gso tipc: add hardware gso include/uapi/linux/tipc_config.h | 6 -- net/tipc/Makefile | 2 +- net/tipc/bearer.c | 23 ++++- net/tipc/core.c | 3 + net/tipc/link.c | 147 +++++++++++++++++++++++++++---- net/tipc/link.h | 29 ++++++ net/tipc/msg.c | 1 + net/tipc/msg.h | 3 + net/tipc/node.c | 15 +++- net/tipc/offload.c | 70 +++++++++++++++ net/tipc/udp_media.c | 18 ++-- 11 files changed, 287 insertions(+), 30 deletions(-) create mode 100644 net/tipc/offload.c -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:26
|
Instead of using 14000 for ipv4/udp mtu, and 1280 for ipv6/udp mtu, this patch to set mtu according to the lower device's mtu at the beginning. The pmtu will be determined by the PLPMTUD probe in the following patches. Signed-off-by: Xin Long <luc...@gm...> --- include/uapi/linux/tipc_config.h | 6 ------ net/tipc/udp_media.c | 8 ++++---- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/include/uapi/linux/tipc_config.h b/include/uapi/linux/tipc_config.h index 4dfc05651c98..7e23b7f438b4 100644 --- a/include/uapi/linux/tipc_config.h +++ b/include/uapi/linux/tipc_config.h @@ -185,12 +185,6 @@ #define TIPC_DEF_LINK_WIN 50 #define TIPC_MAX_LINK_WIN 8191 -/* - * Default MTU for UDP media - */ - -#define TIPC_DEF_LINK_UDP_MTU 14000 - struct tipc_node_info { __be32 addr; /* network address of node */ __be32 up; /* 0=down, 1= up */ diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c index c2bb818704c8..dc4bae965549 100644 --- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -661,7 +661,7 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr) static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, struct nlattr *attrs[]) { - int err = -EINVAL; + int err = -EINVAL, hlen; struct udp_bearer *ub; struct udp_media_addr remote = {0}; struct udp_media_addr local = {0}; @@ -743,7 +743,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, err = -EINVAL; goto err; } - b->mtu = b->media->mtu; + hlen = sizeof(struct iphdr); #if IS_ENABLED(CONFIG_IPV6) } else if (local.proto == htons(ETH_P_IPV6)) { dev = ub->ifindex ? __dev_get_by_index(net, ub->ifindex) : NULL; @@ -760,12 +760,13 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, else udp_conf.local_ip6 = local.ipv6; ub->ifindex = dev->ifindex; - b->mtu = 1280; + hlen = sizeof(struct ipv6hdr); #endif } else { err = -EAFNOSUPPORT; goto err; } + b->mtu = b->media->mtu ?: dev->mtu - hlen - sizeof(struct udphdr); udp_conf.local_udp_port = local.port; err = udp_sock_create(net, &udp_conf, &ub->ubsock); if (err) @@ -851,7 +852,6 @@ struct tipc_media udp_media_info = { .tolerance = TIPC_DEF_LINK_TOL, .min_win = TIPC_DEF_LINK_WIN, .max_win = TIPC_DEF_LINK_WIN, - .mtu = TIPC_DEF_LINK_UDP_MTU, .type_id = TIPC_MEDIA_TYPE_UDP, .hwaddr_len = 0, .name = "udp" -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:32
|
These are 4 constants described in rfc8899#section-5.1.2: MAX_PROBES, MIN_PLPMTU, MAX_PLPMTU, BASE_PLPMTU; And 2 variables described in rfc8899#section-5.1.3: PROBED_SIZE, PROBE_COUNT; And 5 states described in rfc8899#section-5.2: DISABLED, BASE, SEARCH, SEARCH_COMPLETE, ERROR; 'count' and 'raise' are used for two timers' counting: PROBE_TIMER and PMTU_RAISE_TIMER. 'probe_high' is used for finding the optimal value for pmtu. Signed-off-by: Xin Long <luc...@gm...> --- net/tipc/link.c | 13 +++++++++++++ net/tipc/link.h | 20 ++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/net/tipc/link.c b/net/tipc/link.c index cf586840caeb..1aa775cef3bb 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -182,6 +182,14 @@ struct tipc_link { /* Max packet negotiation */ u16 mtu; u16 advertised_mtu; + struct { + u16 probe_size; + u16 probe_high; + u16 pmtu; + u8 count; + u8 state:3; + u8 raise:5; + } pl; /* Sending */ struct sk_buff_head transmq; @@ -984,6 +992,11 @@ void tipc_link_reset(struct tipc_link *l) l->peer_session--; l->session++; l->mtu = l->advertised_mtu; + l->pl.state = TIPC_PL_BASE; + l->pl.pmtu = TIPC_BASE_PLPMTU; + l->pl.probe_size = TIPC_BASE_PLPMTU; + l->pl.count = 0; + l->pl.probe_high = 0; spin_lock_bh(&l->wakeupq.lock); skb_queue_splice_init(&l->wakeupq, &list); diff --git a/net/tipc/link.h b/net/tipc/link.h index a16f401fdabd..30bee2562987 100644 --- a/net/tipc/link.h +++ b/net/tipc/link.h @@ -66,6 +66,26 @@ enum { TIPC_LINK_SND_STATE = (1 << 2) }; +/* PLPMTUD state + */ +enum { + TIPC_PL_DISABLED, + TIPC_PL_BASE, + TIPC_PL_SEARCH, + TIPC_PL_COMPLETE, + TIPC_PL_ERROR, +}; + +#define TIPC_BASE_PLPMTU 1200 +#define TIPC_MAX_PLPMTU 9000 +#define TIPC_MIN_PLPMTU 512 + +#define TIPC_MAX_PROBES 3 +#define TIPC_PROBE_INTERVAL 10 + +#define TIPC_PL_BIG_STEP 32 +#define TIPC_PL_MIN_STEP 4 + /* Starting value for maximum packet size negotiation on unicast links * (unless bearer MTU is less) */ -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:33
|
pl.count will make a timer that 'timeout' every after '10 * node timer interval', where it does state transition in tipc_link_pl_send() and sends probe in tipc_link_build_proto_msg(). For the details, see: https://lwn.net/Articles/860385/ Signed-off-by: Xin Long <luc...@gm...> --- net/tipc/link.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/net/tipc/link.c b/net/tipc/link.c index 414f9cf543ff..3af6c04f82c2 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -292,6 +292,7 @@ static int tipc_link_advance_transmq(struct tipc_link *l, struct tipc_link *r, bool *retransmitted, int *rc); static void tipc_link_update_cwin(struct tipc_link *l, int released, bool retransmitted); +static void tipc_link_pl_send(struct tipc_link *l); /* * Simple non-static link routines (i.e. referenced outside this file) */ @@ -902,6 +903,14 @@ int tipc_link_timeout(struct tipc_link *l, struct sk_buff_head *xmitq) if (state || probe || setup) tipc_link_build_proto_msg(l, mtyp, PROBE_MSTATE, 0, 0, 0, 0, xmitq); + if (probe && tipc_link_is_up(l)) { + l->pl.count++; + if (!(l->pl.count % TIPC_PROBE_INTERVAL)) { + tipc_link_pl_send(l); + tipc_link_build_proto_msg(l, mtyp, PROBE_PLPMTU, 0, 0, 0, 0, xmitq); + } + } + return rc; } @@ -3013,3 +3022,42 @@ int tipc_link_dump(struct tipc_link *l, u16 dqueues, char *buf) return i; } + +static void tipc_link_pl_send(struct tipc_link *l) +{ + pr_debug("%s: PLPMTUD: link: %p, state: %d, pmtu: %d, size: %d, high: %d\n", + __func__, l, l->pl.state, l->pl.pmtu, l->pl.probe_size, l->pl.probe_high); + + if (l->pl.count <= TIPC_MAX_PROBES * TIPC_PROBE_INTERVAL) + return; + + if (l->pl.state == TIPC_PL_BASE) { + if (l->pl.probe_size == TIPC_BASE_PLPMTU) { /* BASE_PLPMTU Confirmation Failed */ + l->pl.state = TIPC_PL_ERROR; /* Base -> Error */ + + l->pl.pmtu = TIPC_MIN_PLPMTU; + l->mtu = l->pl.pmtu; + } + } else if (l->pl.state == TIPC_PL_SEARCH) { + if (l->pl.pmtu == l->pl.probe_size) { /* Black Hole Detected */ + l->pl.state = TIPC_PL_BASE; /* Search -> Base */ + l->pl.probe_size = TIPC_BASE_PLPMTU; + l->pl.probe_high = 0; + + l->pl.pmtu = TIPC_BASE_PLPMTU; + l->mtu = l->pl.pmtu; + } else { /* Normal probe failure. */ + l->pl.probe_high = l->pl.probe_size; + l->pl.probe_size = l->pl.pmtu; + } + } else if (l->pl.state == TIPC_PL_COMPLETE) { + if (l->pl.pmtu == l->pl.probe_size) { /* Black Hole Detected */ + l->pl.state = TIPC_PL_BASE; /* Search Complete -> Base */ + l->pl.probe_size = TIPC_BASE_PLPMTU; + + l->pl.pmtu = TIPC_BASE_PLPMTU; + l->mtu = l->pl.pmtu; + } + } + l->pl.count = TIPC_PROBE_INTERVAL; +} -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:33
|
This patch is to adjust the code in tipc_link_build_proto_msg() to make it able to build probe packet with a specific size for sender, and probe reply packet with mtu set. Note that to send the probe packet, the df flag has to be set. Signed-off-by: Xin Long <luc...@gm...> --- net/tipc/link.c | 38 +++++++++++++++++++++++--------------- net/tipc/link.h | 9 +++++++++ net/tipc/msg.c | 1 + net/tipc/udp_media.c | 3 ++- 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/net/tipc/link.c b/net/tipc/link.c index 1aa775cef3bb..414f9cf543ff 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -273,8 +273,8 @@ static int link_is_up(struct tipc_link *l) static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, struct sk_buff_head *xmitq); -static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, - bool probe_reply, u16 rcvgap, +static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, + u8 ptype, u32 mtu, u16 rcvgap, int tolerance, int priority, struct sk_buff_head *xmitq); static void link_print(struct tipc_link *l, const char *str); @@ -900,7 +900,7 @@ int tipc_link_timeout(struct tipc_link *l, struct sk_buff_head *xmitq) } if (state || probe || setup) - tipc_link_build_proto_msg(l, mtyp, probe, 0, 0, 0, 0, xmitq); + tipc_link_build_proto_msg(l, mtyp, PROBE_MSTATE, 0, 0, 0, 0, xmitq); return rc; } @@ -1862,8 +1862,8 @@ int tipc_link_rcv(struct tipc_link *l, struct sk_buff *skb, return rc; } -static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, - bool probe_reply, u16 rcvgap, +static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, + u8 ptype, u32 mtu, u16 rcvgap, int tolerance, int priority, struct sk_buff_head *xmitq) { @@ -1874,7 +1874,7 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, struct sk_buff *skb; bool node_up = link_is_up(bcl); u16 glen = 0, bc_rcvgap = 0; - int dlen = 0; + int dlen = 0, msg_sz; void *data; /* Don't send protocol message during reset or link failover */ @@ -1884,11 +1884,13 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, if (!tipc_link_is_up(l) && (mtyp == STATE_MSG)) return; - if ((probe || probe_reply) && !skb_queue_empty(dfq)) + if (ptype && !skb_queue_empty(dfq)) rcvgap = buf_seqno(skb_peek(dfq)) - l->rcv_nxt; - skb = tipc_msg_create(LINK_PROTOCOL, mtyp, INT_H_SIZE, - tipc_max_domain_size + MAX_GAP_ACK_BLKS_SZ, + msg_sz = tipc_max_domain_size + MAX_GAP_ACK_BLKS_SZ; + if (ptype == PROBE_PLPMTU) + msg_sz = l->pl.probe_size - INT_H_SIZE; + skb = tipc_msg_create(LINK_PROTOCOL, mtyp, INT_H_SIZE, msg_sz, l->addr, tipc_own_addr(l->net), 0, 0, 0); if (!skb) return; @@ -1915,13 +1917,19 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, msg_set_seq_gap(hdr, rcvgap); bc_rcvgap = link_bc_rcv_gap(bcl); msg_set_bc_gap(hdr, bc_rcvgap); - msg_set_probe(hdr, probe); - msg_set_is_keepalive(hdr, probe || probe_reply); + msg_set_probe(hdr, ptype == PROBE_MSTATE || ptype == PROBE_PLPMTU); + msg_set_is_keepalive(hdr, !!ptype); + if (ptype == PROBE_REPLY) + msg_set_max_pkt(hdr, mtu); if (l->peer_caps & TIPC_GAP_ACK_BLOCK) glen = tipc_build_gap_ack_blks(l, hdr); tipc_mon_prep(l->net, data + glen, &dlen, mstate, l->bearer_id); - msg_set_size(hdr, INT_H_SIZE + glen + dlen); - skb_trim(skb, INT_H_SIZE + glen + dlen); + if (ptype != PROBE_PLPMTU) { + msg_set_size(hdr, INT_H_SIZE + glen + dlen); + skb_trim(skb, INT_H_SIZE + glen + dlen); + } else { + skb->ignore_df = 0; + } l->stats.sent_states++; l->rcv_unacked = 0; } else { @@ -1935,7 +1943,7 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, msg_set_size(hdr, INT_H_SIZE + TIPC_MAX_IF_NAME); skb_trim(skb, INT_H_SIZE + TIPC_MAX_IF_NAME); } - if (probe) + if (ptype == PROBE_MSTATE || ptype == PROBE_PLPMTU) l->stats.sent_probes++; if (rcvgap) l->stats.sent_nacks++; @@ -2329,7 +2337,7 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, skb_queue_empty(&l->deferdq)) rcvgap = peers_snd_nxt - l->rcv_nxt; if (rcvgap || reply) - tipc_link_build_proto_msg(l, STATE_MSG, 0, reply, + tipc_link_build_proto_msg(l, STATE_MSG, PROBE_REPLY, msg_size(hdr), rcvgap, 0, 0, xmitq); released = tipc_link_advance_transmq(l, l, ack, gap, ga, xmitq, diff --git a/net/tipc/link.h b/net/tipc/link.h index 30bee2562987..87b3ebe5b91d 100644 --- a/net/tipc/link.h +++ b/net/tipc/link.h @@ -66,6 +66,15 @@ enum { TIPC_LINK_SND_STATE = (1 << 2) }; +/* Probe Type + */ +enum { + PROBE_NONE, + PROBE_MSTATE, + PROBE_REPLY, + PROBE_PLPMTU, +}; + /* PLPMTUD state */ enum { diff --git a/net/tipc/msg.c b/net/tipc/msg.c index 5c9fd4791c4b..6d8bcc180f8b 100644 --- a/net/tipc/msg.c +++ b/net/tipc/msg.c @@ -75,6 +75,7 @@ struct sk_buff *tipc_buf_acquire(u32 size, gfp_t gfp) skb_put(skb, size); skb->next = NULL; } + skb->ignore_df = 1; return skb; } diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c index dc4bae965549..5078c5b19e81 100644 --- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -174,6 +174,7 @@ static int tipc_udp_xmit(struct net *net, struct sk_buff *skb, local_bh_disable(); ndst = dst_cache_get(cache); if (dst->proto == htons(ETH_P_IP)) { + u8 df = skb->ignore_df ? 0 : htons(IP_DF); struct rtable *rt = (struct rtable *)ndst; if (!rt) { @@ -193,7 +194,7 @@ static int tipc_udp_xmit(struct net *net, struct sk_buff *skb, ttl = ip4_dst_hoplimit(&rt->dst); udp_tunnel_xmit_skb(rt, ub->ubsock->sk, skb, src->ipv4.s_addr, - dst->ipv4.s_addr, 0, ttl, 0, src->port, + dst->ipv4.s_addr, 0, ttl, df, src->port, dst->port, false, true); #if IS_ENABLED(CONFIG_IPV6) } else { -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:43
|
This patch is to receive and process the probe ack by checking msg_max_pkt() == l->pl.probe_size then does state transition in tipc_link_pl_recv(). For the details, see: https://lwn.net/Articles/860385/ Signed-off-by: Xin Long <luc...@gm...> --- net/tipc/link.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/net/tipc/link.c b/net/tipc/link.c index 3af6c04f82c2..241c9378e258 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -293,6 +293,7 @@ static int tipc_link_advance_transmq(struct tipc_link *l, struct tipc_link *r, static void tipc_link_update_cwin(struct tipc_link *l, int released, bool retransmitted); static void tipc_link_pl_send(struct tipc_link *l); +static void tipc_link_pl_recv(struct tipc_link *l); /* * Simple non-static link routines (i.e. referenced outside this file) */ @@ -2333,6 +2334,13 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, break; } + if (!reply && msg_max_pkt(hdr) == l->pl.probe_size) { + tipc_link_pl_recv(l); + if (l->pl.state == TIPC_PL_COMPLETE) + break; + tipc_link_build_proto_msg(l, STATE_MSG, PROBE_PLPMTU, 0, 0, 0, 0, xmitq); + } + /* Receive Gap ACK blocks from peer if any */ glen = tipc_get_gap_ack_blks(&ga, l, hdr, true); @@ -3061,3 +3069,43 @@ static void tipc_link_pl_send(struct tipc_link *l) } l->pl.count = TIPC_PROBE_INTERVAL; } + +static void tipc_link_pl_recv(struct tipc_link *l) +{ + pr_debug("%s: PLPMTUD: link: %p, state: %d, pmtu: %d, size: %d, high: %d\n", + __func__, l, l->pl.state, l->pl.pmtu, l->pl.probe_size, l->pl.probe_high); + + l->pl.pmtu = l->pl.probe_size; + l->pl.count = 0; + if (l->pl.state == TIPC_PL_BASE) { + l->pl.state = TIPC_PL_SEARCH; /* Base -> Search */ + l->pl.probe_size += TIPC_PL_BIG_STEP; + } else if (l->pl.state == TIPC_PL_ERROR) { + l->pl.state = TIPC_PL_SEARCH; /* Error -> Search */ + + l->pl.pmtu = l->pl.probe_size; + l->mtu = l->pl.pmtu; + l->pl.probe_size += TIPC_PL_BIG_STEP; + } else if (l->pl.state == TIPC_PL_SEARCH) { + if (!l->pl.probe_high) { + l->pl.probe_size = min(l->pl.probe_size + TIPC_PL_BIG_STEP, + TIPC_MAX_PLPMTU); + return; + } + l->pl.probe_size += TIPC_PL_MIN_STEP; + if (l->pl.probe_size >= l->pl.probe_high) { + l->pl.probe_high = 0; + l->pl.raise = 0; + l->pl.state = TIPC_PL_COMPLETE; /* Search -> Search Complete */ + + l->pl.probe_size = l->pl.pmtu; + l->mtu = l->pl.pmtu; + } + } else if (l->pl.state == TIPC_PL_COMPLETE) { + l->pl.raise++; + if (l->pl.raise == 30) { + l->pl.state = TIPC_PL_SEARCH; /* Search Complete -> Search */ + l->pl.probe_size += TIPC_PL_MIN_STEP; + } + } +} -- 2.27.0 |
From: Jon M. <jm...@re...> - 2021-09-10 00:04:01
|
On 06/07/2021 14:22, Xin Long wrote: > This patch is to receive and process the probe ack by checking > msg_max_pkt() == l->pl.probe_size then does state transition > in tipc_link_pl_recv(). > > For the details, see: > > https://lwn.net/Articles/860385/ > > Signed-off-by: Xin Long <luc...@gm...> > --- > net/tipc/link.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 48 insertions(+) > > diff --git a/net/tipc/link.c b/net/tipc/link.c > index 3af6c04f82c2..241c9378e258 100644 > --- a/net/tipc/link.c > +++ b/net/tipc/link.c > @@ -293,6 +293,7 @@ static int tipc_link_advance_transmq(struct tipc_link *l, struct tipc_link *r, > static void tipc_link_update_cwin(struct tipc_link *l, int released, > bool retransmitted); > static void tipc_link_pl_send(struct tipc_link *l); > +static void tipc_link_pl_recv(struct tipc_link *l); > /* > * Simple non-static link routines (i.e. referenced outside this file) > */ > @@ -2333,6 +2334,13 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb, > break; > } > > + if (!reply && msg_max_pkt(hdr) == l->pl.probe_size) { > + tipc_link_pl_recv(l); > + if (l->pl.state == TIPC_PL_COMPLETE) > + break; > + tipc_link_build_proto_msg(l, STATE_MSG, PROBE_PLPMTU, 0, 0, 0, 0, xmitq); > + } > + > /* Receive Gap ACK blocks from peer if any */ > glen = tipc_get_gap_ack_blks(&ga, l, hdr, true); > > @@ -3061,3 +3069,43 @@ static void tipc_link_pl_send(struct tipc_link *l) > } > l->pl.count = TIPC_PROBE_INTERVAL; > } > + > +static void tipc_link_pl_recv(struct tipc_link *l) > +{ > + pr_debug("%s: PLPMTUD: link: %p, state: %d, pmtu: %d, size: %d, high: %d\n", > + __func__, l, l->pl.state, l->pl.pmtu, l->pl.probe_size, l->pl.probe_high); Many of these lines will not pass checkpatch. ///jon > + > + l->pl.pmtu = l->pl.probe_size; > + l->pl.count = 0; > + if (l->pl.state == TIPC_PL_BASE) { > + l->pl.state = TIPC_PL_SEARCH; /* Base -> Search */ > + l->pl.probe_size += TIPC_PL_BIG_STEP; > + } else if (l->pl.state == TIPC_PL_ERROR) { > + l->pl.state = TIPC_PL_SEARCH; /* Error -> Search */ > + > + l->pl.pmtu = l->pl.probe_size; > + l->mtu = l->pl.pmtu; > + l->pl.probe_size += TIPC_PL_BIG_STEP; > + } else if (l->pl.state == TIPC_PL_SEARCH) { > + if (!l->pl.probe_high) { > + l->pl.probe_size = min(l->pl.probe_size + TIPC_PL_BIG_STEP, > + TIPC_MAX_PLPMTU); > + return; > + } > + l->pl.probe_size += TIPC_PL_MIN_STEP; > + if (l->pl.probe_size >= l->pl.probe_high) { > + l->pl.probe_high = 0; > + l->pl.raise = 0; > + l->pl.state = TIPC_PL_COMPLETE; /* Search -> Search Complete */ > + > + l->pl.probe_size = l->pl.pmtu; > + l->mtu = l->pl.pmtu; > + } > + } else if (l->pl.state == TIPC_PL_COMPLETE) { > + l->pl.raise++; > + if (l->pl.raise == 30) { > + l->pl.state = TIPC_PL_SEARCH; /* Search Complete -> Search */ > + l->pl.probe_size += TIPC_PL_MIN_STEP; > + } > + } > +} > |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:46
|
Since there's no enough bit in netdev_features_t for NETIF_F_GSO_TIPC_BIT, and tipc is using the simliar code as sctp, this patch will reuse SKB_GSO_SCTP and NETIF_F_GSO_SCTP_BIT for tipc. Signed-off-by: Xin Long <luc...@gm...> --- include/linux/skbuff.h | 2 -- net/tipc/node.c | 15 ++++++++++++++- net/tipc/offload.c | 4 ++-- 3 files changed, 16 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 148bf0ed7336..b2db9cd9a73f 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -599,8 +599,6 @@ enum { SKB_GSO_UDP_L4 = 1 << 17, SKB_GSO_FRAGLIST = 1 << 18, - - SKB_GSO_TIPC = 1 << 19, }; #if BITS_PER_LONG > 32 diff --git a/net/tipc/node.c b/net/tipc/node.c index 9947b7dfe1d2..17e59c8dac31 100644 --- a/net/tipc/node.c +++ b/net/tipc/node.c @@ -2068,7 +2068,7 @@ static bool tipc_node_check_state(struct tipc_node *n, struct sk_buff *skb, * Invoked with no locks held. Bearer pointer must point to a valid bearer * structure (i.e. cannot be NULL), but bearer can be inactive. */ -void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) +static void __tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) { struct sk_buff_head xmitq; struct tipc_link_entry *le; @@ -2189,6 +2189,19 @@ void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) kfree_skb(skb); } +void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) +{ + struct sk_buff *seg, *next; + + if (!skb_is_gso(skb) || !skb_is_gso_sctp(skb)) + return __tipc_rcv(net, skb, b); + + skb_list_walk_safe(skb_shinfo(skb)->frag_list, seg, next) + __tipc_rcv(net, seg, b); + skb_shinfo(skb)->frag_list = NULL; + consume_skb(skb); +} + void tipc_node_apply_property(struct net *net, struct tipc_bearer *b, int prop) { diff --git a/net/tipc/offload.c b/net/tipc/offload.c index d137679f4db0..26e372178635 100644 --- a/net/tipc/offload.c +++ b/net/tipc/offload.c @@ -5,7 +5,7 @@ static struct sk_buff *tipc_gso_segment(struct sk_buff *skb, netdev_features_t features) { - if (!(skb_shinfo(skb)->gso_type & SKB_GSO_TIPC)) + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_SCTP)) return ERR_PTR(-EINVAL); return skb_segment(skb, (features | NETIF_F_HW_CSUM) & ~NETIF_F_SG); @@ -39,7 +39,7 @@ bool tipc_msg_gso_append(struct sk_buff **p, struct sk_buff *skb, u16 segs) skb_shinfo(nskb)->frag_list = head; skb_shinfo(nskb)->gso_segs = 1; - skb_shinfo(nskb)->gso_type = SKB_GSO_TIPC; + skb_shinfo(nskb)->gso_type = SKB_GSO_SCTP; skb_shinfo(nskb)->gso_size = GSO_BY_FRAGS; skb_reset_network_header(head); -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:43
|
This is the base code for tipc gso, and tipc_gso_segment() will only be called after gso packets are built in the next patch. Signed-off-by: Xin Long <luc...@gm...> --- include/linux/skbuff.h | 2 ++ net/tipc/Makefile | 2 +- net/tipc/core.c | 3 +++ net/tipc/msg.h | 2 ++ net/tipc/offload.c | 29 +++++++++++++++++++++++++++++ 5 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 net/tipc/offload.c diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index b2db9cd9a73f..148bf0ed7336 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -599,6 +599,8 @@ enum { SKB_GSO_UDP_L4 = 1 << 17, SKB_GSO_FRAGLIST = 1 << 18, + + SKB_GSO_TIPC = 1 << 19, }; #if BITS_PER_LONG > 32 diff --git a/net/tipc/Makefile b/net/tipc/Makefile index ee49a9f1dd4f..ff276bf78d03 100644 --- a/net/tipc/Makefile +++ b/net/tipc/Makefile @@ -9,7 +9,7 @@ tipc-y += addr.o bcast.o bearer.o \ core.o link.o discover.o msg.o \ name_distr.o subscr.o monitor.o name_table.o net.o \ netlink.o netlink_compat.o node.o socket.o eth_media.o \ - topsrv.o group.o trace.o + topsrv.o group.o trace.o offload.o CFLAGS_trace.o += -I$(src) diff --git a/net/tipc/core.c b/net/tipc/core.c index 3f4542e0f065..1f59371aa036 100644 --- a/net/tipc/core.c +++ b/net/tipc/core.c @@ -186,6 +186,8 @@ static int __init tipc_init(void) if (err) goto out_netlink_compat; + tipc_offload_init(); + pr_info("Started in single node mode\n"); return 0; @@ -210,6 +212,7 @@ static int __init tipc_init(void) static void __exit tipc_exit(void) { + tipc_offload_exit(); tipc_netlink_compat_stop(); tipc_netlink_stop(); tipc_bearer_cleanup(); diff --git a/net/tipc/msg.h b/net/tipc/msg.h index 64ae4c4c44f8..d6c6231b8208 100644 --- a/net/tipc/msg.h +++ b/net/tipc/msg.h @@ -1203,6 +1203,8 @@ bool tipc_msg_pskb_copy(u32 dst, struct sk_buff_head *msg, bool __tipc_skb_queue_sorted(struct sk_buff_head *list, u16 seqno, struct sk_buff *skb); bool tipc_msg_skb_clone(struct sk_buff_head *msg, struct sk_buff_head *cpy); +void tipc_offload_init(void); +void tipc_offload_exit(void); static inline u16 buf_seqno(struct sk_buff *skb) { diff --git a/net/tipc/offload.c b/net/tipc/offload.c new file mode 100644 index 000000000000..f8a81c8886f0 --- /dev/null +++ b/net/tipc/offload.c @@ -0,0 +1,29 @@ +#include <linux/netdevice.h> +#include <linux/skbuff.h> +#include "msg.h" + +static struct sk_buff *tipc_gso_segment(struct sk_buff *skb, + netdev_features_t features) +{ + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_TIPC)) + return ERR_PTR(-EINVAL); + + return skb_segment(skb, (features | NETIF_F_HW_CSUM) & ~NETIF_F_SG); +} + +static struct packet_offload tipc_packet_offload __read_mostly = { + .type = cpu_to_be16(ETH_P_TIPC), + .callbacks = { + .gso_segment = tipc_gso_segment, + }, +}; + +void tipc_offload_init(void) +{ + dev_add_offload(&tipc_packet_offload); +} + +void tipc_offload_exit(void) +{ + dev_remove_offload(&tipc_packet_offload); +} -- 2.27.0 |
From: Xin L. <luc...@gm...> - 2021-07-06 18:22:52
|
TIPC GSO is implemented in the skb frag_list way as SCTP does. We don't need to change much in the tx path, but only create a head skb and append the skbs when there are more than one skb ready to send. In the lower-layer gso_segment(), it does fragmentation by copy eth header or ip/udp header to each skb in the head_skb's frag_list and send them one by one. This supports with both eth media and udp media. Signed-off-by: Xin Long <luc...@gm...> --- net/tipc/bearer.c | 23 +++++++++++++++++++++-- net/tipc/msg.h | 1 + net/tipc/offload.c | 41 +++++++++++++++++++++++++++++++++++++++++ net/tipc/udp_media.c | 7 +++++++ 4 files changed, 70 insertions(+), 2 deletions(-) diff --git a/net/tipc/bearer.c b/net/tipc/bearer.c index 443f8e5b9477..b0321b21bfdc 100644 --- a/net/tipc/bearer.c +++ b/net/tipc/bearer.c @@ -570,8 +570,9 @@ void tipc_bearer_xmit(struct net *net, u32 bearer_id, struct tipc_media_addr *dst, struct tipc_node *__dnode) { + struct sk_buff *head = NULL, *skb, *tmp; struct tipc_bearer *b; - struct sk_buff *skb, *tmp; + u16 segs = 0; if (skb_queue_empty(xmitq)) return; @@ -585,13 +586,31 @@ void tipc_bearer_xmit(struct net *net, u32 bearer_id, if (likely(test_bit(0, &b->up) || msg_is_reset(buf_msg(skb)))) { #ifdef CONFIG_TIPC_CRYPTO tipc_crypto_xmit(net, &skb, b, dst, __dnode); - if (skb) + if (!skb) + continue; #endif + if (!skb->ignore_df) { /* PLPMTUD probe packet*/ b->media->send_msg(net, skb, b, dst); + continue; + } + if (!head) { + segs = 1; + head = skb; + continue; + } + if (tipc_msg_gso_append(&head, skb, segs)) { + segs++; + continue; + } + b->media->send_msg(net, head, b, dst); + segs = 1; + head = skb; } else { kfree_skb(skb); } } + if (head) + b->media->send_msg(net, head, b, dst); rcu_read_unlock(); } diff --git a/net/tipc/msg.h b/net/tipc/msg.h index d6c6231b8208..4d1ff666790c 100644 --- a/net/tipc/msg.h +++ b/net/tipc/msg.h @@ -1205,6 +1205,7 @@ bool __tipc_skb_queue_sorted(struct sk_buff_head *list, u16 seqno, bool tipc_msg_skb_clone(struct sk_buff_head *msg, struct sk_buff_head *cpy); void tipc_offload_init(void); void tipc_offload_exit(void); +bool tipc_msg_gso_append(struct sk_buff **p, struct sk_buff *skb, u16 segs); static inline u16 buf_seqno(struct sk_buff *skb) { diff --git a/net/tipc/offload.c b/net/tipc/offload.c index f8a81c8886f0..d137679f4db0 100644 --- a/net/tipc/offload.c +++ b/net/tipc/offload.c @@ -18,6 +18,47 @@ static struct packet_offload tipc_packet_offload __read_mostly = { }, }; +bool tipc_msg_gso_append(struct sk_buff **p, struct sk_buff *skb, u16 segs) +{ + struct sk_buff *head = *p; + struct sk_buff *nskb; + + if (head->len + skb->len >= 65535) + return false; + + if (segs == 1) { + nskb = tipc_buf_acquire(0, GFP_ATOMIC); + if (!nskb) + return false; + + nskb->ip_summed = CHECKSUM_UNNECESSARY; + nskb->truesize += head->truesize; + nskb->data_len += head->len; + nskb->len += head->len; + TIPC_SKB_CB(nskb)->tail = head; + + skb_shinfo(nskb)->frag_list = head; + skb_shinfo(nskb)->gso_segs = 1; + skb_shinfo(nskb)->gso_type = SKB_GSO_TIPC; + skb_shinfo(nskb)->gso_size = GSO_BY_FRAGS; + skb_reset_network_header(head); + + head = nskb; + *p = head; + } + + head->truesize += skb->truesize; + head->data_len += skb->len; + head->len += skb->len; + TIPC_SKB_CB(head)->tail->next = skb; + TIPC_SKB_CB(head)->tail = skb; + + skb_shinfo(head)->gso_segs++; + skb_reset_network_header(skb); + + return true; +} + void tipc_offload_init(void) { dev_add_offload(&tipc_packet_offload); diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c index 5078c5b19e81..7da02db6a50e 100644 --- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -245,6 +245,13 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb, goto out; } + if (skb_is_gso(skb)) + skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM; + + skb->encapsulation = 1; + skb_reset_inner_mac_header(skb); + skb_reset_inner_network_header(skb); + skb_reset_inner_transport_header(skb); skb_set_inner_protocol(skb, htons(ETH_P_TIPC)); ub = rcu_dereference(b->media_ptr); if (!ub) { -- 2.27.0 |
From: Jon M. <jm...@re...> - 2021-09-09 20:04:45
|
On 06/07/2021 14:22, Xin Long wrote: > This patchset is to implement PLPMTUD and GSO for TIPC, > Patch 1-5 are for PLPMTUD while 6-8 are for GSO. I think this should be posted as two separate series, as they really implement two different features. The problem I see with this is that you reduce MTU in patch #1, so performance will suffer until the second series is adapted. Unless there is a way around this the two series mut at least be applied within the same release. Also, if I understand this correctly, PLTMUD will work also for the Ethernet bearer, so that jumbo frame capability can be detected? I am uncertain about the value of this, since jumbo frame capability is already detected by the endpoint bearers, and I doubt that such frames ever do more than one intra-subnet hop. But maybe I am wrong here? Anyway, this feature cannot do any harm even on Ethernet. ///jon > > It gets some ideas from SCTP as their similarities like > both are reliable datagram packets and possible to run > over IP(v6)/UDP. But also it does some adjustments for > TIPC. > > Xin Long (8): > tipc: set the mtu for bearer properly for udp media > tipc: add the constants and variables for plpmtud > tipc: build probe and its reply in tipc_link_build_proto_msg > tipc: add probe send and state transition > tipc: add probe recv and state transition > tipc: add offload base > tipc: add software gso > tipc: add hardware gso > > include/uapi/linux/tipc_config.h | 6 -- > net/tipc/Makefile | 2 +- > net/tipc/bearer.c | 23 ++++- > net/tipc/core.c | 3 + > net/tipc/link.c | 147 +++++++++++++++++++++++++++---- > net/tipc/link.h | 29 ++++++ > net/tipc/msg.c | 1 + > net/tipc/msg.h | 3 + > net/tipc/node.c | 15 +++- > net/tipc/offload.c | 70 +++++++++++++++ > net/tipc/udp_media.c | 18 ++-- > 11 files changed, 287 insertions(+), 30 deletions(-) > create mode 100644 net/tipc/offload.c > |
From: Jon M. <jm...@re...> - 2021-09-10 00:08:51
|
On 06/07/2021 14:22, Xin Long wrote: > Since there's no enough bit in netdev_features_t for > NETIF_F_GSO_TIPC_BIT, and tipc is using the simliar > code as sctp, this patch will reuse SKB_GSO_SCTP and > NETIF_F_GSO_SCTP_BIT for tipc. > > Signed-off-by: Xin Long <luc...@gm...> > --- > include/linux/skbuff.h | 2 -- > net/tipc/node.c | 15 ++++++++++++++- > net/tipc/offload.c | 4 ++-- > 3 files changed, 16 insertions(+), 5 deletions(-) > > diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h > index 148bf0ed7336..b2db9cd9a73f 100644 > --- a/include/linux/skbuff.h > +++ b/include/linux/skbuff.h > @@ -599,8 +599,6 @@ enum { > SKB_GSO_UDP_L4 = 1 << 17, > > SKB_GSO_FRAGLIST = 1 << 18, > - > - SKB_GSO_TIPC = 1 << 19, > }; > > #if BITS_PER_LONG > 32 > diff --git a/net/tipc/node.c b/net/tipc/node.c > index 9947b7dfe1d2..17e59c8dac31 100644 > --- a/net/tipc/node.c > +++ b/net/tipc/node.c > @@ -2068,7 +2068,7 @@ static bool tipc_node_check_state(struct tipc_node *n, struct sk_buff *skb, > * Invoked with no locks held. Bearer pointer must point to a valid bearer > * structure (i.e. cannot be NULL), but bearer can be inactive. > */ > -void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > +static void __tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > { > struct sk_buff_head xmitq; > struct tipc_link_entry *le; > @@ -2189,6 +2189,19 @@ void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > kfree_skb(skb); > } > > +void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > +{ > + struct sk_buff *seg, *next; > + > + if (!skb_is_gso(skb) || !skb_is_gso_sctp(skb)) > + return __tipc_rcv(net, skb, b); > + > + skb_list_walk_safe(skb_shinfo(skb)->frag_list, seg, next) > + __tipc_rcv(net, seg, b); > + skb_shinfo(skb)->frag_list = NULL; > + consume_skb(skb); > +} > + > void tipc_node_apply_property(struct net *net, struct tipc_bearer *b, > int prop) > { > diff --git a/net/tipc/offload.c b/net/tipc/offload.c > index d137679f4db0..26e372178635 100644 > --- a/net/tipc/offload.c > +++ b/net/tipc/offload.c > @@ -5,7 +5,7 @@ > static struct sk_buff *tipc_gso_segment(struct sk_buff *skb, > netdev_features_t features) > { > - if (!(skb_shinfo(skb)->gso_type & SKB_GSO_TIPC)) > + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_SCTP)) > return ERR_PTR(-EINVAL); > > return skb_segment(skb, (features | NETIF_F_HW_CSUM) & ~NETIF_F_SG); > @@ -39,7 +39,7 @@ bool tipc_msg_gso_append(struct sk_buff **p, struct sk_buff *skb, u16 segs) > > skb_shinfo(nskb)->frag_list = head; > skb_shinfo(nskb)->gso_segs = 1; > - skb_shinfo(nskb)->gso_type = SKB_GSO_TIPC; > + skb_shinfo(nskb)->gso_type = SKB_GSO_SCTP; > skb_shinfo(nskb)->gso_size = GSO_BY_FRAGS; > skb_reset_network_header(head); > > I don´t have much more to add, -it looks good to me, and way simpler than what I was trying a couple of years ago. If you fix the checkpatch issues and, maybe if possible, split it into two series, you have my ack. PS. Did you test this with crypto? ///jon |
From: Xin L. <luc...@gm...> - 2021-09-22 08:35:20
|
On Fri, Sep 10, 2021 at 8:08 AM Jon Maloy <jm...@re...> wrote: > > > > On 06/07/2021 14:22, Xin Long wrote: > > Since there's no enough bit in netdev_features_t for > > NETIF_F_GSO_TIPC_BIT, and tipc is using the simliar > > code as sctp, this patch will reuse SKB_GSO_SCTP and > > NETIF_F_GSO_SCTP_BIT for tipc. > > > > Signed-off-by: Xin Long <luc...@gm...> > > --- > > include/linux/skbuff.h | 2 -- > > net/tipc/node.c | 15 ++++++++++++++- > > net/tipc/offload.c | 4 ++-- > > 3 files changed, 16 insertions(+), 5 deletions(-) > > > > diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h > > index 148bf0ed7336..b2db9cd9a73f 100644 > > --- a/include/linux/skbuff.h > > +++ b/include/linux/skbuff.h > > @@ -599,8 +599,6 @@ enum { > > SKB_GSO_UDP_L4 = 1 << 17, > > > > SKB_GSO_FRAGLIST = 1 << 18, > > - > > - SKB_GSO_TIPC = 1 << 19, > > }; > > > > #if BITS_PER_LONG > 32 > > diff --git a/net/tipc/node.c b/net/tipc/node.c > > index 9947b7dfe1d2..17e59c8dac31 100644 > > --- a/net/tipc/node.c > > +++ b/net/tipc/node.c > > @@ -2068,7 +2068,7 @@ static bool tipc_node_check_state(struct tipc_node *n, struct sk_buff *skb, > > * Invoked with no locks held. Bearer pointer must point to a valid bearer > > * structure (i.e. cannot be NULL), but bearer can be inactive. > > */ > > -void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > > +static void __tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > > { > > struct sk_buff_head xmitq; > > struct tipc_link_entry *le; > > @@ -2189,6 +2189,19 @@ void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > > kfree_skb(skb); > > } > > > > +void tipc_rcv(struct net *net, struct sk_buff *skb, struct tipc_bearer *b) > > +{ > > + struct sk_buff *seg, *next; > > + > > + if (!skb_is_gso(skb) || !skb_is_gso_sctp(skb)) > > + return __tipc_rcv(net, skb, b); > > + > > + skb_list_walk_safe(skb_shinfo(skb)->frag_list, seg, next) > > + __tipc_rcv(net, seg, b); > > + skb_shinfo(skb)->frag_list = NULL; > > + consume_skb(skb); > > +} > > + > > void tipc_node_apply_property(struct net *net, struct tipc_bearer *b, > > int prop) > > { > > diff --git a/net/tipc/offload.c b/net/tipc/offload.c > > index d137679f4db0..26e372178635 100644 > > --- a/net/tipc/offload.c > > +++ b/net/tipc/offload.c > > @@ -5,7 +5,7 @@ > > static struct sk_buff *tipc_gso_segment(struct sk_buff *skb, > > netdev_features_t features) > > { > > - if (!(skb_shinfo(skb)->gso_type & SKB_GSO_TIPC)) > > + if (!(skb_shinfo(skb)->gso_type & SKB_GSO_SCTP)) > > return ERR_PTR(-EINVAL); > > > > return skb_segment(skb, (features | NETIF_F_HW_CSUM) & ~NETIF_F_SG); > > @@ -39,7 +39,7 @@ bool tipc_msg_gso_append(struct sk_buff **p, struct sk_buff *skb, u16 segs) > > > > skb_shinfo(nskb)->frag_list = head; > > skb_shinfo(nskb)->gso_segs = 1; > > - skb_shinfo(nskb)->gso_type = SKB_GSO_TIPC; > > + skb_shinfo(nskb)->gso_type = SKB_GSO_SCTP; > > skb_shinfo(nskb)->gso_size = GSO_BY_FRAGS; > > skb_reset_network_header(head); > > > > > > I don´t have much more to add, -it looks good to me, and way simpler > than what I was trying a couple of years ago. > > If you fix the checkpatch issues and, maybe if possible, split it into > two series, you have my ack. > > PS. Did you test this with crypto? Hi Jon, Sorry for late. Got an urgent problem from a customer recently, and spent quite a few weeks getting things almost done. I need to do more testing for its performance in different scenarios before continuing. I think I did, but I will confirm it will work well over crypto. Thanks a lot for checking! > > ///jon > |