tipc-discussion — User questions & problem reports, project news, developer discussion

Re: [tipc-discussion] Sends Hanging

[Duzan, G. D <Gar...@fi...>]

   It looks like disabling/enabling all my bearers got thing back into a sane state. There had been some syslog activity like:

[Tue Jul 15 08:41:06 2025] tipc: Gacks on 04320114767:eno12399np0-0a0743c2:eno12399np0: 64, ql: 179!

and

[Thu Jul 10 16:23:43 2025] tipc: Resetting link <6c92cf7572e:eno12399np0-1423f2c2ed9:enp129s0f0np0>, link overflow

which indicated that things might not be healthy, and I guess the bearer cycle was enough to clean it up.

   Thanks.

Gary Duzan
IT Architect Senior
GT.M Core Team
________________________________
From: Duzan, Gary D <Gar...@fi...>
Sent: Wednesday, July 16, 2025 9:39 AM
To: Tung Nguyen <tun...@en...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: Re: Sends Hanging

   As for the stuck process, here is a /proc/{pid}/stack (from 5.14.0-570.25.1.el9_6.x86_64):

[<0>] wait_woken+0x50/0x60
[<0>] __tipc_sendmsg+0x28b/0x6d0 [tipc]
[<0>] tipc_sendmsg+0x31/0x50 [tipc]
[<0>] ____sys_sendmsg+0x388/0x3b0
[<0>] ___sys_sendmsg+0x88/0xd0
[<0>] __sys_sendmsg+0x59/0xa0
[<0>] do_syscall_64+0x5c/0xe0
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

Gary Duzan
IT Architect Senior
GT.M Core Team
________________________________
From: Duzan, Gary D <Gar...@fi...>
Sent: Wednesday, July 16, 2025 8:22 AM
To: Tung Nguyen <tun...@en...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: Re: Sends Hanging

Tung Nguyen
7/16/2025 5:20 AM
What command did you use to adjust the window size ?
# tipc link set window 51 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 100 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 500 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link statistics show
[...]
Link <1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
  RX packets:2004240 fragments:634446/44511 bundles:4826/22566
  TX packets:3063887 fragments:2068092/689014 bundles:2035/10681
  TX profile sample:267 packets average:1375 octets
  0-64:69% -256:1% -1024:0% -4096:0% -16384:30% -32768:0% -66000:0%
  RX states:2855437 probes:1363107 naks:8613 defs:92196 dups:19121
  TX states:2771466 probes:1363580 naks:2608352 acks:36820 retrans:2382
  Congestion link:938  Send queue max:0 avg:0

# tipc link get window link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
50

That was on an Ubuntu 24.04.2 server, but I get the same on RHEL 9.6 and Fedora 42.

   Thanks.
Gary Duzan
IT Architect Senior
GT.M Core Team

________________________________
From: Tung Nguyen <tun...@en...>
Sent: Wednesday, July 16, 2025 5:20 AM
To: Duzan, Gary D <Gar...@fi...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: RE: Sends Hanging


>   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity.
It seems a forever link congestion.
> My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?
What command did you use to adjust the window size ?



The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: [tipc-discussion] Sends Hanging

[Duzan, G. D <Gar...@fi...>]

   As for the stuck process, here is a /proc/{pid}/stack (from 5.14.0-570.25.1.el9_6.x86_64):

[<0>] wait_woken+0x50/0x60
[<0>] __tipc_sendmsg+0x28b/0x6d0 [tipc]
[<0>] tipc_sendmsg+0x31/0x50 [tipc]
[<0>] ____sys_sendmsg+0x388/0x3b0
[<0>] ___sys_sendmsg+0x88/0xd0
[<0>] __sys_sendmsg+0x59/0xa0
[<0>] do_syscall_64+0x5c/0xe0
[<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80

Gary Duzan
IT Architect Senior
GT.M Core Team
________________________________
From: Duzan, Gary D <Gar...@fi...>
Sent: Wednesday, July 16, 2025 8:22 AM
To: Tung Nguyen <tun...@en...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: Re: Sends Hanging

Tung Nguyen
7/16/2025 5:20 AM
What command did you use to adjust the window size ?
# tipc link set window 51 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 100 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 500 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link statistics show
[...]
Link <1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
  RX packets:2004240 fragments:634446/44511 bundles:4826/22566
  TX packets:3063887 fragments:2068092/689014 bundles:2035/10681
  TX profile sample:267 packets average:1375 octets
  0-64:69% -256:1% -1024:0% -4096:0% -16384:30% -32768:0% -66000:0%
  RX states:2855437 probes:1363107 naks:8613 defs:92196 dups:19121
  TX states:2771466 probes:1363580 naks:2608352 acks:36820 retrans:2382
  Congestion link:938  Send queue max:0 avg:0

# tipc link get window link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
50

That was on an Ubuntu 24.04.2 server, but I get the same on RHEL 9.6 and Fedora 42.

   Thanks.
Gary Duzan
IT Architect Senior
GT.M Core Team

________________________________
From: Tung Nguyen <tun...@en...>
Sent: Wednesday, July 16, 2025 5:20 AM
To: Duzan, Gary D <Gar...@fi...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: RE: Sends Hanging


>   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity.
It seems a forever link congestion.
> My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?
What command did you use to adjust the window size ?



The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: [tipc-discussion] Sends Hanging

[Duzan, G. D <Gar...@fi...>]

Tung Nguyen
7/16/2025 5:20 AM
What command did you use to adjust the window size ?
# tipc link set window 51 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 100 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link set window 500 link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
# tipc link statistics show
[...]
Link <1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
  RX packets:2004240 fragments:634446/44511 bundles:4826/22566
  TX packets:3063887 fragments:2068092/689014 bundles:2035/10681
  TX profile sample:267 packets average:1375 octets
  0-64:69% -256:1% -1024:0% -4096:0% -16384:30% -32768:0% -66000:0%
  RX states:2855437 probes:1363107 naks:8613 defs:92196 dups:19121
  TX states:2771466 probes:1363580 naks:2608352 acks:36820 retrans:2382
  Congestion link:938  Send queue max:0 avg:0

# tipc link get window link 1423f2c2ed9:enp129s0f0np0-04320114767:eno12399np0
50

That was on an Ubuntu 24.04.2 server, but I get the same on RHEL 9.6 and Fedora 42.

   Thanks.
Gary Duzan
IT Architect Senior
GT.M Core Team

________________________________
From: Tung Nguyen <tun...@en...>
Sent: Wednesday, July 16, 2025 5:20 AM
To: Duzan, Gary D <Gar...@fi...>; Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: RE: Sends Hanging


>   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity.
It seems a forever link congestion.
> My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?
What command did you use to adjust the window size ?



The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: [tipc-discussion] Sends Hanging

[Tung N. <tun...@en...>]

>   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity.
It seems a forever link congestion.
> My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?
What command did you use to adjust the window size ?



The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.

Re: [tipc-discussion] Sends Hanging

[Duzan, G. D <Gar...@fi...>]

   FWIW, here are the statistics of one of the links at issue:

Link <6c92cf7572e:eno12399np0-1423f2c2ed9:enp129s0f0np0>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
  RX packets:12175437 fragments:9688615/3172386 bundles:106357/731415
  TX packets:7688023 fragments:2106920/245873 bundles:202585/676190
  TX profile sample:915 packets average:186 octets
  0-64:90% -256:2% -1024:6% -4096:1% -16384:1% -32768:0% -66000:0%
  RX states:2570054 probes:1152299 naks:100121 defs:1057114 dups:210455
  TX states:2558618 probes:1151705 naks:275121 acks:139955 retrans:973666
  Congestion link:1179509  Send queue max:0 avg:0

And by hang, I mean strace shows the sendmsg() call never returning. I also sometimes see a case where a message goes through one way, but the reply message never arrives, with the same for retransmissions. I do look for the cmsg data on both ends, but I'm not seeing it in these cases.

Gary Duzan
IT Architect Senior
GT.M Core Team
________________________________
From: Duzan, Gary D via tipc-discussion <tip...@li...>
Sent: Tuesday, July 15, 2025 10:11 AM
To: Duzan, Gary D via tipc-discussion <tip...@li...>
Subject: [tipc-discussion] Sends Hanging

   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity. My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?

   Thanks.


Gary Duzan

IT Architect Senior

GT.M Core Team



E:  gar...@fi...

FIS | Advancing the way the world pays, banks and invests™


The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_______________________________________________
tipc-discussion mailing list
tip...@li...
https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/tipc-discussion__;!!AI5hVGByxkJIiw!gZUze6tUrRKCDcInJDYKm1vagC1ieW_kyuGU6w5ahxYwQQ74BR2lGQZJbfo7kkHFFbwX8t8e5tijIBYrQtR2T2B0Dy5SnK4RPidJBPIJ$
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

[tipc-discussion] Sends Hanging

[Duzan, G. D <Gar...@fi...>]

   I'm trying to diagnose an issue with TIPC datagram anycast or group broadcast sends hanging when the receiver processes on another host are waiting in either poll or recvmsg. These are relatively fast machines with 10g ethernet connecting them, and varying amounts of background activity. My first thought was to try adjusting the window settings, but the media/bearer window is maxed out, and the link windows seem to be fixed at 50. Is this the expected behavior?

   Thanks.


Gary Duzan

IT Architect Senior

GT.M Core Team



E:  gar...@fi...

FIS | Advancing the way the world pays, banks and invests™


The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: [tipc-discussion] Question on removing remote IP

[Hou, H. <HH...@rb...>]

Hi Tung,

Thanks for your reply.

In my application, there is a need to dynamically add and remove the remote IP addresses.
Can we add an option to the "tipc bearer" command to delete one of the remote IP? Since we have the "add", it makes sense to have a "delete" to pair with it.

Is there any other way to remove a remote IP from a bearer without disabling the bearer.

Thanks,
Hao


-----Original Message-----
From: Tung Nguyen <tun...@en...> 
Sent: Tuesday, February 4, 2025 7:25 PM
To: Hou, Hao <HH...@rb...>; tip...@li...
Subject: [EXTERNAL] RE: Question on removing remote IP


>TIPC supports adding multiple remote IP addresses to a UDP bearer with "tipc bearer add" command.
>How do we remove a previously added remote IP address from a UDP bearer without impacting the TIPC communications with other existing remote IP addresses in the same UDP bearer? The "tipc bearer --help" doesn't show such option.
No, there is no such option.

The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.

Disclaimer

This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

Re: [tipc-discussion] Question on removing remote IP

[Tung N. <tun...@en...>]

>TIPC supports adding multiple remote IP addresses to a UDP bearer with "tipc bearer add" command.
>How do we remove a previously added remote IP address from a UDP bearer without impacting the TIPC communications with other existing remote IP addresses in the same UDP bearer? The "tipc bearer --help" doesn't show such option.
No, there is no such option.

The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.

[tipc-discussion] Question on removing remote IP

[Hou, H. <HH...@rb...>]

Greetings TIPC team,

TIPC supports adding multiple remote IP addresses to a UDP bearer with "tipc bearer add" command.
How do we remove a previously added remote IP address from a UDP bearer without impacting the TIPC communications with other existing remote IP addresses in the same UDP bearer? The "tipc bearer --help" doesn't show such option.

Thanks,
Hao

Disclaimer

This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

Re: [tipc-discussion] Reg: Need information about TIPC_MCAST subscriber socket

[prakash b. <ps1...@gm...>]

Thank you Jon for your response. In some cases, we might be hitting a
race-condition during the small window that you have mentioned.

To give you more context about our use case which is little different than
you have mentioned in your solution. We have one publisher and many
subscribers.
Typically the publisher is created first and the subscribers may come later
at any time.
The subscriber application needs to pull already published data from the
publisher application.

Currently the subscriber application is doing the pull just after the
bind() system call through a different channel( not pub/sub).
But if some data is published after the pull and before the subscriber
socket becomes ready, then that data will never reach the subscriber
application.

Is there any way to know if the subscriber socket is fully ready inside the
subscriber application ?
Or
If the subscriber application waits for a few milliseconds ( e.g.100ms )
after bind() system call() and  before the pull, would it be a
practical approach to avoid this race condition ?

Kindly note that we are using tipc pub/sub sockets with scope as
"TIPC_NODE_SCOPE" so the window we are talking about here should be
relatively very small.

Thanks and Regards,
Prakash


On Tue, Sep 17, 2024 at 1:01 AM Jon Maloy <jm...@re...> wrote:

>
>
> On 2024-09-16 07:58, prakash bisht wrote:
> > Hi Hi Tung,Xin,John,
> >
> > We are using tipc multicast sockets for publisher/subscriber type of
> > application.
> > Below are the socket details
> > socket type     :  SOCK_RDM
> > tipc addr type  :  TIPC_MCAST
> >
> > We would like to know if the successful bind() call on the subscriber
> > socket guarantees that the subscriber is ready and no publication
> > would be missed ? Or shall we wait for some time before assuming the
> > subscriber socket is ready ?
> There may be (very) a small lag between the return of the bind() call
> until the bind is registered on all the other nodes.
> The proper way to get around this is to let your "publishers" use the
> TIPC subscription mechanism to keep track of your
> "subscriber", i.e., they don´t start publishing until they have received
> a notification that the subscriber is bound
> to the service address/range.
> ///jon
> >
> >
> > Thanks and Regards,
> > Prakash
>
>

Re: [tipc-discussion] Reg: Need information about TIPC_MCAST subscriber socket

[Jon M. <jm...@re...>]

On 2024-09-16 07:58, prakash bisht wrote:
> Hi Hi Tung,Xin,John,
>
> We are using tipc multicast sockets for publisher/subscriber type of 
> application.
> Below are the socket details
> socket type     :  SOCK_RDM
> tipc addr type  :  TIPC_MCAST
>
> We would like to know if the successful bind() call on the subscriber 
> socket guarantees that the subscriber is ready and no publication 
> would be missed ? Or shall we wait for some time before assuming the 
> subscriber socket is ready ?
There may be (very) a small lag between the return of the bind() call 
until the bind is registered on all the other nodes.
The proper way to get around this is to let your "publishers" use the 
TIPC subscription mechanism to keep track of your
"subscriber", i.e., they don´t start publishing until they have received 
a notification that the subscriber is bound
to the service address/range.
///jon
>
>
> Thanks and Regards,
> Prakash

[tipc-discussion] Reg: Need information about TIPC_MCAST subscriber socket

[prakash b. <ps1...@gm...>]

Hi  Hi Tung,Xin,John,

We are using tipc multicast sockets for publisher/subscriber type of
application.
Below are the socket details
socket type     :  SOCK_RDM
tipc addr type  :  TIPC_MCAST

We would like to know if the successful bind() call on the subscriber
socket guarantees that the subscriber is ready and no publication would be
missed ? Or shall we wait for some time before assuming the subscriber
socket is ready ?


Thanks and Regards,
Prakash

Re: [tipc-discussion] Reg: TIPC Error - "Unable to remove publication from failed node"

[Tung N. <tun...@en...>]

Hi Tung,Xin,John,
Any thoughts about the previous issue ? We are seeing this quite often in our system now. Any help would be appreciated.
Does this happen on a specific node or on some nodes of your cluster?  We can only check if you provide syslog (/var/log/messages) of all nodes.

The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Any opinions expressed are mine and do not necessarily represent the opinions of the Company. Emails are susceptible to interference. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is strictly prohibited and may be unlawful. If you have received this message in error, do not open any attachments but please notify the Endava Service Desk on (+44 (0)870 423 0187), and delete this message from your system. The sender accepts no responsibility for information, errors or omissions in this email, or for its use or misuse, or for any act committed or omitted in connection with this communication. If in doubt, please verify the authenticity of the contents with the sender. Please rely on your own virus checkers as no responsibility is taken by the sender for any damage rising out of any bug or virus infection.

Endava plc is a company registered in England under company number 5722669 whose registered office is at 125 Old Broad Street, London, EC2N 1AR, United Kingdom. Endava plc is the Endava group holding company and does not provide any services to clients. Each of Endava plc and its subsidiaries is a separate legal entity and has no liability for another such entity's acts or omissions.

Re: [tipc-discussion] Reg: TIPC Error - "Unable to remove publication from failed node"

[prakash b. <ps1...@gm...>]

Hi Tung,Xin,John,
Any thoughts about the previous issue ? We are seeing this quite often in
our system now. Any help would be appreciated.

Thanks and Regards,
Prakash

On Wed, Jul 17, 2024 at 5:57 PM prakash bisht <ps1...@gm...> wrote:

> Thanks Tung for your response. I verified the kernel logs in our setup.
>> There is no memory error as in your case.
>>
>> I see the below pattern though.
>> 1. There are multiple local publication related messages. But TIPC is not
>> entering the infinite loop immediately.
>>       Jul 16 13:23:31 in-debbld39-pd kernel: [90527.681211] Failed to
>> remove local publication {2,1359237139,1359237139}/0
>> 2. There are some errors of illegal FSM even after which the infinite
>> loop is triggered.
>>     Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396954] Illegal FSM
>> event fa110ede in state d0000 on link
>> 13500451:INTERCC_BEARER-135004a1:INTERCC_BEARER
>>  Not sure if the above error has some relation with hitting "Unable to
>> remove publication from failed node" infinite loop issue.
>>
>>
>> From kernel.log file -
>>
>> /**************************************************************************************************************************************************************************************************************************************/
>> Jul 16 13:23:31 in-debbld39-pd kernel: [90527.681211] Failed to remove
>> local publication {2,1359237139,1359237139}/0
>> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.911632] [UFW BLOCK] IN=eno1
>> OUT= MAC=5c:b9:01:fe:f6:d0:40:a8:f0:3b:ce:40:08:00 SRC=10.220.82.25
>> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=45831 DF PROTO=TCP
>> SPT=46905 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
>> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.913988] [UFW BLOCK] IN=eno1
>> OUT= MAC=5c:b9:01:fe:f6:d0:d4:f5:ef:70:c9:10:08:00 SRC=10.220.82.110
>> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=64852 DF PROTO=TCP
>> SPT=48891 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
>> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.915163] [UFW BLOCK] IN=eno1
>> OUT= MAC=5c:b9:01:fe:f6:d0:ec:b1:d7:83:83:90:08:00 SRC=10.220.82.28
>> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27835 DF PROTO=TCP
>> SPT=60415 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
>> Jul 16 13:23:41 in-debbld39-pd kernel: [90537.913017] Failed to remove
>> local publication {2,1359237139,1359237139}/0
>> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299673] ACPI Error:
>> SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found
>> length 32 (20180810/exfield-393)
>> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299684] ACPI Error: Method
>> parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT
>> (20180810/psparse-516)
>> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299695] ACPI Error:
>> AE_AML_BUFFER_LIMIT, Evaluating _PMM (20180810/power_meter-339)
>> Jul 16 13:23:48 in-debbld39-pd kernel: [90544.825004] Failed to remove
>> local publication {2,1359237139,1359237139}/0
>> Jul 16 13:23:51 in-debbld39-pd kernel: [90547.993176] Failed to remove
>> local publication {2,1359237139,1359237139}/0
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90549.816973] Failed to remove
>> local publication {2,1359237139,1359237139}/0
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396954] Illegal FSM event
>> fa110ede in state d0000 on link
>> 13500451:INTERCC_BEARER-135004a1:INTERCC_BEARER
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396971] Unable to remove
>> publication from failed node
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396971]  (type=0,
>> lower=2701414419, node=0xa1045013, port=0, key=2701414419)
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396976] Unable to remove
>> publication from failed node
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396976]  (type=19398666,
>> lower=1, node=0xa1045013, port=3268044884, key=3268044885)
>> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396978] Unable to remove
>> publication from failed node
>>
>> /**************************************************************************************************************************************************************************************************************************************/
>>
>> Thanks,
>> Prakash
>>
>> On Mon, Jul 8, 2024 at 3:32 PM Tung Quang Nguyen <
>> tun...@de...> wrote:
>>
>>> >*Jun 29 14:45:36 in-debbld-33 kernel: [1724399.196945] Unable to remove
>>> publication from failed node Jun 29 14:45:36 in-debbld-33
>>> >kernel:
>>> >[1724399.196945]  (type=20185106, lower=1, node=0xd1045013,
>>> port=2177385505, key=2177385506) Jun 29 14:45:36 in-debbld-33
>>> >kernel:
>>> >[1724399.196948] Unable to remove publication from failed node Jun 29
>>> >14:45:36 in-debbld-33 kernel: [1724399.196948]  (type=20185106,
>>> lower=1, node=0xd1045013, port=2177385505, key=2177385506)
>>> >Jun 29 14:45:36
>>> >in-debbld-33 kernel: [1724399.196954] Unable to remove publication from
>>> failed node Jun 29 14:45:36 in-debbld-33 kernel:
>>> >[1724399.196954] (type=20185106, lower=1, node=0xd1045013,
>>> port=2177385505, key=2177385506)*
>>> >
>>>
>>> >###############################################################################################################
>>> >######
>>> >
>>> >
>>> >
>>> >Any idea why this would be happening ?
>>> Memory error could cause this. I observed the same thing in my system:
>>> "
>>> 2024-05-27T07:27:22.747+02:00 kernel:[425219.492047] {...}[Hardware
>>> Error]: Hardware error from APEI Generic Hardware Error Source: 0
>>> ...
>>> 2024-05-27T07:27:22.747+02:00 kernel:[425219.517622] {...}[Hardware
>>> Error]:   section_type: memory error
>>> ...
>>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.704107] tipc: Unable to
>>> remove publication from failed node
>>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.704107]  (type=143322,
>>> lower=9, node=0x1001009, port=1295954722, key=1295954723)
>>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.717912] tipc: Unable to
>>> remove publication from failed node
>>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.717912]  (type=143322,
>>> lower=9, node=0x1001009, port=1295954722, key=1295954723)
>>> "
>>>
>>

Re: [tipc-discussion] Reg: TIPC Error - "Unable to remove publication from failed node"

[prakash b. <ps1...@gm...>]

>
> Thanks Tung for your response. I verified the kernel logs in our setup.
> There is no memory error as in your case.
>
> I see the below pattern though.
> 1. There are multiple local publication related messages. But TIPC is not
> entering the infinite loop immediately.
>       Jul 16 13:23:31 in-debbld39-pd kernel: [90527.681211] Failed to
> remove local publication {2,1359237139,1359237139}/0
> 2. There are some errors of illegal FSM even after which the infinite loop
> is triggered.
>     Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396954] Illegal FSM
> event fa110ede in state d0000 on link
> 13500451:INTERCC_BEARER-135004a1:INTERCC_BEARER
>  Not sure if the above error has some relation with hitting "Unable to
> remove publication from failed node" infinite loop issue.
>
>
> From kernel.log file -
>
> /**************************************************************************************************************************************************************************************************************************************/
> Jul 16 13:23:31 in-debbld39-pd kernel: [90527.681211] Failed to remove
> local publication {2,1359237139,1359237139}/0
> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.911632] [UFW BLOCK] IN=eno1
> OUT= MAC=5c:b9:01:fe:f6:d0:40:a8:f0:3b:ce:40:08:00 SRC=10.220.82.25
> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=45831 DF PROTO=TCP
> SPT=46905 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.913988] [UFW BLOCK] IN=eno1
> OUT= MAC=5c:b9:01:fe:f6:d0:d4:f5:ef:70:c9:10:08:00 SRC=10.220.82.110
> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=64852 DF PROTO=TCP
> SPT=48891 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
> Jul 16 13:23:37 in-debbld39-pd kernel: [90533.915163] [UFW BLOCK] IN=eno1
> OUT= MAC=5c:b9:01:fe:f6:d0:ec:b1:d7:83:83:90:08:00 SRC=10.220.82.28
> DST=10.220.82.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27835 DF PROTO=TCP
> SPT=60415 DPT=111 WINDOW=64240 RES=0x00 SYN URGP=0
> Jul 16 13:23:41 in-debbld39-pd kernel: [90537.913017] Failed to remove
> local publication {2,1359237139,1359237139}/0
> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299673] ACPI Error:
> SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found
> length 32 (20180810/exfield-393)
> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299684] ACPI Error: Method
> parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT
> (20180810/psparse-516)
> Jul 16 13:23:42 in-debbld39-pd kernel: [90539.299695] ACPI Error:
> AE_AML_BUFFER_LIMIT, Evaluating _PMM (20180810/power_meter-339)
> Jul 16 13:23:48 in-debbld39-pd kernel: [90544.825004] Failed to remove
> local publication {2,1359237139,1359237139}/0
> Jul 16 13:23:51 in-debbld39-pd kernel: [90547.993176] Failed to remove
> local publication {2,1359237139,1359237139}/0
> Jul 16 13:23:53 in-debbld39-pd kernel: [90549.816973] Failed to remove
> local publication {2,1359237139,1359237139}/0
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396954] Illegal FSM event
> fa110ede in state d0000 on link
> 13500451:INTERCC_BEARER-135004a1:INTERCC_BEARER
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396971] Unable to remove
> publication from failed node
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396971]  (type=0,
> lower=2701414419, node=0xa1045013, port=0, key=2701414419)
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396976] Unable to remove
> publication from failed node
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396976]  (type=19398666,
> lower=1, node=0xa1045013, port=3268044884, key=3268044885)
> Jul 16 13:23:53 in-debbld39-pd kernel: [90550.396978] Unable to remove
> publication from failed node
>
> /**************************************************************************************************************************************************************************************************************************************/
>
> Thanks,
> Prakash
>
> On Mon, Jul 8, 2024 at 3:32 PM Tung Quang Nguyen <
> tun...@de...> wrote:
>
>> >*Jun 29 14:45:36 in-debbld-33 kernel: [1724399.196945] Unable to remove
>> publication from failed node Jun 29 14:45:36 in-debbld-33
>> >kernel:
>> >[1724399.196945]  (type=20185106, lower=1, node=0xd1045013,
>> port=2177385505, key=2177385506) Jun 29 14:45:36 in-debbld-33
>> >kernel:
>> >[1724399.196948] Unable to remove publication from failed node Jun 29
>> >14:45:36 in-debbld-33 kernel: [1724399.196948]  (type=20185106, lower=1,
>> node=0xd1045013, port=2177385505, key=2177385506)
>> >Jun 29 14:45:36
>> >in-debbld-33 kernel: [1724399.196954] Unable to remove publication from
>> failed node Jun 29 14:45:36 in-debbld-33 kernel:
>> >[1724399.196954] (type=20185106, lower=1, node=0xd1045013,
>> port=2177385505, key=2177385506)*
>> >
>>
>> >###############################################################################################################
>> >######
>> >
>> >
>> >
>> >Any idea why this would be happening ?
>> Memory error could cause this. I observed the same thing in my system:
>> "
>> 2024-05-27T07:27:22.747+02:00 kernel:[425219.492047] {...}[Hardware
>> Error]: Hardware error from APEI Generic Hardware Error Source: 0
>> ...
>> 2024-05-27T07:27:22.747+02:00 kernel:[425219.517622] {...}[Hardware
>> Error]:   section_type: memory error
>> ...
>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.704107] tipc: Unable to
>> remove publication from failed node
>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.704107]  (type=143322,
>> lower=9, node=0x1001009, port=1295954722, key=1295954723)
>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.717912] tipc: Unable to
>> remove publication from failed node
>> 2024-05-27T07:27:22.973+02:00 kernel:[425221.717912]  (type=143322,
>> lower=9, node=0x1001009, port=1295954722, key=1295954723)
>> "
>>
>

Re: [tipc-discussion] Reg: TIPC Error - "Unable to remove publication from failed node"

[Tung Q. N. <tun...@de...>]

>*Jun 29 14:45:36 in-debbld-33 kernel: [1724399.196945] Unable to remove publication from failed node Jun 29 14:45:36 in-debbld-33
>kernel:
>[1724399.196945]  (type=20185106, lower=1, node=0xd1045013, port=2177385505, key=2177385506) Jun 29 14:45:36 in-debbld-33
>kernel:
>[1724399.196948] Unable to remove publication from failed node Jun 29
>14:45:36 in-debbld-33 kernel: [1724399.196948]  (type=20185106, lower=1, node=0xd1045013, port=2177385505, key=2177385506)
>Jun 29 14:45:36
>in-debbld-33 kernel: [1724399.196954] Unable to remove publication from failed node Jun 29 14:45:36 in-debbld-33 kernel:
>[1724399.196954] (type=20185106, lower=1, node=0xd1045013, port=2177385505, key=2177385506)*
>
>###############################################################################################################
>######
>
>
>
>Any idea why this would be happening ?
Memory error could cause this. I observed the same thing in my system:
"
2024-05-27T07:27:22.747+02:00 kernel:[425219.492047] {...}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
...
2024-05-27T07:27:22.747+02:00 kernel:[425219.517622] {...}[Hardware Error]:   section_type: memory error
...
2024-05-27T07:27:22.973+02:00 kernel:[425221.704107] tipc: Unable to remove publication from failed node
2024-05-27T07:27:22.973+02:00 kernel:[425221.704107]  (type=143322, lower=9, node=0x1001009, port=1295954722, key=1295954723)
2024-05-27T07:27:22.973+02:00 kernel:[425221.717912] tipc: Unable to remove publication from failed node
2024-05-27T07:27:22.973+02:00 kernel:[425221.717912]  (type=143322, lower=9, node=0x1001009, port=1295954722, key=1295954723)
"

[tipc-discussion] Reg: TIPC Error - "Unable to remove publication from failed node"

[prakash b. <ps1...@gm...>]

Hi All,



We have been using TIPC sockets for a while now. Off late we are facing
issues randomly where the CPU is frozen.

The TIPC module seems to be entering in an infinite loop while trying to
remove publication from a failed node.

This is resulting in log flooding. Almost 50 GB worth of logs written in
1-2 hours.  This also seems to be hogging the CPU  resulting in the
non-responsive system. We had to do a hard shutdown to recover from it.


Below are the logs snippets from the kernel log file.

#####################################################################################################################






*Jun 29 14:45:36 in-debbld-33 kernel: [1724399.196945] Unable to remove
publication from failed node Jun 29 14:45:36 in-debbld-33 kernel:
[1724399.196945]  (type=20185106, lower=1, node=0xd1045013,
port=2177385505, key=2177385506) Jun 29 14:45:36 in-debbld-33 kernel:
[1724399.196948] Unable to remove publication from failed node Jun 29
14:45:36 in-debbld-33 kernel: [1724399.196948]  (type=20185106, lower=1,
node=0xd1045013, port=2177385505, key=2177385506) Jun 29 14:45:36
in-debbld-33 kernel: [1724399.196954] Unable to remove publication from
failed node Jun 29 14:45:36 in-debbld-33 kernel: [1724399.196954]
(type=20185106, lower=1, node=0xd1045013, port=2177385505, key=2177385506)*

#####################################################################################################################



Any idea why this would be happening ?

Would appreciate any help.


*Kernel version : 4.19.0-26-amd64*


Thanks,

Prakash

Re: [tipc-discussion] Tuning/Debugging of "Retransmission failure"

[Duzan, G. D <Gar...@fi...>]

   FWIW, my code changes did manage to fix the problems I was having with idempotent datagram requests, where I could retransmit either way safely. Things got messier for a non-idempotent service with either dropped datagrams or dropped connections due to the link reset. So far, the only thing I have found to mitigate the problem is increasing the UDP media/bearer/link tolerance from 1500 to 5000. At least the tests I have run so far seem happier with it.

                                                  Gary Duzan
                                                  IT Architect Senior
                                                  GT.M Core Team

________________________________
From: Duzan, Gary D via tipc-discussion <tip...@li...>
Sent: Thursday, May 16, 2024 4:46 PM
To: Tung Quang Nguyen <tun...@de...>; tip...@li... <tip...@li...>
Subject: Re: [tipc-discussion] Tuning/Debugging of "Retransmission failure"


Tung Quang Nguyen
5/15/2024 11:56 PM
Not sure what you mean by "pushing TIPC a bit". If it means dropping TIPC messages, then "Retransmission failure" is expected.
   It just means that I increased the amount of traffic across TIPC in the cluster. I only noticed because it appeared from the application level that messages were being dropped. I do have TIPC_DEST_DROPPABLE and TIPC_SRC_DROPPABLE set to zero, but I just realized that I only have the TIPC_ERRINFO handling on one end. I should fix that.

Tung Quang Nguyen
5/15/2024 11:56 PM
If you did not intentionally drop TIPC messages, then issue could be due to packet drop at NIC (in your VM node or host or Switch/Router etc.). You need to do the tunning at NIC.
   Running some more tests, it does appear that my clusters of larger servers with eth bearers are not encountering this issue, but my clusters of smaller servers with udp bearers are. There is also some disparity in net.tipc.tipc_rmem settings, so I should address that.

   So it looks like I have some things to try. I'll follow up if that doesn't address the problem.

   Thanks.

Gary Duzan
IT Architect Senior
GT.M Core Team

________________________________
From: Tung Quang Nguyen <tun...@de...>
Sent: Wednesday, May 15, 2024 11:56 PM
To: Duzan, Gary D <Gar...@fi...>; tip...@li... <tip...@li...>
Subject: RE: Tuning/Debugging of "Retransmission failure"


>   I've started to notice messages like this when pushing TIPC a bit:
Not sure what you mean by "pushing TIPC a bit". If it means dropping TIPC messages, then "Retransmission failure" is expected.
>Is there any tuning I can do to avoid the problem, or other data to collect to better understand it?
>
If you did not intentionally drop TIPC messages, then issue could be due to packet drop at NIC (in your VM node or host or Switch/Router etc.). You need to do the tunning at NIC.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Message Encrypted via TLS connection

_______________________________________________
tipc-discussion mailing list
tip...@li...
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Ftipc-discussion&data=05%7C02%7Cgary.duzan%40fisglobal.com%7Cd4e9b48e25094d3a586508dc75e962c0%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C638514892444800272%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=C%2F7M%2BcPUuQlPRZIQd7k7OicwrZFtRG6vYK9aFKyfP50%3D&reserved=0<https://lists.sourceforge.net/lists/listinfo/tipc-discussion>
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Message Encrypted via TLS connection

Re: [tipc-discussion] Tuning/Debugging of "Retransmission failure"

[Duzan, G. D <Gar...@fi...>]

Tung Quang Nguyen
5/15/2024 11:56 PM
Not sure what you mean by "pushing TIPC a bit". If it means dropping TIPC messages, then "Retransmission failure" is expected.
   It just means that I increased the amount of traffic across TIPC in the cluster. I only noticed because it appeared from the application level that messages were being dropped. I do have TIPC_DEST_DROPPABLE and TIPC_SRC_DROPPABLE set to zero, but I just realized that I only have the TIPC_ERRINFO handling on one end. I should fix that.

Tung Quang Nguyen
5/15/2024 11:56 PM
If you did not intentionally drop TIPC messages, then issue could be due to packet drop at NIC (in your VM node or host or Switch/Router etc.). You need to do the tunning at NIC.
   Running some more tests, it does appear that my clusters of larger servers with eth bearers are not encountering this issue, but my clusters of smaller servers with udp bearers are. There is also some disparity in net.tipc.tipc_rmem settings, so I should address that.

   So it looks like I have some things to try. I'll follow up if that doesn't address the problem.

   Thanks.

Gary Duzan
IT Architect Senior
GT.M Core Team

________________________________
From: Tung Quang Nguyen <tun...@de...>
Sent: Wednesday, May 15, 2024 11:56 PM
To: Duzan, Gary D <Gar...@fi...>; tip...@li... <tip...@li...>
Subject: RE: Tuning/Debugging of "Retransmission failure"


>   I've started to notice messages like this when pushing TIPC a bit:
Not sure what you mean by "pushing TIPC a bit". If it means dropping TIPC messages, then "Retransmission failure" is expected.
>Is there any tuning I can do to avoid the problem, or other data to collect to better understand it?
>
If you did not intentionally drop TIPC messages, then issue could be due to packet drop at NIC (in your VM node or host or Switch/Router etc.). You need to do the tunning at NIC.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Message Encrypted via TLS connection

Re: [tipc-discussion] Tuning/Debugging of "Retransmission failure"

[Tung Q. N. <tun...@de...>]

>   I've started to notice messages like this when pushing TIPC a bit:
Not sure what you mean by "pushing TIPC a bit". If it means dropping TIPC messages, then "Retransmission failure" is expected.
>Is there any tuning I can do to avoid the problem, or other data to collect to better understand it?
>
If you did not intentionally drop TIPC messages, then issue could be due to packet drop at NIC (in your VM node or host or Switch/Router etc.). You need to do the tunning at NIC.

Re: [tipc-discussion] Tuning/Debugging of "Retransmission failure"

[Duzan, G. D <Gar...@fi...>]

   In case it matters, I'm seeing the same thing on older kernels, too.

May 15 11:58:11 rhel04 kernel: tipc: Retransmission failure on link <qqqqqqqqqqqq:rhel04-wwwwwwwwwwww:ubu04>
May 15 11:58:11 rhel04 kernel: tipc: State of link  Link <qqqqqqqqqqqq:rhel04-wwwwwwwwwwww:ubu04> state e
May 15 11:58:11 rhel04 kernel: tipc: XMTQ: 45 [40360-40425], BKLGQ: 0, SNDNX: 40426, RCVNX: 37551
May 15 11:58:11 rhel04 kernel: tipc: Failed msg: usr 12, typ 1, len 13928, err 0
May 15 11:58:11 rhel04 kernel: tipc: sqno 40360, prev: be5645f1, dest: 8356ce57
May 15 11:58:11 rhel04 kernel: tipc: retr_stamp -1694829503, retr_cnt 110

Linux rhel04 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Linux ubu04 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux


Gary Duzan

IT Architect Senior

GT.M Core Team



E:  gar...@fi...

FIS | Advancing the way the world pays, banks and invests™


________________________________
From: Duzan, Gary D
Sent: Wednesday, May 15, 2024 1:07 PM
To: tip...@li... <tip...@li...>
Subject: Tuning/Debugging of "Retransmission failure"

   I've started to notice messages like this when pushing TIPC a bit:

[Wed May 15 11:27:32 2024] tipc: Retransmission failure on link <xxxxxxxxxxxx:deb03-yyyyyyyyyyyy:ubu06>
[Wed May 15 11:27:32 2024] tipc: State of link  Link <xxxxxxxxxxxx:deb03-yyyyyyyyyyyy:ubu06> state e
[Wed May 15 11:27:32 2024] tipc: XMTQ: 15 [30965-31149], BKLGQ: 0, SNDNX: 31150, RCVNX: 27685
[Wed May 15 11:27:32 2024] tipc: Failed msg: usr 12, typ 0, len 13928, err 0
[Wed May 15 11:27:32 2024] tipc: sqno 30965, prev: ace568c, dest: ad7f568c
[Wed May 15 11:27:32 2024] tipc: retr_stamp -1697900680, retr_cnt 73

The host reporting the problem is running:

Linux vlltcgtmdeb03 6.7.9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.9-2 (2024-03-13) x86_64 GNU/Linux

and the other end of the link is:

Linux vlltcgtmubu06 6.5.0-27-generic #28-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar  7 18:21:00 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Is there any tuning I can do to avoid the problem, or other data to collect to better understand it?

   Thanks.


Gary Duzan

IT Architect Senior

GT.M Core Team



E:  gar...@fi...

FIS | Advancing the way the world pays, banks and invests™


The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Message Encrypted via TLS connection

[tipc-discussion] Tuning/Debugging of "Retransmission failure"

[Duzan, G. D <Gar...@fi...>]

   I've started to notice messages like this when pushing TIPC a bit:

[Wed May 15 11:27:32 2024] tipc: Retransmission failure on link <xxxxxxxxxxxx:deb03-yyyyyyyyyyyy:ubu06>
[Wed May 15 11:27:32 2024] tipc: State of link  Link <xxxxxxxxxxxx:deb03-yyyyyyyyyyyy:ubu06> state e
[Wed May 15 11:27:32 2024] tipc: XMTQ: 15 [30965-31149], BKLGQ: 0, SNDNX: 31150, RCVNX: 27685
[Wed May 15 11:27:32 2024] tipc: Failed msg: usr 12, typ 0, len 13928, err 0
[Wed May 15 11:27:32 2024] tipc: sqno 30965, prev: ace568c, dest: ad7f568c
[Wed May 15 11:27:32 2024] tipc: retr_stamp -1697900680, retr_cnt 73

The host reporting the problem is running:

Linux vlltcgtmdeb03 6.7.9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.9-2 (2024-03-13) x86_64 GNU/Linux

and the other end of the link is:

Linux vlltcgtmubu06 6.5.0-27-generic #28-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar  7 18:21:00 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Is there any tuning I can do to avoid the problem, or other data to collect to better understand it?

   Thanks.


Gary Duzan

IT Architect Senior

GT.M Core Team



E:  gar...@fi...

FIS | Advancing the way the world pays, banks and invests™


The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. Message Encrypted via TLS connection

Re: [tipc-discussion] [PATCH net] tipc: fix a possible memleak in tipc_buf_append

[Tung Q. N. <tun...@de...>]

>Subject: [PATCH net] tipc: fix a possible memleak in tipc_buf_append
>
>__skb_linearize() doesn't free the skb when it fails, so move '*buf = NULL' after __skb_linearize(), so that the skb can be freed on the
>err path.
>
>Fixes: b7df21cf1b79 ("tipc: skb_linearize the head skb when reassembling msgs")
>Reported-by: Paolo Abeni <pa...@re...>
>Signed-off-by: Xin Long <luc...@gm...>
>---
> net/tipc/msg.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/net/tipc/msg.c b/net/tipc/msg.c index 5c9fd4791c4b..c52ab423082c 100644
>--- a/net/tipc/msg.c
>+++ b/net/tipc/msg.c
>@@ -142,9 +142,9 @@ int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
> 	if (fragid == FIRST_FRAGMENT) {
> 		if (unlikely(head))
> 			goto err;
>-		*buf = NULL;
> 		if (skb_has_frag_list(frag) && __skb_linearize(frag))
> 			goto err;
>+		*buf = NULL;
> 		frag = skb_unshare(frag, GFP_ATOMIC);
> 		if (unlikely(!frag))
> 			goto err;
>--
>2.43.0
Reviewed-by: Tung Nguyen <tun...@de...>

[tipc-discussion] [PATCH net] tipc: fix a possible memleak in tipc_buf_append

[Xin L. <luc...@gm...>]

__skb_linearize() doesn't free the skb when it fails, so move
'*buf = NULL' after __skb_linearize(), so that the skb can be
freed on the err path.

Fixes: b7df21cf1b79 ("tipc: skb_linearize the head skb when reassembling msgs")
Reported-by: Paolo Abeni <pa...@re...>
Signed-off-by: Xin Long <luc...@gm...>
---
 net/tipc/msg.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/tipc/msg.c b/net/tipc/msg.c
index 5c9fd4791c4b..c52ab423082c 100644
--- a/net/tipc/msg.c
+++ b/net/tipc/msg.c
@@ -142,9 +142,9 @@ int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
 	if (fragid == FIRST_FRAGMENT) {
 		if (unlikely(head))
 			goto err;
-		*buf = NULL;
 		if (skb_has_frag_list(frag) && __skb_linearize(frag))
 			goto err;
+		*buf = NULL;
 		frag = skb_unshare(frag, GFP_ATOMIC);
 		if (unlikely(!frag))
 			goto err;
-- 
2.43.0

Re: [tipc-discussion] Reg: TIPC socket buffer size recommendation

[Tung Q. N. <tun...@de...>]

>1. As TIPC creates a UDP socket for a bearer and multiple TIPC sockets are kind of multiplexed into one UDP socket. Can it slow the
>performance when the incoming traffic is high?
I do not see any performance down in my test when using UDP bearer.

>2. Can we increase the buffer size of the TIPC UDP socket alone without changing the system level default receive buffer size ?
>
>I would like to increase the socket buffer size to reduce the packet drops and be efficient but don't want to change the default value at
>the system level.
There is no packet drop at TIPC socket level when using SOCK_SEQPACKET. You do not need to do anything.