#3399 ntf: deadlock on ntf agent during failover

[PhanTranQuocDat]

When a NTF agent program invokes NTF APIs concurrently with failover, deadlock can happen.
The root cause is MDS thread invokes NTF agent callback during failover, which requires to obtain both MDS lock and NTF agent lock. Concurrently, an API calling thread obtains the NTF agent lock first and require obtaining MDS lock to send message. The encounter rate of this issue increases with the number of threads invoking the NTF API at the time the failover occurs.

[PhanTranQuocDat]

commit 458d58cb1d953bae10b37534fe38671b03be0b7c (HEAD -> develop, origin/develop, ticket-3399)
Author: dat.phan dat.phan@endava.com
Date: Thu Jun 4 07:36:06 2026 +0000

ntf: fix unhandled return code when logging notification [#3399]

When failover happens, new active ntfd must process all queued
notifications. During this failover window, the CLM membership
state on new active ntfd can transiently appear as non-member
to the LOG agent, causing the log client to be marked stale and
return SA_AIS_ERR_UNAVAILABLE when ntfd tries processing
notification. The return code is not handled and gets abort().

Fix by handling SA_AIS_ERR_UNAVAILABLE, requeue the notifcation and
try again.

commit 5ddab129203be7c36e810453650a389cd92e4ea2
Author: dat.phan dat.phan@endava.com
Date: Thu Jun 4 07:27:11 2026 +0000

ntf: add test to verify no deadlock on agent during failover [#3399]

commit 683a39c246a18e6684ed6104813f7a819e14f406
Author: dat.phan dat.phan@endava.com
Date: Thu Jun 4 06:57:34 2026 +0000

ntf: fix deadlock on ntf agent during failover [#3399]

recoverClient() holds cb_lock and calls ncsmds_api() which tries to
acquire gl_mds_library_mutex. Meanwhile, the MDS thread holds
gl_mds_library_mutex and its callback ntfa_mds_rcv() tries to acquire
cb_lock. This creates a circular lock dependency that deadlocks the
NTF agent.

Fix by releasing cb_lock before MDS calls in recoverClient(). A
'recovering' flag is set under cb_lock to prevent concurrent recovery
of the same handle (returns SA_AIS_ERR_TRY_AGAIN). When recovering
associated reader_hdl and subscriber_hdl, reacquire cb_lock during
traverse handle-list to prevent use-after-free.

Addationally, saNtfDispatch() will now properly return the error code
from recoverClient() instead of silently continuing on non-UNAVAILABLE
failures.

Note: cb_lock must never be held when calling MDS APIs (ncsmds_api,
ntfa_mds_msg_sync_send) since MDS callbacks acquire cb_lock under
gl_mds_library_mutex, creating an inherent lock-ordering constraint.