#199 dtsession hangs on the loading screen after login (Motif 2.3.x color-object selection deadlock); dtwm never starts

[Trung Le]

After login, dtsession hangs on the startup/loading screen and never spawns the window manager
(dtwm) or the rest of the desktop. The session appears frozen; nothing is clickable. This
happens with modern Motif (2.3.x as shipped by Linux distributions); it does not reproduce with
the older Motif CDE was originally built against.

Steps to reproduce

1. Build CDE against Motif 2.3.x (e.g. Fedora's motif/motif-devel).
2. Log in through dtlogin.

Expected: dtsession initializes, the color server starts, dtwm and the desktop come up.
Actual: dtsession blocks on the loading screen indefinitely; dtwm is never started.

Root cause
dtsession is the CDE color server. In cde/programs/dtsession/SmMain.c it opens a dedicated display connection for the color server and starts it (InitializeDtcolor(srvDisplay, …), line 379), and only afterwards creates its own top-level widgets (line ~382 onward).

Modern Motif (2.3.x) issues an XConvertSelection on _MOTIF_DEFAULT_COLORS from inside the Initialize method of any widget that pulls in the XmDisplay color object. For every ordinary CDE client this is fine — the request is answered by the color server (dtsession).

But when dtsession creates its own widgets, that selection request is sent to the color server, which is dtsession itself — and dtsession's main event loop has not started yet, so it can never reply to its own request. The result is a self-deadlock during widget initialization. A gdb backtrace shows dtsession blocked in:

_XmColorObjCreate -> XtAppCreateShell -> Initialize -> XtAppProcessEvent -> poll()

Why the fix is scoped to dtsession only

Motif exposes a useColorObj resource that disables this per-process color object. The fix must
be scoped to dtsession, not set globally (e.g. in Dt.ad): dtstyle's Color panel explicitly
checks it and refuses to operate when it is False — cde/programs/dtstyle/ColorMain.c:388:

  if (XmeUseColorObj() == FALSE)
     ErrDialog("The color portion of the Style Manager will not operate
                because the resource 'useColorObj' is set to False.", …);

So a global override would fix dtsession's hang but break the Style Manager's color editor for
everyone.

** Proposed fix**

Disable the Motif color object for dtsession only, in cde/programs/dtsession/Dtsession.src:

  Dtsession*useColorObj:        False

dtsession is the color server itself and does not need to consult a color object; every other
Motif client is unaffected and continues to get colors from the (now functional) color server

[Peter Howkins]

The fundamental issue is not Motif 2.3.x related. Debian/Ubuntu et al use 2.3.8 etc and do not show this issue in dtsession. This needs more investigation.

PS. may I ask, are you using AI to detect and generate these recent tickets and patches (the level of verboseness is unusual)? And if so, how much human oversight there has been in the verification of issues, verifying its conclusions, and the testing of the generated patches?

[Trung Le]

I could re-produce it reliably on Fedora box. Let me try to dig in Debian package to understand why it works there.

[Trung Le]

I stand corrected. The issue lies in my setup; after purging and re-install Motif lib, the issue is gone.

Admin, pleae help me close this ticket. Many thanks

[Jon Trulson]

Closed per request.