In Delphi 10.3, DIALOGS answered "No dialog is up" while the IDE's Options dialog (Tools > Options, TDefaultEnvironmentDialog) was open and in front. FORMS listed it as visible at the same time. Seen twice, about 5 seconds after opening it with MENUCLICK "Tools|Options...". In Delphi XE5, XE8, 10.1 and 10.2, and in 13, the same steps made DIALOGS list the dialog.
How DIALOGS decides: it enumerates the visible, enabled top-level windows of the main thread and lists one if its class is #32770 or if its owner window is disabled (then marked modal). The Options dialog is a VCL form, not a #32770, so in 10.3 the owner test must have failed: either the dialog has no owner window there, or its owner is not disabled while it is shown. Not checked which.
Possible fix: also treat a visible VCL form whose FormState contains fsModal as a dialog, which does not depend on how the IDE sets up the owner. First find out what the owner is in 10.3 (GetWindow(GW_OWNER) and whether it is enabled), to be sure the fix addresses the actual cause.
Correction to the description:
DIALOGSwas only checked in XE5 and 13, where it listed the Options dialog. For XE8, 10.1 and 10.2 onlyFORMSwas asked, so whetherDIALOGSlists it there is not known.