Most menus show a check mark for checked entries, but the items in the "Settings" menu show dots when checked.
Not sure I understand. Screenshot? Some of the settings are mutually exclusive "radio buttons", so you can only select one of them. Those are probably rendered as dots instead of checks. Are they what you are referring to?
Checkboxes are usually for toggling on/off independent options, not option sets.
"Settings" menu with representative "radio buttons"
I guess the dots are supposed to be radio buttons, but the usual white 3D "holes" are not evident for the non-selected items.
See attached screen shot.
Ah I see what you mean now. Yeah, that's basically relying on Tk defaults and it looks like they're just sucking. It should look something more like this: http://docstore.mik.ua/orelly/perl3/tk/figs/mptk.1201.gif
I believe that code is in src/tclscripts/mged/openw.tcl if you're feeling ambitious.. :) Could be a simple matter of setting some attribute/property on the menu widget or could beg for a conversion to the new ttk widgets.