In ooDialog the ButtonControl class has the method style= If you look at the implementation for this method, you can see it is sending the BM_SETSTLYE message to the button control to change the style to what the user specifies.
The whole thing has a number of problems.
(I don't intend to fix this for 3.2.0, just making a note so I don't forget it.)
Looking at the code you see that the method accepts and tries to set a number of button styles like: BS_MULTILANE, BS_TOP, BS_LEFTTEXT, etc.. These types of button stles can not be changed / set using the BM_SETSTYLE message.
The doc does not mention these other styles (which is probably okay, the method does not work for most of them.) However, the method does work for some of the un-mentioned styles, like BS_PUSHLIKE
The styles changes that don't work using the BM_SETSTYLE message, can be changed after the button is created. But it needs to be done by changing the style Window word for the button.
The BM_SETSTYLE message is intended to change the style within a class of buttons. So this should be documented and maybe a check put into the method to enforce it. From MSDN:
The BM_SETSTYLE message changes the style of a button. It is designed for changing button styles within a type (for example, changing a check box to an automatic check box). It is not designed for changing between types (for example, changing a check box to a radio button). An application should not change a button from one type to another.