I see two differences here. In IUP we use LoadImage instead of LoadIcon and WM_SETICON instead of SetClassLong.
I think that the HICON returned in LoadImage and LoadIcon are the same. So the problem should be in the second difference.
But I cannot use SetClassLong or all IupDialogs will have the same Icon, because it will change the Window Class (even using an HWND as parameter) and all IupDialogs use the same Window Class.
In WM_SETICON there is a parameter that specifies the size of the icon, and currently we are using ICON_BIG. The other flag is ICON_SMALL. Unfortunately they cannot be combined, so maybe we have to call WM_SETICON twice. I tried that and it doesn’t seem to work.
Actually I tried many combinations of LoadImage/LoadIcon and SetClassLong/WM_SETICON(SMALL/BIG), and none of them work. Except LoadIcon+SetClassLong which affected all the dialogs.
But I tested only in Windows 8. Maybe in older systems the result is different. Which system are you using?