#69 Segfault in DTAppBuilder
I have fount two places that DTAppBuilder consistently segfaults on two installs I have done.
First setup is a VirtualBox VM running Ubuntu 17.04 minimal with only requirements for CDE installed.
Second is an AWS instance running Ubuntu Server 16.04. Both are 64bit.
In both setups DTBuilder segfaults when clicking on the file menu, and when dragging a widget to a window but not on a panel, ether there is no panel, or dragging to the space with no panel.
Using GDB I found both problems are very similar.
The file menu segfault is in abobj_menu.c at function set_menu_state on line 689. It appears that the line
XtVaGetValues(items[i], XmNuserData, &item_flag, NULL) is changing the value of items to NULL.
If I change the function to check the items[i] and items before and after that line and continue the loop if they are NULL. It seems to work. The file menu appears and I can create a new project.
The second segfault concerning adding the widget is a similar problem in the dtb_utils.c file in function dtb_show_message on line 1634. Here, again the second call to XtVaGetValues is overwriting the value of children and setting it to NULL. I can't verify a check for that because, as noted in the header comments, the file is auto generated and my changes are overwritten when I build the program.
Strangely, in both functions XtVaGetValues is being called in two locations. And in both cases the first call is completed fine.
I am not an experienced Motif programmer, but I would expect XtVaGetValues shouldn't alter the widget that is passed as the first parameter. That, however is what appears to be happening when viewing the variables via GDB.
The problem is, XmNuserData stores a pointer. So the length of this value depends on the word size of the operating system. Retrieving XmNuserData into an 32bit-int-type on a 64bit OS messes up the stack, causing strange things to happen.
Thans Ayron, you got me looking in the right places. Here is what I found out
Changing line 1850 of abmf/lib_fun_strings.c to read:
" long button = -1;\n"
Caused dtb_utils.c to be autogenerated with the fix that solved the problem. This made the change in dtcodegen that resulted in the function being generated in dtappbuilder with the correct size pointer.
Also changing 697 of abobj_menu.c to:
usigned long item_flag;
Fixed that segfault.
I'll comb through the files of dtappbuilder and dtcodegen to find the places that a pointer is assuming a 32bit value and change it be width agnostic. Just changing those lines as I have listed will work for 64 bit procs but not 32. I'll submit a patch when I think I've got it.
Thanks again for the help.
Actually changing those to long should work fine - since long is the same size as a pointer on LP_64 architectures... This would also work on 32b systems. CDE is/was rife with these kinds of problems. Many have been fixed, but AB has been a neglected child and defintely needs some love.
So the position of dtappbuilder maintainer is open?
I will happily support anyone who has the interest and time to "adopt" and maintain a CDE subsystem.
I've fixed a lot of these XmNuserData-problems. Now it doesn't crash, but still doesn't work as it should. This needs some more work.
The Application Builder is a great peace of CDE as it accelerates GUI-development a lot.
Ayron, you seem to have done a lot of work already. Is it OK if I take responsibilty for the Application Builder? I wouldn't, of course, try and dictate what anyone works on. I guess I'll just focus on it and watch out for bugs and requests.
There are already many fixes in dtappbuilder in the current dev branch (cde-fixups-1). I don't know if any of them solved your problems but if not, please feel free to submit patches to the mailing list or submit a pull request.