Great to see that you are using JaCoP.
I looked quickly at your code. I do not see obvious errors.
I recommend the following course of action.
1. Pick up really small problem with two shapes to make sure that you are not using Geost constraint in incorrect manner. If you can get a solution to your very simple artificial problem that had a solution in the first place...
Thanks for bug report in this example. This example was not working in case of black walls.
I have improved this example, but for the moment, I have code only for version 4.0 of JaCoP that is available at github.
The problem contains an example with one black wall.
Before I try to figure out the problem, can I ask you for one thing?
Can you implement your And, Or, XneqC constraints (e.g. Or138) in your first printout as ExtensionalSupport or ExtensionalConflict (whatever has smaller number of tuples or is easier to use for you).
This approach of using Extensional* constraints may help you immensely in solving your problem efficiently...
You need to create an auxiliary variable aSquare that will be constrained to hold square of a.
store.impose(new XmulYeqZ(a, a, aSquare));
The same for bSquare and cSquare
later on you use Sum constraints with *Square variables.