I also recommend using xvfb in your situation.

The problem is probably due to Java itself, not to Jmol : I also encountered this problem in a professional application I have written (when creating gif images from awt components on a JBoss server) and the solution was xvfb for creating a virtual display.