DITA Open Toolkit

There seems to be a fundamental issue with the way the OT has been
designed. The OT launches a single Java process that handles everything.

Unfortunately, a 32bit Windows machine is able to allocate a maximum of 2Gb
for any single process. When generating output using the OT the memory used
by the Java process that handles generation may easily grow beyond this
size. This is particularly true of the memory intensive output types such
as CHM or PDF, though all output types will suffer from this issue given a
large enough map/topics.

The average user generating output from a map referencing a few hundred
topics may never hit this limit, particularly if those topics are "small".
However, it is not uncommon for documentation (for complex software) to
contain many hundreds or even thousands of topics and if those topics are
on the "large" side (= a few full "pages" of content) the 2Gb is easily
reached. If those topics also reference images the memory usage may also
increase, reducing the number of topics needed to hit the limit.

It may be useful to see how quickly this can occur so I am attaching a set
of VBScript files that can be executed on a Windows box to generate a "test
suite" for testing the limits of the OT. Instructions:

1. Save attached zip file and unzip to an EMPTY (!) folder.
2. Run the two VBS files. One generates a single map file containing 2000
topic references. The second generates the 2000 topics that are referenced
by the map. See readme.txt for descriptions on how to modify the VBS to
generate different numbers of topic files and references in the map file.
3. Set ant_opts to use as much memory as your system can dish out:
-Xmx####m (possibly also -Xms####m).
4. Generate output using the map you generated in step 2.
5. Monitor Java process memory and CPU usage using Windows Task Manager or
another monitor.

Repeat steps 2 to 5 after modifying the VBS files so that they generate a
larger map and more topics, or simply larger topics or both, then repeat
steps 3 and 4 until you hit the limit.

To truly test this as described you would need a machine with (about) a
minimum of 3Gb of RAM -- 1Gb for Windows and other miscellaneous processes
and 2Gb that can be dedicated to generating output.

We may wish to blame this on the OS, however, many users may not be able to
move to a 64bit version of Windows (or other OS) and so I believe it is
worth looking into ways to resolve this. That also assumes that the
7152Gb-per-process limit of a 64bit Windows release is enough (perhaps when
the OT supports output to MPEG movies that limit will be hit ;-) But
seriously, making this type of assumption is not good programming -
remember Y2K.

It is not known whether this affects other operating systems. This bug as
submitted is specifically for Windows.

Links:
http://msdn.microsoft.com/msdnmag/issues/01/12/XPKernel/
http://forum.java.sun.com/thread.jspa?threadID=553939&messageID=2712071


Brainstorming:

Break up the generation steps (somehow) so that each one runs in a separate
Java process (perhaps using a .bat file or some other method).

In theory, this might actually improve performance -- stopping one process
after it has completed its OT generation step before launching the next one
could free up any memory allocated to the first Java process. Right now the
single process just continues to grow, only freeing up memory when it has
completed.

Making this change might help other users of the toolkit that are not
worried about hitting any limit by speeding things up (this assumes that
addressing and finding things in such a large block of memory wastes time,
testing may be necessary in that case).

It might also reduce the number of people that need to fiddle with ant_opts
for allocating Java memory. This seems to be an issue that many users of
the OT run into (search a few forums and you'll find many questions about
this). It would be nice if this "just works" with the defaults for more
people and this might help.

Allowing each step in the generation process to complete might allow for
easier debugging (?)

I assume it would not be good to run any of the generation steps in
parallel as then one process would be "robbing" overall system memory from
other Java processes (or would it?)