I second this. Specially for files generated with DOxygen-like programs can be very slow.
The lcl.chm (from the Lazarus project documentation) in the below archive takes 20secs to open on a 2GHz Core2 machine
http://www.stack.nl/~marcov/doc-chm.zip.
If you haven't already done so, please file a bug with a reproducable example at http://bugs.freepascal.org , otherwise it is nearly impossible to guess what is going on.