I think it would be useful to let hsqldb.nio_max_size accept a value of zero to mean that there's no size limit for NIO usage.
This should be safe to use regarding memory exhaustions since the docs say that in this case non-NIO methods are used.
Even if NIO methods might not be extremely reliable in some platforms I think it would be enough to add a warning in the docs.
Reports from widespread deployments have indicated in the past that nio usage for very large database files may be causing issues. We therefore do not want to encourage it beyond commonly available memory sizes.