On Sun, 5 Sep 2004, Robert Ribnitz wrote:
> What is the maximum filesize htdig (3.1.6) can access?
Out of the box under Linux there is a 2 GB limit.
> Any idea how we can acto on such an error?
There are at least a couple issues related to the maximum file size.
The first is ht://Dig's direct handling of on-disk files, which is the one
in play here. The db.wordlist.work file is a text file directly handled
with system calls such as fopen(). Since the ht://Dig package isn't
compiled with largefile support by default, such access is affected by the
2 GB limit. A couple people have reported that adding '-D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64' has successfully provided a workaround for the
limit on db.wordlist.
The second issue is one of support at the Berkeley DB level. The version
provided with 3.1.6 doesn't provide support for large files under Linux.
There is an --enable-bigfile, but its support is limited to AIX, HP/UX,
and Solaris. Some time ago attempts were made to force the option for a
Linux build, but it was reported that the attempts were unsuccessful. A
more recent post implied that the limit at the BDB level is 4 GB.
So by building ht://Dig with largefile support you can most likely bypass
problem with limits on the size of db.wordlist. However there will still
be a limit on the size of the actual databases (presumably 4 GB).