[Spamprobe-users] SpamProbe and 256 MB PBL database limit

[Thomas S. <th...@sc...>]

Hi,

as noted quite a while ago, PBL seems to have problems
with databases exceeding 256 MB. The 256 MB come from
an internal limit for the number of tree blocks per file.
If more blocks are needed, data is put into one or more
additional files, which seems to cause problems (broken
SpamProbe databases, for example).

Most users are fine with this limit, because they never
hit it, but at least on my system, one database exceeds this
limit every now and then, which causes crashes/hangs with
SpamProbe and qmail.

However, there is an easy fix: the maximum number of blocks
per file can be configured at PBL compile time. The default
limit is 0xffff, resulting in 65535*4096 bytes = ~256 MB per
file when the standard 4K blocksize is used.

To increase the maximum size to 1 GB per file, edit
pbl_1_03/makefile and modify the CFLAGS line to include
a modified value for the maximum number of blocks:

CFLAGS=  -Wall -O3 ${IPATH} -DPBLFILEBLOCKS=0x3ffff

Recompile PBL by running "make clean && make", then
recompile SpamProbe using the modified PBL version.
There is no need to rebuild the database files (assuming
they haven't hit the limit). Files made using the original
PBL version can be used with the modified version, but of
course larger databases cannot be used with the old version.

You may require some additional tests to verify this
is working on your system. It seems to work flawlessly
on mine, but I haven't tested it extensively yet.

An alternative way would be to increase the blocksize and leave
the maximum number of blocks per file at the default value
(so use -DPBLDATASIZE=16384), but then obviously all databases
need to be rebuild.

BTW: Why does SpamProbe use PBL ISAM instead of PBL
Keyfile databases? What would be the difference between
the two regarding the database functionality used by SpamProbe?


Regards,
Thomas.