[htdig] input buffer overflow in conf_lex.cxx with large exclude list

[Jonathan S. <sch...@me...>]

Hello,

  I have a rather large exclude list I'm trying to have htsearch use so I
can exclude symlinks in my search results.

  I have been successful in implementing this scenario for a part of this
list at dig time via exclude_urls in the config file, however I do need
the database to contain some of these other symlinked file as they will
be included in search results in various areas of my site and duplicates
there will be controlled via the restrict variable. Works quite well
actually.

  The issue is when searching the main database with no restricts (ie: the
entire site) I get duplicates, of course.......

  I added the "exclude" variable to the config file since it's such a
large list and when I run htsearch I get the following error in my
error_log:
  "input buffer overflow, can't enlarge buffer because scanner uses REJECT"

  I checked through the htdig source and found this message in
htcommon/conf_lex.cxx

  Upon looking further I found a buffer size variable which I tried to
increase and although it did not cause further issues, it did not
resolve the one I'm facing......

  I'm running version 3.2.0b5 (on Red Hat 7.2 kernel 2.4.20-30.7) with
several patches applied including exclude_perform.1 which I recently
added to handle the large list for dig time....

  Does anyone have any ideas on how I can fix this issue? I need to be
able to exclude a large list of symlinks at search time.....any help is
greatly appreciated.

Cheers,
Jonathan Schlackl