Thread: [cvs] SF.net SVN: bogofilter: [6666] trunk/bogofilter
Fast Bayesian spam filter along lines suggested by Paul Graham
Brought to you by:
m-a
From: <m-...@us...> - 2007-07-23 11:11:59
|
Revision: 6666 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6666&view=rev Author: m-a Date: 2007-07-23 04:12:00 -0700 (Mon, 23 Jul 2007) Log Message: ----------- Begin cleanup of CVS -> SVN migration. Modified Paths: -------------- trunk/bogofilter/INSTALL trunk/bogofilter/Makefile.am trunk/bogofilter/NEWS trunk/bogofilter/README trunk/bogofilter/TODO trunk/bogofilter/configure.ac trunk/bogofilter/doc/bogofilter-faq.html Added Paths: ----------- trunk/bogofilter/README.svn Removed Paths: ------------- trunk/bogofilter/README.cvs Modified: trunk/bogofilter/INSTALL =================================================================== --- trunk/bogofilter/INSTALL 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/INSTALL 2007-07-23 11:12:00 UTC (rev 6666) @@ -28,8 +28,8 @@ REMEMBER to read the doc/programmer/README.* file for your operating systems in case of trouble. -NOTE: If you are about to build from code checked out via CVS, then -please read the README.cvs file first before you proceed with this file! +NOTE: If you are about to build from code checked out via SVN, then +please read the README.svn file first before you proceed with this file! make command ============ Modified: trunk/bogofilter/Makefile.am =================================================================== --- trunk/bogofilter/Makefile.am 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/Makefile.am 2007-07-23 11:12:00 UTC (rev 6666) @@ -22,7 +22,7 @@ Doxyfile gpl-3.0.txt gnugetopt \ GETTING.STARTED \ NEWS.0 \ - README.cvs \ + README.svn \ RELEASE.NOTES \ install-staticdblibs.sh Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/NEWS 2007-07-23 11:12:00 UTC (rev 6666) @@ -15,6 +15,12 @@ ------------------------------------------------------------------------------- + 2007-07-23 + + * The upstream repository was migrated to SVN. + In order to check the code out, use this command (one line): + svn co https://bogofilter.svn.sourceforge.net/svnroot/bogofilter/trunk/bogofilter/ bogofilter + 2007-07-22 * The install-staticdblibs.sh script was relicensed under GNU GPL v3, Modified: trunk/bogofilter/README =================================================================== --- trunk/bogofilter/README 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/README 2007-07-23 11:12:00 UTC (rev 6666) @@ -10,7 +10,7 @@ 3. Installing Bogofilter 4. How to contribute 5. How to request features -6. Building & Installing Bogofilter from Source Tarball or CVS +6. Building & Installing Bogofilter from Source Tarball or SVN a. Downloading b. Building & Installing c. Configuring @@ -119,7 +119,7 @@ to the list. -6. Building & Installing Bogofilter from Source Tarball or CVS +6. Building & Installing Bogofilter from Source Tarball or SVN -------------------------------------------------------------- a. Downloading @@ -130,17 +130,12 @@ http://sourceforge.net/projects/bogofilter/ The latest stable version can be downloaded. The development source - is in a CVS repository on SourceForge. To download the latest cvs + is in a Subversion repository on SourceForge. To download the latest SVN source, cd to the directory to which you wish to download and type - the following commands: + the following command: - cvs -d:pserver:ano...@cv...:/cvsroot/bogofilter login + svn co https://bogofilter.svn.sourceforge.net/svnroot/bogofilter/trunk/bogofilter/ bogofilter - When prompted for a password press the RETURN key. - After anonymously logging in: - - cvs -z3 -d:pserver:ano...@cv...:/cvsroot/bogofilter co bogofilter - b. Building & Installing ------------------------ @@ -152,13 +147,15 @@ make all check make install [as root] - To compile and install the standard configuration from CVS, use + To compile and install the standard configuration from SVN, use the following commands: cd bogofilter [change to project directory] ./autogen.sh [add configure options as required] make install [as root] + Be sure to read README.svn for additional preparations required. + You will need a full set of development tools installed to be able to run autogen.sh, including recent automake (1.8) and autoconf (2.59). configure does not have these requirements. Deleted: trunk/bogofilter/README.cvs =================================================================== --- trunk/bogofilter/README.cvs 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/README.cvs 2007-07-23 11:12:00 UTC (rev 6666) @@ -1,26 +0,0 @@ -README.cvs -- How to build bogofilter from CVS -$Id$ - -(C) 2002 by Matthias Andree. Freely distributable according to the terms -of the GNU Free Documentation License 1.0. No front- or back-matter -parts, no invariant parts. ------------------------------------------------------------------------- - -After you have checked out bogofilter from CVS, some files are missing, -for example, configure, ylwrap and others. - -These files can be created automatically with recent autoconf and -automake versions. You will need autoconf 2.53 and automake 1.6 or -newer. - -To recreate these files, run: autoreconf -i -s -f -You can optionally add -v to see what autoreconf is doing. - -Then proceed as usual: ./configure && make && make check -and so on. - -For less verbose output: ./configure --quiet && make -s && make -s check - -Have fun! ------------------------------------------------------------------------- -end of README.cvs Copied: trunk/bogofilter/README.svn (from rev 6665, trunk/bogofilter/README.cvs) =================================================================== --- trunk/bogofilter/README.svn (rev 0) +++ trunk/bogofilter/README.svn 2007-07-23 11:12:00 UTC (rev 6666) @@ -0,0 +1,26 @@ +README.svn -- How to build bogofilter from Subversion (SVN) +$Id$ + +(C) 2002,2007 by Matthias Andree. Freely distributable according to the +terms of the GNU Free Documentation License 1.0. No front- or back-matter +parts, no invariant parts. +------------------------------------------------------------------------- + +After you have checked out bogofilter from SVN, some files are missing, +for example, configure, ylwrap and others. + +These files can be created automatically with recent autoconf and +automake versions. You will need autoconf 2.53 and automake 1.6 or +newer. + +To recreate these files, run: autoreconf -i -s -f +You can optionally add -v to see what autoreconf is doing. + +Then proceed as usual: ./configure && make && make check +and so on. + +For less verbose output: ./configure --quiet && make -s && make -s check + +Have fun! +------------------------------------------------------------------------- +end of README.svn Modified: trunk/bogofilter/TODO =================================================================== --- trunk/bogofilter/TODO 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/TODO 2007-07-23 11:12:00 UTC (rev 6666) @@ -2,6 +2,10 @@ bogofilter TODO list +**** CVS -> SVN migration: + Makefile.am cvs-check target + version.sh adjust to derive proper date + **** If insufficient data is present and the default "undecided" bogosity is added in -p mode, add also a comment stating that bogofilter needs more training first Modified: trunk/bogofilter/configure.ac =================================================================== --- trunk/bogofilter/configure.ac 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/configure.ac 2007-07-23 11:12:00 UTC (rev 6666) @@ -17,7 +17,7 @@ dnl part of the bogofilter source). dnl ******************************************************** dnl -AC_INIT([bogofilter],[1.1.5.cvs]) +AC_INIT([bogofilter],[1.1.5.svn]) dnl AC_PREREQ(2.59) AC_CONFIG_SRCDIR([src/bogofilter.c]) Modified: trunk/bogofilter/doc/bogofilter-faq.html =================================================================== --- trunk/bogofilter/doc/bogofilter-faq.html 2007-07-22 15:09:03 UTC (rev 6665) +++ trunk/bogofilter/doc/bogofilter-faq.html 2007-07-23 11:12:00 UTC (rev 6666) @@ -318,8 +318,9 @@ <td>bog...@li...</td> <td><a href="https://lists.sourceforge.net/lists/listinfo/bogofilter-cvs">[subscribe]</a> <a href="http://sourceforge.net/mailarchive/forum.php?forum=bogofilter-cvs">[archive]</a></td> - <td>Mailing list for announcing code changes to the CVS - archive.</td> + <td>Mailing list for announcing code changes to the SVN + archive. (The CVS name is a leftover from before the migration + for our users' convenience.)</td> </tr> </table> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2007-07-28 13:01:57
|
Revision: 6676 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6676&view=rev Author: m-a Date: 2007-07-28 06:02:00 -0700 (Sat, 28 Jul 2007) Log Message: ----------- Provide svn-check target, replacing cvs-check target. Modified Paths: -------------- trunk/bogofilter/Makefile.am trunk/bogofilter/TODO Modified: trunk/bogofilter/Makefile.am =================================================================== --- trunk/bogofilter/Makefile.am 2007-07-28 12:51:57 UTC (rev 6675) +++ trunk/bogofilter/Makefile.am 2007-07-28 13:02:00 UTC (rev 6676) @@ -26,7 +26,7 @@ RELEASE.NOTES \ install-staticdblibs.sh -.PHONY: check rpm cvs-check +.PHONY: check rpm svn-check # # RPM building - there are some cheats here @@ -104,10 +104,10 @@ - CPPFLAGS="-D__NO_CTYPE -I$(SQPFX)" \ $(RPMBUILD) $(DEF_SQLITE) $(SIGN) -tb $(distdir).tar.gz -cvs-check: +svn-check: rm -rf $(PACKAGE)-$(VERSION)-export - cvs -d "`cat $(srcdir)/CVS/Root`" export -rHEAD -d $(PACKAGE)-$(VERSION)-export $(PACKAGE) - cd $(PACKAGE)-$(VERSION)-export && autoreconf -i -f -s \ + svn export -rBASE $(srcdir) $(PACKAGE)-$(VERSION)-export + cd $(PACKAGE)-$(VERSION)-export && autoreconf -i -f -s -v \ && ./configure && make distcheck # this target expects a .rsyncs file with lines of this format: Modified: trunk/bogofilter/TODO =================================================================== --- trunk/bogofilter/TODO 2007-07-28 12:51:57 UTC (rev 6675) +++ trunk/bogofilter/TODO 2007-07-28 13:02:00 UTC (rev 6676) @@ -3,7 +3,6 @@ bogofilter TODO list **** CVS -> SVN migration: - Makefile.am cvs-check target version.sh adjust to derive proper date **** If insufficient data is present and the default "undecided" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2007-07-28 13:11:38
|
Revision: 6677 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6677&view=rev Author: m-a Date: 2007-07-28 06:11:41 -0700 (Sat, 28 Jul 2007) Log Message: ----------- Update version.sh for Subversion use. Modified Paths: -------------- trunk/bogofilter/TODO trunk/bogofilter/src/version.sh Modified: trunk/bogofilter/TODO =================================================================== --- trunk/bogofilter/TODO 2007-07-28 13:02:00 UTC (rev 6676) +++ trunk/bogofilter/TODO 2007-07-28 13:11:41 UTC (rev 6677) @@ -2,9 +2,6 @@ bogofilter TODO list -**** CVS -> SVN migration: - version.sh adjust to derive proper date - **** If insufficient data is present and the default "undecided" bogosity is added in -p mode, add also a comment stating that bogofilter needs more training first Modified: trunk/bogofilter/src/version.sh =================================================================== --- trunk/bogofilter/src/version.sh 2007-07-28 13:02:00 UTC (rev 6676) +++ trunk/bogofilter/src/version.sh 2007-07-28 13:11:41 UTC (rev 6677) @@ -20,7 +20,7 @@ VERSION=$(grep define.VERSION config.h | awk '{print $3}' | tr -d '"') -SUFFIX=$(echo $VERSION | egrep "\.cvs$") +SUFFIX=$(echo $VERSION | egrep "\.svn$") srcdir=$1 shift @@ -28,23 +28,17 @@ set -e if [ ! -z "$SUFFIX" ]; then - FILES=$(find $srcdir -name CVS -type d -print | while read a ; do find "$a" -name Entries -type f -print ; done) set +e - DATE=CVStime_`perl -MHTTP::Date -e ' - $max = 0; - while (<>) { - split m(/); - $a=str2time($_[3], "GMT"); - $max=$a if $a and $a > $max; - } - $date=HTTP::Date::time2isoz($max); + DATE=$(svn info --xml "$srcdir" | grep date | sed 's,</\?date>,,g' | perl -MHTTP::Date -e ' + $date = str2time(<>); + $date=HTTP::Date::time2isoz($date); $date=~tr/ :Z-/_/d; print $date, "\n"; - ' </dev/null $FILES` || DATE= - if [ "x$FILES" = "x" ] || [ "x$DATE" = "x" ] ; then + ') || DATE= + if [ "x$DATE" = "x" ] ; then DATE=$(env TZ=GMT date "+build_date_%Y%m%d_%Hh") fi -# VERSION="$VERSION.$DATE" + VERSION="$VERSION.$DATE" fi echo "#include \"globals.h\"" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2007-07-28 13:20:40
|
Revision: 6678 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6678&view=rev Author: m-a Date: 2007-07-28 06:20:41 -0700 (Sat, 28 Jul 2007) Log Message: ----------- Update log file format upgrade info (sec. 2.6) for BerkeleyDB 4.5 and 4.6. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/README.db Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-07-28 13:11:41 UTC (rev 6677) +++ trunk/bogofilter/NEWS 2007-07-28 13:20:41 UTC (rev 6678) @@ -15,6 +15,10 @@ ------------------------------------------------------------------------------- + 2007-07-28 + + * doc/README.db was updated to BerkeleyDB 4.6 + 2007-07-23 * The upstream repository was migrated to SVN. Modified: trunk/bogofilter/doc/README.db =================================================================== --- trunk/bogofilter/doc/README.db 2007-07-28 13:11:41 UTC (rev 6677) +++ trunk/bogofilter/doc/README.db 2007-07-28 13:20:41 UTC (rev 6678) @@ -172,9 +172,8 @@ least one of these conditions is true ("-->" means "to") - you upgraded Berkeley DB from a 3.X version --> a 4.Y version -- you upgraded Berkeley DB from 4.0 or 4.1 --> 4.2, 4.3 or 4.4 -- you upgraded Berkeley DB from 4.2 --> 4.3 or 4.4 -- you upgraded Berkeley DB from 4.3 --> 4.4 +- you upgraded Berkeley DB from a 4.X version --> a newer 4.Y version + that means Y > X; EXCEPT if you upgraded from 4.0 to 4.1 Non-transactional databases do not need log file format upgrades as they do not use log files. This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2007-08-14 14:06:50
|
Revision: 6680 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6680&view=rev Author: m-a Date: 2007-08-14 07:06:54 -0700 (Tue, 14 Aug 2007) Log Message: ----------- Further BDB 4.5/4.6 documentation updates. Added section 3.5 on recovering from Logging region out of memory. Added set_lg_regionmax to section 4.2. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/README.db Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-07-28 13:21:58 UTC (rev 6679) +++ trunk/bogofilter/NEWS 2007-08-14 14:06:54 UTC (rev 6680) @@ -15,9 +15,12 @@ ------------------------------------------------------------------------------- - 2007-07-28 + 2007-08-14 * doc/README.db was updated to BerkeleyDB 4.6 + * doc/README.db: section 3.5 was added, with information on how to + resolve "Logging region out of memory; you may need to increase its + size", section 4.2 now documents set_lg_regionmax. 2007-07-23 Modified: trunk/bogofilter/doc/README.db =================================================================== --- trunk/bogofilter/doc/README.db 2007-07-28 13:21:58 UTC (rev 6679) +++ trunk/bogofilter/doc/README.db 2007-08-14 14:06:54 UTC (rev 6680) @@ -378,6 +378,14 @@ A bf_copy script is provided for your convenience. +3.5 OTHER TROUBLESHOOTING + +3.5.1 "Logging region out of memory; you may need to increase its size" + +This happens only for transactional databases. Use the DB_CONFIG file to +set the log region size (set_lg_regionmax) higher and after that, run +database recovery - see section 4.2 for details. + 4. Other Information of Interest --------------------------------------- 4.1 GENERAL INFORMATION @@ -431,13 +439,13 @@ 'bogoutil --db-recover' to become effective. All options are ignored if the database in the same directory as DB_CONFIG is non-transactional. -Here is a list of interesting settings, put one at a line, omitting the +Here is a list of interesting settings. Use one per line, and omit the leading spaces and hyphen: SIZING OPTIONS: - set_cachesize G B C - (valid in Berkeley DB 3.1 - 4.4, requires recovery to change) + (valid in Berkeley DB 3.1 - 4.6, requires recovery to change) sets the cache size to G gigabytes plus B bytes which are spread out in C equally sized caches (all figures are natural numbers). You @@ -451,40 +459,46 @@ will create six caches sized 12500000 bytes (12.5 MB, +25% applied) - set_lg_max 250000 - (valid in Berkeley DB 3.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.1 - 4.6, takes effect immediately) this option configures the maximum log file size, in bytes, before Berkeley DB starts a new log file. The default is 1 MB. + - set_lg_regionmax 262144 + (valid in Berkeley DB 3.3 - 4.6, requires recovery to take effect) + + this option configures the log region size, in bytes, and may + sometimes need to be increased. The default is around 60 kB. + SAFE OPTIONS: - set_flags DB_DIRECT_DB - (valid in Berkeley DB 4.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 4.1 - 4.6, takes effect immediately) this option turns off system buffering of *database* files, to avoid double caching of data. NOT SUPPORTED ON ALL PLATFORMS! - set_flags DB_DIRECT_LOG - (valid in Berkeley DB 4.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 4.1 - 4.6, takes effect immediately) this option turns off system buffering of *log* files, to avoid double caching of data. NOT SUPPORTED ON ALL PLATFORMS! - set_flags DB_DSYNC_LOG - (valid in Berkeley DB 4.3 - 4.4, takes effect immediately) + (valid in Berkeley DB 4.3 - 4.6, takes effect immediately) this option can increase performance on some systems (and decrease on other systems), by using the O_DSYNC POSIX flag rather than a separate function to flush the logs. - set_flags DB_NOMMAP - (valid in Berkeley DB 3.2 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.2 - 4.6, takes effect immediately) this option can reduce memory consumption at run time, particularly with large databases, at some cost of performance - set_flags DB_REGION_INIT - (valid in Berkeley DB 3.2 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.2 - 4.6, takes effect immediately) this option causes all shared memory regions to be "page faulted" into core memory at application start and written at data base @@ -495,11 +509,11 @@ - set_verbose DB_VERB_CHKPOINT (valid in Berkeley DB 3.1 - 4.2, takes effect immediately) - set_verbose DB_VERB_DEADLOCK - (valid in Berkeley DB 3.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.1 - 4.6, takes effect immediately) - set_verbose DB_VERB_RECOVERY - (valid in Berkeley DB 3.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.1 - 4.6, takes effect immediately) - set_verbose DB_VERB_WAITSFOR - (valid in Berkeley DB 3.1 - 4.4, takes effect immediately) + (valid in Berkeley DB 3.1 - 4.6, takes effect immediately) these verbose flags cause extended output for long-lasting operations, ...CHKPOINT prints location information when searching This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2007-11-23 05:27:44
|
Revision: 6684 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6684&view=rev Author: relson Date: 2007-11-22 21:27:48 -0800 (Thu, 22 Nov 2007) Log Message: ----------- Add support for tokyocabinet database. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/configure.ac trunk/bogofilter/src/Makefile.am trunk/bogofilter/src/tests/t.frame Added Paths: ----------- trunk/bogofilter/src/datastore_tc.c Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-11-23 02:50:37 UTC (rev 6683) +++ trunk/bogofilter/NEWS 2007-11-23 05:27:48 UTC (rev 6684) @@ -15,6 +15,11 @@ ------------------------------------------------------------------------------- + 2007-11-22 + + * Support for tokyocabinet datastore added. + (thanks to Pierre Habouzit) + 2007-08-14 * doc/README.db was updated to BerkeleyDB 4.6 Modified: trunk/bogofilter/configure.ac =================================================================== --- trunk/bogofilter/configure.ac 2007-11-23 02:50:37 UTC (rev 6683) +++ trunk/bogofilter/configure.ac 2007-11-23 05:27:48 UTC (rev 6684) @@ -459,7 +459,7 @@ WITH_DB_ENGINE=db AC_ARG_WITH(database, AS_HELP_STRING([--with-database=ENGINE], - [choose database engine {db|qdbm|sqlite3} [[db]]]), + [choose database engine {db|qdbm|sqlite3|tokyocabinet} [[db]]]), [ WITH_DB_ENGINE=$withval ] ) @@ -473,6 +473,27 @@ LIBDB="$LIBSQLITE3" WITH_DB_ENGINE="sqlite3" ;; + xtokyocabinet) + AC_DEFINE(ENABLE_TOKYOCABINET_DATASTORE,1, [Enable tokyocabinet datastore]) + DB_TYPE=tokyocabinet + DB_EXT=.tc + AC_LIB_LINKFLAGS([tokyocabinet]) + LIBDB="$LIBTOKYOCABINET" + saveLIBS="$LIBS" + LIBS="$LIBS $LIBDB" + AC_LINK_IFELSE([AC_LANG_PROGRAM([ +#include <tcutil.h> +#include <tchdb.h> +#include <tcbdb.h> +#include <stdlib.h> +#include <time.h> +#include <stdbool.h> +#include <stdint.h> + ], [ + TCBDB *d = tcbdbnew(); + ])],,AC_MSG_ERROR(Cannot link to tokyocabinet library.)) + LIBS="$saveLIBS" + ;; xqdbm) AC_DEFINE(ENABLE_QDBM_DATASTORE,1, [Enable qdbm datastore]) DB_TYPE=qdbm @@ -489,7 +510,7 @@ ])],,AC_MSG_ERROR(Cannot link to qdbm library.)) LIBS="$saveLIBS" ;; - xdb) + xdb*) dnl check if we need LIBDB at all (it is harmful on BSD since dnl they rename the library including soname) dnl and try to guess if we need -lpthread or -lthread @@ -624,7 +645,7 @@ LIBS="$saveLIBS" ;; *) - AC_MSG_ERROR([Invalid --with-database argument. Supported engines are db, qdbm, sqlite3.]) + AC_MSG_ERROR([Invalid --with-database argument. Supported engines are db, qdbm, sqlite3, tokyocabinet.]) ;; esac @@ -649,6 +670,7 @@ AM_CONDITIONAL(ENABLE_QDBM_DATASTORE, test "x$WITH_DB_ENGINE" = "xqdbm") AM_CONDITIONAL(ENABLE_SQLITE_DATASTORE, test "x$WITH_DB_ENGINE" = "xsqlite3") +AM_CONDITIONAL(ENABLE_TOKYOCABINET_DATASTORE, test "x$WITH_DB_ENGINE" = "xtokyocabinet") dnl Use TRIO to replace missing snprintf/vsnprintf. needtrio=0 Modified: trunk/bogofilter/src/Makefile.am =================================================================== --- trunk/bogofilter/src/Makefile.am 2007-11-23 02:50:37 UTC (rev 6683) +++ trunk/bogofilter/src/Makefile.am 2007-11-23 05:27:48 UTC (rev 6684) @@ -184,6 +184,11 @@ datastore_SOURCE = datastore_sqlite.c datastore_opthelp_dummies.c \ datastore_dummies.c else +if ENABLE_TOKYOCABINET_DATASTORE +datastore_SOURCE = datastore_tc.c \ + datastore_txn_dummies.c datastore_opthelp_dummies.c \ + datastore_dummies.c +else if ENABLE_TRANSACTIONS datastore_SOURCE = datastore_db.c datastore_db_trans.c else @@ -196,6 +201,7 @@ endif endif endif +endif datastore_OBJECT = $(datastore_SOURCE:.c=.o) Added: trunk/bogofilter/src/datastore_tc.c =================================================================== --- trunk/bogofilter/src/datastore_tc.c (rev 0) +++ trunk/bogofilter/src/datastore_tc.c 2007-11-23 05:27:48 UTC (rev 6684) @@ -0,0 +1,333 @@ +/* $Id: datastore_qdbm.c,v 1.49 2005/04/04 11:16:21 relson Exp $ */ + +/***************************************************************************** + +NAME: +datastore_tc.c -- implements the datastore, using tokyocabinet. + +AUTHORS: +Gyepi Sam <gy...@pr...> 2003 +Matthias Andree <mat...@gm...> 2003 +Stefan Bellon <sb...@sb...> 2003-2004 +Pierre Habouzit <mad...@de...> 2007 + +******************************************************************************/ + +#include "common.h" + +#include <tcutil.h> +#include <tchdb.h> +#include <tcbdb.h> +#include <stdlib.h> +#include <time.h> +#include <stdbool.h> +#include <stdint.h> + +#include "datastore.h" +#include "datastore_db.h" +#include "error.h" +#include "paths.h" +#include "xmalloc.h" +#include "xstrdup.h" + +#define UNUSED(x) ((void)&x) + +typedef struct { + char *path; + char *name; + bool locked; + bool created; + TCBDB *dbp; +} dbh_t; + +/* Function definitions */ + +const char *db_version_str(void) +{ + static char v[80]; + if (!v[0]) + snprintf(v, sizeof(v), "TokyoCabinet (version %s, B+tree API)", tcversion); + return v; +} + + +static dbh_t *dbh_init(bfpath *bfp) +{ + dbh_t *handle; + + handle = xmalloc(sizeof(dbh_t)); + memset(handle, 0, sizeof(dbh_t)); /* valgrind */ + + handle->name = xstrdup(bfp->filepath); + + handle->locked = false; + handle->created = false; + + return handle; +} + + +static void dbh_free(/*@only@*/ dbh_t *handle) +{ + if (handle != NULL) { + xfree(handle->name); + xfree(handle->path); + xfree(handle); + } + return; +} + + +/* Returns is_swapped flag */ +bool db_is_swapped(void *vhandle) +{ + UNUSED(vhandle); + + return false; +} + + +/* Returns created flag */ +bool db_created(void *vhandle) +{ + dbh_t *handle = vhandle; + return handle->created; +} + + +/* + Initialize database. + Returns: pointer to database handle on success, NULL otherwise. +*/ +void *db_open(void * dummy, bfpath *bfp, dbmode_t open_mode) +{ + dbh_t *handle; + + bool res; + int open_flags; + TCBDB *dbp; + + UNUSED(dummy); + + if (open_mode & DS_WRITE) + open_flags = BDBOWRITER; + else + open_flags = BDBOREADER; + + handle = dbh_init(bfp); + + if (handle == NULL) return NULL; + + dbp = handle->dbp = tcbdbnew(); + res = tcbdbopen(dbp, handle->name, open_flags); + if (!res && (open_mode & DS_WRITE)) { + res = tcbdbopen(dbp, handle->name, open_flags | BDBOCREAT); + handle->created |= res; + } + + if (!res) + goto open_err; + + if (DEBUG_DATABASE(1)) + fprintf(dbgout, "(tc) tcbdbopen( %s, %d )\n", handle->name, open_mode); + + return handle; + + open_err: + print_error(__FILE__, __LINE__, "(tc) tcbdbopen(%s, %d), err: %d, %s", + handle->name, open_flags, + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); + dbh_free(handle); + + return NULL; +} + + +int db_delete(void *vhandle, const dbv_t *token) +{ + int ret; + dbh_t *handle = vhandle; + TCBDB *dbp; + + dbp = handle->dbp; + ret = tcbdbout(dbp, token->data, token->leng); + + if (ret == 0) { + print_error(__FILE__, __LINE__, "(tc) tcbdbout('%.*s'), err: %d, %s", + CLAMP_INT_MAX(token->leng), + (char *)token->data, + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); + exit(EX_ERROR); + } + ret = ret ^ 1; /* ok is 1 in qdbm and 0 in bogofilter */ + + return ret; /* 0 if ok */ +} + + +int db_get_dbvalue(void *vhandle, const dbv_t *token, /*@out@*/ dbv_t *val) +{ + char *data; + int dsiz; + + dbh_t *handle = vhandle; + TCBDB *dbp = handle->dbp; + + data = tcbdbget(dbp, token->data, token->leng, &dsiz); + + if (data == NULL) + return DS_NOTFOUND; + + if (val->leng < (unsigned)dsiz) { + print_error(__FILE__, __LINE__, + "(tc) db_get_dbvalue( '%.*s' ), size error %lu: %lu", + CLAMP_INT_MAX(token->leng), + (char *)token->data, (unsigned long)val->leng, + (unsigned long)dsiz); + exit(EX_ERROR); + } + + val->leng = dsiz; /* read count */ + memcpy(val->data, data, dsiz); + + free(data); /* not xfree() as allocated by tcbdbget() */ + + return 0; +} + + +/* + Re-organize database according to some heuristics +*/ +static inline void db_optimize(TCBDB *dbp, char *name) +{ + UNUSED(dbp); + UNUSED(name); + + /* The Villa API doesn't need optimizing like the formerly used + Depot API because Villa uses B+ trees and Depot uses hash tables. + Database size may grow larger and could get compacted with + tcbdboptimize() however as the database size with Villa is smaller + anyway, I don't think it is worth it. */ +} + + +int db_set_dbvalue(void *vhandle, const dbv_t *token, const dbv_t *val) +{ + int ret; + dbh_t *handle = vhandle; + TCBDB *dbp = handle->dbp; + + ret = tcbdbput(dbp, token->data, token->leng, val->data, val->leng); + + if (ret == 0) { + print_error(__FILE__, __LINE__, + "(tc) db_set_dbvalue( '%.*s' ) err: %d, %s", + CLAMP_INT_MAX(token->leng), (char *)token->data, + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); + exit(EX_ERROR); + } + + db_optimize(dbp, handle->name); + + return 0; +} + + +/* + Close files and clean up. +*/ +void db_close(void *vhandle) +{ + dbh_t *handle = vhandle; + TCBDB *dbp; + + if (handle == NULL) return; + + if (DEBUG_DATABASE(1)) + fprintf(dbgout, "(tc) tcbdbclose(%s)\n", handle->name); + + dbp = handle->dbp; + + db_optimize(dbp, handle->name); + + if (!tcbdbclose(dbp)) + print_error(__FILE__, __LINE__, "(tc) tcbdbclose for %s err: %d, %s", + handle->name, + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); + + tcbdbdel(dbp); + handle->dbp = NULL; + + dbh_free(handle); +} + + +/* + Flush any data in memory to disk +*/ +void db_flush(void *vhandle) +{ + dbh_t *handle = vhandle; + TCBDB * dbp = handle->dbp; + + if (!tcbdbsync(dbp)) + print_error(__FILE__, __LINE__, "(tc) tcbdbsync err: %d, %s", + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); +} + + +ex_t db_foreach(void *vhandle, db_foreach_t hook, void *userdata) +{ + int ret = 0; + + dbh_t *handle = vhandle; + TCBDB *dbp = handle->dbp; + BDBCUR *cursor; + + dbv_t dbv_key, dbv_data; + int ksiz, dsiz; + char *key, *data; + + cursor = tcbdbcurnew(dbp); + ret = tcbdbcurfirst(cursor); + if (ret) { + while ((key = tcbdbcurkey(cursor, &ksiz))) { + data = tcbdbcurval(cursor, &dsiz); + if (data) { + /* switch to "dbv_t *" variables */ + dbv_key.leng = ksiz; + dbv_key.data = xmalloc(dbv_key.leng+1); + memcpy(dbv_key.data, key, ksiz); + ((char *)dbv_key.data)[dbv_key.leng] = '\0'; + + dbv_data.data = data; + dbv_data.leng = dsiz; /* read count */ + + /* call user function */ + ret = hook(&dbv_key, &dbv_data, userdata); + + xfree(dbv_key.data); + + if (ret != 0) + break; + free(data); /* not xfree() as allocated by dpget() */ + } + free(key); /* not xfree() as allocated by dpiternext() */ + + tcbdbcurnext(cursor); + } + } else { + print_error(__FILE__, __LINE__, "(tc) tcbdbcurfirst err: %d, %s", + tcbdbecode(dbp), tcbdberrmsg(tcbdbecode(dbp))); + exit(EX_ERROR); + } + + tcbdbcurdel(cursor); + return EX_OK; +} + +const char *db_str_err(int e) +{ + return tcbdberrmsg(e); +} Modified: trunk/bogofilter/src/tests/t.frame =================================================================== --- trunk/bogofilter/src/tests/t.frame 2007-11-23 02:50:37 UTC (rev 6683) +++ trunk/bogofilter/src/tests/t.frame 2007-11-23 05:27:48 UTC (rev 6684) @@ -44,6 +44,7 @@ *) DB_TXN=false ;; esac ;; *QDBM*) DB_TXN=false ;; + *Tokyo*) DB_TXN=false ;; *SQLite*) DB_TXN=true ;; *TrivialDB*) DB_TXN=false ;; *) echo >&2 "Unknown data base type in bogofilter -V: $DB_NAME" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2007-11-24 04:33:46
|
Revision: 6685 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6685&view=rev Author: relson Date: 2007-11-23 20:33:51 -0800 (Fri, 23 Nov 2007) Log Message: ----------- Update for 1.1.6 Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/configure.ac Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-11-23 05:27:48 UTC (rev 6684) +++ trunk/bogofilter/NEWS 2007-11-24 04:33:51 UTC (rev 6685) @@ -15,6 +15,8 @@ ------------------------------------------------------------------------------- +1.1.6 2007-11-24 (released) + 2007-11-22 * Support for tokyocabinet datastore added. Modified: trunk/bogofilter/configure.ac =================================================================== --- trunk/bogofilter/configure.ac 2007-11-23 05:27:48 UTC (rev 6684) +++ trunk/bogofilter/configure.ac 2007-11-24 04:33:51 UTC (rev 6685) @@ -17,7 +17,7 @@ dnl part of the bogofilter source). dnl ******************************************************** dnl -AC_INIT([bogofilter],[1.1.5.svn]) +AC_INIT([bogofilter],[1.1.6]) dnl AC_PREREQ(2.59) AC_CONFIG_SRCDIR([src/bogofilter.c]) This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2007-11-25 16:20:38
|
Revision: 6686 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6686&view=rev Author: relson Date: 2007-11-25 08:20:41 -0800 (Sun, 25 Nov 2007) Log Message: ----------- Transaction support added for TokyoCabinet datastore. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/src/Makefile.am trunk/bogofilter/src/datastore_tc.c trunk/bogofilter/src/tests/t.frame Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-11-24 04:33:51 UTC (rev 6685) +++ trunk/bogofilter/NEWS 2007-11-25 16:20:41 UTC (rev 6686) @@ -15,11 +15,14 @@ ------------------------------------------------------------------------------- -1.1.6 2007-11-24 (released) +1.1.6 2007-11-25 (released) + * Transaction support added for TokyoCabinet datastore. + (thanks to Pierre Habouzit) + 2007-11-22 - * Support for tokyocabinet datastore added. + * Support for TokyoCabinet datastore added. (thanks to Pierre Habouzit) 2007-08-14 Modified: trunk/bogofilter/src/Makefile.am =================================================================== --- trunk/bogofilter/src/Makefile.am 2007-11-24 04:33:51 UTC (rev 6685) +++ trunk/bogofilter/src/Makefile.am 2007-11-25 16:20:41 UTC (rev 6686) @@ -186,7 +186,7 @@ else if ENABLE_TOKYOCABINET_DATASTORE datastore_SOURCE = datastore_tc.c \ - datastore_txn_dummies.c datastore_opthelp_dummies.c \ + datastore_opthelp_dummies.c \ datastore_dummies.c else if ENABLE_TRANSACTIONS Modified: trunk/bogofilter/src/datastore_tc.c =================================================================== --- trunk/bogofilter/src/datastore_tc.c 2007-11-24 04:33:51 UTC (rev 6685) +++ trunk/bogofilter/src/datastore_tc.c 2007-11-25 16:20:41 UTC (rev 6686) @@ -40,6 +40,68 @@ TCBDB *dbp; } dbh_t; +/* transaction stuff */ + +static int tc_txn_begin(void *vhandle) { + dbh_t *dbh = vhandle; + if (!dbh->dbp->wmode || tcbdbtranbegin(dbh->dbp)) + return DST_OK; + print_error(__FILE__, __LINE__, "tc_txn_begin(%p), err: %d, %s", dbh->dbp, + tcbdbecode(dbh->dbp), tcbdberrmsg(tcbdbecode(dbh->dbp))); + return DST_FAILURE; +} + +static int tc_txn_abort(void *vhandle) { + dbh_t *dbh = vhandle; + if (!dbh->dbp->wmode || tcbdbtranabort(dbh->dbp)) + return DST_OK; + print_error(__FILE__, __LINE__, "tc_txn_abort(%p), err: %d, %s", dbh->dbp, + tcbdbecode(dbh->dbp), tcbdberrmsg(tcbdbecode(dbh->dbp))); + return DST_FAILURE; +} + +static int tc_txn_commit(void *vhandle) { + dbh_t *dbh = vhandle; + if (!dbh->dbp->wmode || tcbdbtrancommit(dbh->dbp)) + return DST_OK; + print_error(__FILE__, __LINE__, "tc_txn_commit(%p), err: %d, %s", + dbh->dbp, tcbdbecode(dbh->dbp), + tcbdberrmsg(tcbdbecode(dbh->dbp))); + return DST_FAILURE; +} + +static dsm_t dsm_tc = { + /* public -- used in datastore.c */ + &tc_txn_begin, + &tc_txn_abort, + &tc_txn_commit, + + /* private -- used in datastore_db_*.c */ + NULL, /* dsm_env_init */ + NULL, /* dsm_cleanup */ + NULL, /* dsm_cleanup_lite */ + NULL, /* dsm_get_env_dbe */ + NULL, /* dsm_database_name */ + NULL, /* dsm_recover_open */ + NULL, /* dsm_auto_commit_flags*/ + NULL, /* dsm_get_rmw_flag */ + NULL, /* dsm_lock */ + NULL, /* dsm_common_close */ + NULL, /* dsm_sync */ + NULL, /* dsm_log_flush */ + NULL, /* dsm_pagesize */ + NULL, /* dsm_purgelogs */ + NULL, /* dsm_checkpoint */ + NULL, /* dsm_recover */ + NULL, /* dsm_remove */ + NULL, /* dsm_verify */ + NULL, /* dsm_list_logfiles */ + NULL /* dsm_leafpages */ +}; + +dsm_t *dsm = &dsm_tc; + + /* Function definitions */ const char *db_version_str(void) Modified: trunk/bogofilter/src/tests/t.frame =================================================================== --- trunk/bogofilter/src/tests/t.frame 2007-11-24 04:33:51 UTC (rev 6685) +++ trunk/bogofilter/src/tests/t.frame 2007-11-25 16:20:41 UTC (rev 6686) @@ -44,7 +44,7 @@ *) DB_TXN=false ;; esac ;; *QDBM*) DB_TXN=false ;; - *Tokyo*) DB_TXN=false ;; + *Tokyo*) DB_TXN=true ;; *SQLite*) DB_TXN=true ;; *TrivialDB*) DB_TXN=false ;; *) echo >&2 "Unknown data base type in bogofilter -V: $DB_NAME" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2007-11-25 16:59:31
|
Revision: 6687 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6687&view=rev Author: m-a Date: 2007-11-25 08:59:33 -0800 (Sun, 25 Nov 2007) Log Message: ----------- Bump required sqlite version to 3.4.2. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/README.sqlite trunk/bogofilter/src/datastore_sqlite.c Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-11-25 16:20:41 UTC (rev 6686) +++ trunk/bogofilter/NEWS 2007-11-25 16:59:33 UTC (rev 6687) @@ -15,6 +15,11 @@ ------------------------------------------------------------------------------- + 2007-11-25 + + * Bump required sqlite version to 3.4.2, also fixing related compiler + warnings. Bogofilter will complain when used with older versions. + 1.1.6 2007-11-25 (released) * Transaction support added for TokyoCabinet datastore. Modified: trunk/bogofilter/doc/README.sqlite =================================================================== --- trunk/bogofilter/doc/README.sqlite 2007-11-25 16:20:41 UTC (rev 6686) +++ trunk/bogofilter/doc/README.sqlite 2007-11-25 16:59:33 UTC (rev 6687) @@ -20,13 +20,13 @@ 2.1 Compatibility - supported SQLite versions -At this time, only SQLite 3.4.0 and newer are supported. Older versions +At this time, only SQLite 3.4.2 and newer are supported. Older versions back to 3.0.8 may work, but you are advised to carefully review the sqlite3 ChangeLog, because there have been several important bug fixes since 3.0.8, including fixes for bugs that can corrupt the database. Bogofilter prints a warning (but continues to run) when used with SQLite -versions older than 3.4.0. This warning can be suppressed by defining +versions older than 3.4.2. This warning can be suppressed by defining the environment variable BF_USE_OLD_SQLITE to any value, including the empty value. Modified: trunk/bogofilter/src/datastore_sqlite.c =================================================================== --- trunk/bogofilter/src/datastore_sqlite.c 2007-11-25 16:20:41 UTC (rev 6686) +++ trunk/bogofilter/src/datastore_sqlite.c 2007-11-25 16:59:33 UTC (rev 6687) @@ -252,8 +252,8 @@ static void check_sqlite_version(void) { - const wmaj = 3, wmin = 4, wpl = 0; /* desired version of sqlite3 library */ - unsigned int vmaj, vmin, vpl; /* actual version of sqlite3 library */ + const unsigned int wmaj = 3, wmin = 4, wpl = 2; /* desired version of sqlite3 library */ + unsigned int vmaj, vmin, vpl; /* actual version of sqlite3 library */ static int complained; const char *v; This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-01-05 20:31:06
|
Revision: 6698 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6698&view=rev Author: relson Date: 2008-01-05 12:31:02 -0800 (Sat, 05 Jan 2008) Log Message: ----------- Fix bf_compact problem. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/src/bf_compact.in Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2007-12-26 10:54:43 UTC (rev 6697) +++ trunk/bogofilter/NEWS 2008-01-05 20:31:02 UTC (rev 6698) @@ -15,6 +15,11 @@ ------------------------------------------------------------------------------- + + 2008-01-05 + + * bf_compact problem fixed. Reported by Thomas Novin. + 1.1.6 2007-11-25 (released) * Transaction support added for TokyoCabinet datastore. Modified: trunk/bogofilter/src/bf_compact.in =================================================================== --- trunk/bogofilter/src/bf_compact.in 2007-12-26 10:54:43 UTC (rev 6697) +++ trunk/bogofilter/src/bf_compact.in 2008-01-05 20:31:02 UTC (rev 6698) @@ -66,7 +66,7 @@ # reload files for FILE in $FILES ; do NAME=`basename $FILE` - bogoutil -d $FILE | bogoutil --db-transaction=no -l $BOGOTEMP/$NAME + bogoutil -d $BOGOHOME/$FILE | bogoutil --db-transaction=no -l $BOGOTEMP/$NAME done if [ $TXN = "yes" ] ; then This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-02-08 12:54:47
|
Revision: 6703 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6703&view=rev Author: m-a Date: 2008-02-08 04:54:52 -0800 (Fri, 08 Feb 2008) Log Message: ----------- Bump required sqlite version to 3.5.4, earlier versions could sometimes corrupt the database. Update install-staticdblibs.sh. Bogofilter will complain when used with older versions. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/README.sqlite trunk/bogofilter/src/datastore_sqlite.c Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-02-08 12:51:55 UTC (rev 6702) +++ trunk/bogofilter/NEWS 2008-02-08 12:54:52 UTC (rev 6703) @@ -15,7 +15,12 @@ ------------------------------------------------------------------------------- + 2008-02-08 + * Bump required sqlite version to 3.5.4, earlier versions could + sometimes corrupt the database. Update install-staticdblibs.sh. + Bogofilter will complain when used with older versions. + 2008-01-05 * bf_compact problem fixed. Reported by Thomas Novin. Modified: trunk/bogofilter/doc/README.sqlite =================================================================== --- trunk/bogofilter/doc/README.sqlite 2008-02-08 12:51:55 UTC (rev 6702) +++ trunk/bogofilter/doc/README.sqlite 2008-02-08 12:54:52 UTC (rev 6703) @@ -20,13 +20,13 @@ 2.1 Compatibility - supported SQLite versions -At this time, only SQLite 3.4.2 and newer are supported. Older versions +At this time, only SQLite 3.5.4 and newer are supported. Older versions back to 3.0.8 may work, but you are advised to carefully review the sqlite3 ChangeLog, because there have been several important bug fixes since 3.0.8, including fixes for bugs that can corrupt the database. Bogofilter prints a warning (but continues to run) when used with SQLite -versions older than 3.4.2. This warning can be suppressed by defining +versions older than 3.5.4. This warning can be suppressed by defining the environment variable BF_USE_OLD_SQLITE to any value, including the empty value. Modified: trunk/bogofilter/src/datastore_sqlite.c =================================================================== --- trunk/bogofilter/src/datastore_sqlite.c 2008-02-08 12:51:55 UTC (rev 6702) +++ trunk/bogofilter/src/datastore_sqlite.c 2008-02-08 12:54:52 UTC (rev 6703) @@ -252,7 +252,7 @@ static void check_sqlite_version(void) { - const unsigned int wmaj = 3, wmin = 4, wpl = 2; /* desired version of sqlite3 library */ + const unsigned int wmaj = 3, wmin = 5, wpl = 4; /* desired version of sqlite3 library */ unsigned int vmaj, vmin, vpl; /* actual version of sqlite3 library */ static int complained; const char *v; This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-03-19 11:20:33
|
Revision: 6705 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6705&view=rev Author: m-a Date: 2008-03-19 04:20:40 -0700 (Wed, 19 Mar 2008) Log Message: ----------- Overhaul bogoupgrade, verify databases before dumping. Fixes Debian Bug#226643 and Debian Bug#226646. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/TODO trunk/bogofilter/src/bogoupgrade.in Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-02-24 11:33:47 UTC (rev 6704) +++ trunk/bogofilter/NEWS 2008-03-19 11:20:40 UTC (rev 6705) @@ -15,6 +15,17 @@ ------------------------------------------------------------------------------- + 2008-03-19 + + * Bogoupgrade now verifies databases before dumping them, to avoid + getting into unterminated loop and burning all memory when the + database is corrupt. + This should fix Debian Bug#226643 and Debian Bug#226646. + + * Bogoupgrade now uses Pod::Usage to print usage/help/manpage, prints + error messages that are a bit more concise and validates arguments a + bit stricter. + 2008-02-08 * Bump required sqlite version to 3.5.4, earlier versions could Modified: trunk/bogofilter/TODO =================================================================== --- trunk/bogofilter/TODO 2008-02-24 11:33:47 UTC (rev 6704) +++ trunk/bogofilter/TODO 2008-03-19 11:20:40 UTC (rev 6705) @@ -2,10 +2,6 @@ bogofilter TODO list -**** Make bogoupgrade check databases before attempting upgrades, - <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=226643> - <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=226646> (merged) - **** If insufficient data is present and the default "undecided" bogosity is added in -p mode, add also a comment stating that bogofilter needs more training first Modified: trunk/bogofilter/src/bogoupgrade.in =================================================================== --- trunk/bogofilter/src/bogoupgrade.in 2008-02-24 11:33:47 UTC (rev 6704) +++ trunk/bogofilter/src/bogoupgrade.in 2008-03-19 11:20:40 UTC (rev 6705) @@ -1,14 +1,11 @@ -#! @PERL@ +#! @PERL@ -w =pod -Name: +=head1 NAME + bogoupgrade -- upgrade a bogofilter database to current version. -Author: -Gyepi Sam <gy...@pr...> -David Relson <re...@os...> - =cut # bogofilter-0.3 through bogofilter-0.6.3 @@ -33,14 +30,17 @@ require 5.005_03; use strict; +use filetest 'access'; +use Pod::Usage; sub convert_double_to_single(); sub convert_format_A(); sub convert_format_B(); +sub db_verify($); -my $VERSION = '0.4.2'; +my $VERSION = '0.5.0'; -my ($in, $out, $help, $dir, $yday, $msg_count_token); +my ($in, $out, $dir, $yday, $msg_count_token); my $bogoutil = 'bogoutil'; @@ -58,25 +58,28 @@ } elsif ($arg eq '-y'){ $yday = "-y $ARGV[++$i]"; } elsif ($arg eq '-h' or $arg eq '--help'){ - help(); - exit(0); + pod2usage(-verbose => 1, -exitstatus => 0); + } elsif ($arg eq '--man') { + pod2usage(-verbose => 2, -exitstatus => 0); } else { - usage(); - exit(1); + pod2usage(-verbose => 0, -exitstatus => 1); } } my $db_ext = $ENV{DB_EXT} || 'db'; -if ( $dir ) { +if ($dir) { + if ($in || $out) { + pod2usage("Cannot use -d at the same time as -i or -o!"); + } convert_double_to_single(); } else { - die "Missing input filename\n" unless $in; - die "Missing output filename\n" unless $out; + pod2usage("Missing input filename.\nAborting") unless $in; + pod2usage("Missing output filename.\nAborting") unless $out; my $msg_count_token = '.MSG_COUNT'; - open(F, "< $in") or die "Cannot open input file [$in]. $!.\n"; + open(F, "< $in") or die "Cannot open input file [$in]. $!.\nAborting"; my $sig = <F>; chomp($sig); if ($sig =~ m/^\# bogofilter wordlist \(format version A\):\s(\d+)$/){ @@ -92,15 +95,24 @@ } } +sub db_verify($) { + my ($f) = shift; + if (!-r $f) { + die "Database $f is not readable: $!\nAborting"; + } + my $ret = system ($bogoutil, '--db-verify', $f); + die "Database $f is corrupt.\nAborting" if $ret; +} + sub convert_format_A() { my $msg_count = $1; my $cmd = "$bogoutil $yday -l $out"; - open(OUT, "| $cmd") or die "Cannot run command \"$cmd\": $!\n"; + open(OUT, "| $cmd") or die "Cannot run command \"$cmd\": $!\nAborting"; while(<F>){ print OUT $_; } print OUT "$msg_count_token $msg_count\n"; - close(OUT) or die "Error executing command \"$cmd\": $!\n"; + close(OUT) or die "Error executing command \"$cmd\": $!\nAborting"; close(F); } @@ -116,11 +128,12 @@ } my $cmd = "$bogoutil $yday -l $out"; - open(OUT, "| $cmd") or die "Cannot run command \"$cmd\": $!\n"; + open(OUT, "| $cmd") or die "Cannot run command \"$cmd\": $!\nAborting"; close(F); + db_verify($in_db); $cmd = "$bogoutil $yday -d $in_db"; - open(F, "$cmd |") or die "Cannot run command \"$cmd\": $!\n"; + open(F, "$cmd |") or die "Cannot run command \"$cmd\": $!\nAborting"; while(<F>){ if (m/^\.count\s+(\d+)$/){ @@ -147,15 +160,16 @@ my @tmp; my $cmd; - die "need array context in cvs2to1_get" unless wantarray; + die "need array context in cvs2to1_get.\nAborting" unless wantarray; + db_verify($filename); $cmd = "$bogoutil -d $filename"; - open(I, "$cmd |") or die "cannot run \"$cmd\": $!"; + open(I, "$cmd |") or die "cannot run \"$cmd\": $!\nAborting"; while (<I>) { chomp; my @a = split; push @tmp, sprintf($format, $a[0], $a[1], $a[2]); } - close I or die "error running \"$cmd\": $!"; + close I or die "error running \"$cmd\": $!\nAborting"; return @tmp; } @@ -166,52 +180,77 @@ my $spam = "$dir/spamlist.$db_ext"; my $good = "$dir/goodlist.$db_ext"; - if (-e $word) { die "$word already exists."; } + if (-e $word) { die "$word already exists.\nAborting"; } push @tmp, cvt2to1_get($spam, "%s %d 0 %d"); push @tmp, cvt2to1_get($good, "%s 0 %d %d"); my $cmd = "$bogoutil -l $word"; - open(O, "| $cmd") or die "cannot run \"$cmd\": $!"; + open(O, "| $cmd") or die "cannot run \"$cmd\": $!\nAborting"; print O join("\n", sort @tmp), "\n"; - close O or die "error executing \"$cmd\": $!"; + close O or die "error executing \"$cmd\": $!\nAborting"; } exit(0); -sub usage { - warn "usage: $0 [ -d <bogofilter directory> ]\n [ -i <input text file> -o <output db file> [ -b <path to bogoutil>] ]\n [ -h ]\n"; -} +__END__ +=pod -sub help { - print <<EOF; -$0 -- upgrades bogofilter database to current version. -Options: - -d <directory> +=head1 SYNOPSIS - Name of directory containing database files. Old files will be read and new - files will be written. + bogoupgrade [options] -d <bogofilter directory> + bogoupgrade [options] -i <input text file> -o <output db file> + bogoupgrade {-h|--help} + bogoupgrade {--man} - -i <input file> + Options: + -b <path to bogoutil> - Text file containing message count, and possibly wordlist data records. If - the file only contains a message count but no word list records, there must - be a database file, in the same directory as the text file, which contains - the word list data. +=head1 DESCRIPTION - -o <output file> +B<bogoupgrade> updates bogofilter databases from older formats to the current +format. - Output database file. Use the appropriate file extension for your version - of bogofilter, i.e. '.db' for Berkeley DB or '.sql' for SQLite3. +=head1 OPTIONS - -b <path to bogoutil program> +=over 8 - Defaults to 'bogoutil', in the hopes that your shell will find it. +=item B<-d <directory>> - -h help +Name of directory containing database files. Old files will be read and new +files will be written. - You are reading it. -EOF - exit(0); -} +=item B<-i <input file>> +Text file containing message count, and possibly wordlist data records. If the +file only contains a message count but no word list records, there must be a +database file, in the same directory as the text file, which contains the word +list data. + +=item B<-o <output file>> + +Output database file. Use the appropriate file extension for your version of +bogofilter, i.e. '.db' for Berkeley DB and SQLite3, or '.qdbm' for QDBM. + +=item B<-b <path to bogoutil program>> + +Defaults to 'bogoutil', in the hopes that your shell will find it. + +=item B<-h> | B<--help> + +Print the help - you are currently reading it. + +=item B<--man> + +Print the full manual page. + +=back + +=head1 AUTHORS + + Gyepi Sam <gy...@pr...> + David Relson <re...@os...> + Matthias Andree <mat...@gm...> + +=cut + # vim: set filetype=perl ai: This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-03-19 11:32:34
|
Revision: 6706 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6706&view=rev Author: m-a Date: 2008-03-19 04:32:35 -0700 (Wed, 19 Mar 2008) Log Message: ----------- bf_compact: Verify databases before compaction. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/src/bf_compact.in Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-03-19 11:20:40 UTC (rev 6705) +++ trunk/bogofilter/NEWS 2008-03-19 11:32:35 UTC (rev 6706) @@ -17,9 +17,12 @@ 2008-03-19 + * bf_compact now verifies databases before dumping them, to avoid + getting into an unterminated loop and wasting all diskspace. + * Bogoupgrade now verifies databases before dumping them, to avoid - getting into unterminated loop and burning all memory when the - database is corrupt. + getting into an unterminated loop and burning all memory or disk + space when the database is corrupt. This should fix Debian Bug#226643 and Debian Bug#226646. * Bogoupgrade now uses Pod::Usage to print usage/help/manpage, prints Modified: trunk/bogofilter/src/bf_compact.in =================================================================== --- trunk/bogofilter/src/bf_compact.in 2008-03-19 11:20:40 UTC (rev 6705) +++ trunk/bogofilter/src/bf_compact.in 2008-03-19 11:32:35 UTC (rev 6706) @@ -20,7 +20,7 @@ shift if [ ! -d "$BOGOHOME" ] ; then - echo $BOGOHOME must be a directory, not a file + echo "$BOGOHOME must be a directory, not a file" exit 1 fi @@ -32,15 +32,17 @@ esac done +export BOGOHOME + # find wordlists if [ -n "$1" ] ; then FILES="$@" else - DIR=`bogofilter -QQ | grep ^bogofilter_dir | @AWK@ '{print $3}'` + DIR=$(bogofilter -QQ | grep ^bogofilter_dir | @AWK@ '{print $3}') if [ "$BOGOHOME" != "$DIR" ] ; then - FILES=`ls "$BOGOHOME"/*.db` + FILES=$(ls "$BOGOHOME"/*.db) else - FILES=`bogofilter -QQ | grep ^wordlist | cut -f3 -d,` + FILES=$(bogofilter -QQ | grep ^wordlist | cut -f3 -d,) fi fi @@ -57,7 +59,7 @@ fi # determine transactions -if test "`find "$BOGOHOME/" -name "log.??????????" -print`" = "" ; then +if test "$(find "$BOGOHOME/" -name "log.??????????" -print)" = "" ; then TXN=no else TXN=yes @@ -65,17 +67,18 @@ # reload files for FILE in $FILES ; do - NAME=`basename $FILE` - bogoutil -d "$BOGOHOME"/$FILE | bogoutil --db-transaction=no -l $BOGOTEMP/$NAME + NAME="$(basename $FILE)" + bogoutil --db-verify "$FILE" || { echo "Aborting." ; rm -r "$BOGOTEMP" ; exit 1 ; } + bogoutil -d "$FILE" | bogoutil --db-transaction=no -l "$BOGOTEMP/$NAME" done if [ $TXN = "yes" ] ; then #create database environment files - bogofilter -e -C -d $BOGOTEMP --db-transaction=yes < /dev/null + bogofilter -e -C -d "$BOGOTEMP" --db-transaction=yes < /dev/null fi # remove $BOGOHOME.old so we don't move the new backup *into* it # rather than renaming the backup to it. rm -rf "$BOGOHOME.old" mv "$BOGOHOME" "$BOGOHOME.old" -mv $BOGOTEMP "$BOGOHOME" +mv "$BOGOTEMP" "$BOGOHOME" This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-03-21 15:07:08
|
Revision: 6708 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6708&view=rev Author: m-a Date: 2008-03-21 08:07:12 -0700 (Fri, 21 Mar 2008) Log Message: ----------- Enable bf_compact to work with other databases than Berkeley DB. Put bf_compact under test coverage. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/src/bf_compact.in trunk/bogofilter/src/tests/Makefile.am trunk/bogofilter/src/tests/t.frame Added Paths: ----------- trunk/bogofilter/src/tests/t.bf_compact Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-03-21 15:06:11 UTC (rev 6707) +++ trunk/bogofilter/NEWS 2008-03-21 15:07:12 UTC (rev 6708) @@ -14,7 +14,11 @@ !!!!!!!! READ THE RELEASE.NOTES !!!!!!!! ------------------------------------------------------------------------------- + 2008-03-21 + * bf_compact now supports compacting databases that use QDBM, Tokyo + Cabinet or SQLite3 and is covered by the test suite. + 2008-03-19 * bf_compact now verifies databases before dumping them, to avoid @@ -25,9 +29,9 @@ space when the database is corrupt. This should fix Debian Bug#226643 and Debian Bug#226646. - * Bogoupgrade now uses Pod::Usage to print usage/help/manpage, prints - error messages that are a bit more concise and validates arguments a - bit stricter. + * Bogoupgrade now uses Pod::Usage to print usage/help, prints error + messages that are a bit more concise and validates arguments a bit + stricter. 2008-02-08 Modified: trunk/bogofilter/src/bf_compact.in =================================================================== --- trunk/bogofilter/src/bf_compact.in 2008-03-21 15:06:11 UTC (rev 6707) +++ trunk/bogofilter/src/bf_compact.in 2008-03-21 15:07:12 UTC (rev 6708) @@ -10,6 +10,9 @@ set -e # die on errors +: ${BOGOFILTER:=bogofilter} +: ${BOGOUTIL:=bogoutil} + if [ -z "$1" ] ; then echo 'usage: bf_compact source_dir [wordlist_name...]' exit 1 @@ -38,11 +41,11 @@ if [ -n "$1" ] ; then FILES="$@" else - DIR=$(bogofilter -QQ | grep ^bogofilter_dir | @AWK@ '{print $3}') + DIR=$($BOGOFILTER -QQ | grep ^bogofilter_dir | @AWK@ '{print $3}') if [ "$BOGOHOME" != "$DIR" ] ; then - FILES=$(ls "$BOGOHOME"/*.db) + FILES=$(ls "$BOGOHOME"/*@DB_EXT@) else - FILES=$(bogofilter -QQ | grep ^wordlist | cut -f3 -d,) + FILES=$($BOGOFILTER -QQ | grep ^wordlist | cut -f3 -d,) fi fi @@ -58,23 +61,34 @@ cp -p "$BOGOHOME"/DB_CONFIG "$BOGOTEMP"/ fi -# determine transactions -if test "$(find "$BOGOHOME/" -name "log.??????????" -print)" = "" ; then - TXN=no -else - TXN=yes -fi +case "$($BOGOFILTER -V | grep Database:)" in + *"Berkeley DB"*) + # determine transactions + if test "$(find "$BOGOHOME/" -name "log.??????????" -print)" = "" ; then + TXN=no + else + TXN=yes + fi + ;; + *) + TXN=noarg + ;; +esac # reload files for FILE in $FILES ; do NAME="$(basename $FILE)" - bogoutil --db-verify "$FILE" || { echo "Aborting." ; rm -r "$BOGOTEMP" ; exit 1 ; } - bogoutil -d "$FILE" | bogoutil --db-transaction=no -l "$BOGOTEMP/$NAME" + $BOGOUTIL --db-verify "$FILE" \ + || { echo "$FILE corrupted, aborting." ; rm -r "$BOGOTEMP" ; exit 1 ; } + $BOGOUTIL -d "$FILE" | case $TXN in + no|yes) $BOGOUTIL --db-transaction=no -l "$BOGOTEMP/$NAME" ;; + noarg) $BOGOUTIL -l "$BOGOTEMP/$NAME" ;; + esac done if [ $TXN = "yes" ] ; then #create database environment files - bogofilter -e -C -d "$BOGOTEMP" --db-transaction=yes < /dev/null + $BOGOFILTER -e -C -d "$BOGOTEMP" --db-transaction=yes < /dev/null fi # remove $BOGOHOME.old so we don't move the new backup *into* it Modified: trunk/bogofilter/src/tests/Makefile.am =================================================================== --- trunk/bogofilter/src/tests/Makefile.am 2008-03-21 15:06:11 UTC (rev 6707) +++ trunk/bogofilter/src/tests/Makefile.am 2008-03-21 15:07:12 UTC (rev 6708) @@ -46,7 +46,7 @@ t.rfc2047_broken t.rfc2047_folded \ t.message_addr t.message_id t.queue_id -WORDLIST_TESTS = t.dump.load t.nonascii.replace t.maint t.robx t.regtest t.upgrade.subnet.prefix t.multiple.wordlists t.probe +WORDLIST_TESTS = t.dump.load t.nonascii.replace t.maint t.robx t.regtest t.upgrade.subnet.prefix t.multiple.wordlists t.probe t.bf_compact SCORING_TESTS = t.query.config t.score1 t.score2 t.systest t.grftest t.wordhist Added: trunk/bogofilter/src/tests/t.bf_compact =================================================================== --- trunk/bogofilter/src/tests/t.bf_compact (rev 0) +++ trunk/bogofilter/src/tests/t.bf_compact 2008-03-21 15:07:12 UTC (rev 6708) @@ -0,0 +1,32 @@ +#! /bin/sh + +# Note: When run via "make check", test output files are automatically deleted. +# When run from the command line, output files are left in directory +# wordhist.YYYYMMDD. This is useful when something is different. +# +# ./inputs - test inputs + +NODB=1 . ${srcdir=.}/t.frame + +[ -d "$TMPDIR"/tests ] && rm -rf "$TMPDIR"/tests +mkdir -p "$TMPDIR"/tests + +OPTS="-C -y 0" + +$BOGOFILTER $OPTS -s < "$SYSTEST/inputs/spam.mbx" +$BOGOFILTER $OPTS -n < "$SYSTEST/inputs/good.mbx" + +( +$SHELL $BF_COMPACT $BOGOFILTER_DIR + +$BOGOUTIL -d "$WORDLIST" >$BOGOFILTER_DIR/cpc +$BOGOUTIL -d "$BOGOFILTER_DIR/wordlist.$DB_EXT" >$BOGOFILTER_DIR/org + +if [ $verbose -eq 0 ] ; then + cmp $BOGOFILTER_DIR/org $BOGOFILTER_DIR/cpc +else + diff $DIFF_BRIEF $BOGOFILTER_DIR/org $BOGOFILTER_DIR/cpc +fi +) && a=0 || a=$? +rm -rf "$BOGOFILTER_DIR.old" +exit $a Property changes on: trunk/bogofilter/src/tests/t.bf_compact ___________________________________________________________________ Name: svn:executable + * Modified: trunk/bogofilter/src/tests/t.frame =================================================================== --- trunk/bogofilter/src/tests/t.frame 2008-03-21 15:06:11 UTC (rev 6707) +++ trunk/bogofilter/src/tests/t.frame 2008-03-21 15:07:12 UTC (rev 6708) @@ -26,10 +26,13 @@ BOGOLEXER="$VAL${relpath}/bogolexer$EXE_EXT" BOGOTUNE="$VAL${relpath}/bogotune$EXE_EXT" BOGOUTIL="$VAL${relpath}/bogoutil$EXE_EXT" +BF_COMPACT="${relpath}/bf_compact" export BOGOFILTER export BOGOLEXER +export BOGOTUNE export BOGOUTIL +export BF_COMPACT # Cater for GNU coreutils "compatibility" _POSIX2_VERSION=199209 This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-04-19 03:20:50
|
Revision: 6713 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6713&view=rev Author: relson Date: 2008-04-18 20:20:54 -0700 (Fri, 18 Apr 2008) Log Message: ----------- Use uniform protype for all versions of process_arg() and call proper version for each program (bug #1943136). Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/src/bogoconfig.c trunk/bogofilter/src/bogoconfig.h trunk/bogofilter/src/bogolexer.c trunk/bogofilter/src/bogotune.c trunk/bogofilter/src/bogoutil.c Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/NEWS 2008-04-19 03:20:54 UTC (rev 6713) @@ -14,6 +14,21 @@ !!!!!!!! READ THE RELEASE.NOTES !!!!!!!! ------------------------------------------------------------------------------- + 2008-04-18 + + * Function process_arg now has the same prototype for + bogofilter, bogolexer, bogoutil, and bogotune. The proper + version is called by function read_config_file for all + programs. Problem reported by Roman Trunov. + + 2008-04-17 + + * Update Doxyfile for doxygen v1.5.5 + + 2008-04-16 + + * Fixed syntax errors in t.valgrind test + 2008-03-21 * bf_compact now supports compacting databases that use QDBM, Tokyo Modified: trunk/bogofilter/src/bogoconfig.c =================================================================== --- trunk/bogofilter/src/bogoconfig.c 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/src/bogoconfig.c 2008-04-19 03:20:54 UTC (rev 6713) @@ -230,7 +230,7 @@ return ok; } -void comma_parse(char opt, const char *arg, double *parm1, double *parm2, double *parm3) +static void comma_parse(char opt, const char *arg, double *parm1, double *parm2, double *parm3) { char *parse = xstrdup(arg); char *copy = parse; @@ -478,7 +478,7 @@ return; } -void process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) +int process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) { switch (option) { @@ -740,6 +740,8 @@ exit(EX_ERROR); } } + + return 0; } #define Q1 if (query >= 1) Modified: trunk/bogofilter/src/bogoconfig.h =================================================================== --- trunk/bogofilter/src/bogoconfig.h 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/src/bogoconfig.h 2008-04-19 03:20:54 UTC (rev 6713) @@ -25,6 +25,6 @@ extern rc_t query_config(void); extern void process_parameters(int argc, char **argv, bool warn_on_error); -extern void process_arg(int option, const char *name, const char *arg, priority_t precedence, arg_pass_t pass); +extern int process_arg(int option, const char *name, const char *arg, priority_t precedence, arg_pass_t pass); #endif Modified: trunk/bogofilter/src/bogolexer.c =================================================================== --- trunk/bogofilter/src/bogolexer.c 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/src/bogolexer.c 2008-04-19 03:20:54 UTC (rev 6713) @@ -161,7 +161,7 @@ } } -void process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) +int process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) { pass = 0; /* suppress compiler warning */ @@ -283,6 +283,8 @@ exit(EX_ERROR); } } + + return 0; } static int count=0; Modified: trunk/bogofilter/src/bogotune.c =================================================================== --- trunk/bogofilter/src/bogotune.c 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/src/bogotune.c 2008-04-19 03:20:54 UTC (rev 6713) @@ -54,6 +54,7 @@ #include "bogoconfig.h" #include "bogoreader.h" +#include "bool.h" #include "collect.h" #include "datastore.h" #include "longoptions.h" @@ -187,16 +188,24 @@ bool fMakeCheck = false; /* allows quick & dirty regression testing */ uint cMakeCheck = 50; /* ... for 50 cycles */ +const char *logtag = NULL; const char *spam_header_place = ""; /* Function Declarations */ -static void process_bogotune_arg(int option); - /* Function Definitions */ static void bt_trap(void) {} +static bool get_bool(const char *name, const char *arg) +{ + bool b = str_to_bool(arg); + if (DEBUG_CONFIG(2)) + fprintf(dbgout, "%s -> %s\n", name, + b ? "Yes" : "No"); + return b; +} + static int get_cnt(double fst, double lst, double amt) { int cnt = (fabs(lst - fst) + EPS) / (fabs(amt) - EPS) + 1; @@ -972,7 +981,7 @@ name = (option_index == 0) ? argv[this_option_optind] : longopts_bogotune[option_index].name; - process_bogotune_arg(option); + process_arg(option, NULL, NULL, PR_NONE, PASS_1_CLI); } if (ds_flag == DS_NONE) /* default is "wordlist on disk" */ @@ -996,10 +1005,15 @@ return count; } -static void process_bogotune_arg(int option) +int process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) { static int lastmode = -1; + (void) name; /* suppress compiler warning */ + (void) val; /* suppress compiler warning */ + (void) precedence; /* suppress compiler warning */ + (void) pass; /* suppress compiler warning */ + if (option == 1) { /* If getopt's RETURN_IN_ORDER behavior */ switch (lastmode) { @@ -1015,41 +1029,54 @@ switch (option) { case 'c': + case O_CONFIG_FILE: read_config_file(optarg, false, false, PR_CFG_USER, longopts_bogotune); - /* FALLTHROUGH */ + /*@fallthrough@*/ + /* fall through to suppress reading config files */ + case 'C': suppress_config_file = true; break; + case 'd': ds_path = xstrdup(optarg); ds_flag = (ds_flag == DS_NONE) ? DS_DSK : DS_ERR; break; + case 'D': ds_flag = (ds_flag == DS_NONE) ? DS_RAM : DS_ERR; break; + case 'e': exit_zero = true; break; + case 'E': esf_flag ^= true; break; + case 'M': bogolex_file = optarg; break; + case 'n': lastmode = 'n'; filelist_add(ham_files, optarg); break; + case 'q': quiet = true; break; + case 'r': user_robx = atof(optarg); break; + case 's': lastmode = 's'; filelist_add(spam_files, optarg); break; + #ifdef TEST case 't': test += 1; @@ -1058,12 +1085,15 @@ case 'T': coerced_target = atoi(optarg); break; + case 'v': verbose += 1; break; + case 'V': print_version(); exit(EX_OK); + case 'x': if (strcmp(optarg, "MakeCheck") == 0) fMakeCheck = true; @@ -1087,10 +1117,20 @@ multi_token_count=atoi(optarg); break; + case O_BLOCK_ON_SUBNETS: + block_on_subnets = get_bool(name, val); + break; + + case O_REPLACE_NONASCII_CHARACTERS: + replace_nonascii_characters = get_bool(name, val); + break; + default: help(); exit(EX_ERROR); } + + return 0; } static double get_robx(void) Modified: trunk/bogofilter/src/bogoutil.c =================================================================== --- trunk/bogofilter/src/bogoutil.c 2008-04-19 02:33:29 UTC (rev 6712) +++ trunk/bogofilter/src/bogoutil.c 2008-04-19 03:20:54 UTC (rev 6713) @@ -21,6 +21,7 @@ #include "getopt.h" +#include "bogoconfig.h" #include "bogofilter.h" #include "bogohist.h" #include "bool.h" @@ -54,10 +55,6 @@ const char *spam_header_place = ""; -/* Function Prototypes */ - -static int process_arg(int option, const char *name, const char *arg); - /* Function Definitions */ /* dummies to avoid score.o */ @@ -632,7 +629,7 @@ break; name = (option_index == 0) ? argv[this_option_optind] : longopts_bogoutil[option_index].name; - count += process_arg(option, name, optarg); + count += process_arg(option, name, optarg, PR_NONE, PASS_1_CLI); } if (max_token_len != 0 && max_multi_token_len == 0) { @@ -653,10 +650,13 @@ return count; } -static int process_arg(int option, const char *name, const char *val) +int process_arg(int option, const char *name, const char *val, priority_t precedence, arg_pass_t pass) { int count = 0; + (void) precedence; /* suppress compiler warning */ + (void) pass; /* suppress compiler warning */ + switch (option) { case '?': fprintf(stderr, "Unknown option '%s'.\n", name); This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-04-20 16:39:51
|
Revision: 6717 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6717&view=rev Author: relson Date: 2008-04-20 08:34:59 -0700 (Sun, 20 Apr 2008) Log Message: ----------- Credit Roman Trunov. Modified Paths: -------------- trunk/bogofilter/AUTHORS trunk/bogofilter/NEWS Modified: trunk/bogofilter/AUTHORS =================================================================== --- trunk/bogofilter/AUTHORS 2008-04-20 15:32:22 UTC (rev 6716) +++ trunk/bogofilter/AUTHORS 2008-04-20 15:34:59 UTC (rev 6717) @@ -54,3 +54,4 @@ Jason A. Smith Marco Bozzolan Paul Mangan +Roman Trunov Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-04-20 15:32:22 UTC (rev 6716) +++ trunk/bogofilter/NEWS 2008-04-20 15:34:59 UTC (rev 6717) @@ -14,6 +14,12 @@ !!!!!!!! READ THE RELEASE.NOTES !!!!!!!! ------------------------------------------------------------------------------- + + 2008-04-20 + + * In function process_arg use the val parameter rather than optarg. + Patch provided by Roman Trunov. + 2008-04-18 * Function process_arg now has the same prototype for This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-04-28 23:26:40
|
Revision: 6720 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6720&view=rev Author: relson Date: 2008-04-28 16:26:47 -0700 (Mon, 28 Apr 2008) Log Message: ----------- Add maildir training info. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/bogofilter-faq-fr.html trunk/bogofilter/doc/bogofilter-faq.html Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-04-26 12:51:17 UTC (rev 6719) +++ trunk/bogofilter/NEWS 2008-04-28 23:26:47 UTC (rev 6720) @@ -15,9 +15,19 @@ ------------------------------------------------------------------------------- + 2008-04-28 + + * Added maildir training info to English and French FAQs. + Thanks to Karl Schmidt and to Mouss. + + 2008-04-26 + + * Fix uninitialized variable in lexer.c when unicode is disabled. + Patch provided by Roman Trunov. + 2008-04-20 - * In function process_arg use the val parameter rather than optarg. + * In process_arg functions use the val parameter rather than optarg. Patch provided by Roman Trunov. 2008-04-18 Modified: trunk/bogofilter/doc/bogofilter-faq-fr.html =================================================================== --- trunk/bogofilter/doc/bogofilter-faq-fr.html 2008-04-26 12:51:17 UTC (rev 6719) +++ trunk/bogofilter/doc/bogofilter-faq-fr.html 2008-04-28 23:26:47 UTC (rev 6720) @@ -71,6 +71,7 @@ Questions op\xE9rationnelles <ul> <li><a href="#training">Comment d\xE9buter l'apprentissage de Bogofilter?</a></li> + <li><a href="#training_maildirs">Comment lancer l'entrainement avec des formats mbox et maildir?</a></li> <li><a href="#production">Comment conserver une bonne d'acuit\xE9?</a></li> <li><a href="#mboxformats">Quels formats de boites aux lettres (fichier) Bogofilter comprend-il?</a></li> <li><a href="#vvv">Que signifie la sortie verbeuse de Bogofilter?</a></li> @@ -444,6 +445,30 @@ <hr> + <h2 id="training_maildirs">Comment lancer l'entrainement avec des formats mbox et maildir?</a></li> + + <h3>Entrainement initial avec un format mbox:</h3> + +<pre> bogofilter -M -s -I ~/mail/Spam + bogofilter -M -n -I ~/mail/NonSpam</pre> + + <h3>Entrainement initial avec un format maildir:</h3> + +<pre> bogofilter -s -B ~/Maildir/.Spam + bogofilter -n -B ~/Maildir/.NonSpam</pre> + + <h3>Entrainement sur erreur avec un format mbox:</h3> + +<pre> bogofilter -M -Ns -I ~/mail/Missed_Spam + bogofilter -M -Sn -I ~/mail/False_Spam</pre> + + <h3>Entrainement sur erreur avec un format maildir:</h3> + +<pre> bogofilter -s -B ~/Maildir/.Missed_Spam + bogofilter -n -B ~/Maildir/.False_Spam</pre> + + <hr> + <h2 id="production">Comment puis-je conserver une bonne acuit\xE9?</h2> <p>Bogofilter fera des erreurs de temps en temps. C'est pourquoi un Modified: trunk/bogofilter/doc/bogofilter-faq.html =================================================================== --- trunk/bogofilter/doc/bogofilter-faq.html 2008-04-26 12:51:17 UTC (rev 6719) +++ trunk/bogofilter/doc/bogofilter-faq.html 2008-04-28 23:26:47 UTC (rev 6720) @@ -80,6 +80,7 @@ Operational Questions <ul> <li><a href="#training">How do I start my bogofilter training?</a></li> + <li><a href="#training_maildirs">How do I train using maildirs?</a></li> <li><a href="#production">How can I keep the scoring accuracy high?</a></li> <li><a href="#mboxformats">What mailbox (file) formats does bogofilter understand?</a></li> <li><a href="#vvv">What does bogofilter's verbose output mean?</a></li> @@ -446,6 +447,30 @@ <hr> + <h2 id="training_maildirs">How do I train using maildirs?</a></li> + + <h3>Initial training from mbox:</h3> + +<pre> bogofilter -M -s -I ~/mail/Spam + bogofilter -M -n -I ~/mail/NonSpam</pre> + + <h3>Initial training from maildir:</h3> + +<pre> bogofilter -s -B ~/Maildir/.Spam + bogofilter -n -B ~/Maildir/.NonSpam</pre> + + <h3>Corrective training from mbox:</h3> + +<pre> bogofilter -M -Ns -I ~/mail/Missed_Spam + bogofilter -M -Sn -I ~/mail/False_Spam</pre> + + <h3>Corrective training from maildir:</h3> + +<pre> bogofilter -s -B ~/Maildir/.Missed_Spam + bogofilter -n -B ~/Maildir/.False_Spam</pre> + + <hr> + <h2 id="production">How can I keep the scoring accuracy high?</h2> <p>Bogofilter will make mistakes once in a while. So ongoing This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-04-30 22:12:03
|
Revision: 6722 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6722&view=rev Author: relson Date: 2008-04-30 15:12:08 -0700 (Wed, 30 Apr 2008) Log Message: ----------- Update bogofilter-milter.pl for 1.1.7 Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/contrib/bogofilter-milter.pl Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-04-29 00:19:02 UTC (rev 6721) +++ trunk/bogofilter/NEWS 2008-04-30 22:12:08 UTC (rev 6722) @@ -15,6 +15,11 @@ ------------------------------------------------------------------------------- + 2008-04-30 + + * Updated sendmail milter contrib/bogofilter-milter.pl to v1.45 + (thanks to Jonathan Kamens) + 2008-04-28 * Added maildir training info to English and French FAQs. Modified: trunk/bogofilter/contrib/bogofilter-milter.pl =================================================================== --- trunk/bogofilter/contrib/bogofilter-milter.pl 2008-04-29 00:19:02 UTC (rev 6721) +++ trunk/bogofilter/contrib/bogofilter-milter.pl 2008-04-30 22:12:08 UTC (rev 6722) @@ -24,8 +24,9 @@ # You will need the following non-standard Perl modules installed to # use this script: Sendmail::Milter, Mail::Alias, Proc::Daemon, -# IO::Stringy. Before using this script, search for CONFIGURABLE -# SETTINGS and configure them appropriately for your site. +# IO::Stringy, Socket, Net::CIDR. Before using this script, search +# for CONFIGURABLE SETTINGS and configure them appropriately for your +# site. # # Inserts "X-Bogosity: Spam, tests=bogofilter" into messages that # appear to be spam (or "Ham" into ones that don't). If the message is @@ -60,12 +61,19 @@ # This script logs various informational, warning and error messages # to the "mail" facility. +# Thanks to Tom Anderson <neo...@or...> +# for the IP whitelisting changes and for several other useful +# suggestions and bug fixes. + # BEGIN CONFIGURABLE SETTINGS # If this string appears in the Subject of a message (case # insensitive), the message won't be filtered. my $magic_string = '[no-bogofilter]'; +# Set the syslog facility you wish to log messages to. +my $log_facility = 'LOG_MAIL'; + # These settings control exactly what error sendmail sends back to the # sender if a message is rejected. You can leave them as-is, or # customize them as desired. @@ -75,6 +83,18 @@ "If it isn't, resend it with $magic_string " . "in the Subject line."; +# Whitelist any IP addresses or ranges from this filter. +# For example: +#my(@whitelist) = ("127.0.0.1", "10.127.0.1-10.127.0.9", "192.168.0.0/16"); +my(@ip_whitelist) = (); + +# If you want to whitelist any addresses which have authenticated +# via poprelayd (i.e. remote workstations of users on your server) +# set $dbfile to your popip.db location, else set it to undef. +# For example: +#my $ip_whitelist_db = "/etc/mail/popip.db"; +my $ip_whitelist_db = undef; + # The largest message to keep in memory rather than writing to a # temporary file. my $MAX_INCORE_MSG_LENGTH = 1000000; @@ -86,6 +106,23 @@ # there, but not here). my $socket = '/var/run/bogofilter-milter.sock'; +# The following two settings give more granular control over whether +# bogofilter is used for any particular user and what configuration +# settings are used when it is. +# - If $bogofilter_cf is set, then the script will look for a file +# with that name in the user's home directory. If it finds it, then +# bogofilter will be called with "-c $HOME/$bogofilter_cf" so that the +# specified configuration file is used rather than the default, +# .bogofilter.cf. +# - If $require_cf is true, then the specified configuration file +# *must* exist for bogofilter to be used for this user. In other +# words, rather than only looking for the .bogofilter subdirectory of +# the user's home directory, the script will look for both the +# .bogofilter subdirectory *and* the config file. +# - Note that $require_cf is ignored if $bogofilter_cf is unset. +my $bogofilter_cf = undef; +my $require_cf = undef; + # If a file with this name exists in the user's .bogofilter directory, # then that user's mail will be filtered in training mode. This means # that the message will be filtered and registered as spam or non-spam @@ -111,6 +148,21 @@ # into an mbox format file. my $cyrus_deliver = '/usr/lib/cyrus-imapd/deliver'; +# If you would like to use a shared bogofilter database for everyone, +# rather than separate per-user databases, then create a user on your +# system to be used as a home for the shared database, and set +# $database_user to that user's username. +# +# If you set $database_user, then all the logic described above for +# deciding whether to run bogofilter, whether to run in training mode +# or real mode, and whether to archive spam still applies, so make +# sure you configure $database_user's account properly. +# +# If you set $database_user, then $aliases_file, $sendmail_canon, +# $sendmail_prog, $recipient_cache_expire, and +# $recipient_cache_check_interval do NOT apply and are ignored. +my $database_user = undef; + # Mail::Alias is used to expand SMTP recipient addresses into local # mailboxes to determine if any of them have bogofilter databases. If # someone sends E-mail to a mailing list or alias whose expansion @@ -137,6 +189,35 @@ my $sendmail_canon = 1; my $sendmail_prog = '/usr/sbin/sendmail'; +# @discard_control is an array of anonymous arrays. Each sub-array +# contains a pair of entries, a control pattern and an action, either +# "discard" or "reject". The action corresponding to the first +# matching control pattern determines what happens to the messages. +# If @discard_control is empty or none of its control patterns match, +# the default action is "reject". The following control patterns are +# valid: + +# "addr:a.b.c.d" matches if the sending host has the indicated IP address +# "netblock:a.b.c.d/e" matches if the sending host is in the indicated netblock +# "host:fqdn" matches if the IP address of the sending host resolves +# to the indicated host name +# "domain:fqdn" matches if the IP address of the sending host resolves +# to a host name in the indicated domain +# "mx" matches if one of the MX servers for the recipient's +# domain resolves to the IP address of the sending host +# "*" always matches + +# The default @discard_control setting discards messages from MX +# servers to prevent this script from contributing to spam "blowback", +# which occurs when a spammer forges someone's real email address as +# the return address on spam, and then that person has to deal with +# tons of bounce messages from sites that reject the spam. +my(@discard_control) = + ( + ["mx" => "discard"], + ["*" => "reject"], + ); + # You can configure how long addresses will stay in the cache of # addresses that have been been expanded against the virtual user # table (if $sendmail_canon is set above), then expanded against the @@ -164,19 +245,24 @@ use strict; use warnings; -use Sendmail::Milter; +use DB_File; +use Data::Dumper; +use English '-no_match_vars'; +use Fcntl qw(:flock :seek); use File::Basename; use File::Temp qw(tempfile); -use Fcntl qw(:flock :seek); -use Mail::Alias; -use User::pwent; -use Sys::Syslog qw(:DEFAULT :macros setlogsock); -use English '-no_match_vars'; -use Proc::Daemon; use Getopt::Long; use IO::Scalar; use IPC::Open2; -use Data::Dumper; +use Mail::Alias; +use Net::CIDR; +use Net::DNS; +use POSIX; +use Proc::Daemon; +use Sendmail::Milter; +use Socket; +use Sys::Syslog qw(:DEFAULT :macros setlogsock); +use User::pwent; $Data::Dumper::Indent = 0; @@ -196,8 +282,12 @@ 'body' => \&my_body_callback, 'eom' => \&my_eom_callback, 'abort' => \&my_abort_callback, + 'close' => \&my_close_callback, ); +$my_milter_callbacks{'connect'} = \&my_connect_callback + if (@ip_whitelist || $ip_whitelist_db || @discard_control); + dia $usage if (! GetOptions('daemon' => \$run_as_daemon, 'debug' => \$debug, 'help|h|?' => \$get_help)); @@ -213,12 +303,45 @@ my $magic_string_re = $magic_string; $magic_string_re =~ s/(\W)/\\$1/g; +# convert whitelist into CIDR notation +{ + my(@whitelist_cidr); + + foreach my $IP (@ip_whitelist) { + if (not eval {@whitelist_cidr = + Net::CIDR::cidradd($IP, @whitelist_cidr)}) { + &die("Error processing whitelist: \"$IP\" is not a valid IP ", + "address or range."); + } + } + @ip_whitelist = @whitelist_cidr; +} + +# open popip database for reading +my %ip_whitelist_db; + +&opendb_read if ($ip_whitelist_db); + setlogsock('unix'); -openlog($whoami, 'pid', 'mail'); +openlog($whoami, 'pid', $log_facility); if (! $debug) { - setlogmask(&LOG_UPTO(LOG_INFO)); + # I'd really like to to this, but it doesn't work wit Sys::Syslog + # 0.13 in Perl 5.8.8. + # setlogmask(&LOG_UPTO(LOG_INFO)); + eval " + no warnings 'redefine'; + sub debuglog { + } + "; } +if ($database_user) { + $aliases_file = $sendmail_canon = $sendmail_prog = + $recipient_cache_expire = $recipient_cache_check_interval = undef; + syslog("info", "Using shared bogofilter database under %s's account", + $database_user); +} + if (! (open(PIDFILE, '+<', $pid_file) || open(PIDFILE, '+>', $pid_file))) { &die("open($pid_file): $!\n"); @@ -242,8 +365,57 @@ Sendmail::Milter::main($milter_interpreters); +&closedb; + +sub my_connect_callback { + my $ctx = shift; # milter context object + my $hostname = shift; # The connection's host name. + my $sockaddr_in = shift; # AF_INET portion of the host address, + # from getpeername(2) syscall + my $hash = $ctx->getpriv(); + + my ($port, $ipaddr) = Socket::unpack_sockaddr_in($sockaddr_in) or + &die("Could not unpack socket address: $!"); + $ipaddr = Socket::inet_ntoa($ipaddr); # translates into standard IPv4 addr + + &debuglog("my_connect_callback: entering with hostname=$hostname, ", + "ipaddr=$ipaddr, port=$port"); + + # check if the connecting server is listed in the whitelist + if (@ip_whitelist) { + if (eval {Net::CIDR::cidrlookup($ipaddr, @ip_whitelist)}) { + syslog('info', '%s', "$ipaddr is whitelisted, so this email is " . + "being accepted unfiltered."); + $ctx -> setpriv(undef); + return SMFIS_ACCEPT; + } + else { + &debuglog("$ipaddr is not in the whitelist"); + } + } + + # check if connecting server is listed in the popip database + if ($ip_whitelist_db) { + if ($ip_whitelist_db{$ipaddr}) { + syslog('info', '%s', "$ipaddr is authenticated via poprelayd, " . + "so this email is being accepted unfiltered."); + $ctx -> setpriv(undef); + return SMFIS_ACCEPT; + } + else { + &debuglog("$ipaddr is not in the popip database"); + } + } + + $hash->{'ipaddr'} = $ipaddr; + $ctx->setpriv($hash); + &debuglog("my_connect_callback: return CONTINUE with hash"); + return SMFIS_CONTINUE; +} + sub my_rcpt_callback { my $ctx = shift; + my $envrcpt = shift; my $hash = $ctx->getpriv(); &debuglog("my_rcpt_callback: entering with " . Data::Dumper->Dump([$hash], [qw(hash)])); @@ -260,6 +432,7 @@ if (&filtered_dir($rcpt)) { $hash->{'rcpt'} = $rcpt; + $hash->{'envrcpt'} = $envrcpt; $ctx->setpriv($hash); &debuglog("my_rcpt_callback: return CONTINUE with hash"); return SMFIS_CONTINUE; @@ -406,8 +579,11 @@ elsif (! $pid) { &die("couldn't restrict permissions") if (! &restrict_permissions($hash->{'rcpt'}, 1));; - exec('bogofilter', '-u', '-d', $dir) || - &die("exec(bogofilter): $!\n"); + my(@cmd) = ('bogofilter', '-u', '-d', $dir); + if ($bogofilter_cf && -f "$dir/$bogofilter_cf") { + push(@cmd, '-c', "$dir/$bogofilter_cf"); + } + exec(@cmd) || &die("exec(bogofilter): $!\n"); # &die had better not return! } @@ -442,7 +618,9 @@ } $ctx->addheader('X-Bogosity', 'Spam, tests=bogofilter'); my $from = $ctx->getsymval('{mail_addr}'); - syslog('info', '%s', ($training ? "would reject" : "rejecting") . + my $which = &reject_or_discard($hash); + my($verb) = ($which == SMFIS_REJECT) ? "reject" : "discard"; + syslog('info', '%s', ($training ? "would $verb" : "${verb}ing") . " likely spam from $from to " . $hash->{'rcpt'} . " based on $dir"); if (! $training) { my($archive, $link); @@ -524,6 +702,7 @@ while (<$fh>) { s/\r\n/\n/; + s/^From />From /; if (! print(MBOX $_)) { syslog('warning', '%s', "write($archive): $!"); goto close_archive; @@ -549,7 +728,7 @@ permissions_already_unrestricted: $ctx->setreply($rcode, $xcode, $reject_message); $ctx->setpriv(undef); - return SMFIS_REJECT; + return $which; } } else { @@ -575,6 +754,23 @@ return SMFIS_CONTINUE; } +sub my_close_callback { + my($ctx) = shift; + my $hash = $ctx->getpriv(); + + &debuglog("my_close_callback: entering with " . Data::Dumper->Dump([$hash], [qw(hash)])); + + if ($hash) { + if ($hash->{'fn'}) { + unlink $hash->{'fn'}; + } + } + + $ctx->setpriv(undef); + &debuglog("my_close_callback: returning CONTINUE with undef"); + return SMFIS_CONTINUE; +} + sub filtered_dir { my($uid, $gid, $dir) = &expand_recipient($_[0]); $dir; @@ -638,6 +834,10 @@ } } + if ($database_user) { + $rcpt = $database_user; + } + if (defined($cached_recipients{$rcpt})) { return(@{$cached_recipients{$rcpt}}); } @@ -666,9 +866,14 @@ $pw ? ($pw->uid, $pw->gid, undef, $now, $stripped) : (undef, undef, undef, $now, undef); if ($pw && $pw->dir && &restrict_permissions($orig) && - -d ($dir = $pw->dir . "/.bogofilter")) { + -d ($dir = $pw->dir . "/.bogofilter") && + ! ($bogofilter_cf && $require_cf && ! -f "$dir/$bogofilter_cf")) { $cached_recipients{$orig}->[2] = $dir; } + elsif ($database_user) { + syslog("warning", "Shared database user %s is not configured " . + "properly for bogofilter", $database_user); + } &unrestrict_permissions; return(@{$cached_recipients{$orig}}); } @@ -713,9 +918,18 @@ } } +sub opendb_read { + tie(%ip_whitelist_db, "DB_File", $ip_whitelist_db, O_RDONLY, 0, $DB_HASH) or &die("Can't open $ip_whitelist_db: $!"); +} + +sub closedb { + untie %ip_whitelist_db; +} + sub die { my(@msg) = @_; + &closedb; syslog('err', '%s', "@msg"); exit(1); } @@ -723,3 +937,113 @@ sub debuglog { syslog('debug', "DEBUG: " . join("", @_)); } + +my(%mx_cache); + +sub reject_or_discard { + my($hash) = @_; + my $hostname; + + foreach my $i (0..@discard_control-1) { + my($pattern, $action) = @{$discard_control[$i]}; + my $ret; + if ($action =~ /^reject$/i) { + $ret = SMFIS_REJECT; + } + elsif ($action =~ /^discard$/i) { + $ret = SMFIS_DISCARD; + } + else { + &die("Invalid action $action ", + "for discard control pttern $pattern\n"); + } + if ($pattern =~ /^addr:(.*)$/i) { + my $addr = $1; + &die("Invalid IP address in discard control pattern $pattern\n") + if ($addr !~ /^\d+\.\d+\.\d+\.\d+$/); + if ($hash->{'ipaddr'} eq $addr) { + &debuglog("reject_or_discard: addr match $addr: $action"); + return $ret; + } + } + elsif ($pattern =~ /^netblock:(.*)$/i) { + my $netblock = $1; + &die("Invalid netblock in discard control pattern $pattern\n") + if ($netblock !~ /^\d+\.\d+\.\d+\.\d+\/\d+$/); + if (Net::CIDR::cidrlookup($hash->{'ipaddr'}, $netblock)) { + &debuglog("reject_or_discard: netblock match ", + "$hash->{ipaddr} in $netblock: $action"); + return $ret; + } + } + elsif ($pattern =~ /^host:(.*)$/i) { + my $match_host = lc $1; + $hostname = lc gethostbyaddr(inet_aton($hash->{ipaddr}), AF_INET) + if (! $hostname); + if ($match_host eq $hostname) { + &debuglog("reject_or_discard: ", + "host match $hostname for $hash->{ipaddr}: ", + "$action and cache"); + splice(@discard_control, $i, 0, + [ "addr:$hash->{ipaddr}", $action ]); + return $ret; + } + } + elsif ($pattern =~ /^domain:(.*)$/i) { + my $match_domain = lc $1; + $hostname = lc gethostbyaddr(inet_aton($hash->{ipaddr}), AF_INET) + if (! $hostname); + if ($match_domain eq $hostname or + (substr($hostname, -1-length($match_domain)) eq + ".$match_domain")) { + &debuglog("reject_or_discard: domain match ", + "$hostname for $hash->{ipaddr} in $match_domain: ", + "$action and cache"); + splice(@discard_control, $i, 0, + [ "addr:$hash->{ipaddr}", $action ]); + return $ret; + } + } + elsif ($pattern =~ /^mx$/i) { + my $mx_domain = lc $hash->{'envrcpt'}; + if (! $mx_domain) { + &debuglog("reject_or_discard: no envrcpt\n"); + next; + } + $mx_domain =~ s/.*\@(.*[^\>])\>?/$1/; + my %mx_ips; + if ($mx_cache{$mx_domain} and + # refetch MX records once per hour + time - $mx_cache{$mx_domain}->[0] < 60 * 60) { + %mx_ips = %{$mx_cache{$mx_domain}->[1]}; + } + else { + my %mx_ips; + foreach my $mx (mx($mx_domain)) { + my($name, $aliases, $addrtype, $length, @addrs) = + gethostbyname($mx->exchange); + foreach my $addr (@addrs) { + $mx_ips{inet_ntoa($addr)} = 1; + } + } + $mx_cache{$mx_domain} = [time, \%mx_ips]; + &debuglog("reject_or_discard: cached MX IPs ", + join(" ", sort keys %mx_ips), + " for domain $mx_domain"); + } + if ($mx_ips{$hash->{'ipaddr'}}) { + &debuglog("reject_or_discard: MX addr match ", + "$hash->{ipaddr} for domain $mx_domain: $action"); + return $ret; + } + } + elsif ($pattern eq "*") { + return $ret; + } + else { + &die("Unrecognized discard control pattern: $pattern"); + } + } + + return SMFIS_REJECT; +} This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <re...@us...> - 2008-05-04 12:13:45
|
Revision: 6723 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6723&view=rev Author: relson Date: 2008-05-04 05:13:44 -0700 (Sun, 04 May 2008) Log Message: ----------- Update for 1.1.7 Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/configure.ac Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-04-30 22:12:08 UTC (rev 6722) +++ trunk/bogofilter/NEWS 2008-05-04 12:13:44 UTC (rev 6723) @@ -15,6 +15,8 @@ ------------------------------------------------------------------------------- +1.1.7 2008-05-04 (released) + 2008-04-30 * Updated sendmail milter contrib/bogofilter-milter.pl to v1.45 Modified: trunk/bogofilter/configure.ac =================================================================== --- trunk/bogofilter/configure.ac 2008-04-30 22:12:08 UTC (rev 6722) +++ trunk/bogofilter/configure.ac 2008-05-04 12:13:44 UTC (rev 6723) @@ -17,7 +17,7 @@ dnl part of the bogofilter source). dnl ******************************************************** dnl -AC_INIT([bogofilter],[1.1.6]) +AC_INIT([bogofilter],[1.1.7]) dnl AC_PREREQ(2.59) AC_CONFIG_SRCDIR([src/bogofilter.c]) This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-06-04 19:08:26
|
Revision: 6732 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6732&view=rev Author: m-a Date: 2008-06-04 12:08:31 -0700 (Wed, 04 Jun 2008) Log Message: ----------- Add .gitignore files. Added Paths: ----------- trunk/bogofilter/.gitignore trunk/bogofilter/src/.gitignore Added: trunk/bogofilter/.gitignore =================================================================== --- trunk/bogofilter/.gitignore (rev 0) +++ trunk/bogofilter/.gitignore 2008-06-04 19:08:31 UTC (rev 6732) @@ -0,0 +1,31 @@ +*~ +.deps +.rsyncs +ChangeLog +Makefile +Makefile.in +aclocal.m4 +autogen.log +autom4te.cache +bogofilter-*.tar.gz +bogofilter.pdf +bogofilter.spec +bogolexer.pdf +bogoutil.pdf +build* +compile +config.cache +config.guess +config.in +config.log +config.status +config.sub +configtest +configure +cscope.out +depcomp +doxygen +install-sh +missing +mkinstalldirs +ylwrap Added: trunk/bogofilter/src/.gitignore =================================================================== --- trunk/bogofilter/src/.gitignore (rev 0) +++ trunk/bogofilter/src/.gitignore 2008-06-04 19:08:31 UTC (rev 6732) @@ -0,0 +1,5 @@ +bf_compact +bf_copy +bf_tar +config.h +stamp-h1 This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-06-18 15:32:23
|
Revision: 6734 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6734&view=rev Author: m-a Date: 2008-06-18 08:32:26 -0700 (Wed, 18 Jun 2008) Log Message: ----------- Drop mysetvbuf.?, associated tests and calls. Obsolete. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/programmer/RISC-OS/Makefile trunk/bogofilter/src/Makefile.am trunk/bogofilter/src/passthrough.c Removed Paths: ------------- trunk/bogofilter/src/mysetvbuf.c trunk/bogofilter/src/mysetvbuf.h Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/NEWS 2008-06-18 15:32:26 UTC (rev 6734) @@ -15,6 +15,10 @@ ------------------------------------------------------------------------------- + * Drop support for systems that reverse setvbuf arguments. The last + systems to do that are reported to be shipped in 1987 by the autoconf + manual, so ditch them. + 1.1.7 2008-05-04 (released) 2008-04-30 Modified: trunk/bogofilter/doc/programmer/RISC-OS/Makefile =================================================================== --- trunk/bogofilter/doc/programmer/RISC-OS/Makefile 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/doc/programmer/RISC-OS/Makefile 2008-06-18 15:32:26 UTC (rev 6734) @@ -95,7 +95,6 @@ o.mime \ o.msgcounts \ o.mxcat \ - o.mysetvbuf \ o.passthrough \ o.paths \ o.prob \ Modified: trunk/bogofilter/src/Makefile.am =================================================================== --- trunk/bogofilter/src/Makefile.am 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/src/Makefile.am 2008-06-18 15:32:26 UTC (rev 6734) @@ -150,8 +150,7 @@ xstrlcpy.h xstrlcpy.c \ $(MEMDEBUG_SRC) \ xatox.h xatof.c xatoi.c \ - datastore_db.h \ - mysetvbuf.c mysetvbuf.h + datastore_db.h libbogofilter_a_SOURCES = \ version.c \ Deleted: trunk/bogofilter/src/mysetvbuf.c =================================================================== --- trunk/bogofilter/src/mysetvbuf.c 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/src/mysetvbuf.c 2008-06-18 15:32:26 UTC (rev 6734) @@ -1,21 +0,0 @@ -/* - * mysetvbuf.c - * (C) 2002 by Matthias Andree - * LICENSE: GNU LESSER GENERAL PUBLIC LICENSE - * - * offer a consistent wrapper around broken setvbuf calls - */ - -#include "config.h" -#include <stdio.h> - -#include "mysetvbuf.h" - -int mysetvbuf(FILE *stream, char *buf, int mode, size_t size) -{ -#ifdef SETVBUF_REVERSED - return setvbuf(stream, mode, buf, size); -#else - return setvbuf(stream, buf, mode, size); -#endif -} Deleted: trunk/bogofilter/src/mysetvbuf.h =================================================================== --- trunk/bogofilter/src/mysetvbuf.h 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/src/mysetvbuf.h 2008-06-18 15:32:26 UTC (rev 6734) @@ -1,10 +0,0 @@ -#ifndef MYSETVBUF_H -#define MYSETVBUF_H - -#include <stddef.h> -#include <stdio.h> - -/* from mysetvbuf.c */ -int mysetvbuf(FILE *stream, char *buf, int mode, size_t size); - -#endif Modified: trunk/bogofilter/src/passthrough.c =================================================================== --- trunk/bogofilter/src/passthrough.c 2008-06-11 11:15:34 UTC (rev 6733) +++ trunk/bogofilter/src/passthrough.c 2008-06-18 15:32:26 UTC (rev 6734) @@ -24,7 +24,6 @@ #include "format.h" #include "textblock.h" #include "xmalloc.h" -#include "mysetvbuf.h" #include "lexer.h" /* need have_body */ @@ -365,7 +364,7 @@ /* if we're not in passthrough mode, set line buffered mode just in * case some program that calls uses waits for our output in -T mode */ if (!passthrough) { - mysetvbuf(fpo, NULL, _IOLBF, BUFSIZ); + setvbuf(fpo, NULL, _IOLBF, BUFSIZ); } } This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <m-...@us...> - 2008-07-01 08:47:18
|
Revision: 6735 http://bogofilter.svn.sourceforge.net/bogofilter/?rev=6735&view=rev Author: m-a Date: 2008-07-01 01:47:26 -0700 (Tue, 01 Jul 2008) Log Message: ----------- Update doc/integrating-with-postfix. The script adds -G to sendmail command line (gateway submission, defeats some local expansions that don't make sense for external mail), and we suggest flags=Rq (added q) as argument to pipe in master.cf. Modified Paths: -------------- trunk/bogofilter/NEWS trunk/bogofilter/doc/integrating-with-postfix Modified: trunk/bogofilter/NEWS =================================================================== --- trunk/bogofilter/NEWS 2008-06-18 15:32:26 UTC (rev 6734) +++ trunk/bogofilter/NEWS 2008-07-01 08:47:26 UTC (rev 6735) @@ -15,6 +15,12 @@ ------------------------------------------------------------------------------- + * Update doc/integrating-with-postfix: the script now suggests sendmail + -G -i (where -G will be ignored by Postfix before 2.3) to tell + Postfix it's a gateway submission, not an original injection; the + filter pipe(8) magic for master.cf now suggests flags=Rq (was + flags=R), as per Postfix's FILTER_README. + * Drop support for systems that reverse setvbuf arguments. The last systems to do that are reported to be shipped in 1987 by the autoconf manual, so ditch them. Modified: trunk/bogofilter/doc/integrating-with-postfix =================================================================== --- trunk/bogofilter/doc/integrating-with-postfix 2008-06-18 15:32:26 UTC (rev 6734) +++ trunk/bogofilter/doc/integrating-with-postfix 2008-07-01 08:47:26 UTC (rev 6735) @@ -49,7 +49,12 @@ # messages truncated at the first period that is alone on a line # (which can happen with several kinds of messages, particularly # quoted-printable) - POSTFIX="/usr/sbin/sendmail -i" + # -G is ignored before Postfix 2.3 and tells it that the message + # does not originate on the local system (Gateway submission), + # so Postfix avoids some of the local expansions that can leave + # misleading traces in headers, such as local address + # canonicalizations. + POSTFIX="/usr/sbin/sendmail -G -i" export BOGOFILTER_DIR=/home/bogofilter # Exit codes from <sysexits.h> @@ -88,7 +93,7 @@ At the end of the file, add the following two lines: filter unix - n n - - pipe - flags=R user=filter argv=/home/bogofilter/postfix-filter.sh -f ${sender} -- ${recipient} + flags=Rq user=filter argv=/home/bogofilter/postfix-filter.sh -f ${sender} -- ${recipient} - Now, every incoming message will have the header line This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |