Menu
▾
▴
Re: [Assp-user] HMM-Check has given less than 6 results - using monitoring mode only
|
From: Thomas E. <Tho...@th...> - 2016-02-02 04:48:50
|
>Jan-28-16 09:28:22 Spam Weight : 3,264,527 >Jan-28-16 09:28:22 Not-Spam Weight: 3,265,258 >Jan-28-16 09:28:22 Corpus norm: 0.9998 - (very good - balanced) >Jan-28-16 09:28:22 Corpus confidence: 0.06250000 Corpus confidence: 0.06250000 - this value is impossible (expected is 1.000) if -> Corpus norm: 0.9998 - (very good - balanced). I think your Berkeley-DB ENV or DB is damaged for some or all BDB files - but at least for HMMdb. - shutdown assp - remove all files (__*.* , *.bdb) from assp/tmpDB/HMMdb - do the same for spamdb - remove assp/hmmdb.bdb and assp/spamdb.bdb - start assp - import any avalable backup for both DB's - or run a rebuildspamdb - restart assp to force a recalculation of the used BDB cache If you use any *nix - KEEP in MIND! Your init.d script for assp (stop case) has to wait until assp has been finished - otherwise your BerkeleyDB files WILL BE DESTROYED! To be clear - I mean 'WILL BE DESTROYED' - not 'may be' or 'possibly' Some damaging of BDB files is fixed by an assp internal BDB repair mechanism - some , not all! Thomas Von: Dossy Shiobara <do...@pa...> An: ass...@li... Datum: 28.01.2016 17:17 Betreff: [Assp-user] HMM-Check has given less than 6 results - using monitoring mode only Recently, it seems my HMM and Bayes checks are no longer working? In mail log, I see: "HMM-Check has given less than 6 results - using monitoring mode only" I'll include my latest rebuildrun.txt, which looks like it ran successfully. Why is this happening? I'm running ASSP 2.4.7(16004). Also, it seems like if I get this error, it doesn't even perform Bayesian scoring -- basically, spam that was previously being blocked is now being let through... ---rebuildrun.txt--- Jan-28-16 09:05:00 RebuildSpamDB-thread rebuildspamdb-version 7.26 started in ASSP version 2.4.7(16004) Jan-28-16 09:05:00 RebuildSpamDB uses BerkeleyDB for temporary hashes Jan-28-16 09:05:00 RebuildSpamDB uses BerkeleyDB-ENV with 62.50 MByte Jan-28-16 09:05:00 RebuildSpamDB will create a Hidden Markov Model Jan-28-16 09:05:00 RebuildSpamDB will create unicode enabled databases Jan-28-16 09:05:00 RebuildSpamDB will process all words as Sequence of UAX #29 Grapheme Clusters Jan-28-16 09:05:00 RebuildSpamDB will normalize unicode characters Jan-28-16 09:05:00 RebuildSpamDB will use the ASSP_WordStem engine Jan-28-16 09:05:00 ---ASSP Settings--- Jan-28-16 09:05:00 Do Not Collect Messages with RedListed address: Enabled **Messages with RedListed addresses will be removed from the corpus!** Jan-28-16 09:05:00 Do Not Collect RedRe Messages: Enabled **Messages matching the RedRe will be removed from the corpus!** Jan-28-16 09:05:00 Use Subject as Maillog Names: True Jan-28-16 09:05:00 Maxbytes: 4,000 Jan-28-16 09:05:00 RebuildFileTimeLimit: 1 5 Jan-28-16 09:05:00 RebuildFileTimeLimit: files will be moved away from the corpus if their processing takes longer than 5 second(s) Jan-28-16 09:05:00 /data/assp/errors/spam Jan-28-16 09:05:00 File Count: 11 Jan-28-16 09:05:00 Processing... errors/spam with 11 files Jan-28-16 09:05:00 ignore and remove files older than Sep-11-88 10:05:00 in folder errors/spam Jan-28-16 09:05:01 Imported Files for HeloBlackList: 10 Jan-28-16 09:05:01 Imported Files for Bayes/HMM: 10 Jan-28-16 09:05:01 Finished in 1 second(s) Jan-28-16 09:05:01 /data/assp/errors/notspam Jan-28-16 09:05:01 File Count: 1 Jan-28-16 09:05:01 Processing... errors/notspam with 1 files Jan-28-16 09:05:01 ignore and remove files older than Sep-11-88 10:05:01 in folder errors/notspam Jan-28-16 09:05:01 Imported Files for HeloBlackList: 0 Jan-28-16 09:05:01 Imported Files for Bayes/HMM: 0 Jan-28-16 09:05:01 Finished in 1 second(s) Jan-28-16 09:05:01 info: corpusnorm after processing errors/spam and errors/notspam is Spam Weight: 8280 / Not-Spam Weight: 0 => norm: 10.000 Jan-28-16 09:05:01 info: require approx. 6,726 files (3,255,584 words) from folder spam to get the wanted corpusnorm (1.000) Jan-28-16 09:05:01 /data/assp/spam Jan-28-16 09:05:01 File Count: 11,195 Jan-28-16 09:05:01 Processing... spam with 11,195 files Jan-28-16 09:05:01 ignore and remove files older than Dec-28-15 09:05:01 in folder spam Jan-28-16 09:15:31 Removed Old: 5 Jan-28-16 09:15:31 Imported Files for HeloBlackList: 11,190 Jan-28-16 09:15:31 Imported Files for Bayes/HMM: 6,672 Jan-28-16 09:15:31 Finished in 630 second(s) Jan-28-16 09:15:31 info: require approx. all files (3,264,527 words) from folder notspam to get the wanted corpusnorm (1.000) Jan-28-16 09:15:31 /data/assp/notspam Jan-28-16 09:15:31 File Count: 7,009 Jan-28-16 09:15:31 Processing... notspam with 7,009 files Jan-28-16 09:15:31 ignore and remove files older than Dec-28-15 09:15:31 in folder notspam Jan-28-16 09:25:53 Removed Old: 7 Jan-28-16 09:25:53 Imported Files for HeloBlackList: 7,002 Jan-28-16 09:25:53 Imported Files for Bayes/HMM: 6,992 Jan-28-16 09:25:53 Finished in 622 second(s) Jan-28-16 09:25:53 Generating weighted Bayesian tuplets Jan-28-16 09:26:10 populating Spamdb 503166 records - Bayesian check is now disabled Jan-28-16 09:26:23 done - populating Spamdb records - 503166 - Bayesian check is now enabled Jan-28-16 09:26:23 done - Generating weighted Bayesian tuplets Jan-28-16 09:26:23 Bayesian Pairs: 503,166 now in list Jan-28-16 09:26:23 Generating consolidated Hidden-Markov-Model database from 3,772,337 record model Jan-28-16 09:28:22 HMM sequences: 1,848,357 now in list Jan-28-16 09:28:22 generating Spamdb.helo records from 7,502 collected HELO's Jan-28-16 09:28:22 cleaning old Spamdb.helo records Jan-28-16 09:28:22 done - cleaning old Spamdb.helo records Jan-28-16 09:28:22 HELO Blacklist: 4 new, 0 now in list Jan-28-16 09:28:22 Spam Weight : 3,264,527 Jan-28-16 09:28:22 Not-Spam Weight: 3,265,258 Jan-28-16 09:28:22 Corpus norm: 0.9998 - (very good - balanced) Jan-28-16 09:28:22 Corpus confidence: 0.06250000 Jan-28-16 09:28:27 Start populating Hidden Markov Model. HMM-check is disabled for this time! Jan-28-16 09:28:27 start populating Hidden Markov Model with 1,848,357 records! Jan-28-16 09:28:59 Finished populating Hidden Markov Model with 1,848,357 records! Jan-28-16 09:28:59 Finished populating Hidden Markov Model. HMM-check is now enabled again! Jan-28-16 09:28:59 Total processing time: 1,439 second(s) Jan-28-16 09:28:59 Total processing data: 118.85 MByte Jan-28-16 09:28:59 Rebuild processed 14.52 files per second. Jan-28-16 09:28:59 After finishing the Rebuild process, the /data/assp/tmpDB folder contains 791.45 MByte. Jan-28-16 09:28:59 After finishing the Rebuild process, the drive that contains the /data/assp/tmpDB folder has 1.22 GByte free space from total 1.90 GByte. Jan-28-16 09:28:59 building new GripList records and bounce report Jan-28-16 09:28:59 processing Logfile /data/assp/logs/maillog.txt Jan-28-16 09:28:59 processing Logfile /data/assp/logs/16-01-27.maillog.txt Jan-28-16 09:29:01 processing Logfile /data/assp/logs/16-01-26.maillog.txt Jan-28-16 09:29:02 processing Logfile /data/assp/logs/16-01-25.maillog.txt Jan-28-16 09:29:03 processing Logfile /data/assp/logs/16-01-24.maillog.txt Jan-28-16 09:29:03 processing Logfile /data/assp/logs/16-01-23.maillog.txt Jan-28-16 09:29:03 skipping bounce report because 'DoNotCollectBounces' is switched ON Jan-28-16 09:29:03 Uploading Griplist via Direct Connection Jan-28-16 09:29:04 Submitted 6,924 bytes: 0 IPv6 addresses, 768 IPv4 addresses Jan-28-16 09:29:04 Trashlist was saved to /data/assp/trashlist.db -- Dossy Shiobara | "He realized the fastest way to change do...@pa... | is to laugh at your own folly -- then you http://panoptic.com/ | can let go and quickly move on." (p. 70) * WordPress * jQuery * MySQL * Security * Business Continuity * ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 _______________________________________________ Assp-user mailing list Ass...@li... https://lists.sourceforge.net/lists/listinfo/assp-user DISCLAIMER: ******************************************************* This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ******************************************************* |
