Re: [Assp-user] HMM-Check has given less than 6 results - using monitoring mode only

[Dossy S. <do...@pa...>]

Also: I have BayesAfterHMM blank, and the log doesn't show any Bayes
scoring happening (I have DoHMM and DoBayesian both set to "score").


On 1/28/16 10:48 AM, Dossy Shiobara wrote:
> Recently, it seems my HMM and Bayes checks are no longer working?  In
> mail log, I see:
>
> "HMM-Check has given less than 6 results - using monitoring mode only"
>
> I'll include my latest rebuildrun.txt, which looks like it ran successfully.
>
> Why is this happening?  I'm running ASSP 2.4.7(16004).  Also, it seems
> like if I get this error, it doesn't even perform Bayesian scoring --
> basically, spam that was previously being blocked is now being let
> through...
>
>
> ---rebuildrun.txt---
>
> Jan-28-16 09:05:00 RebuildSpamDB-thread rebuildspamdb-version 7.26
> started in ASSP version 2.4.7(16004)
>
> Jan-28-16 09:05:00 RebuildSpamDB uses BerkeleyDB for temporary hashes
>
> Jan-28-16 09:05:00 RebuildSpamDB uses BerkeleyDB-ENV with 62.50 MByte
>
> Jan-28-16 09:05:00 RebuildSpamDB will create a Hidden Markov Model
>
> Jan-28-16 09:05:00 RebuildSpamDB will create unicode enabled databases
>
> Jan-28-16 09:05:00 RebuildSpamDB will process all words as Sequence of
> UAX #29 Grapheme Clusters
>
> Jan-28-16 09:05:00 RebuildSpamDB will normalize unicode characters
>
> Jan-28-16 09:05:00 RebuildSpamDB will use the ASSP_WordStem engine
>
> Jan-28-16 09:05:00 ---ASSP Settings---
> Jan-28-16 09:05:00 Do Not Collect Messages with RedListed address: Enabled
> **Messages with RedListed addresses will be removed from the corpus!**
>
> Jan-28-16 09:05:00 Do Not Collect RedRe Messages: Enabled
> **Messages matching the RedRe will be removed from the corpus!**
>
> Jan-28-16 09:05:00 Use Subject as Maillog Names: True
> Jan-28-16 09:05:00 Maxbytes: 4,000
> Jan-28-16 09:05:00 RebuildFileTimeLimit: 1 5
> Jan-28-16 09:05:00 RebuildFileTimeLimit: files will be moved away from
> the corpus if their processing takes longer than 5 second(s)
>
> Jan-28-16 09:05:00 /data/assp/errors/spam
> Jan-28-16 09:05:00 File Count:  11
> Jan-28-16 09:05:00 Processing... errors/spam with 11 files
> Jan-28-16 09:05:00 ignore and remove files older than Sep-11-88 10:05:00
> in folder errors/spam
> Jan-28-16 09:05:01 Imported Files for HeloBlackList:    10
> Jan-28-16 09:05:01 Imported Files for Bayes/HMM:        10
> Jan-28-16 09:05:01 Finished in 1 second(s)
>
> Jan-28-16 09:05:01 /data/assp/errors/notspam
> Jan-28-16 09:05:01 File Count:  1
> Jan-28-16 09:05:01 Processing... errors/notspam with 1 files
> Jan-28-16 09:05:01 ignore and remove files older than Sep-11-88 10:05:01
> in folder errors/notspam
> Jan-28-16 09:05:01 Imported Files for HeloBlackList:    0
> Jan-28-16 09:05:01 Imported Files for Bayes/HMM:        0
> Jan-28-16 09:05:01 Finished in 1 second(s)
> Jan-28-16 09:05:01 info: corpusnorm after processing errors/spam and
> errors/notspam is Spam Weight: 8280 / Not-Spam Weight: 0 => norm: 10.000
> Jan-28-16 09:05:01 info: require approx. 6,726 files (3,255,584 words)
> from folder spam to get the wanted corpusnorm (1.000)
>
> Jan-28-16 09:05:01 /data/assp/spam
> Jan-28-16 09:05:01 File Count:  11,195
> Jan-28-16 09:05:01 Processing... spam with 11,195 files
> Jan-28-16 09:05:01 ignore and remove files older than Dec-28-15 09:05:01
> in folder spam
> Jan-28-16 09:15:31 Removed Old: 5
> Jan-28-16 09:15:31 Imported Files for HeloBlackList:    11,190
> Jan-28-16 09:15:31 Imported Files for Bayes/HMM:        6,672
> Jan-28-16 09:15:31 Finished in 630 second(s)
> Jan-28-16 09:15:31 info: require approx. all files (3,264,527 words)
> from folder notspam to get the wanted corpusnorm (1.000)
>
> Jan-28-16 09:15:31 /data/assp/notspam
> Jan-28-16 09:15:31 File Count:  7,009
> Jan-28-16 09:15:31 Processing... notspam with 7,009 files
> Jan-28-16 09:15:31 ignore and remove files older than Dec-28-15 09:15:31
> in folder notspam
> Jan-28-16 09:25:53 Removed Old: 7
> Jan-28-16 09:25:53 Imported Files for HeloBlackList:    7,002
> Jan-28-16 09:25:53 Imported Files for Bayes/HMM:        6,992
> Jan-28-16 09:25:53 Finished in 622 second(s)
>
> Jan-28-16 09:25:53 Generating weighted Bayesian tuplets
> Jan-28-16 09:26:10 populating Spamdb 503166 records - Bayesian check is
> now disabled
> Jan-28-16 09:26:23 done - populating Spamdb records - 503166 - Bayesian
> check is now enabled
> Jan-28-16 09:26:23 done - Generating weighted Bayesian tuplets
>
> Jan-28-16 09:26:23 Bayesian Pairs: 503,166 now in list
>
> Jan-28-16 09:26:23 Generating consolidated Hidden-Markov-Model database
> from 3,772,337 record model
> Jan-28-16 09:28:22 HMM sequences: 1,848,357 now in list
>
> Jan-28-16 09:28:22 generating Spamdb.helo records from 7,502 collected
> HELO's
> Jan-28-16 09:28:22 cleaning old Spamdb.helo records
> Jan-28-16 09:28:22 done - cleaning old Spamdb.helo records
>
> Jan-28-16 09:28:22 HELO Blacklist: 4 new, 0 now in list
>
> Jan-28-16 09:28:22 Spam Weight    :   3,264,527
> Jan-28-16 09:28:22 Not-Spam Weight:   3,265,258
>
> Jan-28-16 09:28:22 Corpus norm: 0.9998 - (very good - balanced)
> Jan-28-16 09:28:22 Corpus confidence:   0.06250000
>
> Jan-28-16 09:28:27 Start populating Hidden Markov Model. HMM-check is
> disabled for this time!
> Jan-28-16 09:28:27 start populating Hidden Markov Model with 1,848,357
> records!
> Jan-28-16 09:28:59 Finished populating Hidden Markov Model with
> 1,848,357 records!
> Jan-28-16 09:28:59 Finished populating Hidden Markov Model. HMM-check is
> now enabled again!
>
> Jan-28-16 09:28:59 Total processing time: 1,439 second(s)
>
> Jan-28-16 09:28:59 Total processing data: 118.85 MByte
>
>
> Jan-28-16 09:28:59 Rebuild processed 14.52 files per second.
>
> Jan-28-16 09:28:59 After finishing the Rebuild process, the
> /data/assp/tmpDB folder contains 791.45 MByte.
>
> Jan-28-16 09:28:59 After finishing the Rebuild process, the drive that
> contains the /data/assp/tmpDB folder has 1.22 GByte free space from
> total 1.90 GByte.
>
> Jan-28-16 09:28:59 building new GripList records and bounce report
> Jan-28-16 09:28:59 processing Logfile /data/assp/logs/maillog.txt
> Jan-28-16 09:28:59 processing Logfile /data/assp/logs/16-01-27.maillog.txt
> Jan-28-16 09:29:01 processing Logfile /data/assp/logs/16-01-26.maillog.txt
> Jan-28-16 09:29:02 processing Logfile /data/assp/logs/16-01-25.maillog.txt
> Jan-28-16 09:29:03 processing Logfile /data/assp/logs/16-01-24.maillog.txt
> Jan-28-16 09:29:03 processing Logfile /data/assp/logs/16-01-23.maillog.txt
>
> Jan-28-16 09:29:03 skipping bounce report because 'DoNotCollectBounces'
> is switched ON
>
> Jan-28-16 09:29:03 Uploading Griplist via Direct Connection
> Jan-28-16 09:29:04 Submitted 6,924 bytes: 0 IPv6 addresses, 768 IPv4
> addresses
>
> Jan-28-16 09:29:04 Trashlist was saved to /data/assp/trashlist.db
>

-- 
Dossy Shiobara         |      "He realized the fastest way to change
do...@pa...     |   is to laugh at your own folly -- then you
http://panoptic.com/   |   can let go and quickly move on." (p. 70) 
  * WordPress * jQuery * MySQL * Security * Business Continuity *