SpamAssassin

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-25 10:58 -------
Subject: Re: [SAdev]  msgcount should be msgs learned, not processed

On Fri, Apr 25, 2003 at 09:56:44AM -0700, bugzilla-daemon@... wrote:
> Thoughts?  I think it's a really great idea, but I'd like to hear what other
> folks think.

Oh, in short, this idea takes the best features of Dan's "use time_t"
proposal (to determine which tokens to expire), and the best features
of the current message count method (choosing when to expire).

Yea. :)




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From quinlan@...  2003-04-26 01:03 -------
I really like the proposal.

One nit.  I believe that:

> - rename bayes_expiry_min_db_size to be more representative of what it does:
> bayes_expiry_token_count

is and will be very non-intuitive to users (and developers too).  I'd prefer if
we stuck to having configuration variables that are meaningful and useful.  I
really doubt that token_count is meaningful to anyone except the most hard-code
developer and even then, do you really care about this setting or just maximum
disk size?  So, can we just get rid of the variable or, failing that, find
a way to make it more meaningful.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From jm@...  2003-04-26 16:00 -------
sounds good.  I agree with Dan BTW, min_db_size is more usable for end-users and
admins, so stick with that nomenclature.

I'd like to test it with a bayes-10pcv run, but they're hard to set up, and I'm
firefighting at work at the mo so I won't have time to do it.   However, it
sounds good to me intuitively.

BTW why keep a "learn count"?  AFAICS, using time_t means we don't have to count
that.  Am I missing something?



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-26 17:17 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Sat, Apr 26, 2003 at 04:00:58PM -0700, bugzilla-daemon@... wrote:
> sounds good.  I agree with Dan BTW, min_db_size is more usable for end-users and
> admins, so stick with that nomenclature.

Ok, whatever.  I just thought it didn't make sense the way it was named.
But that's not the important part of the suggestion. ;)

> BTW why keep a "learn count"?  AFAICS, using time_t means we don't have to count
> that.  Am I missing something?

Well, it keeps the best part of a message count (figuring out when to
do the expire) and the best part of using time_t (figuring out which
tokens to keep when doing the expire).

The current problem (and the issue when we were trying to figure out
count vs. atime) was that we were single minded -- either count or atime
to do both when and which.  By merging the two together we should get
better operation. :)

A learn count, BTW, would also let us get rid of the msgcount file since
it's really used as a journal for msgcount when scanning (which should
probably just go in the bayes journal to begin with ...)





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From jm@...  2003-04-26 19:49 -------
btw this is planned for 2.6x right?  I reckon we could get a 2.6x release ready
pretty soon, but if we change the expiry/etc. scheme in 2.5x, we'll wind up
spending lots of time supporting that instead.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-26 20:06 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Sat, Apr 26, 2003 at 07:49:48PM -0700, bugzilla-daemon@... wrote:
> btw this is planned for 2.6x right?  I reckon we could get a 2.6x release ready
> pretty soon, but if we change the expiry/etc. scheme in 2.5x, we'll wind up
> spending lots of time supporting that instead.

oh yeah, this is all 2.60.  I wouldn't make a major change to the bayes
format/etc in a stable release.


So the question is, besides that config option change that you guys
don't like, how does the rest sound?





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From jm@...  2003-04-26 20:17 -------
I'm still not sure a learn count is needed -- we could just do expiry when
ntokens goes over the threshold by a certain amount (let's say when ntokens >=
110% of bayes_expiry_min_db_size).  In that case, we don't need to count learn
ops -- expiry will just happen once sufficient learning has happened anyway...

All else I'm very happy with BTW, makes sense...



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From parkerm@...  2003-04-26 22:00 -------
Subject: Re: [SAdev]  msgcount should be msgs learned, not processed

On Sat, Apr 26, 2003 at 04:00:58PM -0700, bugzilla-daemon@... wrote:
> ------- Additional Comments From jm@...  2003-04-26 16:00 -------
> sounds good.  I agree with Dan BTW, min_db_size is more usable for end-users and
> admins, so stick with that nomenclature.

What does min_db_size mean in the SQL sense?

Michael





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-27 05:45 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Sat, Apr 26, 2003 at 10:00:10PM -0700, bugzilla-daemon@... wrote:
> What does min_db_size mean in the SQL sense?

In the current sense, it's talking about tokens, so it would have to be
the same in SQL land.  (SQL expires will be pretty sweet.  Something like
'SELECT atime FROM bayes_toks ORDER BY atime DESC LIMIT $min_db,$min_db',
if it doesn't return an error (ie: there are at least $min_db entries),
take the 1 returned atime and put into $min_atime, then 'DELETE FROM
bayes_toks WHERE atime < $min_atime'.  Then you can do simple INSERTs
to update oldest token and token count.)





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-27 13:58 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Sat, Apr 26, 2003 at 08:17:54PM -0700, bugzilla-daemon@... wrote:
> I'm still not sure a learn count is needed -- we could just do expiry when
> ntokens goes over the threshold by a certain amount (let's say when ntokens >=
> 110% of bayes_expiry_min_db_size).  In that case, we don't need to count learn
> ops -- expiry will just happen once sufficient learning has happened anyway...

Well, yeah, I guess it depends what would make more sense.  Message count
has the possibility of having more tokens before an expire.  Token count
means the DB will stay smaller.

I still like the idea of message count, but will go either way.  We should
decide on the format, figure out if there is anything else we'd like
to do, then implement.  I've already implemented one new DB format,
and we're discussing another.  I don't want to do a third pre-2.60. ;)

> All else I'm very happy with BTW, makes sense...

Yea. :)





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821

felicity@... changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |spamassassin-
                   |                            |devel@...
         AssignedTo|spamassassin-               |felicity@...
                   |devel@... |



------- Additional Comments From felicity@...  2003-04-28 17:57 -------
> I'm still not sure a learn count is needed -- we could just do expiry when
> ntokens goes over the threshold by a certain amount (let's say when ntokens >=
> 110% of bayes_expiry_min_db_size).  In that case, we don't need to count learn
> ops -- expiry will just happen once sufficient learning has happened anyway...

So I was working on this today.  It's doable, but there's one issue I hit: when
doing an expire, how do we figure out which tokens to expire?  Currently we go
through in DB order, and check current_count - 5000 vs atime, and expire the
ones that are older.

Of course, keeping learn_count doesn't solve this either.  Still churning on it.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From graham@...  2003-04-28 23:01 -------
I can think of 2 solutions.

1) Always expire back to the database size (eg 10000 tokens) 

2) Merge back part of the original Bayes mechanism and make any tokens with an
atime older than (now - min_token_age) (or whatever the parameter was called, I
don't remember) a candidate for expiry.



------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From quinlan@...  2003-04-29 00:24 -------
Theo, I have an idea stemming from our earlier discussion:

When deciding how many tokens to expire, why not do this:

  - after an expire is done, store the following values:
      1. the change in the disk usage of the DB (from before to after the
         expire)
      2. the atime delta (newest token purged - oldest token purged)
      3. the atime of the oldest token retained
  
  - the next time you want to expire, it's an easy calculation to estimate
    the new atime delta based on #1 and #2 and then using #3, you know what
    limit to use.  If the DB expired 500k of data last time and needs to
    expire 750k of data this time, then just multiply the delta by 1.5.

This should achieve the single pass goal as well as allowing "each" to be used.
I would also suggest going with Justin's idea to aim for expiring back to 75%
(or whatever) of the max_size.  If it's not exact, no big deal, but it should
be pretty close once it gets going since it's basing the calculation directly
on the last expire.




------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-30 09:00 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Mon, Apr 28, 2003 at 11:01:21PM -0700, bugzilla-daemon@... wrote:
> 1) Always expire back to the database size (eg 10000 tokens) 

We're already doing that.  The issue is figuring out which tokens to expire.

> 2) Merge back part of the original Bayes mechanism and make any tokens with an
> atime older than (now - min_token_age) (or whatever the parameter was called, I
> don't remember) a candidate for expiry.

We could do that, but it doesn't work out so well with atime of time_t in
18hr chunks.  Do you specify that any tokens older than 6 months expire?





------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-04-30 09:53 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Tue, Apr 29, 2003 at 12:24:07AM -0700, bugzilla-daemon@... wrote:
> When deciding how many tokens to expire, why not do this:
> 
>   - after an expire is done, store the following values:
>       1. the change in the disk usage of the DB (from before to after the
>          expire)
>       2. the atime delta (newest token purged - oldest token purged)
>       3. the atime of the oldest token retained
>   
>   - the next time you want to expire, it's an easy calculation to estimate
>     the new atime delta based on #1 and #2 and then using #3, you know what
>     limit to use.  If the DB expired 500k of data last time and needs to
>     expire 750k of data this time, then just multiply the delta by 1.5.

Well, #2 is a chicken and egg problem: you need to do an expire to figure
out when to do the first expire.  And how do we know the size of what
needs to be expired this time?  You can guestimate by saying the average
datasize is token_length + 11, but that really isn't true (it could also
be token_length + 3 depending on if it's a "small use" token).

I was thinking of something like storing oldest and newest atime values.
When we need to expire (ntoks > min_db_size * 140%) we simply do something
like "$expire_atime = ($newest-$oldest) / 140%" and come out with the
approximate atime to expire before.

However, if someone in a single 18hr stretch learns enough tokens to
expire, all the atimes will be the same and it won't be able to expire
any of them.





------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From quinlan@...  2003-04-30 15:11 -------
> Well, #2 is a chicken and egg problem: you need to do an expire to figure
> out when to do the first expire.  And how do we know the size of what
> needs to be expired this time?

You know when to do the first expire because you exceed the maximum size.

To determine which time delta to use for the first expire, you can walk
the database.  The extra walk is only needed for the first expire.

> You can guestimate by saying the average
> datasize is token_length + 11, but that really isn't true (it could also
> be token_length + 3 depending on if it's a "small use" token).

A guess would be fine for the first expire, but walking would also do the
job.

> I was thinking of something like storing oldest and newest atime values.
> When we need to expire (ntoks > min_db_size * 140%) we simply do something
> like "$expire_atime = ($newest-$oldest) / 140%" and come out with the
> approximate atime to expire before.

I guess that might work, but I'd rather use something size-based as the
limit rather than a percentage of the age differences.

> However, if someone in a single 18hr stretch learns enough tokens to
> expire, all the atimes will be the same and it won't be able to expire
> any of them.

If the atimes are all the same, you just have to expire based on count
or randomly, I think.  (Whichever works best, I feel like we're in danger
of slipping far behind the spambayes people in terms of methodology.  I don't
really care what seems clean or not; I want us to use something fast and
something accurate.)




------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

http://www.hughes-family.org/bugzilla/show_bug.cgi?id=1821





------- Additional Comments From felicity@...  2003-05-04 17:42 -------
Subject: Re:  msgcount should be msgs learned, not processed

On Wed, Apr 30, 2003 at 03:11:32PM -0700, bugzilla-daemon@... wrote:
> You know when to do the first expire because you exceed the maximum size.

Actually, I've been thinking about this ...

When the expire occurs, why not try expiring all tokens < last expire
time.  We can then (if necessary) reprieve tokens back to min_db_size.

I've also been thinking that instead of min_db_size, we should have a
db_size_limits (min,max) option.





------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.