sqlgrey-users

[Sqlgrey-users] Problem with count-subroutines

[Michael S. <Mic...@lr...>]

Hi Lionel and Dan,

since every record in connect and the AWLs has an (implizit)
expiration/purge-date, all routines, which check if a record is in one of
the tables, construct the select-statement in a way that expired records
are not found:

last_seen > $self->past_tstamp($self->{sqlgrey}{awl_age},'DAY')

However, this is not the case with all of the count subroutines, though
they are semantically and syntactically nearly identical.

Correct are:

- is_in_from_awl
- is_in_domain_awl
- in_connect
- recently_in_connect

Wrong are:

- count_from_awl
- count_src_domain_awl
- count_src_from_awl
- count_src_connect

Maybe correct, even without checking expiration ???:

- get_first_seen_in_from_awl
- get_reconnect_delay


Michael Storz
--
======================================================
Leibniz-Rechenzentrum      |    <mailto:St...@lr...>
Boltzmannstr. 1            |    Fax: +49 89 35831-9700
85748 Garching / Germany   |    Tel: +49 89 35831-8840
======================================================

Re: [Sqlgrey-users] Problem with count-subroutines

[Dan F. <da...@ha...>]

Michael Storz wrote:
> last_seen > $self->past_tstamp($self->{sqlgrey}{awl_age},'DAY')
>
> However, this is not the case with all of the count subroutines, though
> they are semantically and syntactically nearly identical.
>   
How on earth did you stumble upon this? ;) Ive just looked through the 
source and i see what you mean, but i dont see it causing any problems. 
Or am i wrong?
It would be "cleaner" if this is fixed, so ill put it on my TODO list..

- Dan

Re: [Sqlgrey-users] Problem with count-subroutines

[Michael S. <Mic...@lr...>]

On Thu, 15 Feb 2007, Dan Faerch wrote:

> Michael Storz wrote:
> > last_seen > $self->past_tstamp($self->{sqlgrey}{awl_age},'DAY')
> >
> > However, this is not the case with all of the count subroutines, though
> > they are semantically and syntactically nearly identical.
> >
> How on earth did you stumble upon this? ;) Ive just looked through the

Through code analysis ;)

> source and i see what you mean, but i dont see it causing any problems.
> Or am i wrong?

It is normally not a severe problem. But imagine the case of throttling.
If you cleanup your database only once every 24 hours, e.g midnight, then
you can have the case, that throttling for an IP address is done 24 hours
longer than it should.

Actually the problem was, that I got inconsistent results between using
memcached and the SQL database.

> It would be "cleaner" if this is fixed, so ill put it on my TODO list..
>
> - Dan

Michael Storz
--
======================================================
Leibniz-Rechenzentrum      |    <mailto:St...@lr...>
Boltzmannstr. 1            |    Fax: +49 89 35831-9700
85748 Garching / Germany   |    Tel: +49 89 35831-8840
======================================================

Re: [Sqlgrey-users] Problem with count-subroutines

[Dan F. <da...@ha...>]

Michael Storz wrote:
> you can have the case, that throttling for an IP address is done 24 hours
> longer than it should.
>
>   
I see what you mean..
> Actually the problem was, that I got inconsistent results between using
> memcached and the SQL database.
>
>   
Which is why i asked how you stumbled upon this, as i couldnt really 
imagine what obscure situation that had occurred for you to notice this ;)..

It definitely should be fixed in next release. If you've already fixed 
this id be really glad to receive a patch ;).

- Dan

Re: [Sqlgrey-users] Problem with count-subroutines

[Michael S. <Mic...@lr...>]

On Fri, 16 Feb 2007, Dan Faerch wrote:

> Michael Storz wrote:
> > you can have the case, that throttling for an IP address is done 24 hours
> > longer than it should.
> >
> >
> I see what you mean..
> > Actually the problem was, that I got inconsistent results between using
> > memcached and the SQL database.
> >
> >
> Which is why i asked how you stumbled upon this, as i couldnt really
> imagine what obscure situation that had occurred for you to notice this ;)..
>
> It definitely should be fixed in next release. If you've already fixed
> this id be really glad to receive a patch ;).
>

Hi Dan,

did you get my patches, I sent you off-list?

Michael Storz
--
======================================================
Leibniz-Rechenzentrum      |    <mailto:St...@lr...>
Boltzmannstr. 1            |    Fax: +49 89 35831-9700
85748 Garching / Germany   |    Tel: +49 89 35831-8840
======================================================

Re: [Sqlgrey-users] Problem with count-subroutines

[Dan F. <da...@ha...>]

Michael Storz wrote:
> On Fri, 16 Feb 2007, Dan Faerch wrote:
>   
> Hi Dan,
>
> did you get my patches, I sent you off-list?
>   

You mean the 2 patches you sent to me Friday, right? Yes i did, thank 
you..  As you requested, ill look into testing the timing stuff on our 
cluster, i just need to find a "stale" moment at work to do this and 
this week definitely wont be it, as i'm back to work after a weeks 
vacation and everything is on fire ;)
I am also very interested in looking at the timings myself.

- Dan

Re: [Sqlgrey-users] Problem with count-subroutines

[Michael S. <Mic...@lr...>]

On Wed, 21 Feb 2007, Dan Faerch wrote:

> Michael Storz wrote:
> > On Fri, 16 Feb 2007, Dan Faerch wrote:
> >
> > Hi Dan,
> >
> > did you get my patches, I sent you off-list?
> >
>
> You mean the 2 patches you sent to me Friday, right? Yes i did, thank

correct.

> you..  As you requested, ill look into testing the timing stuff on our
> cluster, i just need to find a "stale" moment at work to do this and
> this week definitely wont be it, as i'm back to work after a weeks
> vacation and everything is on fire ;)
> I am also very interested in looking at the timings myself.

Great to hear. Hope you'll have time soon.

Michael Storz
--
======================================================
Leibniz-Rechenzentrum      |    <mailto:St...@lr...>
Boltzmannstr. 1            |    Fax: +49 89 35831-9700
85748 Garching / Germany   |    Tel: +49 89 35831-8840
======================================================