#399 Ease into revived DB

Slash 2.5/3.0
MySQL (16)

When a DB in the dbs table switches from isalive no to yes, it
pretty much gets slammed by the whole website, as every process
notices in short order that it's available again.

MySQL typically will be slow to respond at first, but then as the
incoming queries load indexes into RAM and fall into the query
cache, it gets up to full speed.

This morning I got reader28 back up and running, and let the db-
angel turn its isalive back on. Once it did, the DB went from
inactivity to suddenly being wedged on a number of different
queries. It took about 20-30 seconds for that initial crush of
queries to complete (some finished earlier than others). But once
that first boggage was cleared, the DB had everything cached and
was running fine.

I wonder if we couldn't implement a layer of weighting over the
existing weight parameter, such that for the first minute a DB
comes back alive, it is fed a gradually increasing amount of traffic,
instead of slamming it with its fair share all at once.

The code could keep track of when a DB came back alive. When
the check for aliveness is re-done, all DBs that were not-alive less
than 60 seconds ago could have their weights adjusted. So if it
came alive t seconds ago and its weight in the dbs table is w, then:
w = w*(t/60) if t < 60

Of course, on the flip side, when a DB goes down, it should go
down instantly, that behavior shouldn't change.


  • Chris Nandor

    Chris Nandor - 2004-08-13

    Logged In: YES

    Check it out, lemme know any thoughts you have.

  • Chris Nandor

    Chris Nandor - 2004-08-13
    • assigned_to: pudge --> jamiemccarthy
  • Jamie McCarthy

    Jamie McCarthy - 2005-10-31

    Logged In: YES

    Er. I just saw this. Or if I saw it a year ago I don't remember. Doh.
    Check what out? Did you fix this?

    I'm removing the duplicate issue <https://sourceforge.net/tracker/?
    func=detail&aid=1080158&group_id=4421&atid=354421> which had
    these thoughts:

    The behavior I just observed is that a DB whose isalive is set to
    'no' still has its weight_adjust adjusted by balance_readers, which
    is not what I'd expect (I thought my code would have it just be
    ignored, guess not).

    What it does now is return a dead DB's weight_adjust to 1.0.

    What it should do it return it to a preset value like 0.1. That way,
    when the isalive is set back to 'yes', the DB doesn't suddenly get
    slammed. And once it is 'yes' again, balance_readers.pl can start
    raising its value again (assuming it handles the small load fine,
    isn't bogged, etc.)

  • Jamie McCarthy

    Jamie McCarthy - 2005-11-04

    Logged In: YES

    Oh I see, pudge just meant he was assigning it to me. Never mind. I'll look
    into this...


Log in to post a comment.