Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#27 write locks deny database reads

closed-fixed
nobody
None
5
2004-08-16
2002-12-13
David Saez
No

We are doing every day a long bogofilter -s/-n (about two consecutive hours a day)
when this process is finishing (when write locks are acquired on the database) all
bogofilter -p operations are put on hold until the locks are released. As we use
bogofilter -p in mail processing, exim just kills this waiting process (timed out).
When looking at datastore_db.c I noticed that locks are acquired with fcntl on
all the database, maybe it could be better to use Berkeley db own's locking (?)

Discussion

  • Logged In: YES
    user_id=2788

    I don't think the way the bogofilter data base access is currently implemented would profit from locking changes, because AFAICT, we only allow one writer exclusively, or any amount of readers.

    Gyepi is the DB expert, and he may comment if it would be necessary to switch to concurrent mode and comment how much an effort this would be. I forwarded this to him and the bogofilter-dev mailing list.

     
  • David Saez
    David Saez
    2002-12-14

    Logged In: YES
    user_id=268278

    the problem is that bogofilter does only one long write lock at the end to update the database,
    it will better to have many fast write locks (i.e. updating the database just after every message is
    processed) as this will allow lock reads between every message. Having write locks that last
    two hours and don't let any access to the database is not a good thing. This will also allow to
    use Berkeley DB locking (it only locks the read/written registries) instead of having to lock the
    whole database file, as I see on SleepyCat homepage, this is very easy to implement.

     
  • David Saez
    David Saez
    2002-12-14

    Logged In: YES
    user_id=268278

    further and better tests shown that write locks are hold for a small amount of
    time compared to the whole bogofilter process (about 3-4 minutes for a whole
    process of about 2 hours), so the problem is not as worse as I figure at the
    first impression. BTW, I've made a patch that allows to configure the maximum
    amount of processed words before database updated (will update the database
    as soon as this limit is reached). If you feel this could be useful I could upload it.

     
  • Logged In: YES
    user_id=2788

    Setting the environment variable BOGOFILTER_CONCURRENT_DATA_STORE
    before running bogofilter programs (perhaps in your ~/.profile?) should
    improve your situation in current bogofilter releases such as 0.92.4, it uses
    Berkeley DB's own locking to allow for concurrent access.

    Please let us know if this helps.

     
    • status: open --> closed-fixed