#15 DBD::RAM per-machine comment cache

closed
Rob Malda
None
2
2003-08-26
2002-01-04
Jamie McCarthy
No

With a few caveats, I think krow's plan of installing a DBD::AnyData daemon server on each webserver, and using that per-machine process as the central caching place for all comments, is a good one. It'll be a bit of work and it's pretty grungy work too; all this is is writing a wrapper around the module and unix-domain sockets to provide a server.

One of the interesting questions is where and how to cache comments. This server has to be forking, since we can't bottleneck an entire machine's requests. But children forked off which learn new comment texts can't store that new data back in the parent. The best solution might be for the parent at startup to just snag the last week's worth of comments and then, every few minutes, request from the server all comment_text WHERE cid > $current_max_known_cid. Many requests of its children will then have to hit the DB for the same handful of comments over and over, but that'll be a pretty light load.

Obviously the apache children will have to gracefully handle the server's being down.

Another interesting question is whether this should be started up with "init.d/slash start" along with slashd, or should it be hooked into the "apachectl start" script somehow? I lean toward the latter but there's no good mechanism provided with apache for this and we have no good mechanism as yet for patching "apachectl".

The main advantage of this system will be dramatically reducing the RAM load on our 9 webservers. Speed will probably stay about the same, and I wouldn't be stunned to see it actually get a bit slower. But that will be more than made up for by having httpds that don't keep getting killed off when they bloat to 45 MB.

Discussion

  • Jamie McCarthy
    Jamie McCarthy
    2002-06-21

    Logged In: YES
    user_id=3889

    More recently, we've been thinking about installing a mysqld
    on each webserver that replicates the comments,comment_text
    tables. That might entirely eliminate the need for comment
    caching by the httpd children. The question is whether that
    would save more RAM than it uses.

     
  • Jamie McCarthy
    Jamie McCarthy
    2002-06-21

    • milestone: --> 191917
     
  • Jamie McCarthy
    Jamie McCarthy
    2003-02-04

    • labels: 310780 -->
     
  • Jamie McCarthy
    Jamie McCarthy
    2003-02-04

    Logged In: YES
    user_id=3889

    Recent data have suggested that this wouldn't do much good
    at all. Might be fun to try it someday and see what happens,
    though.

     
  • Jamie McCarthy
    Jamie McCarthy
    2003-02-04

    • assigned_to: jamiemccarthy --> nobody
    • priority: 6 --> 2
    • milestone: 191917 -->
     
  • Jamie McCarthy
    Jamie McCarthy
    2003-08-25

    Logged In: YES
    user_id=3889

    Memcached kicks ass all over this idea.

     
  • Jamie McCarthy
    Jamie McCarthy
    2003-08-25

    • assigned_to: nobody --> cmdrtaco
     
  • Rob Malda
    Rob Malda
    2003-08-26

    • status: open --> closed