Is it possible to limit the stored headers?

  • Booji

    Booji - 2005-03-04

    I would really like to limit the number of headers that are saved in the cache files to only a couple of days.  My news server has quite a bit of retention, but I don't want to keep all of them as I monitor a large set of groups.

    Is there anyway to reduce this?  The ability to discard anything over a certain amount of time would be ideal at least for my situation.

    I looked at the source briefly, but my c++ skills are not that good.  It looks like there are a couple of defines that might be the start of this, but maybe it's not all hooked up yet?

    Any help, even a pointer where I could try and change this would be much appreciated.


    • Matthew Mueller

      Matthew Mueller - 2005-05-12

      CVS now has a maxheaders option that allows you to limit based upon number of headers.

      • Booji

        Booji - 2005-05-28

        Is it a lot harder to limit based on age?

        I really only need to keep a couple of days worth of headers, but I don't want to try and guess how many headers that is.  In some groups it will be a lot and otehrs very little.

        This will still help, but age would be ideal.


    • Frederick Bruckman

      That still hasn't shown up on anonymous cvs, nor on CVSweb.

      • Matthew Mueller

        Matthew Mueller - 2005-05-16

        Strange, there isn't any relevant cvs issues in the site status.  I submitted a SF support request about it, so hopefully they'll get the anon cvs updating again soon.

    • TH

      TH - 2005-05-23

      Could you possibly zip up the files and post them? Easynews seem to have stopped expiring articles entirely, and now have more than 8M articles in some groups. It is killing my server.

    • Frederick Bruckman

      Yes, it's a lot harder. The protocol supports a simple way to do it -- the NEWNEWS command, but essentially zero news servers support that.  News servers all have an overview file for each group, which contains selected headers for each message, in message ID (arrival) order.  You can get this as fast as you can ftp a file. Any other request is a lot more work for a server, so it's going to be a lot slower, if it even supports it at all.

      I think the main reason folks have been asking to limit stored headers is because nget is running out of memory trying to handle many millions of headers. (That was my reason.) The memory used depends on the number of headers stored, so a simple number is useful here.

      If your host has enough memory to handle all the headers your sever serves up, just mark all articles read after you've looked them over, so you'll only see the new articles.

    • Booji

      Booji - 2005-05-29

      Right.  I don't need it when getting the headers.  It is the saved cache files that kill me.

      I can use a header count to limit, but for some groups saving 1 million headers is a waste where others that may only be a day or two. 

      My scripts run daily.  I only need to keep a couple of days worth of headers to account for a day of network outages.  That is why a limit on age when saving the cache is ideal.  This way I will have a consistent view across all groups, where a strict header count will not get me that, and may actually not give me the entire view for a couple of days if I make it too low for some groups.


Log in to post a comment.