SQLite backend?

2005-02-21
2013-03-21
  • Dieter Shirley
    Dieter Shirley
    2005-02-21

    How much interest would there be for a patch to nget that used SQLite as the backend for article storage?  My machine has only 256M of RAM but since it's only used as a file/download server, this is plenty.... for everything EXCEPT nget; it wants >300M for some groups, even with my reduced memory patches.

    Pros:

    - significant reduction of caching code which should end up being much easier to maintain
    - huge decrease in memory footprint
    - a potentially significant speedup when dealing with large newsgroups (and newsgroups are getting bigger all the time...)

    Cons:

    - this would either increase the distribution size (by including sqlite statically), or add a dependency
    - the cache files would be much larger; I estimate 10x larger than compressed (this could be mitigated with gzip at startup/shutdown, but not efficiently)

    It's possible I'll make that change locally, just for myself, but I'm more likely to do it if others show interest.

    It should be possible to clean up cache.h so that both solutions could co-exist, chosen by a ./configure switch, but I'm not really keen on doing ALL that myself.  If someone else were to undertake cleaning up cache.h to completely encapsulate the implementation, I'd be willing to commit to doing the SQLite rework.  Any takers?

     
    • Travis
      Travis
      2005-03-01

      I am very interested in it and have been thinking about it for some time now. though, the option to use other SQL's would be ideal, (mysql, postgresql, etc). It would allow alot of options for sorts and queries from external scripts

      maybe make a generic interface so different people can make a module for a specific sql lib.

      I would also be interested in helping with this option (testing, coding, etc)

       
    • I guess it might be helpful to consider this issue together with threaded downloads. If the application is split into a header-download instance, one or more download instances and a decoder-instance, then they could communicate using the shared database.

       
      • Travis
        Travis
        2005-03-18

        I have been thinking about that too. I guess the first area to work on is a good schema for the db. I'll try to work on some ideas myself as time permits. I believe  this single feature will make the frontend developers and scripters go crazy... or maybe just me...  lol