MOre Notify patch for 1.1.0
Status: Beta
Brought to you by:
bboy
Menu
▾
▴
#16 MOre Notify patch for 1.1.0
This is a major mega patch to 1.1.0 - it changes about 20% of the code and critically affects the task scheduler.
This is from a posting to the users list about these changes -
I am a long way into rewriting the task scheduler for the changes needed for notification on start up and periodic tasks (e.g. garbage collection) which is resulting in another mega patch. IT looks to me that this should be a new release of the mydns code. How do I get access to the CVS store to publish new data.
The changes I am undertaking include:
* Separate scheduling queues for 'High Priority Tasks ', 'UDP query tasks - Normal tasks', 'IO driven tasks - e.g. TCP Queries, Notifies, ...', 'Time driven tasks e.g. garbage collection, stats reporting, ...', 'Low priority Tasks'.
* Clean shutdown - I have removed the shutdown code from the signal handler and now catch children dying a bit more reliably.
* Handle would block events cleanly - allowing more threads to run.
* Send Notifies on start at a measured rate so that all of the resources of MYDNS does not get consumed by its own actions - Notify -> SOA request -> IXFR request -> AXFR request.
* Added PTR updates. Works buts needs some attention to make it completely clean.
* Release TCP sockets under error conditions - there is a number of leakage paths in the current code where TCP sockets can get consumed.
I am currently looking at the task data structure to ensure it is memory/time efficient. This will mean most of the src/mydns files will have to change.
I am also looking to add the following:
* IXFR responses so that AXFR's can be reduced in use.
* Limiting number of concurrent AXFR's by placing AXFR requests on the LOW priority queue - once forked they will run at full priority
* Starting replacement slaves when they die. The 1.1.0 release caused the master to give up and leave any other slaves running, the current patches I have just collect the slave but keep on running in the master.
* Add code in the slaves to die if the master does without informing the slave.
* Limit the rate of creation of notify tasks at start up to keep the number of file descriptors used to a reasonable level - a system with say 50,000 zones would blow the number of descriptors rapidly if the slaves servers are a distance away, or are busy or are down. Current Linux kernels are built with 1024 FD's and I propose to limit the notify pool to 10% of this at any time. In a local area network environment the master process will queue about 6 notify tasks when debug is running.
* Add facilities to reply to Notify messages if received - will just ignore them for now as I am not interested in adding slave facilities to MYDNS at this point.
* Dynamic creation of slaves if the backlogs within the master or current slaves get too large. This needs a master/slave communication mechanism which may mean the master is finally just that and does not process any queries.
* Optimise the MYSQL interface to use pre-compiled queries as well as ensuring that the query interface can recover from network partitions and database crashes - eventually I would like to support multiple instances of MYSQL/POSTGRESQL backends so that the system can cope with the loss of a Database engine. I will publish some thoughts on this later.
All of the code I am developing is running on i386 architecture 2.6.2x kernels (Fedora 7+) using MYSQL only and thus will need to be validated on different distributions and different OS/architectures and Postgresql. I will publish the current set of changes as an incremental patch to my other patches, mega patch to 1.1.0 and as a tarball (1.2.0?) if anybody wants to try them out. Also, if there are any 'C'/Linux programmers out there who would like to read my somewhat quirky code style and suggested improvements or additions I would be grateful for any comments.
A word of warning, the debug additions in this patch are extensive. I will tidy them up and make them more generally available as I get the other parts of the system cleaned up.
My IXFR model will require a change to the schema of the database as well as code. I will publish a conversion routine probably written in PERL or if simple enough in SHELL for an MYSQL database. Again I will need someone to verify the Postgresql equivalent.
Incremental patch to 1.1.0 for more notify changes etc.