Re: [Nagios-devel] RFC/RFP Nagios command workers
Nagios network monitoring software is enterprise server monitoring
Brought to you by:
egalstad,
sawolf-nagios
From: Andreas E. <ae...@op...> - 2011-06-29 08:51:46
|
On 06/28/2011 05:13 PM, Matthieu Kermagoret wrote: > Hi list, > > First of all, sorry for the delayed response, last month was pretty > crazy at work :-p > > On Mon, May 23, 2011 at 12:38 PM, Andreas Ericsson<ae...@op...> wrote: >> On 05/23/2011 11:37 AM, Matthieu Kermagoret wrote: >> Because shipping an official module that does it would mean not only >> supporting the old complexity, but also the new one. Having a single >> default system for running checks would definitely be preferrable to >> supporting multiple ones. >> > > I agree with you when you say that a single system is better than two. > However I fear that the worker system would need very more code than a > simpler system (and less code usually means less bugs) and that the > worker system would destabilize Nagios. Quite the opposite, really. The amount of backflips we're doing right now to make sure the core is threadsafe is huge, so it's likely this patch will even reduce the LoC count in Nagios. > For years it's been Nagios' > development team's policy not to include features that could be > written as modules. I liked it that way. > Everything can be written as modules. The worker process thing will have the nice sideeffect that modules can register sockets that core Nagios will listen to events from, with a special callback when there's data available on the socket. This reduces complexity of a lot of modules by a fair bit. With worker-processes instead of multiple threads it's also trivial to write modules with regards to thread-safety, and potential leaks in worker modules (such as embedded perl) can be ignored, since we can just kill the worker process and spawn a new one once it's done some arbitrary number of checks. This is how Apache handles leaky modules and we could do far worse than using the world's most popular webserver as an example. There's also another thing. Mozilla Firefox has been accused of feature stagnation in the core since they let addon writers handle adding new features, and far from everybody uses modules. Google Chrome has taken a fair share of users from Firefox lately, partly because it implements some of the more popular modules directly in-core. Nagios has also been accused of feature stagnation, even though broker module development has flourished in recent years (nagios with modules is nothing like the old nagios without them), so it makes sense to add certain selected module capabilities to the core. >>> 1) Remove the multiple fork system to execute a command. The Nagios >>> Core process forks directly the process that will exec the command >>> (more or less sh's parsing of command line, don't really know if this >>> could/should be integreted in the Core). >>> >> >> This really can't be done without using multiple threads since the >> core can't wait() for input and children while at the same time >> issuing select() calls to multiplex the new output of currently >> running checks. >> > > What about a signal handler on SIGCHLD that would wait() terminated > process and a select() on pipe FDs connected to child processes, with > a timeout to kill non-responding checks ? > Highly impractical for shortlived children and with so many pipes to listen to. It would mean we'd be iterating over the entire childstack several hundred times per second just to read new output. We're forced to do that, since pipes can't contain an infinite amount of data. The child's write() call will fail when the pipe is full and the children won't exit while waiting to write. Doing so many select() calls means the scheduler will suffer greatly, along with modules that wish to run code in the main thread every now and then. With sockets, we can let each worker handle a smaller number of checks at the time, and since they have no scheduling responsibilities the master process is free to just await new input. >>> 2) The root process and the subprocess are connected with a pipe() so >>> that the command output can be fetched by reading the pipe. Nagios >>> will maintain a list of currently running commands. >>> >> >> Pipes are limited in that they only guarantee 512 bytes of atomic >> writes and reads. TCP sockets don't have this problem. There's also > > It is my understanding of Posix that the core standard defines a > 512-byte minimal limit for atomic I/O operations but I cannot find any > section enforcing atomicity on I/O operations on TCP sockets, so pipes > would be better indeed. Were you refering to the XSI Streams or could > you point me to the appropriate section ? > No. TCP sockets don't enforce atomicity beyond the 512 bytes already specified, but they do enforce ordering, which pipes don't. This is actually a real problem (although an unusual one) when several processes tries to write data to Nagios' command pipe and one of them writes more than the atomic limit on whatever system it's being written on. The fact that pipes use fixed-size buffers for pipes (requiring a full kernel recompile to change) and the fact that a program can change the size of its socket buffers with a simple system call makes sockets an even bigger winner. >> the fact that a lot of modules already use sockets, so we can get >> rid of a lot of code in those modules and let them re-use Nagios' >> main select() loop and get inbound events on "their" sockets as a >> broker callback event. Much neater that way. >> > > A pretty API would definitely be great, no doubt. > >>> 3) The event loop will multiplex processes' I/O and process them as necessary. >>> >> >> That's what the worker processes will do and then feed the results >> back to the nagios core through the sequential socket, which will >> guarantee read and write operations large enough to never truncate >> any of the data necessary for the master process to do proper book- >> keeping. >> > > I'm not very fond of the "large buffer" approach because I'm not > really sure that I/O operations are atomic (see above). > They needn't be atomic so long as they don't stall the writer or the reader and maintain ordering. TCP ensures ordering and select() or ioctl() calls ensure non-stalling behaviour. UDP sockets would have more or less the same problem as pipes, but TCP avoids them completely at the expense of slightly more overhead. For local TCP operations the overhead is comparable to one set operation and one if() statement inside the kernel code and doesn't even require a context switch. >>> The worker system could still be implemented and used only by users >>> who need it (but that's what DNX and mod_gearman do). I believe it is >>> better to leave the default command execution system as simple as it >>> is right now (but improve it) and leave distribution algorithms to >>> modules. I can imagine multiple reasons for which one would want to >>> distribute checks among workers : >>> >> >> The only direction we can improve it is to remove it and rebuild it. >> Removing one fork() call would mean implementing the multiplexer >> anyway, and once that's done we're 95% done with the worker process >> code anyways. >> > > I guess that you shouldn't be far from a complete worker system but > (again :-) ) writing it as a module wouldn't be difficult either. > Not very, no. I've got some proof-of-concept code that does pretty well. One of the best things with the worker-process thing is that one can run modules or even external programs to execute the checks in place of in-core code. That will serve as an excellent testing ground for new algorithms without requiring huge patches to Nagios. I digress though, but letting modules handle core functionality would imo be a very good idea indeed. That way we can let hackers test new core functionality as modules and the most efficient one gets implemented in the core to become default eventually. Organic software development with all the benefits of evolution. Since each subsystem should be more or less self-contained and dependant only on a few core API's anyway, there's no reason replacing one should even be difficult. >>> So in fact you plan removing the old FIFO and doing all stuffs through >>> the socket ? What about acknowledgements or downtimes ? Could they be >>> sent through the socket too or would there be another system ? >>> >> >> They could be sent through the socket, but for a while at least we'll >> have to support both pipe and socket, so addon developers have some >> time to adjust to the new world order. >> > > All right, thanks for the explanations. A status returned by Nagios > after any external command execution would be a nice feature indeed. > Yes, it's one of the most liked features of mk-livestatus. -- Andreas Ericsson and...@op... OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace. |