#32 evlog causes clients to crash or hang
Hi, in our case we have a client that writes to evlog via the client evlog library. We write a lot so to keep the logfile at a resonable size, we occasionally use logrotate to rotate the log files. After a rotation we restart evlogd because that appears to be the only way to tell it to create and continue writing to a new file. After this, we also restart the client so that it gets connected to the new evlogd.
Now, when stressing the system (in our case, we wrote 234 byte message every 0.5 milliseconds, and restarted evlogd every 7.2 seconds), we get into problems after running successfully about 8 hours or so. Under normal circumstances, when evlogd starts and forks a child for handling kernel events. This child connects and dies because we don't log kernel events. Thereafter our client connects and everything works fine. However, sometimes the following order can be observed: After starting evlogd, the client connects and logs some events, thereafter the forked child connects and dies as we do not log kernel events. In this case, the parent process gets a sigchild, and usually we are so unlucky that we get the sigchild when reading an event from the client. But the read doesn't handle interrupts, and so the evlogd closes the socket to the client and the clients dies.
We tried to solve the problem by changing all read and write of sockets in evlogd.c to handle interrupts and continue reading/writing when it happens. This did indeed help in the sense that evlogd doesn't any longer close the client socket when it gets a sigchild. However, now we could observe something else. Namely, after the sigchild was handled, the evlogd didn't write the single byte character back to the client that informs it that the event has been processed. This caused the client to get stucked in a read. We tried to change the evlog library code also so that it handles interrupted reads and writes. And we tried to not block all signals but just sigpipe. This didn't help.
What we are trying just now is to remove the forking all together. Probably it will work now. However, I suggest that the code of evlogd is rewritten to cope better with signals and heavy load on system. It might also be a good idea if evlogd was able to start a new log file without having to restart it. Perhaps it could do that when receiving a SIGHUP.
Regards,
drago.krznaric@se.transport.bombardier.com
