Memcached failure causes cascading database failures
Status: Beta
Brought to you by:
worden
Menu
▾
▴
#518 Memcached failure causes cascading database failures
Because when the background jobs code can't reach memcached, it sleeps 1 second and polls again, forever. So we accumulate more and more apache processes, waiting and polling. But keeping all these apache processes alive keeps database connections open, I think, so pretty soon you can't open any more database connections and all wiki pages start to fail.
Discussion
Anonymous
Solution: quit after polling 100 times. Done.
Actually let's quit after like 3 times, because nobody wants to wait 100 seconds. It would be best to have it return an error if it never gets the job listing. Right now it's just returning an empty list, which means in the browser the list of background jobs will temporarily vanish, and reappear later. So I think I'll reopen this ticket until I get that done right.
Looks like the cascade of failed db connections may have been actually caused somehow by a disk failure on one of the cluster nodes, but regardless it's good that I fixed this issue.