[Persistentperl-users] too many perperl_backend processes problem

[Milt E. <mep...@ui...>]

Hi.  We're using PersistentPerl for some CGI scripts and twice in the
last week we've run into a problem where there were hundreds (close to a
thousand) perperl_backend processes running and some of the CGI
invocations were failing (with Internal Server Errors).  I was hoping
someone here might be able to give suggestions as to what could be going
wrong and/or even leads as to what to investigate, because so far I
haven't found anything.

Here's some more info:

- We're using version 2.22, and I note that the Changes file indicates a
  fix (in version 2.21) for what sounds like the problem we're running
  into.  It mentions temp-file corruption, and in fact the Apache
  error_log is reporting lots of temp-file corrruption errors.  The
  errors come in pairs, a "temp file is corrupt" followed immediately by
  a "Premature end of script headers".

- We actually have a cluster of three servers running these scripts --
  all essentially set up the same -- and the problem has shown up on two
  of them.

- A similar problem occurred a little over two years ago.  The root
  cause was ostensibly traced to some latency issues on another server
  that the scripts have to communicate with.  The servers were running
  in the interim without incident, and no such similar issues have been
  found that might explain the current incidents (the earlier latency
  issues have been addressed).  I could not find anything that has
  changed recently on the servers where the problem showed up.

- In both cases, a reboot was done and the problem cleared up.

- The scripts in question (actually, it looks like it's just one script
  that was involved) sets MaxRuns to 1.  It also sets MaxBackends, but
  it's being set via a call to the PersistentPerl's module's setopt()
  method, which according to the module documentation is not a valid way
  to set that option (it needs to be set via a command line option;
  we're not using the Apache module).  Addressing this might help
  prevent the problem, but it doesn't explain why it occurred.

Thanks for any help you can provide.

Milt Epstein
Research Programmer
Integration and Software Engineering (ISE)
Campus Information Technologies and Educational Services (CITES)
University of Illinois at Urbana-Champaign (UIUC)
mep...@ui...