[chiPHPug-discuss] Long Running Batch Job

[Jason R. <ja...@ho...>]

Hey gang,

   Below is the code I used to illustrate taking an apache daemon off to 
do what people often do in cron jobs. There are dangers in this method 
but also a few benefits as well, least of which is leveraging existing 
code base and memcache/DB/FS connection and resource pools already 
designed into your app (and not duplicating them for CLI cron purposes).

   You can also still trigger this execution via cron with a very simple 
curl call.

   I quickly hacked together the "locking" method for Named jobs which 
obviously only works on single or dedicated cron machine, but I figured 
I'd add it to illustrate what I was talking about regarding coordination 
of long running jobs.  You should replace it with perhaps a call to a 
centralized memcache or DB store if you wanted to do locking across 
multiple machines.  Doing all that right was to complex to add into 
simple example.

   Also keep in mind the balancing you have to do with memory if you 
allow the job to up its mem limit to something high.  As a refresher, a 
typical tuned apache/mysql config might be like:

4gb RAM on server
8mb PHP memlimit + 2mb apache module overhead = 10mb/per process
normal OS running 256mb = 3,840mb available

apache MaxClients = 384
(assuming you've tweaked apache headers prioir to recompile to allow 
 >256 on linux)

Now if you use one of these daemons to do a job that takes 120mb memory 
to run AND you get hit with large spike of heavy traffic you could send 
your web server into thrashing state (allocated mem > RAM -> swap to 
disk)...  So I might tune apache to accommodate this model by dropping 
MaxClients down to 373 (384 - (120mb / 10mb per daemon - 1) )..

Anyways, I'm sure I've got errors in the above but you get the idea..

Here is code:
-------------------

<?php
function LogIt($msg=null){
  print $msg;
  return TRUE;
}
 
//---------------------------------------------------------------------------
  // BEGIN StartBatchJob
  // NEED TO ADD NAMED JOBS, LOCKING, RECOVER, SEPARATE LOGGING
 
//---------------------------------------------------------------------------
  public function StartBatchJob($Name='',$MemLimit=NULL){
   $MeFile=__FILE__;
   $MeClass=__CLASS__;
   $MeFunction=__FUNCTION__;
   $Me="$MeFile:$MeClass:$MeFunction";

   // QUICK HACK FOR LOCKING BATCH JOB
   if($Name){
    $LockDir="/tmp/$Name";
    $PID=posix_getpid();
    if(!mkdir($LockDir,0770,TRUE)){
     $Files=scandir($LockDir);
     $FoundPid='';
     if($Files){
      foreach($Files as $File){if($File!='.'&&$File!='..'){
       $FoundPid=$File;
      }}
     }
     if($FoundPid){
      // CHECK IF PID RUNNING
      if(file_exists("/proc/$FoundPid")){
       print "\n<br>\nBATCH JOB STILL RUNNING\n<br>\n";
       LogIt("$Me BATCH JOB STILL RUNNING $LockDir");
       return FALSE;
      }else{
       print "\n<br>\n PREVIOUS  BATCH JOB DEAD, CLEARING LOCK\n<br>\n";
       LogIt("$Me PREVIOUS BATCH JOB DEAD, CLEARING LOCK $LockDir");
       unlink("$LockDir/$FoundPid");
      }
     }
    }
    touch("$LockDir/$PID");
   }

   // FLUSH CONTENT AND LET CLIENT DISCONNECT
   ini_set('zlib.output_compression','0');
   ignore_user_abort(1);
   header('Content-type: text/html; charset=ISO-8859-1');
   header('Connection: close');
   $size = ob_get_length();
   header("Content-Length: $size");
   while(ob_get_length()>0){
    ob_end_flush();
    ob_flush();
    flush();
   }

   // SET EXECUTION PARAMETERS AND RETURN
   ini_set('max_execution_time','0');
   ini_set('child_terminate','1');
   if($MemLimit!=NULL){ini_set('memory_limit', $MemLimit);}
   set_time_limit(0);
   return TRUE;

   // DEFAULT ERROR
   LogIt("$Me DEFAULT ERROR. SHOULD NOT BE HERE");
   return FALSE;
  } // END StartBatchJob


 
//---------------------------------------------------------------------------
  // BEGIN EndBatchJob
  // NEED TO ADD NAMED JOBS, LOCKING, RECOVER, SEPARATE LOGGING
 
//---------------------------------------------------------------------------
  public function EndBatchJob($Name=''){
   $MeFile=__FILE__;
   $MeClass=__CLASS__;
   $MeFunction=__FUNCTION__;
   $Me="$MeFile:$MeClass:$MeFunction";

   // HACK UNLOCK
   if($Name){
    $LockDir="/tmp/$Name";
    if(is_dir($LockDir)){
     $Files=scandir($LockDir);
     if($Files){
      foreach($Files as $File){
       if($File!='.'&&$File!='..'){unlink("$LockDir/$File");}
      }
     }
     rmdir($LockDir);
    }
   }

   // TELL APACHE TO KILL THIS DAEMON FOR MEMORY CLEANUP
   ini_set('child_terminate','1');
   apache_child_terminate();
   exit(0);

   LogIt("$Me DEFAULT ERROR. SHOULD NOT BE HERE");
   return FALSE;
  } // END StartBatchJob

?>
<html>
<head>
<body><center>
<h1>Batch Job Runner Example</h1>
<?php
$var='';
if(isset($_REQUEST['Job'])){$Job=$_REQUEST['Job'];}
$PID=posix_getpid();
print "$Job will be run by daemon $PID\n<br>\n";
?>
</center></body></html>
<?php
if($Job){
  if($HTTPTrans->StartBatchJob($Job)){
   // DO LONG RUNNING WORK
   sleep(10);
   $HTTPTrans->EndBatchJob($Job);
  }
}
?>