[Fez-developers] Inserting large files

[Jauslin K. <kai...@li...>]

Hello Fez developers,

I have a PHP offline script inserting and synchronizing MODS objects =
into the Fez index. Unfortunately, there seems to be a problem with =
large files (videos, >300MB). The problem is in Misc::processURL with =
curl_exec. PHP reports an out of memory error trying to allocate 320MB, =
even though the memory limit is set to 1.5GB. PHP version 5.2.4.

To work around this problem, I made a small change to class.reindex and =
class.misc. That way, CURL will write the file if needed directly to =
disk (without using any PHP memory). The code does exactly the same as =
before, but will write directly for managed datastreams in =
indexFezFedora objects. My first impression is that this is now much =
faster than before.

Any comments are appreciated.

Cheers, Kai


-- class.misc:
function processURL($url,$passthru=3Dfalse, $filehandle=3Dnull) {
	    if (empty($url)) { return ""; }
	    $url=3Dstr_replace('&','&',$url);
	    $ch=3Dcurl_init();
	    Logger::debug("CURL_URL=3D".$url);
	    curl_setopt($ch, CURLOPT_URL, $url);
	    	   =20
	    if ($filehandle !=3D null) {	    		    	    =09
	    	curl_setopt($ch, CURLOPT_FILE, $filehandle);
	    	curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
	   		curl_setopt($ch, CURLOPT_BUFFERSIZE, 64000);
	    } else {
		    if (!$passthru) {		    =09
				curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
			}=09
	    }
	   =20
	    if (APP_HTTPS_CURL_CHECK_CERT =3D=3D "OFF")  {
          curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
          curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
        }
	=09
        $data =3D curl_exec ($ch);
	    if ($data) {
		  $info =3D curl_getinfo($ch);=20
	      curl_close ($ch);
		} else {
			$info =3D array();
			Error_Handler::logError(curl_error($ch)." ".$url,__FILE__,__LINE__);
			curl_close ($ch);
		}	  =20
 	    return array($data,$info); =20
	}

-- class.reindex (in function indexFezFedoraObjects)

$handle =3D fopen(APP_TEMP_DIR.$new_dsID, "w");                    =20
$urlReturn =3D Misc::ProcessURL($urldata, false, $handle);               =
        =20
//fwrite($handle, $urlReturn[0]);
fclose($handle);


--=20
Kai Jauslin, Dipl. Informatik-Ing. ETH, ETH Z=FCrich, ETH-Bibliothek, =
R=E4mistrasse 101, CH-8092 Z=FCrich
kai...@li..., Tel +41-44-6324972, B=FCro STB F19