Re: [Fez-developers] Inserting large files

[Christiaan K. <c.k...@li...>]

As a side note it would be nice if we or someone else could integrate
something like this into Fez:
http://blog.joshuaeichorn.com/archives/2006/03/14/php-ajax-file-upload-progr
ess-meter-updates/

Christiaan


On 16/10/07 1:16 PM, "Lynette Rayle" <el...@cs...> wrote:

> 
> I'm seeing a similar size issue when adding a record through Fez.  The record
> includes an upload of a photoshop file that is 52M.  Smaller files (< 1M)
> upload very fast.  The large file spins for a long time and finally dies.
> The error message I see isn't helpful because it occurs when fez tries to
> access the object that was not successfully uploaded.
> 
> Rev 1066
> 
> Lynette
> 
> -----Original Message-----
> From: fez...@li...
> [mailto:fez...@li...] On Behalf Of Christiaan
> Kortekaas
> Sent: Monday, October 15, 2007 7:20 PM
> To: fez...@li...
> Subject: Re: [Fez-developers] Inserting large files
> 
> Hi Kai
> 
> That looks like a good solution. Curl is certainly a better way to go than
> fwrite for speed and efficiency. We'll test it out and add it to the trunk.
> 
> Thanks again!
> 
> Cheers,
> Christiaan
> 
> 
> On 16/10/07 2:20 AM, "Jauslin  Kai" <kai...@li...> wrote:
> 
>> Hello Fez developers,
>> 
>> I have a PHP offline script inserting and synchronizing MODS objects into
> the
>> Fez index. Unfortunately, there seems to be a problem with large files
>> (videos, >300MB). The problem is in Misc::processURL with curl_exec. PHP
>> reports an out of memory error trying to allocate 320MB, even though the
>> memory limit is set to 1.5GB. PHP version 5.2.4.
>> 
>> To work around this problem, I made a small change to class.reindex and
>> class.misc. That way, CURL will write the file if needed directly to disk
>> (without using any PHP memory). The code does exactly the same as before,
> but
>> will write directly for managed datastreams in indexFezFedora objects. My
>> first impression is that this is now much faster than before.
>> 
>> Any comments are appreciated.
>> 
>> Cheers, Kai
>> 
>> 
>> -- class.misc:
>> function processURL($url,$passthru=false, $filehandle=null) {
>>    if (empty($url)) { return ""; }
>>    $url=str_replace('&amp;','&',$url);
>>    $ch=curl_init();
>>    Logger::debug("CURL_URL=".$url);
>>    curl_setopt($ch, CURLOPT_URL, $url);
>>    
>>    if ($filehandle != null) {
>> curl_setopt($ch, CURLOPT_FILE, $filehandle);
>> curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
>> curl_setopt($ch, CURLOPT_BUFFERSIZE, 64000);
>>    } else {
>>    if (!$passthru) {
>> curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
>> } 
>>    }
>>    
>>    if (APP_HTTPS_CURL_CHECK_CERT == "OFF")  {
>>           curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
>>           curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
>>         }
>> 
>>         $data = curl_exec ($ch);
>>    if ($data) {
>>  $info = curl_getinfo($ch);
>>      curl_close ($ch);
>> } else {
>> $info = array();
>> Error_Handler::logError(curl_error($ch)." ".$url,__FILE__,__LINE__);
>> curl_close ($ch);
>> }    
>>    return array($data,$info);
>> }
>> 
>> -- class.reindex (in function indexFezFedoraObjects)
>> 
>> $handle = fopen(APP_TEMP_DIR.$new_dsID, "w");
>> $urlReturn = Misc::ProcessURL($urldata, false, $handle);
>> //fwrite($handle, $urlReturn[0]);
>> fclose($handle);
>> 

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Christiaan Kortekaas
Senior Library Open Sorcerer
Library Technology Service
The University of Queensland, Australia QLD 4072
Telephone : (+61) (7) 3346 4337
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~