Re: [Exist-open] Retrieving binary files

[nsincaglia <nsi...@nu...>]

I have been conducting some more testing on this issue. To further isolate
the issue, I have removed the MD5 hash sum calculation using the
expath-crypto module from the scenario and I am able to get the same errors.

 So, to restate the issue we are experiencing:
1). We send a httpclient:get() request to S3 to get a XML listing of the
contents of our S3 bucket.
2). We enter a 'for' loop which reads the file information from this S3 XML
file listing and calls a httpclient:get() function to retrieve these files.
These files are mostly binary files (50 MB or less)

For testing purposes, we are not doing anything with these files. We are
just retrieving them and going through the loop again to retrieve another
file. 

Things work fine for a while but as the log file snippet I attached in my
earlier post shows, the TemporaryFileManager.java tries to delete something:

2012-11-19 16:58:20,056 [eXistThread-352] INFO  (TemporaryFileManager.java
[returnTemporaryFile]:106) - Deleted temporary file:
/mnt/data/exist-2.0/tools/wrapper/bin/../../../tools/jetty/tmp/_mmtfm_c9386701-a399-48bb-b319-e68352310437/mmtf_13531746842905597142882655243385.tmp 

but repeatably is unable to:

2012-11-19 16:58:20,057 [eXistThread-352] WARN  (TemporaryFileManager.java
[returnTemporaryFile]:108) - Could not delete temporary file:
/mnt/data/exist-2.0/tools/wrapper/bin/../../../tools/jetty/tmp/_mmtfm_c9386701-a399-48bb-b319-e68352310437/mmtf_13531746842905597142882655243385.tmp.
Returning to stack for re-use. 
2012-11-19 16:58:20,057 [eXistThread-352] WARN  (TemporaryFileManager.java
[returnTemporaryFile]:108) - Could not delete temporary file:
/mnt/data/exist-2.0/tools/wrapper/bin/../../../tools/jetty/tmp/_mmtfm_c9386701-a399-48bb-b319-e68352310437/mmtf_13531746842905597142882655243385.tmp.
Returning to stack for re-use. 
2012-11-19 16:58:20,058 [eXistThread-352] WARN  (TemporaryFileManager.java
[returnTemporaryFile]:108) - Could not delete temporary file:
/mnt/data/exist-2.0/tools/wrapper/bin/../../../tools/jetty/tmp/_mmtfm_c9386701-a399-48bb-b319-e68352310437/mmtf_13531746842905597142882655243385.tmp.
Returning to stack for re-use. 
2012-11-19 16:58:20,058 [eXistThread-352] WARN  (TemporaryFileManager.java
[returnTemporaryFile]:108) - Could not delete temporary file:
/mnt/data/exist-2.0/tools/wrapper/bin/../../../tools/jetty/tmp/_mmtfm_c9386701-a399-48bb-b319-e68352310437/mmtf_13531746842905597142882655243385.tmp.
Returning to stack for re-use. 

I suspect we are running out of temporary memory. Our Xquery program
continues to make httpclient:get() requests that return these binary files
that are stored in a variable and then we run through our loop again.

Assuming this is the issue, does anyone understand why the system can not
delete these temporary files? 

Is this a setting or system permissions issue that we can tweak? 

is there anything we can do in our XQuery 'for' loop to signal to the system
that after we are finished with the file we have retrieved and no longer
need it to be temporarily stored? 

Any help would be greatly appreciated.

Nick



--
View this message in context: http://exist.2174344.n4.nabble.com/Retrieving-binary-files-tp4657215p4657249.html
Sent from the exist-open mailing list archive at Nabble.com.