[Xena-devel] [ xena-Feature Requests-2996354 ] Stop using File.deleteOnExit

[SourceForge.net <no...@so...>]

Feature Requests item #2996354, was opened at 2010-05-04 16:31
Message generated for change (Settings changed) made by grooverdan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=577092&aid=2996354&group_id=85722

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
>Category: Web Site
Group: None
Status: Open
Resolution: None
>Priority: 4
Private: No
Submitted By: Justin Waddell (jwaddell)
>Assigned to: Matthew Oliver (matthewoliver)
Summary: Stop using File.deleteOnExit

Initial Comment:
We use deleteOnExit when creating a lot of our temporary files. It turns out this is a bad idea as deleteOnExit uses memory to store references to the files to be deleted, and this memory is not freed while the JVM is running (even if the file is manually deleted after the call has been made).

Given that DPR could conceivably not be shut down for long periods this represents a significant memory leak, especially given the amount of temporary files we produce when normalising, viewing and exporting files.

We need to remove all references to deleteOnExit, while still ensuring that all our temporary files are deleted after use.

----------------------------------------------------------------------

Comment By: Justin Waddell (jwaddell)
Date: 2010-05-05 09:16

Message:
Apparently each entry uses up about 1kB of memory. This memory used is not
on the Java Heap, but in the JVM process itself, so I think that means it
would be limited by the amount of RAM available. So our biggest transfer
job so far of 300,000 records would take up about 300MB which would
probably be OK for now.

In general I think we've tried to delete temporary records after they are
finished with as we previously ran in to disk space issues. I don't think
it would be too difficult to strengthen the code so that it always delete
the temporary files unless there is a JVM crash, power outage etc (which is
the same behaviour as using deleteOnExit).

----------------------------------------------------------------------

Comment By: John (vombatus)
Date: 2010-05-05 08:48

Message:
Justin

Our new, improved security model requires us to shut down the machines in
the lab when not in use and lock away the hard drives. Even before this
came in, I was in the habit of shutting down the machines when not in use
to save energy.

This means that the Quarantine and Preservation instances of DPR are
regularly shut down and restarted.

Also, I do not leave DPR running on the Digital Repository when it is not
actually doing anything. No use in leaving that particular security hole
wide open.

Obviously, eliminating memory leaks is a good thing, but reality is, in
production, DPR is regularly shut down and restarted. If we continue to use
deleteOnExit, I assume that we might run the risk of having a transfer with
a very large number of files causing out of memory exceptions (which we
have not got since we changed to the new architecture). Do we know what the
upper limit might be? We can always restructure jobs to ensure that any
limit is not exceeded (not a perfect solution, but an easy one).

I assume that it would take a fair bit of work to reinvent the
deleteOnExit functionality so that it will not cause memory leaks. I also
assume that the risk of leaving a load of temporary files hanging around
increases if we remove it.

Is it worth doing so at the moment, when the chance of the memory leaks
occurring is minimal?

John



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=577092&aid=2996354&group_id=85722