Re: [ww-users] cleaning out old directories

[Lee W. <wor...@gm...>]

Hi Peter!

So the recommended treatment for big costly files is to isolate their 
make rules to be run only in background jobs:
http://lalashan.mcmaster.ca/theobio/projects/index.php/Background_Jobs#Background_jobs_design_pattern

If you do that, they won't be remade unless you do it explicitly by 
creating a background job.

Or you can just make sure to use <project-file ... make=false/> and 
write your make rules so that nothing that makes automatically has a 
dependency on any of your costly targets.

If we did have a "do not remake" flag I'd hesitate to assume that 
anything not tagged could be erased, because it's common to put off 
housekeeping tasks like making sure all the files are tagged when you 
add new ones.

A way to change things to source files might be useful - or something 
like Jonathan has suggested (if I remember right), where project files 
that are being archived in the wiki get written back out to the working 
directory, so they're preserved even if the whole directory gets 
removed.  But those kind of ideas are confusing and troubling to me and 
I don't know what to do with them.

Thanks-
lw

On 03/29/11 10:24, Peter L. Ralph wrote:
> I always feel nervous when there are large, costly files that WW might
> still decide to go ahead and re-make without my intending to.  Perhaps
> what we need is some easy mechanism to change the status of such files
> -- change them to 'source files' for instance.  This would be similar to
> a "DNE" flag, but would also flag "do not re-make"; so re-making would
> require first manual deletion.  Then any files not flagged as such could
> be deleted without problem.
>
> --peter
>
> On Mon, Mar 28, 2011 at 08:24:01PM -0700, Lee Worden wrote:
>> I have some open questions about WorkingWiki's features, where I would
>> appreciate your perspective as people who use WW, or even run your own
>> WW sites.  Please feel free to ignore this if you're busy or not interested.
>>
>> For quite a while, we've had a need to clean out old data that piles up
>> and isn't in use - not in the wiki pages themselves, but in the working
>> directories that we use behind the scenes to compute the output of
>> latex, R, etc.
>>
>> Most obvious is the preview sessions - any time you preview a page that
>> includes WW data while editing, it makes a copy of the data in the back
>> end to keep it separate from the unedited page's data.  When you save,
>> it gets rid of the copied files by merging with the saved ones, but if
>> you abandon the changes without saving (a completely reasonable thing to
>> do) the copy is left sitting there, and needs to be cleaned out sometime
>> later.  These can be quite large - we've seen project directories that
>> take up 4GB or even more.  That cleanout has been on my to do list, and
>> now it's getting done.
>>
>> More controversially, there are also old projects that eventually need
>> to disappear.  For instance, if someone creates a project and then
>> change its name, the old working directory just sits there abandoned.
>> Or if a page once had some WW files on it and now it doesn't, the
>> project directory is abandoned.  But also if I use the inline latex
>> features to add something like $$\alpha + \beta$$ to a page, then change
>> it to something else, a project is created to process that latex code,
>> and it needs to not be kept forever.  So at some point project
>> directories need to be cleared away or the disk will eventually fill up
>> with files that no one wants.  I'll probably do this by erasing things
>> that haven't been touched in over 3 months or something.
>>
>> Generally, this should be harmless even if I erase files that someone is
>> using, because they can be remade from the source files - it will just
>> mean waiting a minute or two (maybe more...) for them to be made.
>> Unfortunately,  in the worst case it could mean erasing a directory full
>> of output files that can't be easily recreated.  I could implement a "Do
>> Not Erase" feature to mark particular projects that are sensitive and
>> should never be erased.
>>
>> My first question: Is there a better way to protect project files that
>> should be permanent?  Does anyone have strong feelings about all this?
>>
>> Finally, I think background jobs should generally be left alone for as
>> long as it takes for people to decide whether to erase them.  But there
>> is a slight danger: suppose I create a project and run a background job,
>> then erase or rename the project.  The background job becomes orphaned,
>> and it won't show up in any listings.  So I should probably do something
>> to erase things like that.
>>
>> One way to address both of these things is to check whether each project
>> is actually connected to current pages in one of the wikis - that would
>> clear up whether it's orphaned or not.  But the directory cleaning is
>> done in a separate back-end component ("ProjectEngine") when requested
>> by the front end ("WorkingWiki"), and I'm trying to avoid two-way
>> communication where ProjectEngine has to ask questions of WorkingWiki
>> while it's completing a request, so I'm looking for an alternative...
>>
>> Lee
>>
>> ------------------------------------------------------------------------------
>> Enable your software for Intel(R) Active Management Technology to meet the
>> growing manageability and security demands of your customers. Businesses
>> are taking advantage of Intel(R) vPro (TM) technology - will your software
>> be a part of the solution? Download the Intel(R) Manageability Checker
>> today! http://p.sf.net/sfu/intel-dev2devmar
>> _______________________________________________
>> workingwiki-users mailing list
>> wor...@li...
>> https://lists.sourceforge.net/lists/listinfo/workingwiki-users