Re: [Radmind-users] shadow filesystems

[Preston H. <san...@gm...>]

I'm basically using your option 3 but with a partition instead of a  
disk image.  Trying to fool installer with the modified ENV tricks

In the ideal:

1. You want to monitor the whole file system, but you are only  
interested in changes made by your installation process.

2. You want to be able to do this on a "dirty" system, that is, one  
that may have unradminded changes (I say this is the ideal as you can  
skip the "clean via a fresh radmind step" common in most loadset  
building workflows).

3. You want to create the loadset quickly (both in terms of scanning  
time, and reviewing for incorrectly included items)

Using a diskimage or non-boot volume as the target of installer takes  
care of 1 & 2 pretty well

A drag and drop install is ideal as you can just fsdiff that app  
folder/pkg.

But there are times when installer won't work correctly on a non-boot  
volume, and you still have to fsdiff the entire drive (I know - its  
not that long - but I'm impatient).

Really FSEvents is key here because it alone can give you changes  
limited by a certain process (installer).

So I did a little experiment this morning and it looks promising.

It relies on FSEventer [1] because anything else right now requires  
too much ugly mucking around.

It uses a pair of remarkably short - but not entirely simple python  
scripts.

The process goes something like this:

start logging with fseventer

run the first python script which starts installer, but monitors the  
system for any decedent process ids spawned by installer (any of which  
could be making changes)

when this install is done - it spits out a list of process ids that  
were involved in the install

stop fseventer and export its log

the second python script then takes the list of pids and the log and  
exports a radmind transcript based only FS changes made by one of the  
participating pids

fseventerlog2transcript.py installer.log | lsort > readyToGo.T

The advantages of this are:
  - You can limit captured changes to those made by installer on the  
boot volume of any system (it doesn't even have to be a managed machine)

  - The creation of a transcript after installation takes only seconds

The disadvantages are:
  - relies on FSEvents which in turn relies on a private low level  
FSEvents API  - and it has a 10k limit on its event buffer

  - uses a three step, human required, workflow (fseventer is not  
scriptable...)

- does not use defined radmind excludes (although that could be rolled  
into the python script pretty easily)

There is an embedded tool in fseventer called leotool - but I couldn't  
glean anything useful about its usage (an inquiry was sent to the  
author) the killer tool would be one that spits stuff out to stdout -  
then this could all be rolled into a single killer script!

If anyone is interested in the scripts in their current draft form -  
I'm happy to share.

-Preston

[1]  http://www.fernlightning.com/doku.php?id=software:fseventer:start



On Mar 26, 2009, at 3:36 PM, Karl Kuehn wrote:

>
> On Mar 26, 2009, at 3:16 PM, Preston Holmes wrote:
>
>> Unfortuneately - there I can't see any way to access just the  
>> contents
>> of the shadow file.
>
> <snip>
>
> 	In a shadow mount the shadow file only has the block-level changes  
> in it, it does not contain "real" files, so needs the original image  
> that it was shadowing in order to be useful.
>
>
>> All the original contents - even the stuff I didn't write out - is  
>> still being preserved in the shadow - which seems to be noticed in  
>> this post
>>
>> http://forums.macosxhints.com/showthread.php?t=22694
>>
>> So unless someone has a lead on working with these shadow files -  
>> looks like they won't fit the original bill of being able to  
>> capture the contents of what an installer writes out.
>
> 	In re-writing InstaDMG I got very familiar with shadow files, and  
> they are great for some thing, but I don't think that they are worth  
> pursuing for anything Radmind related. However, there are three  
> other areas that might provide more fertile ground:
>
> 1) Using the fs_events event log to backtrack all of the folders  
> that have changes since a time marker or using fs_events in live  
> mode to try and filter the stream of events as they come in  
> (requires that the monitor be running at the time of changes).
>
> 2) Use DMG's and "union" mount them on top of folders that you want  
> to monitor. Union mounting means that any file changes/replacements  
> will go into the DMG as whole files. I don't know off-hand how  
> deletions are handled. Union mounts are not well documented, but  
> there are snippets in the man pages for hdiutil and mount.
>
> 3) A final tool to consider in this space is something like we do in  
> InstaDMG: mount an entirely separate filesystem (a DMG that we  
> install os + updates + applications + settings to). That would allow  
> you to do things to your space without worrying about polluting your  
> booted OS. I will warn anyone going in this directions that there  
> are lots of gotchas about how installers work with this sort of  
> environment. It is something I want to and work with more (chroot  
> and Apple's sandbox), but need to get my head above water first.
>
> --
> 		Karl Kuehn
> 			la...@so...
>
>
>