Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> Read it again.  Every VMU file has an extra file "header" that is not
> considered part of the file data, but is used when
> displaying/parsing/validating the file's contents.  There should be two
> fields that immediately stick out in that header: CRC and "Number of bytes
> of actual file data following header". =20

You could look at the 'header' section of a file as a seperate file, but
the pages you referenced don't and that doesn't fit in with the way things
are layed out.  The CRC is no problem--just expost it and
let the user set it to a intelligable value or don't let them and calculate
it the fs driver.

> On the proceeding page, there is a field in the root directory structure
> labeled "offset of header (in blocks) from file start".  Although this could
> be assumed to always be 0 for data files, it isn't a constant so there is a
> possibility it could change.

The pages referenced make it seem like this is a constant depending on what
kind of file the directory entry is for.  There isn't really much more game
development for the DreamCast at this point, is there?  So, can't it be
considered static, now?  Look at the values used by existing software and
let's see if it truely is dynamic or not.

This is a really poorly designed filesystem.  The more I read about it, the
more I wonder what they were smoking.  I realize it's for an embedded and
custom applicaiton, but there are better ways to organize the data they 
want.  Why does the header need to be randomly placeable in the file?  Is that
for the game files which might want to put it at the end so they don't have
to jump over it when running vs the game files which might have a dozen
8K+ true color bitmaps for their icon animation?  Strange.

> Outside of vmufs, there is absolutely no way to update that data using
> standard Unix VFS semantics.  On vmufs of course, the driver can keep it as
> part of it's private inode data and update it dynamically whenever the file
> is written to.

But then we get to the point of why?  What's the point of sort of making it
look like a normal filesystem if you can't safely manipuate anything?  if
you want to store data on it, format it cramfs.  If you want to use it as
a proper VMU, then use some user space tools that deal with the raw block
device and know all the rules.  I guess I don't see the value of a middle
ground.

> I'm surprised you asked that question since those fields are the reason Adr=
> ian
> asked his question in the first place.

Two things.  One is that his quesiton didn't make that clear.  Two is that
it is possible to put all of the rules and parsing in the filesystem driver--
if you really want to.  I'm just not sure why one would want to.

> It was not my intention to start a pissing match - I posed that question as
> a valid one since you seem to be missing the big picture, esp. on VFS
> issues.  I apologize if my statements were peceived as such.

No problem.  It's always hard to tell peoples intentions in a written media
and I just wanted to clarify that I wasn't here to be arguementative, I would
really like to find a good solution to this.  I'm glad we're all in agreement
on that.

I realize that the VFS has always had problems with stashing data in inodes
when a filesystem wanted to keep it private--there are no good guarentees
that it'll be kept around for the durration.  That's why things like HFS
make them explicitly visible.  Sort of "okay, I'm not going to hide them
and play some kind of shell game where we just hope it doesn't get lost or
out of date.  You can see it, *you* deal with it.  I'm just a filesystem..."

So, the question, as I saw it was "can we do a similar trick with the VMU
data."  It's looking more and more like *mostly*, but it's getting ugly
and there is very little benefit, so why bother?  If there is some specific
use that someone has for this vmufs, making that clear might help at this
point. :)  If you can't safely manipulate them as UNIX files, why try to 
make them look like that?  So I can also stick a file of my own on there
that coexists with the existing structure?  If that's the case, how about
putting a header on them automatically that makes them invisible to the
Dreamcast's browser and *not* displaying all of the ones that *are* normal
Dreamcast files intended for the Dreamcast browser?  If the two can't
safely meet, don't let them.  How's that?

> Sorry if I misunderstood you, but in your preceeding post you proposed
> changing the first block of the VMU file data into a non-standard format.

Oh, sorry, I meant to make up the first block of the file when read on
the VFS side *from* the directory metadata--and stuff it back there on
writes.  That would allow everyone to do the "it's just a stream of bytes"
trick which would allow files from the VMU to be coppied off and back on
and they should still be valid--but it doesn't make the bits of that
data editable.  That didn't seem good at the time, but it is sure looking
better, now. :)

> > The kernel does not care what the fs driver does with the data it's given
> > nor how it comes up with the data it presents.  It just wants to give
> > it paths and files and perform some basic fops on them or get errors tryi=
> ng.
> 
> You forget that the kernel does care how the file data are represented.
> Please see struct inode in include/linux/fs.h.

Ahh, you're still thinking of hiding stuff in the inode.  I'm very much
recommending agains that.  It's only going to lead to a lot of pain.

> > I'll go into detail since this appears not to be to your liking:
> >=20
> [...]
> > It looks like all of the data in the header will work well if exposed as
> > seperate files in the .header directory associated with the file in quest=
> ion.
> >=20
> > Shall I go into more detail?
> >=20
> 
> Please look at the big picture.  Your above scheme is beautiful for
> everything that just does vmufs <-> vmufs.  But what happens when I want to
> create a brand new file on the mounted vmu, and I do a "touch myfile"?
> Where does the header come from?  In order to remain compatible it must
> either come from the user or have a default supplied by the vmufs driver.
> Both scenarios aren't pretty but the latter makes it easier for the end
> user to use the VMU as a generic storage device.  Note that without the
> file header it's possible that the VMU will break for anything outside of
> Linux (that hasn't been confirmed yet, but all DC software that uses the
> vmu accesses the header).

The vmufs will create a directory entry with a creation time and default
the other entries.  All of these other 'metafiles' are made up by the
vmufs driver when you do a directory lookup or an fstat--they don't exist
as separate files on disk.

If the user writes a block of zeroes to the VMU ramdomly, it'll trash it,
right? (well, depending on what they hit)  Do we protect against that?

I don't think that defaults are going to be a big issue.  See below.

> The case that Adrian mentioned is "cp oldvmu/file newvmu/file".  Under your
> proposed scheme this quickly become unmanageable as you now have:
> 
> $ cp oldvmu/file newvmu/file
> $ cp oldvmu/.header/file newvmu/.header/file

So make the 'file' itself look like a unix directory and put all of the
header and data chunks inside it.  So it becomes:

$ cp -r oldvmu/file newvmu/file

If they used 'cp' on a directory, they'd get a nice normal unix error and
they'd either realize they need to copy the contents and/or the whole
directory tree.

> and so on for each file that you want to copy.  Can you see the problem with
> that?  Everytime you add a new virtual file that has to be tagged along
> when the real file leaves vmufs, you decrease your flexibility.  Even
> copying the entire directory tree(s) becomes problematic because as soon as
> you copy, say the global VMU color you've destroyed what was on the other
> VMU.  And unless you have everything under directories (since vmufs can't
> physically support subdirectories) any special file in the root directory
> pollutes your namespace.

Sure, I do.  The solution is just to keep the metadata as close to the file
as possible, like in the scheme above where they're all files in a common
directory.

If the files specifying the properties of the VMU are in hidden 'dot' files,
you'll have to explicitly copy the.  cp won't pick them up unless told to.

With my proposed scheme, you can tar up or cp-r a whole VMU onto another
VMU and everything would be put back where it belongs.

I guess we're having a semantic difference, you and I.  What is presented 
by the vmufs to the kernel by the VFS layer *does not have to have anything
to do with the structure of data on whatever media the FS is accessing*.

The VMU directory structure does not need to support subdirs--they are
made up by the vmufs when asked for things through the VFS.  Here's an
example:

filesystem as seen by VFS:
an opeation on root/datafiles/BigBobBowls tells the vmufs to look for a 
directory entry where name == "BigBobBowls" and which is a normal data
file(filetype==0x33), not a game file (filetype==0xcc).  stats of it
will return a ctime from the 'BCD timestamp' field.  Atime and Mtime
can be initalized in the inode from that value and then just left to
themselves--let the VFS play with them if it wants, but changes to them
do not persist in the filesystem.

an operation on root/datafiles/BigBobBowls/header will return the normally
'hidden' directory information formatted in nice little text fields or
as a binary chunk--up the the implementer.  Either way reads come from
the two fields that a fs can't deal with by itself--'copy protect' and
'offset of header'.  The length in blocks can be dealt with by the vmufs
as it sees fit.  The actual contents of the blocks of the file can
be accessed by operations on root/datafiles/BigBobBowls/data.

This is the simplifies "don't try to mess with file headers" version
of my proposal.

> To me, your scheme sounds more like you would have both a vmufs
> driver that exposed the normal Unix files, and a /proc-like implementation
> that exposed global VMU metadata.

Pretty much, but in one filesystem.  The only part that needs the proc
treatment is the stuff hidden in the root block and the two fields from
the directory block.  The rest is pretty straight forward.

> Unfortunately, there is no precedence for exposing vmufs data in this way
> other than MacOS-specific filesystems, which have always needed hacks to
> play nice with PC-based OSes (data/rsrc crap comes to mind).  I can't think
> of any other filesystem that doesn't do a simple map of file, directory, and
> attributes on top of the VFS.

proc, devfs, usbdevfs, all of the directory and other metadata for any
advance journaling FS, and HFS. :)  If we do the minimal amount of this
faking, I don't think we're going to cross into uncharted terratory.
I don't see Linus or Alan sending us a nastygram. :)

> > That's the same kind of thing I'm proposing.  If you chose to rm -rf the
> > .header data in the vmufs, you're going to mess things up.  So, uhh, don't
> > do that if you don't want to mess up the VMU.
> 
> I wasn't on HFS/AppleShare when I did the rm.  vmufs can enforce the policy
> that you can't remove .header/, but what other filesystem will?

Oh, I got'cha!  You're right, none will.  But, if they come back with just the
data portion of a file, they get nada.  They shouldn't have messed with it.

What happens if someone coppies just the .exe file for a game and leaves
behind the .dlls and the various (ID-ish) .pak files?  Surprise, the game
doesn't work.  Silly user.  You need to copy the *whole* game.

If you see this as a huge issue, how about mapping the first block of the
file as the 'hidden' directory fields?  That way a whole file on the VMU
*looks* like one file at the VFS and has all of the information--except
wether it's a game or a save file--which we would have to trust the user
to remember.  I just think we want to avoid storing the 'is it a game'
inside the file as someone trying to edit that value in the file (no-no)
will expect it to work, but the file will still be in the wrong area of
the VMU.  If the directory structure is used to store that little bit of 
info, they we can set the value depending on what dir the file is placed
in--and we can disallow moves of files from one dir to another--or we
can reallocate and move the file if that were to happen--at the discretion
(free time) of the vmufs author.

> > The .afpvol trick was just a way for the Mac to translate the way it saw
> > things into the way the unix box saw things.  If the rules on how this
> > 'hidden' or 'extra' information is handled is built into the filesystem
> > so that you can't mess it up, you have a very useful tool.  If you just
> > want to make it look like some simple files, you're going to allow people
> > more chance to rm -rf your .afpvol so-to-speak.
> 
> The point is to make it look like simple files (the vmufs files
> themselves).  A frequently requested feature of vmufs is (was, before it was
> written) to a allow the VMU to function as a generic storage device.
> Enforcing such policies within the filesystem driver makes this impossible.

So see my previous comments about hiding *all* of the real VMU files and
stuffing our files/filesystem into it as a file that the Dreamcast will
ignore.  Never mix the two kinds of files.

> > Yes, touching raw blocks when you don't now what you're doing is a bad id=
> ea.
> > With the 'make it look like a bunch of files' method, the user never gets
> > a chance to do that.
> 
> Again, I think your scheme works well when using the VMU as strictly a
> VMU-to-VMU transfer device.  I don't think it can hold up to other uses
> without the vmufs driver making sweeping decisions over what data the VMU
> can hold.  This is what Adrian is attempting to avoid in his design of
> vmufs.

It can't witstand bits of the tree being removed or horribly manged and
then expecting the game/savedata to still work.  So, you're quire correct.

> (Sorry if it looks like I'm speaking for Adrian in these posts, but this is
> just rehashed stuff we've discussed on and off the list in the last few
> weeks.)

Ahh, I didn't see it on the list.  If Adrian wants to take offence at you
speaking for him, that's his call, I don't think he needs me to take offence
for him over you speaking for him--talk about convoluted... :)

I guess the part missing here is a clean statement of what we would
like to accomplish with the vmufs.  Do we want to be able to stash
our files on a VMU and have them safely coexist with the native
files *or* do we want to look at, copy, and manipulate (assuming we know
how) the VMU files?  I just want to make sure I'm solving the problem
asked, so context is important.

Take care.

Cheers,
David