Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> >=20
> > That's the advantage of these.  Normal tools will copy the 'special' files
> > as well.  Tar up the whole thing and you will save the special info with
> > the normal info.
> >=20
> 
> Ok, first off before *anyone* else responds to this thread, read the pages
> at http://mc.pp.se/dc/vms/flashmem.html and
> http://mc.pp.se/dc/vms/fileheader.html to gain an understanding of what's
> being discussed.

I have read that in the past and I just reread it as you requested.

> Your above paragraph: think about it.  If I copy a file from VMU, say to
> /tmp, and perform an edit or some other operation that changes the size of
> the file, what happens to that extra info that was stored elsewhere?

*what* extra info?

> It's invalid and the file (w.r.t to the VMU) is worthless.  The only way
> that works is if the editing tool understands the extra little files and
> directorires and can update them as soon as the main file is updated.  That
> isn't realistic.

The filesystem *is* the editing tool.  It displays the card formatted data
in a way that it is editable with standard unix tools.

> >=20
> > This requires a whole suite of cross-platform tools that understand how
> > one hokey little system mangles its data., ick.
> >=20
> 
> Heh, have you done Linux kernel development, in particular VFS development?
> Do you know the headaches that Adrian has been through getting his driver
> to work?  Doing a userspace (cross-platform is an extra bonus)
> implementation would simplify things greatly - all data remains intact when
> transferring between vmus and the code becomes more than trivial.  Besides,
> what do you think the VFS driver is doing with this "hokey little system"?

Yes.  Take a look at the SCSI subsystem.  I've been a linux developer
since version .12 in fall of '92.  I'm not here to start some kind os
pissing match and I'll ask you to respect that.

I'm not complaining one bit about the effort that Adrian has gone through.
I'm just answering a question posed to the list about how the vmufs could
work to make it easier on the user.

The way you put it, there is no point in *having* a vmufs.  Just leave it
as a block device and let a custom user space tool manipulate it--one that
knows all of the rules to its structure.

What I am suggesting is to have the vmufs format the data in such a was
as it will appear as normal files to the user and be editable as such.
I don't see anything in the description of the directory nor the files
that makes this unimplementable.

> The only disadvantage is a non-intuitive way to access file data, but at
> least it's compatible with every other vmufs implementation (including the
> DC browser itself) outside of Linux.

Pardon?  I'm not suggesting changing *one byte* on the card.  We're talking
about the vfs, right?  That's the interface in the kernel between high-ish
level calls (open, write, read, close, dlookup, etc.) and the code which
determines what is allowed and where it goes/comes from on the backing media.

The kernel does not care what the fs driver does with the data it's given
nor how it comes up with the data it presents.  It just wants to give
it paths and files and perform some basic fops on them or get errors trying.

> >=20
> > Okay, how does this sound.  Make the first block of the file be the speci=
> al
> > data for that file?  When the first block is read/written, the fs knows t=
> o=20
> > generate that info/save that info in the 'special' area.  All other blocks
> > work normally.
> >=20
> 
> See my last couple of paragraphs.

I'll go into detail since this appears not to be to your liking:

Starting at the root of the vmufs filesystem we have a root block that 
contains some properties of the filesystem.  It contains the date the
card was formatted.  This doesn't fit well into Ctime, Atime, nor Mtime--maybe
Ctime would be closest.  So, make the Ctime of the fs root directory entry
the format time.  Then it contains color components for the VMU.  These
can appear in a file called .color off of the root dir of the fs.  It can
contain five files called 'custom', 'blue', 'green', 'red', and 'alpha'. Each 
file is one byte in length and just contains the byte that specifies the
property that its name implies.  Writes beyond one byte get an ENOSPACE error
and the file gets truncated.

All of the other properties stored in that block are fs internal and don't
serve any purpose being exposed except for icon shape.  Something similar
to .color can be done with this.  Or color and shape can be combined in an
.icon directory off of root.

FAT is internal and doesn't need exposing.

The directory information is a little more comples and there are a few
ways to do it.  I would suggest that games and data files be though of seperately.
From the root there could be data and game subdirs.  If a file is put into
game, it will be placed starting at block 0 and will be stored contiguously.
This will also allow the fs to fill in the directory info fields file type
and header offset.

If a file is put in data, the files are stored starting at 199 and counting
down.  The fs will also know how to fill in the header offset and file type
from this.

All of the rest of the directory properties for a file are internal and
need not be exposed except for the 'copy protect' flag which can become a
property stored in the .header directory associated with the file.

It looks like all of the data in the header will work well if exposed as
seperate files in the .header directory associated with the file in question.

Shall I go into more detail?

> >=20
> > If the filesystem is HFS (say a disk from a Mac that's been stuck into
> > a linux box) then on disk the data is in one big 'file' chunk that one has
> > to understand how to split into code and resource 'forks'.  The VFS makes=
>  it
> > look like seperate files.  If the Linux box serves it out to an AppleShare
> > client, it makes it look like one file again.
> >=20
> 
> At my former university, every student had an account and disk space on a
> Unix system.  In each lab, the workstations would map this disk space so
> that students could save documents, etc. there.  On the Macs, because they
> use "extra" info for their filesystems, they tended to create an ".afpvol"
> directory that was hidden as long as you accessed that share on a Mac.
> 
> I spent a lot of my time under Unix and I typically trimmed my directory
> because of the quota imposed.  What do you think I did with the Mac-created
> =2Eafpvol?  rm -rf.  Could the Mac HFS do anything with this info any more?
> No.  Did it become problematic when accessing that share on the Mac again?
> Just a little, since it held type/creator information and other
> HFS-specific bits that made the file recognizable under the Finder.
> 
> I say all that to hopefully get you to see that vmufs-specific data will
> only ever make sense or be usuable by vmufs - other filesystems could care
> less about that extra information, _wherever_ it's stored.

That's the same kind of thing I'm proposing.  If you chose to rm -rf the
.header data in the vmufs, you're going to mess things up.  So, uhh, don't
do that if you don't want to mess up the VMU.

The .afpvol trick was just a way for the Mac to translate the way it saw
things into the way the unix box saw things.  If the rules on how this
'hidden' or 'extra' information is handled is built into the filesystem
so that you can't mess it up, you have a very useful tool.  If you just
want to make it look like some simple files, you're going to allow people
more chance to rm -rf your .afpvol so-to-speak.  Is there much use in
anything between what I propose and just leaving it as a raw block device
that only special user mode tools know how to deal with?  Either way
would be valid and the latter would, probably, be cleaner.  I'm not advocating
either.  I'm just suggesting a way to implement the former and cautioning
that anything between the two would be problematic.

> > The VFS just wants to say 'here is a path, open this file and give me
> > blocks of it'.  The filesystem driver has complete control over what
> > goes to and comes from the backing store--disk, flash, tape, etc.  Some
> > filesytems are completely 'made up'--proc, devfs...=20
> >=20
> 
> vmufs is a block-oriented filesystem.  You can't do magic things that are
> persistent because in the end, you're writing to a block device: whether
> it's the VMU mapped over MTD, a raw 128k dump of the VMU, or something
> else.

Every time you manipulate a file on a unix box, you do something magic
to it that is persistent.  Are you writing to a block of the disk?  No.
You're handing a block off to the FS via calls through the VFS and
it is probalby just holding it in memory unitl it feels like doing something
with it.  It decides when it gets written to disk.  No different than
the vmufs could.  It doesn't make to write anyting until you unmount the
filesystem.

> As soon as you modify the raw block data in a non-consistent manner, you no
> longer have vmufs.

Yes, touching raw blocks when you don't now what you're doing is a bad idea.
With the 'make it look like a bunch of files' method, the user never gets
a chance to do that.

> You can always present a special interface to the user, but see above as to
> why this won't persist across filesystems.

Let me see if I can understand what you're getting at.  Let's say you do the
'make it all look like files' trick and you recursively copy the presented
filesystem to another directory.  You edit it there and then try to copy
it back, right?  At that point your cp-r or tar is going to get errors when
things you're trying to write are in violation of what are allowed on a vmufs.
Just as if you had edited them *on* the vmufs--just delayed until you try to
copy them back.  The vmufs enforced these rules.  Think of these files like
the files in some of the 'proc' filesystem directories.  You can't write
to some, some that you can write to don't read as the data you put into them--
think of the scsi attach and detach commands on the scsi section of the 
proc filesystem.  Writes to a file get parsed and end up becoming function
calls into the scsi subsystem.  The 'file' is just a convenient metaphor
for the user and the kernel to share so they can meaningfully communicate
with the other.

Cheers,
David