linuxdc-dev

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> On Sat, 7 Dec 2002 10:37:39 +0000
> Adrian McMenamin <ad...@mc...> wrote:
> 
> > (a) Ban all interaction between vmufs and any other filesystem (ie
> > assume that all an any file is a vmufs file)
> > 
> > (b) Ignore the problem because it is probably not too serious in any
> > case
> 
> Test (b). Is it possible to "fake" the extra info somehow ? 

Yes, I can see two sort of clean ways to do it.
One is to make the files look just like they do now and hide the 
extra directory info in a file called .filename or in another
directory--from the root of the fs--with the same directory
structure that has a 'filename' file that has the info in it, so:

filesystemrootdir/supergame/bob is our file
filesystemrootdir/supergame/.bob is the extra info or:
filesystemrootdir/extradirinfo/supergame/bob is the extra info

Or make each file a directory and have files in it with the 'main'
chunk and the 'extra' chunk:

filesystemrootdir/supergame/bob/main is our file and
filesystemrootdir/supergame/bob/extra is the extra info

They're all pretty easy to implement as the 'extra' directories
and files are faked by the filessytem.  To make life easy, you
can ban directory creates in the 'extra' info directory structures.
You can ban the creation of . files if you use that scheme.  Pick
a scheme above and implement some rules like that to make life easier--or
you can make it a bit tougher and more flexable and make a dir create
in the 'extra' dirs actually make one in the 'real' dirs--which will 
implicitly create on in the 'extra' dirs.

Does any of this make sense to anyone?  Or did I explain it poorly?
(could be both.)

> > PS Today is my birthday... all together now....
> 
> C O N G R A T U L A T I O N S !

C O N G R A T U L A T I O N S !

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[Matt T. <xw...@xw...>]

I haven't really paid attention much, but is this extra file information 
what would be considered metadata?  Does Linux have no support for 
metadata without going about it in a hackish way?  Or is it just that 
the filesystem driver actually needs to use this metadata and the way 
the vfs is, it can't easily be done?

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> I haven't really paid attention much, but is this extra file information 
> what would be considered metadata?  Does Linux have no support for 
> metadata without going about it in a hackish way?  Or is it just that 
> the filesystem driver actually needs to use this metadata and the way 
> the vfs is, it can't easily be done?

To the people who designed Unix, metadata is a hack.  The mantra is
'everything is a file'.  If confused, repeat. :)

Neither windows nor Unix have a good way of exposing it.  The tricks
that unix has done with 'proc' filesystems and the like have gone one
way and the Windows "require a special tool to manipulate it" theory
has gone another.

To each their own, I guess.  The 'everything is a file' school gets
you into some interesting situations when the underlying data has
strange dependencies--like the VMU fs does.  I'm not even sure why
we want such a thing.  I'm inclined to just treat the memory in the
VMU as a single chunk and make a user mode tool to manipulate it.
That keeps strange policy out of the kernel and removes the chance
that someone will not realize that these files and directories presented
by the vmufs are 'special' and muck things up.

Users, life would be so much simpler without them. ;)

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> Read it again.  Every VMU file has an extra file "header" that is not
> considered part of the file data, but is used when
> displaying/parsing/validating the file's contents.  There should be two
> fields that immediately stick out in that header: CRC and "Number of bytes
> of actual file data following header". =20

You could look at the 'header' section of a file as a seperate file, but
the pages you referenced don't and that doesn't fit in with the way things
are layed out.  The CRC is no problem--just expost it and
let the user set it to a intelligable value or don't let them and calculate
it the fs driver.

> On the proceeding page, there is a field in the root directory structure
> labeled "offset of header (in blocks) from file start".  Although this could
> be assumed to always be 0 for data files, it isn't a constant so there is a
> possibility it could change.

The pages referenced make it seem like this is a constant depending on what
kind of file the directory entry is for.  There isn't really much more game
development for the DreamCast at this point, is there?  So, can't it be
considered static, now?  Look at the values used by existing software and
let's see if it truely is dynamic or not.

This is a really poorly designed filesystem.  The more I read about it, the
more I wonder what they were smoking.  I realize it's for an embedded and
custom applicaiton, but there are better ways to organize the data they 
want.  Why does the header need to be randomly placeable in the file?  Is that
for the game files which might want to put it at the end so they don't have
to jump over it when running vs the game files which might have a dozen
8K+ true color bitmaps for their icon animation?  Strange.

> Outside of vmufs, there is absolutely no way to update that data using
> standard Unix VFS semantics.  On vmufs of course, the driver can keep it as
> part of it's private inode data and update it dynamically whenever the file
> is written to.

But then we get to the point of why?  What's the point of sort of making it
look like a normal filesystem if you can't safely manipuate anything?  if
you want to store data on it, format it cramfs.  If you want to use it as
a proper VMU, then use some user space tools that deal with the raw block
device and know all the rules.  I guess I don't see the value of a middle
ground.

> I'm surprised you asked that question since those fields are the reason Adr=
> ian
> asked his question in the first place.

Two things.  One is that his quesiton didn't make that clear.  Two is that
it is possible to put all of the rules and parsing in the filesystem driver--
if you really want to.  I'm just not sure why one would want to.

> It was not my intention to start a pissing match - I posed that question as
> a valid one since you seem to be missing the big picture, esp. on VFS
> issues.  I apologize if my statements were peceived as such.

No problem.  It's always hard to tell peoples intentions in a written media
and I just wanted to clarify that I wasn't here to be arguementative, I would
really like to find a good solution to this.  I'm glad we're all in agreement
on that.

I realize that the VFS has always had problems with stashing data in inodes
when a filesystem wanted to keep it private--there are no good guarentees
that it'll be kept around for the durration.  That's why things like HFS
make them explicitly visible.  Sort of "okay, I'm not going to hide them
and play some kind of shell game where we just hope it doesn't get lost or
out of date.  You can see it, *you* deal with it.  I'm just a filesystem..."

So, the question, as I saw it was "can we do a similar trick with the VMU
data."  It's looking more and more like *mostly*, but it's getting ugly
and there is very little benefit, so why bother?  If there is some specific
use that someone has for this vmufs, making that clear might help at this
point. :)  If you can't safely manipulate them as UNIX files, why try to 
make them look like that?  So I can also stick a file of my own on there
that coexists with the existing structure?  If that's the case, how about
putting a header on them automatically that makes them invisible to the
Dreamcast's browser and *not* displaying all of the ones that *are* normal
Dreamcast files intended for the Dreamcast browser?  If the two can't
safely meet, don't let them.  How's that?

> Sorry if I misunderstood you, but in your preceeding post you proposed
> changing the first block of the VMU file data into a non-standard format.

Oh, sorry, I meant to make up the first block of the file when read on
the VFS side *from* the directory metadata--and stuff it back there on
writes.  That would allow everyone to do the "it's just a stream of bytes"
trick which would allow files from the VMU to be coppied off and back on
and they should still be valid--but it doesn't make the bits of that
data editable.  That didn't seem good at the time, but it is sure looking
better, now. :)

> > The kernel does not care what the fs driver does with the data it's given
> > nor how it comes up with the data it presents.  It just wants to give
> > it paths and files and perform some basic fops on them or get errors tryi=
> ng.
> 
> You forget that the kernel does care how the file data are represented.
> Please see struct inode in include/linux/fs.h.

Ahh, you're still thinking of hiding stuff in the inode.  I'm very much
recommending agains that.  It's only going to lead to a lot of pain.

> > I'll go into detail since this appears not to be to your liking:
> >=20
> [...]
> > It looks like all of the data in the header will work well if exposed as
> > seperate files in the .header directory associated with the file in quest=
> ion.
> >=20
> > Shall I go into more detail?
> >=20
> 
> Please look at the big picture.  Your above scheme is beautiful for
> everything that just does vmufs <-> vmufs.  But what happens when I want to
> create a brand new file on the mounted vmu, and I do a "touch myfile"?
> Where does the header come from?  In order to remain compatible it must
> either come from the user or have a default supplied by the vmufs driver.
> Both scenarios aren't pretty but the latter makes it easier for the end
> user to use the VMU as a generic storage device.  Note that without the
> file header it's possible that the VMU will break for anything outside of
> Linux (that hasn't been confirmed yet, but all DC software that uses the
> vmu accesses the header).

The vmufs will create a directory entry with a creation time and default
the other entries.  All of these other 'metafiles' are made up by the
vmufs driver when you do a directory lookup or an fstat--they don't exist
as separate files on disk.

If the user writes a block of zeroes to the VMU ramdomly, it'll trash it,
right? (well, depending on what they hit)  Do we protect against that?

I don't think that defaults are going to be a big issue.  See below.

> The case that Adrian mentioned is "cp oldvmu/file newvmu/file".  Under your
> proposed scheme this quickly become unmanageable as you now have:
> 
> $ cp oldvmu/file newvmu/file
> $ cp oldvmu/.header/file newvmu/.header/file

So make the 'file' itself look like a unix directory and put all of the
header and data chunks inside it.  So it becomes:

$ cp -r oldvmu/file newvmu/file

If they used 'cp' on a directory, they'd get a nice normal unix error and
they'd either realize they need to copy the contents and/or the whole
directory tree.

> and so on for each file that you want to copy.  Can you see the problem with
> that?  Everytime you add a new virtual file that has to be tagged along
> when the real file leaves vmufs, you decrease your flexibility.  Even
> copying the entire directory tree(s) becomes problematic because as soon as
> you copy, say the global VMU color you've destroyed what was on the other
> VMU.  And unless you have everything under directories (since vmufs can't
> physically support subdirectories) any special file in the root directory
> pollutes your namespace.

Sure, I do.  The solution is just to keep the metadata as close to the file
as possible, like in the scheme above where they're all files in a common
directory.

If the files specifying the properties of the VMU are in hidden 'dot' files,
you'll have to explicitly copy the.  cp won't pick them up unless told to.

With my proposed scheme, you can tar up or cp-r a whole VMU onto another
VMU and everything would be put back where it belongs.

I guess we're having a semantic difference, you and I.  What is presented 
by the vmufs to the kernel by the VFS layer *does not have to have anything
to do with the structure of data on whatever media the FS is accessing*.

The VMU directory structure does not need to support subdirs--they are
made up by the vmufs when asked for things through the VFS.  Here's an
example:

filesystem as seen by VFS:
an opeation on root/datafiles/BigBobBowls tells the vmufs to look for a 
directory entry where name == "BigBobBowls" and which is a normal data
file(filetype==0x33), not a game file (filetype==0xcc).  stats of it
will return a ctime from the 'BCD timestamp' field.  Atime and Mtime
can be initalized in the inode from that value and then just left to
themselves--let the VFS play with them if it wants, but changes to them
do not persist in the filesystem.

an operation on root/datafiles/BigBobBowls/header will return the normally
'hidden' directory information formatted in nice little text fields or
as a binary chunk--up the the implementer.  Either way reads come from
the two fields that a fs can't deal with by itself--'copy protect' and
'offset of header'.  The length in blocks can be dealt with by the vmufs
as it sees fit.  The actual contents of the blocks of the file can
be accessed by operations on root/datafiles/BigBobBowls/data.

This is the simplifies "don't try to mess with file headers" version
of my proposal.

> To me, your scheme sounds more like you would have both a vmufs
> driver that exposed the normal Unix files, and a /proc-like implementation
> that exposed global VMU metadata.

Pretty much, but in one filesystem.  The only part that needs the proc
treatment is the stuff hidden in the root block and the two fields from
the directory block.  The rest is pretty straight forward.

> Unfortunately, there is no precedence for exposing vmufs data in this way
> other than MacOS-specific filesystems, which have always needed hacks to
> play nice with PC-based OSes (data/rsrc crap comes to mind).  I can't think
> of any other filesystem that doesn't do a simple map of file, directory, and
> attributes on top of the VFS.

proc, devfs, usbdevfs, all of the directory and other metadata for any
advance journaling FS, and HFS. :)  If we do the minimal amount of this
faking, I don't think we're going to cross into uncharted terratory.
I don't see Linus or Alan sending us a nastygram. :)

> > That's the same kind of thing I'm proposing.  If you chose to rm -rf the
> > .header data in the vmufs, you're going to mess things up.  So, uhh, don't
> > do that if you don't want to mess up the VMU.
> 
> I wasn't on HFS/AppleShare when I did the rm.  vmufs can enforce the policy
> that you can't remove .header/, but what other filesystem will?

Oh, I got'cha!  You're right, none will.  But, if they come back with just the
data portion of a file, they get nada.  They shouldn't have messed with it.

What happens if someone coppies just the .exe file for a game and leaves
behind the .dlls and the various (ID-ish) .pak files?  Surprise, the game
doesn't work.  Silly user.  You need to copy the *whole* game.

If you see this as a huge issue, how about mapping the first block of the
file as the 'hidden' directory fields?  That way a whole file on the VMU
*looks* like one file at the VFS and has all of the information--except
wether it's a game or a save file--which we would have to trust the user
to remember.  I just think we want to avoid storing the 'is it a game'
inside the file as someone trying to edit that value in the file (no-no)
will expect it to work, but the file will still be in the wrong area of
the VMU.  If the directory structure is used to store that little bit of 
info, they we can set the value depending on what dir the file is placed
in--and we can disallow moves of files from one dir to another--or we
can reallocate and move the file if that were to happen--at the discretion
(free time) of the vmufs author.

> > The .afpvol trick was just a way for the Mac to translate the way it saw
> > things into the way the unix box saw things.  If the rules on how this
> > 'hidden' or 'extra' information is handled is built into the filesystem
> > so that you can't mess it up, you have a very useful tool.  If you just
> > want to make it look like some simple files, you're going to allow people
> > more chance to rm -rf your .afpvol so-to-speak.
> 
> The point is to make it look like simple files (the vmufs files
> themselves).  A frequently requested feature of vmufs is (was, before it was
> written) to a allow the VMU to function as a generic storage device.
> Enforcing such policies within the filesystem driver makes this impossible.

So see my previous comments about hiding *all* of the real VMU files and
stuffing our files/filesystem into it as a file that the Dreamcast will
ignore.  Never mix the two kinds of files.

> > Yes, touching raw blocks when you don't now what you're doing is a bad id=
> ea.
> > With the 'make it look like a bunch of files' method, the user never gets
> > a chance to do that.
> 
> Again, I think your scheme works well when using the VMU as strictly a
> VMU-to-VMU transfer device.  I don't think it can hold up to other uses
> without the vmufs driver making sweeping decisions over what data the VMU
> can hold.  This is what Adrian is attempting to avoid in his design of
> vmufs.

It can't witstand bits of the tree being removed or horribly manged and
then expecting the game/savedata to still work.  So, you're quire correct.

> (Sorry if it looks like I'm speaking for Adrian in these posts, but this is
> just rehashed stuff we've discussed on and off the list in the last few
> weeks.)

Ahh, I didn't see it on the list.  If Adrian wants to take offence at you
speaking for him, that's his call, I don't think he needs me to take offence
for him over you speaking for him--talk about convoluted... :)

I guess the part missing here is a clean statement of what we would
like to accomplish with the vmufs.  Do we want to be able to stash
our files on a VMU and have them safely coexist with the native
files *or* do we want to look at, copy, and manipulate (assuming we know
how) the VMU files?  I just want to make sure I'm solving the problem
asked, so context is important.

Take care.

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[Adrian M. <ad...@mc...>]

On Monday 09 Dec 2002 5:27 pm, David Willmore wrote:

[all snipped]

David, as I understand it your basic question is "why bother?" Well my answer 
is "why not?" coupled with "because you can". Sure developing a Linux system 
on a Dreamcast is just a game, a way of wasting time. But aren't lots of 
hacker projects just the same?

Adrian

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> David, as I understand it your basic question is "why bother?" Well my answer 
> is "why not?" coupled with "because you can". Sure developing a Linux system 
> on a Dreamcast is just a game, a way of wasting time. But aren't lots of 
> hacker projects just the same?

Wow, I don't know if that could be more out of context. :)

The meaning of my comment was what's the motivation for making a file
system driver that will always be broken?  We should either make it
complete or spend the time developing some userspace tools that do a
better job of it--like Mr. Brown first suggested.

I'm all for seemingly pointless hacking projects.  I've got a stack
of WebPals here that I've just got running Linux and I'm trying to
install debian on them--I've never installed it on anything before
and maybe starting on a strange box was not a good idea... :)

Since you're the author of the fs in question and since I've got you. :)
Could you say just what it is you want the vmufs to do?  The two ways
I could see it looking and being useful are:
1: full blown filesystem with metadata exposed as files (copyable, tarable,
editable, etc.)
2: just able to 'stash' a file on it and get it back--don't need to look
at or edit the DC files.  Also, keep the DC from choking on the files
we produce.

There is always #3 which is to stop all of this and just develop a nice
suite of userspace tools like mtools (for MSDOS filesystems).  But, I
assumed that we had already passed that fork in the road by talking
about a FS to start with. :)

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[Adrian M. <ad...@mc...>]

On Monday 09 Dec 2002 8:41 pm, David Willmore wrote:
> > David, as I understand it your basic question is "why bother?" Well my
> > answer is "why not?" coupled with "because you can". Sure developing a
> > Linux system on a Dreamcast is just a game, a way of wasting time. But
> > aren't lots of hacker projects just the same?
>
> Wow, I don't know if that could be more out of context. :)
>
> The meaning of my comment was what's the motivation for making a file
> system driver that will always be broken?  We should either make it
> complete or spend the time developing some userspace tools that do a
> better job of it--like Mr. Brown first suggested.
>
> I'm all for seemingly pointless hacking projects.  I've got a stack
> of WebPals here that I've just got running Linux and I'm trying to
> install debian on them--I've never installed it on anything before
> and maybe starting on a strange box was not a good idea... :)
>

What is a "webpal"?

> Since you're the author of the fs in question and since I've got you. :)
> Could you say just what it is you want the vmufs to do?  The two ways
> I could see it looking and being useful are:
> 1: full blown filesystem with metadata exposed as files (copyable, tarable,
> editable, etc.)
> 2: just able to 'stash' a file on it and get it back--don't need to look
> at or edit the DC files.  Also, keep the DC from choking on the files
> we produce.
>

The rea aim, such as there was one, was to allow game players to swap files 
easily, I suppose. But it's a toy, so there are other things - eg I thought 
of hosting apache off one!

> There is always #3 which is to stop all of this and just develop a nice
> suite of userspace tools like mtools (for MSDOS filesystems).  But, I
> assumed that we had already passed that fork in the road by talking
> about a FS to start with. :)
>
Well I have a working fs now - albeit it (mainly, it also deletes) being a 
read-only one. As for the options I am not sure what to do now...I need to 
give it some more thought and decide whether to continue pusuing the rw 
option for the fs or just stop where we are now...

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> What is a "webpal"?

It's an early era web browsing set top box.  They weren't tied to a
particular ISP, so they sold for full price--$400(USD).  They didn't
sell well and now there a ton of them available for almost nothing.
I paid $8(USD) each for mine.

The box has a cirrus CL-PS7500FE which is an arm 7 core processor 
running at 40MHz.  There is one DRAM 72 pin SIMM slot and one FLASH
72 pin SIMM slot.  It comes with 4MB and 1MB resprectively.  You can
put 2k refresh simms in it.  Mine has 32MB and there is a simple mod
to allow 4k page refresh simms, as well.  It has sound out, parallel, 
PS/2 mouse & KB, and VGA out on the back pannel.  It's also got one 
semi-functional ISA slot internally.  On the front is the IR sensor, 
three LEDs and a smart card slot.  Internally, it has an IDE connector.
You can graft on a 5V connector, a short IDE cable and a 3.5" to 2.5"
IDE adapter and you can stuff in a laptop HD of your choice.

So, after some soldering and a lot of software hacking, one is left with
a little ARM box running Linux.  (Oh, forgot, it has composite and S-video
out as well--same content as the VGA out.)

Cute little box.  I'm thinking of grafting a GPS receiver board (Motorola
OnCore GT+) into it with a small patch antenna and making an NNTP server
for my house/ham shack.

> > Since you're the author of the fs in question and since I've got you. :)
> > Could you say just what it is you want the vmufs to do?  The two ways
> > I could see it looking and being useful are:
> > 1: full blown filesystem with metadata exposed as files (copyable, tarable,
> > editable, etc.)
> > 2: just able to 'stash' a file on it and get it back--don't need to look
> > at or edit the DC files.  Also, keep the DC from choking on the files
> > we produce.
> 
> The rea aim, such as there was one, was to allow game players to swap files 
> easily, I suppose. But it's a toy, so there are other things - eg I thought 
> of hosting apache off one!

:)  Okay, then to satisfy that, how about just making the first block of
the file (as seen by the VFS side) completely fake and containing the 'hidden'
directory info.  On write it gets stuffed into the directory entry.  It'll
also tell the vmufs how to allocate the file--most everyone will write the
file block by block from the beginning, so you'll have the data to start
with.

Does that solve things or are there other issues I'm missing?

> > There is always #3 which is to stop all of this and just develop a nice
> > suite of userspace tools like mtools (for MSDOS filesystems).  But, I
> > assumed that we had already passed that fork in the road by talking
> > about a FS to start with. :)

> Well I have a working fs now - albeit it (mainly, it also deletes) being a 
> read-only one. As for the options I am not sure what to do now...I need to 
> give it some more thought and decide whether to continue pusuing the rw 
> option for the fs or just stop where we are now...

My simple solution--above--does make us Linux users unable to simply write
*our* DC-unaware files to it as the first block will get trashed and the
stuff we though we wrote will put some junk in the directory entry.  Of 
course, you could get 'clever' and look at the first block and see if the
unused chunks were left alone and if so, assume it's a proper DC file,
otherwise assume it's a native Linux file and fake some safe directory 
info.

It just doesn't seem like we can have both.

But wait!  Inspiration strikes.  Okay, a hybrid.  From the root of the fs,
there is a directory called 'NativeDC' and it contains the first-block-
faked stuff from above.  There's another directory (seen from the VFS)
caled 'NativeLinux' and it contains the unix style files.  Here's the
trick.  The Linux files are stored *on the VMU* as one file with some
stock DC compatible header.  So, basically the complete Linux filesystem
gets hidden on one DC file on the VMU.  

The cool thing about that is we can copy the DC files with unix--from/to
a VMU or other device--and all the special info gets preserved.  Also,
we can have normal unix stuff hidden in out one little file--permissions,
directories, etc.  *And* the DC browser can copy our whole FS from one
VMU to another because the header/directory entry it's hidden in looks
legit.

I'm typeing this fast, so I might not be making sense.  Let me know and
I'll slow down and get more detailed.  I think this will work and it'll
do *everything* we want. :)

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[Adrian M. <ad...@mc...>]

On Sunday 08 Dec 2002 6:10 pm, David Willmore wrote:
> > On Sat, 7 Dec 2002 10:37:39 +0000

> Does any of this make sense to anyone?  Or did I explain it poorly?
> (could be both.)
>
Sorry, but you totally lost me :-<

I think the point is that the Linux VFS knows nothing of this extra directory 
information and so I can see no way of writing a Linux fs driver that can use 
it. I could obviously just write a bespoke device driver or userland app that 
hacked it all, but that is not as elegant.

Furthermore I want to be able to save any file I like on the vmu.

I think I'm just going to have to live with it.

Adrian

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> On Sunday 08 Dec 2002 6:10 pm, David Willmore wrote:
> > > On Sat, 7 Dec 2002 10:37:39 +0000
> 
> > Does any of this make sense to anyone?  Or did I explain it poorly?
> > (could be both.)
> 
> Sorry, but you totally lost me :-<
> 
> I think the point is that the Linux VFS knows nothing of this extra directory 
> information and so I can see no way of writing a Linux fs driver that can use 
> it. I could obviously just write a bespoke device driver or userland app that 
> hacked it all, but that is not as elegant.
> 
> Furthermore I want to be able to save any file I like on the vmu.
> 
> I think I'm just going to have to live with it.

You Linux VFS sees the files, but the filesystem driver makes them up from the 
info in the directories (the hidden pointers and such).  I'll write a better 
description when I get a moment--have to go finish makeing dinner. :)

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[M. R. B. <mr...@0x...>]

* David Willmore <dav...@ia...> on Sun, Dec 08, 2002:

>=20
> You Linux VFS sees the files, but the filesystem driver makes them up fro=
m the=20
> info in the directories (the hidden pointers and such).  I'll write a bet=
ter=20
> description when I get a moment--have to go finish makeing dinner. :)
>=20

How does this help you when transferring files to/from a filesystem that
doesn't understand this information, like NFS?  Or when transferring saves
around across the network?

The solution would be a user space tool that understood and could transfer
files between full VMU images (whether dumped or mapped via MTD).  This is =
the
only way you can persist the nonstandard info and be flexible enough to
allow other uses of the VMU save data.  MTOOLS (the MSDOS userspace
filesystem utilities?) comes to mind.

Such tools would be a hell of a lot easier to implement than a full-blown
(and unfortunately incompatible) VFS driver.

M. R.

Re: [linuxdc-dev] VMU filesystem question

[Callum L. <se...@ha...>]

M. R. Brown wrote:
> * David Willmore <dav...@ia...> on Sun, Dec 08, 2002:
> 
> 
>>You Linux VFS sees the files, but the filesystem driver makes them up from the 
>>info in the directories (the hidden pointers and such).  I'll write a better 
>>description when I get a moment--have to go finish makeing dinner. :)
>>
> 
> 
> How does this help you when transferring files to/from a filesystem that
> doesn't understand this information, like NFS?  Or when transferring saves
> around across the network?

You copy the entire directory tree.

Re: [linuxdc-dev] VMU filesystem question

[M. R. B. <mr...@0x...>]

* Callum Lerwick <se...@ha...> on Sun, Dec 08, 2002:

> M. R. Brown wrote:
> >* David Willmore <dav...@ia...> on Sun, Dec 08, 2002:
> >
> >
> >>You Linux VFS sees the files, but the filesystem driver makes them up=
=20
> >>from the info in the directories (the hidden pointers and such).  I'll=
=20
> >>write a better description when I get a moment--have to go finish makei=
ng=20
> >>dinner. :)
> >>
> >
> >
> >How does this help you when transferring files to/from a filesystem that
> >doesn't understand this information, like NFS?  Or when transferring sav=
es
> >around across the network?
>=20
> You copy the entire directory tree.
>=20

Reread Adrian's post.  The data he is talking about can't be "copied"
outside of the vmufs driver.  There is no way to persist custom data in the
Linux VFS without making all other filesystems understand that information.

M. R.

Re: [linuxdc-dev] VMU filesystem question

[M. R. B. <mr...@0x...>]

* M. R. Brown <mr...@0x...> on Sun, Dec 08, 2002:

> * Callum Lerwick <se...@ha...> on Sun, Dec 08, 2002:
>=20
> >=20
> > You copy the entire directory tree.
> >=20
>=20
> Reread Adrian's post.  The data he is talking about can't be "copied"
> outside of the vmufs driver.  There is no way to persist custom data in t=
he
> Linux VFS without making all other filesystems understand that informatio=
n.
>=20

Let me clarify.  Even if you devise a brilliant scheme to expose vmufs
directory info or metadata in other "files" or "directories", as _soon_ as
that data leaves vmufs it becomes meaningless.  There is no way you can
assure that data will remain together or intact as it travels to the four
corners of the Earth.  Thus, you are back to square one and you have
incomplete file info.  You could store it as part of the file, but then you
lose compatability because that single file can't be transferred to
non-Linux-vmufs systems who won't know what to do with the extra
information (or even know that it exists).  And be realistic, copying an
entire directory tree for one file is ridiculous...

Besides that, vmufs doesn't support directories and you can get pretty
flexible with the filename (it's plain ASCII).  Whatever filenaming scheme
you enforce would also cause vmufs to force policy on the rest of the syste=
m.

M. R.

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> * David Willmore <dav...@ia...> on Sun, Dec 08, 2002:
> 
> >=20
> > You Linux VFS sees the files, but the filesystem driver makes them up fro=
> m the=20
> > info in the directories (the hidden pointers and such).  I'll write a bet=
> ter=20
> > description when I get a moment--have to go finish makeing dinner. :)
> >=20
> 
> How does this help you when transferring files to/from a filesystem that
> doesn't understand this information, like NFS?  Or when transferring saves
> around across the network?

That's the advantage of these.  Normal tools will copy the 'special' files
as well.  Tar up the whole thing and you will save the special info with
the normal info.

> The solution would be a user space tool that understood and could transfer
> files between full VMU images (whether dumped or mapped via MTD).  This is =
> the
> only way you can persist the nonstandard info and be flexible enough to
> allow other uses of the VMU save data.  MTOOLS (the MSDOS userspace
> filesystem utilities?) comes to mind.

This requires a whole suite of cross-platform tools that understand how
one hokey little system mangles its data., ick.

> Such tools would be a hell of a lot easier to implement than a full-blown
> (and unfortunately incompatible) VFS driver.

There is nothing incompatable about it--as far as being a VFS driver goes.

Okay, how does this sound.  Make the first block of the file be the special
data for that file?  When the first block is read/written, the fs knows to 
generate that info/save that info in the 'special' area.  All other blocks
work normally.

For inspiration on how I see this working, look at the AppleShare system.
How do unix systems serve files to Macs which expect a file to have a 
code and a resource fork?  They do it by having this split and mirrored
directory structure.

rootoffilesystem/code/pathtostuff/myfile is the code 'fork' of the file
rootoffilesystem/res/pathtostuff/myfile is the resource 'fork' of the file.

If the filesystem is HFS (say a disk from a Mac that's been stuck into
a linux box) then on disk the data is in one big 'file' chunk that one has
to understand how to split into code and resource 'forks'.  The VFS makes it
look like seperate files.  If the Linux box serves it out to an AppleShare
client, it makes it look like one file again.

The VFS just wants to say 'here is a path, open this file and give me
blocks of it'.  The filesystem driver has complete control over what
goes to and comes from the backing store--disk, flash, tape, etc.  Some
filesytems are completely 'made up'--proc, devfs... 

I must be describing this poorly and for that I'm sorry.  It's really
a clever way of making the two systems see the same thing differently/
different things the same.

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[M. R. B. <mr...@0x...>]

* David Willmore <dav...@ia...> on Sun, Dec 08, 2002:

>=20
> That's the advantage of these.  Normal tools will copy the 'special' files
> as well.  Tar up the whole thing and you will save the special info with
> the normal info.
>=20

Ok, first off before *anyone* else responds to this thread, read the pages
at http://mc.pp.se/dc/vms/flashmem.html and
http://mc.pp.se/dc/vms/fileheader.html to gain an understanding of what's
being discussed.

Your above paragraph: think about it.  If I copy a file from VMU, say to
/tmp, and perform an edit or some other operation that changes the size of
the file, what happens to that extra info that was stored elsewhere?

It's invalid and the file (w.r.t to the VMU) is worthless.  The only way
that works is if the editing tool understands the extra little files and
directorires and can update them as soon as the main file is updated.  That
isn't realistic.

>=20
> This requires a whole suite of cross-platform tools that understand how
> one hokey little system mangles its data., ick.
>=20

Heh, have you done Linux kernel development, in particular VFS development?
Do you know the headaches that Adrian has been through getting his driver
to work?  Doing a userspace (cross-platform is an extra bonus)
implementation would simplify things greatly - all data remains intact when
transferring between vmus and the code becomes more than trivial.  Besides,
what do you think the VFS driver is doing with this "hokey little system"?

The only disadvantage is a non-intuitive way to access file data, but at
least it's compatible with every other vmufs implementation (including the
DC browser itself) outside of Linux.

>=20
> Okay, how does this sound.  Make the first block of the file be the speci=
al
> data for that file?  When the first block is read/written, the fs knows t=
o=20
> generate that info/save that info in the 'special' area.  All other blocks
> work normally.
>=20

See my last couple of paragraphs.

>=20
> If the filesystem is HFS (say a disk from a Mac that's been stuck into
> a linux box) then on disk the data is in one big 'file' chunk that one has
> to understand how to split into code and resource 'forks'.  The VFS makes=
 it
> look like seperate files.  If the Linux box serves it out to an AppleShare
> client, it makes it look like one file again.
>=20

At my former university, every student had an account and disk space on a
Unix system.  In each lab, the workstations would map this disk space so
that students could save documents, etc. there.  On the Macs, because they
use "extra" info for their filesystems, they tended to create an ".afpvol"
directory that was hidden as long as you accessed that share on a Mac.

I spent a lot of my time under Unix and I typically trimmed my directory
because of the quota imposed.  What do you think I did with the Mac-created
=2Eafpvol?  rm -rf.  Could the Mac HFS do anything with this info any more?
No.  Did it become problematic when accessing that share on the Mac again?
Just a little, since it held type/creator information and other
HFS-specific bits that made the file recognizable under the Finder.

I say all that to hopefully get you to see that vmufs-specific data will
only ever make sense or be usuable by vmufs - other filesystems could care
less about that extra information, _wherever_ it's stored.

> The VFS just wants to say 'here is a path, open this file and give me
> blocks of it'.  The filesystem driver has complete control over what
> goes to and comes from the backing store--disk, flash, tape, etc.  Some
> filesytems are completely 'made up'--proc, devfs...=20
>=20

vmufs is a block-oriented filesystem.  You can't do magic things that are
persistent because in the end, you're writing to a block device: whether
it's the VMU mapped over MTD, a raw 128k dump of the VMU, or something
else.

As soon as you modify the raw block data in a non-consistent manner, you no
longer have vmufs.

You can always present a special interface to the user, but see above as to
why this won't persist across filesystems.

M. R.

Re: [linuxdc-dev] VMU filesystem question

[David W. <dav...@ia...>]

> >=20
> > That's the advantage of these.  Normal tools will copy the 'special' files
> > as well.  Tar up the whole thing and you will save the special info with
> > the normal info.
> >=20
> 
> Ok, first off before *anyone* else responds to this thread, read the pages
> at http://mc.pp.se/dc/vms/flashmem.html and
> http://mc.pp.se/dc/vms/fileheader.html to gain an understanding of what's
> being discussed.

I have read that in the past and I just reread it as you requested.

> Your above paragraph: think about it.  If I copy a file from VMU, say to
> /tmp, and perform an edit or some other operation that changes the size of
> the file, what happens to that extra info that was stored elsewhere?

*what* extra info?

> It's invalid and the file (w.r.t to the VMU) is worthless.  The only way
> that works is if the editing tool understands the extra little files and
> directorires and can update them as soon as the main file is updated.  That
> isn't realistic.

The filesystem *is* the editing tool.  It displays the card formatted data
in a way that it is editable with standard unix tools.

> >=20
> > This requires a whole suite of cross-platform tools that understand how
> > one hokey little system mangles its data., ick.
> >=20
> 
> Heh, have you done Linux kernel development, in particular VFS development?
> Do you know the headaches that Adrian has been through getting his driver
> to work?  Doing a userspace (cross-platform is an extra bonus)
> implementation would simplify things greatly - all data remains intact when
> transferring between vmus and the code becomes more than trivial.  Besides,
> what do you think the VFS driver is doing with this "hokey little system"?

Yes.  Take a look at the SCSI subsystem.  I've been a linux developer
since version .12 in fall of '92.  I'm not here to start some kind os
pissing match and I'll ask you to respect that.

I'm not complaining one bit about the effort that Adrian has gone through.
I'm just answering a question posed to the list about how the vmufs could
work to make it easier on the user.

The way you put it, there is no point in *having* a vmufs.  Just leave it
as a block device and let a custom user space tool manipulate it--one that
knows all of the rules to its structure.

What I am suggesting is to have the vmufs format the data in such a was
as it will appear as normal files to the user and be editable as such.
I don't see anything in the description of the directory nor the files
that makes this unimplementable.

> The only disadvantage is a non-intuitive way to access file data, but at
> least it's compatible with every other vmufs implementation (including the
> DC browser itself) outside of Linux.

Pardon?  I'm not suggesting changing *one byte* on the card.  We're talking
about the vfs, right?  That's the interface in the kernel between high-ish
level calls (open, write, read, close, dlookup, etc.) and the code which
determines what is allowed and where it goes/comes from on the backing media.

The kernel does not care what the fs driver does with the data it's given
nor how it comes up with the data it presents.  It just wants to give
it paths and files and perform some basic fops on them or get errors trying.

> >=20
> > Okay, how does this sound.  Make the first block of the file be the speci=
> al
> > data for that file?  When the first block is read/written, the fs knows t=
> o=20
> > generate that info/save that info in the 'special' area.  All other blocks
> > work normally.
> >=20
> 
> See my last couple of paragraphs.

I'll go into detail since this appears not to be to your liking:

Starting at the root of the vmufs filesystem we have a root block that 
contains some properties of the filesystem.  It contains the date the
card was formatted.  This doesn't fit well into Ctime, Atime, nor Mtime--maybe
Ctime would be closest.  So, make the Ctime of the fs root directory entry
the format time.  Then it contains color components for the VMU.  These
can appear in a file called .color off of the root dir of the fs.  It can
contain five files called 'custom', 'blue', 'green', 'red', and 'alpha'. Each 
file is one byte in length and just contains the byte that specifies the
property that its name implies.  Writes beyond one byte get an ENOSPACE error
and the file gets truncated.

All of the other properties stored in that block are fs internal and don't
serve any purpose being exposed except for icon shape.  Something similar
to .color can be done with this.  Or color and shape can be combined in an
.icon directory off of root.

FAT is internal and doesn't need exposing.

The directory information is a little more comples and there are a few
ways to do it.  I would suggest that games and data files be though of seperately.
From the root there could be data and game subdirs.  If a file is put into
game, it will be placed starting at block 0 and will be stored contiguously.
This will also allow the fs to fill in the directory info fields file type
and header offset.

If a file is put in data, the files are stored starting at 199 and counting
down.  The fs will also know how to fill in the header offset and file type
from this.

All of the rest of the directory properties for a file are internal and
need not be exposed except for the 'copy protect' flag which can become a
property stored in the .header directory associated with the file.

It looks like all of the data in the header will work well if exposed as
seperate files in the .header directory associated with the file in question.

Shall I go into more detail?

> >=20
> > If the filesystem is HFS (say a disk from a Mac that's been stuck into
> > a linux box) then on disk the data is in one big 'file' chunk that one has
> > to understand how to split into code and resource 'forks'.  The VFS makes=
>  it
> > look like seperate files.  If the Linux box serves it out to an AppleShare
> > client, it makes it look like one file again.
> >=20
> 
> At my former university, every student had an account and disk space on a
> Unix system.  In each lab, the workstations would map this disk space so
> that students could save documents, etc. there.  On the Macs, because they
> use "extra" info for their filesystems, they tended to create an ".afpvol"
> directory that was hidden as long as you accessed that share on a Mac.
> 
> I spent a lot of my time under Unix and I typically trimmed my directory
> because of the quota imposed.  What do you think I did with the Mac-created
> =2Eafpvol?  rm -rf.  Could the Mac HFS do anything with this info any more?
> No.  Did it become problematic when accessing that share on the Mac again?
> Just a little, since it held type/creator information and other
> HFS-specific bits that made the file recognizable under the Finder.
> 
> I say all that to hopefully get you to see that vmufs-specific data will
> only ever make sense or be usuable by vmufs - other filesystems could care
> less about that extra information, _wherever_ it's stored.

That's the same kind of thing I'm proposing.  If you chose to rm -rf the
.header data in the vmufs, you're going to mess things up.  So, uhh, don't
do that if you don't want to mess up the VMU.

The .afpvol trick was just a way for the Mac to translate the way it saw
things into the way the unix box saw things.  If the rules on how this
'hidden' or 'extra' information is handled is built into the filesystem
so that you can't mess it up, you have a very useful tool.  If you just
want to make it look like some simple files, you're going to allow people
more chance to rm -rf your .afpvol so-to-speak.  Is there much use in
anything between what I propose and just leaving it as a raw block device
that only special user mode tools know how to deal with?  Either way
would be valid and the latter would, probably, be cleaner.  I'm not advocating
either.  I'm just suggesting a way to implement the former and cautioning
that anything between the two would be problematic.

> > The VFS just wants to say 'here is a path, open this file and give me
> > blocks of it'.  The filesystem driver has complete control over what
> > goes to and comes from the backing store--disk, flash, tape, etc.  Some
> > filesytems are completely 'made up'--proc, devfs...=20
> >=20
> 
> vmufs is a block-oriented filesystem.  You can't do magic things that are
> persistent because in the end, you're writing to a block device: whether
> it's the VMU mapped over MTD, a raw 128k dump of the VMU, or something
> else.

Every time you manipulate a file on a unix box, you do something magic
to it that is persistent.  Are you writing to a block of the disk?  No.
You're handing a block off to the FS via calls through the VFS and
it is probalby just holding it in memory unitl it feels like doing something
with it.  It decides when it gets written to disk.  No different than
the vmufs could.  It doesn't make to write anyting until you unmount the
filesystem.

> As soon as you modify the raw block data in a non-consistent manner, you no
> longer have vmufs.

Yes, touching raw blocks when you don't now what you're doing is a bad idea.
With the 'make it look like a bunch of files' method, the user never gets
a chance to do that.

> You can always present a special interface to the user, but see above as to
> why this won't persist across filesystems.

Let me see if I can understand what you're getting at.  Let's say you do the
'make it all look like files' trick and you recursively copy the presented
filesystem to another directory.  You edit it there and then try to copy
it back, right?  At that point your cp-r or tar is going to get errors when
things you're trying to write are in violation of what are allowed on a vmufs.
Just as if you had edited them *on* the vmufs--just delayed until you try to
copy them back.  The vmufs enforced these rules.  Think of these files like
the files in some of the 'proc' filesystem directories.  You can't write
to some, some that you can write to don't read as the data you put into them--
think of the scsi attach and detach commands on the scsi section of the 
proc filesystem.  Writes to a file get parsed and end up becoming function
calls into the scsi subsystem.  The 'file' is just a convenient metaphor
for the user and the kernel to share so they can meaningfully communicate
with the other.

Cheers,
David

Re: [linuxdc-dev] VMU filesystem question

[M. R. B. <mr...@0x...>]

* David Willmore <dav...@ia...> on Mon, Dec 09, 2002:

>=20
> I have read that in the past and I just reread it as you requested.
>=20
[...]
>=20
> *what* extra info?
>=20

Read it again.  Every VMU file has an extra file "header" that is not
considered part of the file data, but is used when
displaying/parsing/validating the file's contents.  There should be two
fields that immediately stick out in that header: CRC and "Number of bytes
of actual file data following header". =20

On the proceeding page, there is a field in the root directory structure
labeled "offset of header (in blocks) from file start".  Although this could
be assumed to always be 0 for data files, it isn't a constant so there is a
possibility it could change.

Outside of vmufs, there is absolutely no way to update that data using
standard Unix VFS semantics.  On vmufs of course, the driver can keep it as
part of it's private inode data and update it dynamically whenever the file
is written to.

I'm surprised you asked that question since those fields are the reason Adr=
ian
asked his question in the first place.

>=20
> Yes.  Take a look at the SCSI subsystem.  I've been a linux developer
> since version .12 in fall of '92.  I'm not here to start some kind os
> pissing match and I'll ask you to respect that.
>=20

It was not my intention to start a pissing match - I posed that question as
a valid one since you seem to be missing the big picture, esp. on VFS
issues.  I apologize if my statements were peceived as such.

>=20
> What I am suggesting is to have the vmufs format the data in such a was
> as it will appear as normal files to the user and be editable as such.
> I don't see anything in the description of the directory nor the files
> that makes this unimplementable.

Check LinuxDC CVS, vmufs already exists and does what you mention.  The
problem is not how to expose the content of the files, it's what to do with
file header data that can't be transferred outside of vmufs.

>=20
> > The only disadvantage is a non-intuitive way to access file data, but at
> > least it's compatible with every other vmufs implementation (including =
the
> > DC browser itself) outside of Linux.
>=20
> Pardon?  I'm not suggesting changing *one byte* on the card.  We're talki=
ng
> about the vfs, right?  That's the interface in the kernel between high-ish
> level calls (open, write, read, close, dlookup, etc.) and the code which
> determines what is allowed and where it goes/comes from on the backing me=
dia.
>=20

Sorry if I misunderstood you, but in your preceeding post you proposed
changing the first block of the VMU file data into a non-standard format.

> The kernel does not care what the fs driver does with the data it's given
> nor how it comes up with the data it presents.  It just wants to give
> it paths and files and perform some basic fops on them or get errors tryi=
ng.
>=20

You forget that the kernel does care how the file data are represented.
Please see struct inode in include/linux/fs.h.

>=20
> I'll go into detail since this appears not to be to your liking:
>=20
[...]
> It looks like all of the data in the header will work well if exposed as
> seperate files in the .header directory associated with the file in quest=
ion.
>=20
> Shall I go into more detail?
>=20

Please look at the big picture.  Your above scheme is beautiful for
everything that just does vmufs <-> vmufs.  But what happens when I want to
create a brand new file on the mounted vmu, and I do a "touch myfile"?
Where does the header come from?  In order to remain compatible it must
either come from the user or have a default supplied by the vmufs driver.
Both scenarios aren't pretty but the latter makes it easier for the end
user to use the VMU as a generic storage device.  Note that without the
file header it's possible that the VMU will break for anything outside of
Linux (that hasn't been confirmed yet, but all DC software that uses the
vmu accesses the header).

The case that Adrian mentioned is "cp oldvmu/file newvmu/file".  Under your
proposed scheme this quickly become unmanageable as you now have:

$ cp oldvmu/file newvmu/file
$ cp oldvmu/.header/file newvmu/.header/file

and so on for each file that you want to copy.  Can you see the problem with
that?  Everytime you add a new virtual file that has to be tagged along
when the real file leaves vmufs, you decrease your flexibility.  Even
copying the entire directory tree(s) becomes problematic because as soon as
you copy, say the global VMU color you've destroyed what was on the other
VMU.  And unless you have everything under directories (since vmufs can't
physically support subdirectories) any special file in the root directory
pollutes your namespace.

To me, your scheme sounds more like you would have both a vmufs
driver that exposed the normal Unix files, and a /proc-like implementation
that exposed global VMU metadata.

Unfortunately, there is no precedence for exposing vmufs data in this way
other than MacOS-specific filesystems, which have always needed hacks to
play nice with PC-based OSes (data/rsrc crap comes to mind).  I can't think
of any other filesystem that doesn't do a simple map of file, directory, and
attributes on top of the VFS.

>=20
> That's the same kind of thing I'm proposing.  If you chose to rm -rf the
> .header data in the vmufs, you're going to mess things up.  So, uhh, don't
> do that if you don't want to mess up the VMU.
>=20

I wasn't on HFS/AppleShare when I did the rm.  vmufs can enforce the policy
that you can't remove .header/, but what other filesystem will?

> The .afpvol trick was just a way for the Mac to translate the way it saw
> things into the way the unix box saw things.  If the rules on how this
> 'hidden' or 'extra' information is handled is built into the filesystem
> so that you can't mess it up, you have a very useful tool.  If you just
> want to make it look like some simple files, you're going to allow people
> more chance to rm -rf your .afpvol so-to-speak.

The point is to make it look like simple files (the vmufs files
themselves).  A frequently requested feature of vmufs is (was, before it was
written) to a allow the VMU to function as a generic storage device.
Enforcing such policies within the filesystem driver makes this impossible.

Study the source in the fs/ directory, and see how other filesystems do it.
There is only a limited set of information that is transferable between
filesystems, and no .header/ scheme will ever overcome that.

>=20
> Yes, touching raw blocks when you don't now what you're doing is a bad id=
ea.
> With the 'make it look like a bunch of files' method, the user never gets
> a chance to do that.
>=20

Again, I think your scheme works well when using the VMU as strictly a
VMU-to-VMU transfer device.  I don't think it can hold up to other uses
without the vmufs driver making sweeping decisions over what data the VMU
can hold.  This is what Adrian is attempting to avoid in his design of
vmufs.

(Sorry if it looks like I'm speaking for Adrian in these posts, but this is
just rehashed stuff we've discussed on and off the list in the last few
weeks.)

M. R.