My comments inline:
Drake Wilson <drake@...> writes:
> In 030-C-struct.muf.t, I see:
> > "muf" inPackage
> How does muq-db-c (or related subprocesses) know to put this library
> in the "MUF" database? And (how) is it possible to use this feature
> myself? Putting my miscellaneous custom libraries in a "FOO" (or,
> "foo" -- which is recommended?) database would be nice, but is awkward
> because of having to put a new file in to create the database.
You -are- persistent! :)
I believe you're the first person to get this far before
deciding the entry barrier to getting Muq to do something
useful was just too high to be worth the effort.
The multi-db support was one of the last things I put into
Muq: much of the 10th year of development was devoted to it
(along with making the .db files portable between 32 and 64
bit and big/little endian machines).
As a user of TinyMUCK, and an administrator of a >kilouser
virtual world, I was really sick of the pain and mess
involved in installing system library updates, in moving
a user's data from one server to another, &tc &tc.
I wanted something cleaner, and I couldn't find any
very good prior art (other than Unix disk partitions,
which can be un/mounted, moved between systems &tc),
so I had to pretty much roll my own. This is one
thing which I don't think MOO, Smalltalk, Lisp, or
any of the similar systems I've looked at address
at all well.
We already had packages, and Occam's Razor suggested
db==package as the first try, to keep things simple,
but upon consideration, that just didn't seem practical.
One will often want lots of little packages for namespace
cleanliness without wanting to have a separate db file
for every damn little one of them, especially with the
dangers that poses of loading a mixture of .db files
which don't really belong together.
I/we could have set up an 'inDb' &tc series of operators
parallel to the existing 'inPackage' &tc series, but that
seemed like too much of a good thing, to me. Muq already
has enough stuff to learn without users having that batch
of operators thrown at them as well.
So I thought on balance the best way to handle the issue
was to mandate user==db: One .db per user, and one user
per .db, in general. This is simple, it does the right
thing 90% of the time, it sweeps all the .db issues
under the rug for the typical user, and the sysadmin
(who we suppose can handle a bit more complexity) can
create one fictitious "user" per unique system library
that s/he wants in a separate .db file.
So the short answer to your question runs so. The
1) There is one logical .db per user.
2) We create extra fictional 'users' for libraries
and packages which we want in separate .db files,
for independent distribution.
3) We select the .db file in which to place a package
by logging in as that user.
One nice side-effect of this system is that since
Muq automatically assigns every user a unique private
and public key, every separately distributable .db
file has its own public/private key identity, which
can be used for signing / validating / &tc releases.
This does require that we not distribute the private
key as part of the package, of course... :)
Since these facilities have never been tested in
a production environment, just coded up based on
my experience in a production environment, there
are undoubtedly some rough edges to be smoothed
out. But I think we have basically a pretty
workable concept here -- a sound foundation on
which to build. And that's what I was trying to
get in place for release 0.0.0.
It may be that in the long run this "db==user" concept will
prove too simplistic, and we'll have to add something like
"inDb" operators to the virtual machine.
But it is much better to start too simple and then add stuff
as needed, than to start out too complex -- since it is
extremely hard to remove functionality from a virtual
machine once it is in reasonably wide use. ForTran had
"deprecated" a lot of stuff over the last half century, but
last I heard they had still never actually removed anything
from the language. Just too much risk of huge, ancient
programs breaking as a result.
> Also, distributing libraries, I suppose, should be done in both dbfile
> and source format; however, how the source files should be organized
> in a MUF library distribution seems to be currently undefined --
> coordinating three-digit numbers across all MUF library developers may
> be reasonable now, but doesn't scale well at all. I should write some
> installation scripts to help with this, but I'd like feedback on how
> the sources should be organized before I start anything.
> ---> Drake Wilson
You're exactly right.
In the long run, I'd like to have the source code retained
in the compiled packages, and have the .db files be the
preferred distribution unit. And to have a squeak-like
interactive programming environment within Muq for coding
in style. :)
In the short run, the compiler doesn't retain source code,
I don't think (people were bugging me about the .db files
being too big for their taste) and we don't have the above
sort of programming environment yet, so distributing source
separately is the most practical idea, I think, and then
distributing .db files for the things that don't make
sense to distribute as source, such as chunks of living
The numbering system is of course a hack, pure and simple.
I could have taken time to write a full-blown package
management system, but I thought 10 years before first beta
was already long enough, and that this was one example of a
problem where a quick-and-dirty fix would do fine initially:
It isn't committal, so we can go back and retrofit something
more sophisticated later without breaking backward
compatability. (My mantra for the last year or two of Muq
pre-beta development was to work on only those things which
would break backwards compatability if I did them after
In my opinion (no doubt biased, since I'm a heavy Debian
Linux user) the Debian package system is the most
successful, well-tested prior art which we can rely on
on this front.
It is possible that we should figure out a way to use
the existing Debian package infrastructure as is,
rather than re-inventing that wheel.
But I expect that will prove awfully awkward and unpleasnt,
and that we'll want to craft something more suited to Muq
and the virtual world environment. If so, we should study
and steal what makes .deb files work well in terms of
fundamental structure. I think that is mostly 'requires',
'conflicts' &tc relationships between packages, plus
pre/post install/remove scripts, but I haven't looked
carefully at this problem. Then we should steal those
wheels for the Muq .db situation, plus do some things right
from the very start such as having proper signatures to
validate .dbs as being from who they say they are.
For the moment, just having the build scripts load the
source files in alphabetic order and then using the current
numbering scheme to (crudely!) express dependency
relationships between them is a very cost-effective way to
get started: It takes virtually no coding and gets the job
done for now.
My next planned short-term improvement on that front was
to break up the contents of muq/pkg into subdiretories,
one per logical system (such as system libraries, virtual
world server &tc).
There are some only semi-resolved issues involving
cross-references between .db files. The current logic does,
I believe, a good job of not crashing hard in the case of
dangling links &tc, but what is lacking is a clear contract
between consumer and producer packages: What should each of
them do to ensure that the system keeps working sanely after
a library update? If you rebuild a library from scratch and
everything has a different identity, how do links from the
client .db files find the intuitively proper destination?
My interim answer for now is that client packages should
only maintain pointers to package symbols in producer
libraries, and maintainers of producer libraries should be
at pains to never destroy and recreate such symbols. (Or
the package objects in which they live.) The producer
library maintainer is, however, free to destroy and recreate
any other objects in the .db file as part of maintainance,
and the consumer library which stores pointers to them
-will- break and has nothing to blame but itself.
For a Generation II version of the .db filesystem support in
Muq, I'd like to go a bit further than this. Without making
any hard and fast promises, I would like it to make a
good-faith effort to do significantly better than the above.
Specifically, when you unmount a .db file, I would like the
server to essentially convert "forbidden" cross-pointers
from other .dbs into the interior of the dismounted .db with
the moral equivalent of unix filesystem symlinks where-ever
possible: It will search for a path from the root package on
the .db to the referenced object, and replace the forbidden
hard link with a symlink-like proxy object if possible.
This way, when an updated version of the dismounted .db is
mounted in place of it, the converted crosslinks can (often)
still be resolved to the appropriate object (even if its
low-level binary identifier is now different).
(Perhaps I should mention that the current implementation
already guards such cross-links with an 8-bit or so
generation number, so that with moderately probability,
dangling links get detected and resolved to something like
NIL links. Even in the cases where this safety-measure
fails, there shouldn't be any C-server level crasher
bugs, just some unexpected stuff going on in softcode.)
Anyhow, that's sort of where we are. This is one of
those fronts on which (imho) Muq has at least a pretty
good first cut at a solution, where other systems have
yet to even arrive at the question, really.
Was any of that in the least intelligible? :)