#114 Versioning symlinks
There are times when it would be desirable to have
symbolic links in a directory under revision control
with Vesta. One example is when checking in OS
component packages for use during tool invocations.
Currently our pkg2vesta script
(/vesta/vestasys.org/vesta/extras/pkg2vesta) will
convert symlinks to copies if they refer to files in
the same OS component, or to pieces of SDL in root.ves
if the symlink refers to files in another OS component.
Users may also want to do this for various reasons.
One example is migrating from another system which does
allow versioning of symlinks.
See also the RFE "symlink in volatile directory" which
covers a lot of implementation details in this area:
https://sourceforge.net/tracker/index.php?func=detail&aid=1166276&group_id=34164&atid=410430
We should probably complete the implementation of
support for symlinks in volatile directories first.
Also, the evaluator would need some changes to handle
reading immutable directories and converting their
entries to SDL values, as currently only files and
directories are handled.
Logged In: YES
user_id=95236
How do you want symlinks in source to look to an SDL
program? Should it see them as some new data type, or should
they be automatically indirected through and look like the
files or directories (texts or bindings) that they point to?
The latter seems problematic to me, with lots of open
questions.
You write "Currently our pkg2vesta script will convert
symlinks to copies if they refer to files in the same OS
component, or to pieces of SDL in root.ves if the symlink
refers to files in another OS component." Are you aiming for
a symlink feature that would let you simplify pkg2vesta and
just check in the symlinks as symlinks? That suggests you
want the symlinks to be automatically indirected through
when SDL encounters one.
Logged In: YES
user_id=304837
No, I don't think the evaluator should automatically
de-reference them. I agree that would add a lot of
additional complexity and that it's unclear how some things
would work.
I think the SDL value representation of them should be a new
type, as we discussed in the "symlink in volatile directory"
RFE. (The last conclusion we came to about this is that we
would use a singleton list of a text value with the link
target.) This should be the same representation used for
symlinks created by tools in the return value of _run_tool
and symlinks created in SDL before tools are run.
Simplifying pkg2vesta is not actually my primary concern.
We have a situation at Intel where we're importing from
another revision control system and this other system does
allow for symlinks to be under revision control. We'd like
to make this as seamless as possible, so this was raised as
a concern. (Plus the issue has come up from time to time,
and it's worth at least getting it in the tracker.) Making
it a simpler process to import an OS component package would
be a nice additional benefit.
We could also impose some restrictions on symbolic links in
immutable directories, such as not allowing them to point
outside the immutable directory. Of course such a
restriction would again necessitate some massaging of links
in pkg2vesta, so maybe it should be optional (maybe a
vadvance pre trigger that could be disabled for working
copies created by pkg2vesta).
Logged In: YES
user_id=95236
I'm still not sure if I grok all of the concept. Is this right?
1) A symlink in an immutable or mutable directory is
manifested through NFS as a symlink, so editors and other
tools run against the NFS export get standard symlink
behavior -- i.e., the symlink is dereferenced by default but
you can choose not to dereference by using lstat,
O_NOFOLLOW, etc.
2) Symlinks in immutable directories that are visible to SDL
are manifested within the evaluation as something special,
say a singleton list.
3) A tool can create a symlink in a volatile directory.
This gets returned in the result using the same singleton
list convention as in (2).
4) Presumably you can supply a symlink to a _run_tool by
putting a singleton list of text into its root.
5) Presumably if you ship a binding with a singleton list of
text in it, a symlink is created.
This all seems fairly plausible. In some cases symlinks
should "just work": you could create a symlink in a mutable
directory, check it in, have a tool see it and use it, and
even ship it if needed And/or a tool could create a
symlink, return it in a result, and you could ship it.
There are a couple of aspects that aren't so nice, but I
don't have a suggestion as to how to fix them. (The
existence of problems like this is why we didn't support
symlinks in Vesta from the start.)
A) Although _run_tool sees a symlink as a symlink, SDL code
doesn't, so if you move functionality from a tool into SDL
or do some kind of preprocessing in SDL, having a symlink in
the source could break things that people would have
expected to work. You can probably get around this in
specific cases by writing SDL code that explicitly tests for
links and dereferences them, but this is messy, and you
could easily miss a spot where it's needed.
B) SDL programs do a lot of sophisticated custom assembly of
namespaces. Each tool invocation gets its own namespace
assembled from various pieces. It's quite possible that a
symlink value to mean different things at different steps in
its lifecycle (some subset of mutable dir -> immutable dir
-> SDL -> _run_tool -> more SDL -> ship destination). The
problem isn't too bad for short relative links, but the
longer the path and the more ..'s, the more likely the
meaning is to shift. For absolute links it seems even more
likely that the meaning will vary.
Logged In: YES
user_id=304837
Originator: YES
The lack of support for symlinks seems to have become a bit
of a thorn in the side of some Intel users. I'm not making
any immediate promises, but it seems more likely that we'll
move forward with implementing this some time this year.
(Thankfully we already have a plan that's pretty complete.)
I'd like to talk a little about one thing Tim had to say:
> A) Although _run_tool sees a symlink as a symlink, SDL
> code doesn't, so if you move functionality from a tool
> into SDL or do some kind of preprocessing in SDL, having a
> symlink in the source could break things that people would
> have expected to work. You can probably get around this
> in specific cases by writing SDL code that explicitly
> tests for links and dereferences them, but this is messy,
> and you could easily miss a spot where it's needed.
This issue does seem a bit troubling to me. Suppose one
piece of SDL code puts a directory and a symlink to it into
./root with something like:
. ++= [ root = [ foo = [ ... ],
bar = <"foo"> ] ];
Suppose some other piece of code comes along and tries to
put something into the directory "/bar" that's really a
symlink:
. ++= [ root/bar/x = ... ];
Given the semantics of ++, that will replace what used to be
a symlink to a directory with a completely separate
directory.
I think that once we add support for symlinks SDL writers
will want some way to augment a binding that represents a
filesystem with semantics that follow symlinks in the same
way normal filesystem operations would. I don't think we
should change the semantics of the ++ operator, but adding
new primitive functions might not be a bad idea.
This could be implemented in a library of SDL functions.
However I'm a bit concerned that such functions might be
difficult to write and could have unfortunate performance
and dependency characteristics. That concern is based on
some similar work that I've done in SDL for the make_compat
bridge (which I still consider a prototype):
/vesta/beta.vestasys.org/bridges/make_compat
That includes functions for working with text string paths
(i.e. "/foo/x/bar/y"). "get_path" looks up some text string
path in a filesystem. "place_path" creates a partial
filesystem with a value at a particular text string path.
Those two functions would become much more complicated if
they were to handle symlinks.
I'm not going to try and write a spec right now, but we may
want to add primitive functions for at least these two
operations:
- Overlay a partial filesystem onto another in the way ++
would but also follow symlinks if any are present in the
left-hand operand.
- Get the value at some path, following symlinks if they are
encountered.
Given that manipulating bindings representing encapsulated
filesystems is so common in SDL, I don't see adding a few
primitive functions to support