YAML Ain't Markup Language

On Thu, Nov 08, 2001 at 07:59:04PM +0200, Oren Ben-Kiki wrote:
| >  2.  For a top-level scalar, no indentation is 
| >      required and a leading separator is mandatory.
| 
...

| I guess I'll just have to get used to:
| 
| --- |
|     #!/bin/perl
| # Throwaway comment
|     # perl comment
|     print "Oh well.\n";

Sorry... but IMHO, if we go this far, we should 
just make the top level production a plain, unadorned
map xor list and then we don't need sections or 
any other breaks from the core syntax.  

I'm growing convinced that this section stuff is 
extra complexity that we don't need.  If someone 
wants sections, let them use RFC 2045-2049.  The
MIME RFC's do a wonderful, standardized job of
handling multi-part content.  Let's not re-invent
the wheel.

If we want... we can require YAML processors to
be able to handle a MIME document.... This would
give Brian his multiple document syntax...

 str$ = yaml_encode($a,$b)
 ($a,$b) = yaml_decode(str$)

If more than one argument is given to yaml_encode or
yaml_decode then it uses a MIME wrapper.

Best,

Clark

Sorry for my non-constructive response.

On Thu, Nov 08, 2001 at 07:59:04PM +0200, Oren Ben-Kiki wrote:
| >  1.  The document separator is '--' UNIQUE, where 
| >      UNIQUE can be '-', giving the '---' separator.
| 
| Possibly we can get away with just '---' - see below.
| 
| >  2.  For a top-level scalar, no indentation is 
| >      required and a leading separator is mandatory.
| 
| A problem with it:
| 
| --UNIQUE !perl-code |
| #!/bin/perl
| # Is this a throwaway comment
| # or part of the perl code?
| print "S**t!\n";
| --UNIQUE
|
| It is *so close*. Is there any way around this? Turning off throwaways in
| top-level blocks would work... but is so arbitrary... Sigh.

Ok.  A non-arbitrary solution.  Allow throw-away comments
within lists and maps at any indentation level, but not
within scalar values.

#throw away
map:
	#throw away
	one: |
		# not throw away
		this is code
	# throw away
	two:
		# throw away
		- one
		# throw away
		- two
		# throw away
	#throw away
#throw away

This solves that problem?

| It seems that due to *just this tiny problem* we'll have to indent all
| top-level scalars. Well, at least if we do indent them, we can go back to
| using a simple '---' separator, so there's some silver lining in
| this cloud.

IMHO, if this new syntax ('---') isn't going to give me any 
additional power (like easy cut&paste), then we should just
use a single dash ('-') and make the top level production a
sequence.

| --- |
|     #!/bin/perl
| # Throwaway comment
|     # perl comment
|     print "Oh well.\n";
| ---

This isn't even intuitive.  You don't comment scalars.

     const char *boogle = "this is /* not a */ comment ";

| >  3.  The separator resets all anchor references,
| >      preventing one document from referencing another.
| 
| Yes.

On retro-spective, this won't work.  Imagine a map object
X and a list object Y, both of which point to a scalar Z.

$a = <points to X, scalar Z is shared>
$b = <points to X, scalar Z is shared>
$str = yaml_encode($a,$b)
$a, $b = yaml_decode($str)

Now $a and $b have their *own* copy of Z.  I think
that this isn't intuitive with Brian's use case above.

Sighs,

Clark

Clark C . Evans wrote:
> Ok.  A non-arbitrary solution.  Allow throw-away comments
> within lists and maps at any indentation level, but not
> within scalar values.

Yes! Thank you! That's exactly the right thing. Problem solved. And it looks
so much better, too! This is a great idea regardless of the separator issue.

> IMHO, if this new syntax ('---') isn't going to give me any 
> additional power (like easy cut&paste), then we should just
> use a single dash ('-') and make the top level production a
> sequence.

But it does give us extra power:

> | >  3.  The separator resets all anchor references,
> | >      preventing one document from referencing another.
>
> On retro-spective, this won't work.  Imagine a map object
> X and a list object Y, both of which point to a scalar Z.
>
> $a = <points to X, scalar Z is shared>
> $b = <points to X, scalar Z is shared>
> $str = yaml_encode($a,$b)
> $a, $b = yaml_decode($str)
>
> Now $a and $b have their *own* copy of Z.  I think
> that this isn't intuitive with Brian's use case above.

Here I rely on the experience from Java. They have the same problem, when
serializing multiple Java objects to a stream. They recognized the need to
"reset the anchors" as you put it, and there is explicit API to control
that. Pseudo-code (classes and method names are wrong):

serializer = new Serializer(writer);
serializer.serialize(obj1);
serializer.serialize(obj2); // Share
serialier.resetAnchors();
serializer.serialize(obj3); // Don't share

Consider 'serialize' to be the equivalent of my proposed 'append_to_list'
call. As I said, I'm not driven by Brian's Perl magic needs, though I think
that an anchor-reset separator solves them (I mentioned this explicitly and
he didn't seem to mind it).

I think that explicit, separate append-to-list, end/start-document, and
write-as-whole-document methods are both useful and *necessary* to make YAML
a useful serialization format. Even in Perl :-) This requires separators.

Have fun,

    Oren Ben-Kiki

On Thu, Nov 08, 2001 at 05:03:37PM +0200, Oren Ben-Kiki wrote:
| Or, for that matter, a '---' :-) Also note that using a 
| '---' you wouldn't need to indent the scalar!
| 
| --- !typed \
| Non-indented - I love YAML!
| ---

ok.  I suppose that you don't mean '---', but you
are talking about '--' UNIQUE, where UNIQUE='-' 
is this correct?  Otherwise the above scheme 
falls apart when the scalar contains a 
<eol> '---' <eol> string.

| --- !map-type
| !key-type key : value
| --- \
| unindented scalar
| ---
| - list entry
| --- \
| "another scalar"
| ---
| "another
| key" : value
| 
| Back references are not allowed to go across '---' for safe splitting.
| I think this is clean, readable, solves Brian's problem, the concatenation
| problem, the splitting problem, and even the version problem:
| 
| --- YAML-1.0 !vector
| ...
| --- YAML-9.5 ...
| ...

Ok.  So a YAML document is a series of fragments.
And each fragment is a map, seq, or scalar?  This
is a change to the information model...  I'll go
with it if Brian agrees.

Best,

Clark

I think one more restriction would be needed...
the starting separator is optional if the YAML 
text contains a map or a list; but is mandatory
if the text contains a scalar.

On Thu, Nov 08, 2001 at 10:24:31AM -0500, Clark C . Evans wrote:
| On Thu, Nov 08, 2001 at 05:03:37PM +0200, Oren Ben-Kiki wrote:
| | Or, for that matter, a '---' :-) Also note that using a 
| | '---' you wouldn't need to indent the scalar!
| | 
| | --- !typed \
| | Non-indented - I love YAML!
| | ---
| 
| ok.  I suppose that you don't mean '---', but you
| are talking about '--' UNIQUE, where UNIQUE='-' 
| is this correct?  Otherwise the above scheme 
| falls apart when the scalar contains a 
| <eol> '---' <eol> string.
| 
| | --- !map-type
| | !key-type key : value
| | --- \
| | unindented scalar
| | ---
| | - list entry
| | --- \
| | "another scalar"
| | ---
| | "another
| | key" : value
| | 
| | Back references are not allowed to go across '---' for safe splitting.
| | I think this is clean, readable, solves Brian's problem, the concatenation
| | problem, the splitting problem, and even the version problem:
| | 
| | --- YAML-1.0 !vector
| | ...
| | --- YAML-9.5 ...
| | ...
| 
| Ok.  So a YAML document is a series of fragments.
| And each fragment is a map, seq, or scalar?  This
| is a change to the information model...  I'll go
| with it if Brian agrees.
| 
| Best,
| 
| Clark
| 
| 
| _______________________________________________
| Yaml-core mailing list
| Yaml-core@...
| https://lists.sourceforge.net/lists/listinfo/yaml-core

Clark C . Evans [mailto:cce@...] wrote:
> I think one more restriction would be needed...
> the starting separator is optional if the YAML 
> text contains a map or a list; but is mandatory
> if the text contains a scalar.

Yes; because a top-level scalar requires an indicator, and an indicator
requires a separator.

> | ok.  I suppose that you don't mean '---', but you
> | are talking about '--' UNIQUE, where UNIQUE='-' 
> | is this correct?  Otherwise the above scheme 
> | falls apart when the scalar contains a 
> | <eol> '---' <eol> string.

Yes.

> | Ok.  So a YAML document is a series of fragments.
> | And each fragment is a map, seq, or scalar? This
> | is a change to the information model...

I prefer the current draft wording. A YAML stream is a series of YAML
documents, each document being a map, sequence or scalar. This leaves the
information model untouched.

> | I'll go
> | with it if Brian agrees.

Great. Brian?

Have fun,

	Oren Ben-Kiki

On Thu, Nov 08, 2001 at 11:15:41AM +0200, Oren Ben-Kiki wrote:
| I think I missed the point of the latest ideas.

I believe Brian's need for a new top level separator
construct is that he has two different "types" of 
sequences: lists and arrays; or as you call them 
vector and array.  Rather than introducing a brand-new 
top level syntax for vectors (which would only apply 
at the top level).   I was proposing that he use typing
rather than introducing a new syntax.

| I've no problem with renaming "list" to "sequence",
| though I think that:
| 
| empty: !sequence
| 
| Is a bit too much. Would using 'seq' be acceptable?

Fine.

| As a part-time Perl programmer I've never noticed the 
| difference between list and array, and I *did* read the 
| Camel book cover-to-cover. For example, how does one 
| declare a list variable?

From Brian's examples, the "list" type seems to be
introduced during function calls and returns; my
father calls these fellas "transfer vector" as this
was standard IBM 360 word used to describe the
variables put on and retrieved from the stack
during a function call.

| Python tuples vs. lists can be neatly covered by an
| explicit type (as would Perl arrays vs. lists if there
| is such a thing). If you feel that we should
| define two additional core/common types, 'list' 
| and 'tuple', in addition to 'seq', so be it - as long 
| as you can come up with a good language-independent 
| definition of seq, list and tuple which would make
| sense to, say, a Java or a C++ programmer :-)

ok.

| About attaching descriptors to the top-level. That's
| covered in today's spec:
| 
| !typed
| - top-level list

ok.  One question.  If I have two of these buggers 
and then I concatinate them I get...

!typed
- top level list
!typed
- top level list

Is this valid?  This is the reason why I chose the 
comment syntax.

| !typed
| top-level : map

I'm confused with this one.  Is the top-level
key typed, or is the map?  Another good reason
why the comment syntax works.

| 
| !typed \
| 	top-level scalar

This is a good example of why I don't like top-level 
scalars.  One could just as easily add a "-" in front, 
and then we don't have to have special production
rules... this works especially well if the type of
the top level list is a "transfer vector".

| !typed top-level map key : with value

Kinda subtle hunh?


!typed "top level
map key" : with value?

| No need for magical comments.

if you don't like the magical comments, that's
ok.  But how do we do concatination of docs 
in this case?

| Also, adding descriptors to the top-level prevents 
| concatenation (unless you allow for separators), be 
| it using the above current syntax or using magical
| type comments (Ugh!). Speaking of which, I *strongly*
| feel that placing *any* standard YAML semantics in
| '#...' comments is a serious mistake.

Ok.  Then let's burn a "special" indicator to 
mark version and give discriptos to top level
constructs.  Let's use, for example the ?

?YAML v1.0 !vector

The advantage of this is that we don't need
another sequential construct.  We add a single
"roadsign" (later road signs must be identical
or it is an error), remove top level scalars,
and we are done.

Best,

Clark

I'd rather stick to top level sequence (-) xor map (:)
syntax and add a "?YAML v1.0 !vector" declaration.

/ Separators: I'm warming up towards the separator syntax.
| 
| - It allows safe splitting:
| 
| Note Splitting != Concatenation. Separators make *splitting* safe, in the
| sense that it is always safe to textually cut a YAML document surrounded by
| separators and feed it into a YAML system on its own. This is *not* true for
| a top-level list entry, because it may contain references to nodes in
| previous list entries. *Concatenating* is safe for both - you can always
| attach another top-level list entry at the end, same as you can attach
| another separated document.
| 
| - It allows for concatenation even if there are top-level descriptors.
| 
| --- !type1
| ...
| --- !type2
| ...
| ---
| 
| - It allows us to *NOT* indent top level scalars:
| 
| --- \
| Unindented top-level scalar, hurrah!
| ---
| 
| In fact I'd make the separator mandatory if there are top-level descriptors
| or indicators (there's your place to add a YAML version if we ever need
| one). I rather like this and I think that so would Brian. The beauty of it
| that the above is almost exactly covered by the current spec.
| 
| Repeated map keys - I like Clark's idea of making the first key win (for
| overriding configuration file entries). I think this should be reported as a
| warning by the parser, however.
| 
| Indentation, I gather Brian's idea is "use tab for two 4-space levels". I
| see a big problem with it. What happens when one writes SP SP SP SP TAB: is
| that 2 or 3 levels of indentation? An editor/printer will display is as two,
| but it would actually be three (A possibly unrelated fact is that really old
| testament books were written without white space, which lends some credit to
| Brian's opinion of where white space comes from :-)
| 
| At any rate, I think the only valid options we have are tabs only or 4
| spaces only. Anything else is a mess. I don't care which one we choose, as
| along as we inhume this vampire (chop off its head, put some dirt in its
| moth, a stake through its heart, and bury it under a crossroad).
| 
| In short, I'm for keeping things as they are in the current draft. It is
| better then we give it credit for. With minor changes:
| 
| - indentation (maybe),
| - changing 'list' to 'sequence' (and *maybe* adding list/tuple),
| - going through the production to verify the above top-level
| descriptors/indicators idea works,
| 
| I think it would make for a good release candidate.