YAML Ain't Markup Language

On 21/05/02 15:08 -0400, Oren Ben-Kiki wrote:
> Clark C . Evans wrote:
> Tempting, especially since we can boil it down to two styles - a *big*
> simplification. The problem is, it *isn't* very readable, and it *isn't*
> very nice to edit manually. This is a clear tradeoff between being
> manual-editing friendly and simple-elegant.
> 
> Compromise: Support just *three* styles.

I like this proposal. It allows us to add 'mixed' later on. I think we'll
find this to be big use case. Think "installation instructions".

> Folded: The most widely used style - "Just text".
> 
> Block: Used less often - "What you see is what you get".

I really think you'll find that Block is by far the largest use case. At
least I do. YAML itself is a block format. As data becomes more
self-referential in YAML, you'll see block used everywhere. I already do. I
almost never want folded.

> Escaped: The style of last resort - "Can represent anything".
> 
> I'd further restrict the styles so that trailing empty lines are stripped.
> If you want them - use the escaped style:

I like that. Why didn't we think of it before?

> Indicators:
> 
> folded: |]
>    This is folded. It
>    ends with a line break.
>    Just one, even if there
>    are multiple empty lines
>    trailing it, like in here.
> 
> 
> block: |
>     This is a block. No
>     folding. Again, it
>     always ends with one
>     line break, and empty
>     trailing lines are ignored.
> 
> 
> escaped: |\
>     Like a block except that
>     \\ escape sequences are
>     expanded, and escaped \
>     line breaks are ignored.
>     Again, empty trailing lines
>     are stripped. If you want
>     them (RARE!), write:
> 
> 
>     \
> 
> Advantages:
> 
> - Only 3 styles. Much simpler than today's 8.
> 
> - "It is better to be restrictive up front". Let's just support the *useful*
> cases at the start.
> 
> - If we change our mind in the future (deciding more styles are necessary),
> the |<stuff> indicator format allows us to add them seamlessly. This format
> is clearer (| == multi-line scalar, no exceptions; what follows next says
> what special processing is done).
> 
> - We were there before, and got lured by the promise of a "generic" model to
> the current 8-styles mess. In retrospect, that was probably a mistake. I
> think YAML's current 8 scalar styles is its most "scary" feature to a newbie
> (what do all these line noise characters mean? Yikes!).
> 
> Thoughts?

We still allow the indentation number, right?

   --- |]3
        An indented
      paragraph.
   ...

Cheers, Brian

Brian Ingerson [21/05/02 13:19 -0700]:
> > Escaped: The style of last resort - "Can represent anything".
> > 
> > I'd further restrict the styles so that trailing empty lines are
> > stripped. If you want them - use the escaped style:
> 
> I like that. Why didn't we think of it before?

Because it requires unbounded lookahead if you want to do it in a streaming
application. In order to know if an empty line is "significant", you have to
keep reading until you find either another non-empty line in the same block,
or some token that means the block has ended. If you're in a non-streaming
application, you can treat all empty lines as significant, then do a second
pass on the loaded string and remove any trailing newlines.

One solution I thought of is to keep a running count of consecutive empty
lines as you're reading in the block. If you see a non-blank line, you
"flush" the empty line buffer and continue reading. If you get to the end of
the block, you just forget about those trailing newlines.

Later,
Neil

On Tue, May 21, 2002 at 01:19:37PM -0700, Brian Ingerson wrote:
| > Compromise: Support just *three* styles.

Fold, Block, Escape; no Chomp and no Escaped Fold?

| I like this proposal. It allows us to add 'mixed' later on. I think we'll
| find this to be big use case. Think "installation instructions".

Ok.  By Mixed, you mean that it is Folded when lines begin at
the first column, and Block if the lines are intended?  Currently,
I don't have a use case for indented chunks; so why not just 
re-define "fold" behavior to only apply at the first column?

  --- |]
     This is a folded line where this -->
     <-- carriage return is a space.

        However, this is blocked since 
        it is indented.  It has three items:
        - one
        - two
        - three

     This, once again is a folded paragraph
     so the new line is not significant.

Which would be equivalent to...

   --- "This is a folded line where this--> <-- carriage return is a space.\n   However, this is blocked since\n   it is indented.  It has three items:\n   - one\n   - two\n   - three\nThis, once again is a folded paragraph so the new line is not significant.\n"

Yes?

| I really think you'll find that Block is by far the largest use case.

Probably, depends on how you count it.  ;)

| > I'd further restrict the styles so that trailing empty lines 
| > are stripped. If you want them - use the escaped style:
| 
| I like that. Why didn't we think of it before?

We were allowing comments to appear within a scalar, which
would then create unbounded look-ahead in some cases.  I 
like Neil's approach of counting new lines.

|    --- |]3
|         An indented
|       paragraph.

How about | is blocked, |\ is escaped, and |; is folded.  For
some reason the ] is kinda grinding me. 

;) Clark

On Tue, May 21, 2002 at 05:37:43PM +0200, Rolf Veen wrote:
| http://www.helide.com/g/yaml/

Cool production summary and XML base.  You make me want
to write an XSLT tool to generate the appropriate HTML
version of our spec.  Time... 

Best,

Clark

hi rolf,

i have downloaded it and will have
a look on it tonight.

ciao robertj
------------------------------------------------------------
Robert Kuzelj  
Gaissacherstrasse 7           email: robert_kuzelj@...
81371 Muenchen                tel:   0177/5302230

the trinity of desirables of (software) architecture:
Firmitas, Utilitas, Venustas (marcus vitruvius 20 BC)
strength, utility, beauty

robert_kuzelj@... wrote:

> i would like to make a port to c#
> from java.
> 

My Java YAML parser is a work in progress. I plan
to setup a web page for it, because attachments
don't survive the mail archive. To be announced.

In any case, it is not an example of a theoretically
based design, simply because I don't know parser
theory. Nevertheless, that is not an excuse for
not doing a good job.

I think Oren is, quietly, doing another
implementation, so maybe he can say
something about his plans.

:-) Rolf.

hi ralf,

> My Java YAML parser is a work in progress. I plan
> to setup a web page for it, because attachments
> don't survive the mail archive. To be announced.

> In any case, it is not an example of a theoretically
> based design, simply because I don't know parser
> theory. 
since i dont know parser-theory either i wouldnt mind.

>Nevertheless, that is not an excuse for
> not doing a good job.
maybe having somebody doing a port to a semantically
very similar language would be benefical for both of us.
just like having a sparingspartner.

> I think Oren is, quietly, doing another
> implementation, so maybe he can say
> something about his plans.
oren, any comments?

ciao robertj

------------------------------------------------------------
Robert Kuzelj  
Gaissacherstrasse 7           email: robert_kuzelj@...
81371 Muenchen                tel:   0177/5302230

the trinity of desirables of (software) architecture:
Firmitas, Utilitas, Venustas (marcus vitruvius 20 BC)
strength, utility, beauty

Hello Robert. 

If C# has a native interface like Java's JNI, I would 
propose that you join our "C" project and make a C# 
binding of the "C" language parser.  This would probably 
be the quickest path to a working parser, and as Neil's
parser matures and gets for features it would be relatively
easy to update the C# bindings to take advantage of the
new features (as we add YPATH, etc. to libyaml).  For those
doing Java, I'd suggest the same.  By focusing on a single
code base you'd also help us find bugs (or assumptions that
don't work with Java/C#) early on and thus make it better 
for all of us.

As for a native C#/Java parser, I can probably help some
(as I've written parsers before).  There are two approaches,
one is to hand-code your own lexer/parser or to use a 
LEX/YACC like tool.  There are both C# and Java generators
for ANTLR, for example, so a semi-joint C#/Java project may 
make some sense...   I've never used LEX/YACC or ANTLR, but
I'd guess that learning ANTLR can't hurt!  Neil used YACC
(bison) but not LEX.  Neil wrote his own lexer in C.  So,
if you want to hand-code your lexer, you could probably 
work directly from the "C" code.  Also, the BISON grammer
is probably useful as well... even if you are going to hand
code your parser.

In any case, if you want to keep me in the loop I can 
help review code and make suggestions.  Although I don't
know C# and my Java is very dated (1997).

The API Nazi,

Clark

I don't see what the problem is with ']'. Just make it a modifier:

    --- |
    block
    --- |]
    folded
    block
    --- |-
    chomped block
    --- |]-
    chomped
    folded
    block

And so on. Isn't that simple? The trick to avoiding the counting problems is
to parse the |* tokens atomically. The C libyaml currently has to maintain an
"open count" to balance things correctly -- but as a modifier it's simpler,
since you just look for '|', then the characters '-\]' in any order, setting
a few flags along the way.

This has the following advantages:

   (a) not very different from the spec: s/]([-\\\d]*)/|]$1/g
   (b) eliminates the ambiguity between [a, b, c] and ]
   (c) ']' still implies folding. Probably more so than ^ and >.

Let's try and keep things simple. ;-\ is not simple ;-\1

Later,
Neil

> That is, using |; |;\ |-; |;-\ (Hmmm - inconsistency here - 
> does the '-' come before or after the '-'?

I meant before the ';' of course...

Have fun,

	Oren Ben-Kiki

Brian Ingerson [mailto:ingy@...] wrote:
> I think we're missing the whole point of what I really want here.
> Clark alluded to it but, what I really want is the ability to extend
> the multiline leaf formats that we support. One very useful format
> would be a mix of block and folded. Lines beginning with whitespace
> (as content) would be block. Lines beginning with non-whitespace would
> be folded. I see this pattern in textual data all the time. (If you
> know Perl, think POD)

Hmmm. Even if you make this a format, extending it won't be trivial (it
effects parsers/loaders everywhere). And IMVHO str is too basic a type to
extend it like that later on.

> Anyway, we'll probably want to extend our multiline styles.

I, for one, would be happy if we could have *less* styles. The trouble is, I
see the sense in all our three variables (chomping, folding and escaping).

> I proposed
> to Clark that 'folded' was merely another string format like 'mixed'.
> Everything could be specified using our current block format with some
> kind of format or style modifier. And the reformatting would 
> be done at the Loader level, not by the Parser.

That means that we can only apply these formats to !str. On the up side, it
means that we would have just one "style" of multi-line scalars (block). For
type families such as binary, it is easy enough to specify that all white
space is ignored.

I said this is a show stopper before, but if there's a real advantage in
doing this... Let's see:

this: !str|folded
  Is horribly verbose!

Possible solution: How about we allow !|<format> as a shorthand for
!<implicit>|<format> (or at least as a shorthand for !str|<format>)?

this: !|-\) # type: 'str' format: '|-\)'
  Folded ')', chomped '-', escaped '\\'.

This would move all the escaping/folding/chomping code to the string loader
code (and its description to the transfer method section - Eeek!), using
named formats (empty/default format : block, - : chomped, \ : escaped, ) or
> or ^ or ] or whatever : folded, and their combinations.)

Hmmm. This just trades off one special mechanism by another, and reduces the
power (since it becomes impossible to use, say, escaped format for non-!str
types).

I don't know. I think at this point I'd basically keep things as they are,
unless someone comes up with a distinct advantage in doing things
differently. Notation wise, I agree it makes sense to make the folded
indicator a modifier for the multi-line indicator. Ans we agree about using
'-' for chomped and '\' for escaped. This leaves us with two characters -
one for the block indicator (currently |) and one for the folded indicator
(currently ]). What character pair should be used (| or ; for multi-line and
), ], > or ^ for folded) is just a matter of taste. I'll leave the decision
to Clark. My personal choice would be to keep things as they are (| for
block, ] for folded).

Whatever we choose, I think it is vital that the following combinations
would have terse, clear notation:

Block: |
   Seems clean enough
Folded: |]
   Clearer than today's standalone ]
Escaped folded: |\]
   Not too bad.

I think these would cover most of the actual uses.

> How about:
> 
>     --- <multi style="mixed" chomp="no">

Shudder :-)

Have fun,

	Oren Ben-Kiki

On Tue, May 21, 2002 at 10:13:08AM -0400, Oren Ben-Kiki wrote:
| Brian Ingerson [mailto:ingy@...] wrote:
| > I think we're missing the whole point of what I really want here.
| > Clark alluded to it but, what I really want is the ability to extend
| > the multiline leaf formats that we support. One very useful format
| > would be a mix of block and folded. Lines beginning with whitespace
| > (as content) would be block. Lines beginning with non-whitespace would
| > be folded. I see this pattern in textual data all the time. (If you
| > know Perl, think POD)

Ok.  First, there are two things we can't move into the format,
the first one is "scalar" indicator since we must have a way to
know that the given node is not a mapping.   Second, we need
the escaped flag.  YAML has a limited character set and thus
escaping is needed so that the full unicode can be represented.
Third, we need chomp since there is no way to signal if the 
trailing new line is significant.

Both escape and chomp are flags that need to apply to every 
scalar value and handled at the parser level.  Since format is
handled by the loader and specific to given scalar values, we
cannot use escape and chomp within the format.

The question then becomes, is folded a style or can it be moved
into a format.  As a format it would be handled by the loader
and not the parsre like escape and chomp and applies only to
one family, !str, by default.  Other families would have to have
some application-specific knowlege to know that folded is valid.

The use case for folded is rather simple.  In much business data,
paragraphs (say a warranty description) are used without explicit
carriage returns to let the form wrap the content as appropriate.
This is also typical of HTML.  In HTML carriage returns are not
significant and can be inserted into the stream to enhance
readability.  Now, we don't strictly need folded, one could just 
put in these paragaphs/html blocks as one very long line; or could
even use the escape mechanism to break the long line into several
smaller lines.  Using the escape in this manner is good enough for
me, but it isn't the most readable.

So, given that my use case is _primarly_ the ability to break
a line at any given point so that it fits in 76 columns for
"pretty printing" and given that most content of this sort won't
typically have embedded \'s which have to be escaped, this is
perfectly acceptable to me.    In other words,

replace: ]
  This paragraph which is
  folded.
with: |\
  This paragraph which is \
  folded.

So, I'm willing to just "strike" folded from the styles, since
escaping handles my use case (just not as readable).

| Hmmm. Even if you make this a format, extending it won't be trivial (it
| effects parsers/loaders everywhere). And IMVHO str is too basic a type to
| extend it like that later on.

If we make it a format, it will be impossible for the parser to
know that it also applies to an HTML type (as I said earlier).  
Thus, a filter which doesn't know about the type's ability to
have that format can't use folded to reformat the text for pretty
printing.  Thus, it is kinda useless.  Let's just let application
specific types define it in this case, no point in even having
it in the YAML specification if it doesn't apply to everything.

| I, for one, would be happy if we could have *less* styles. The trouble is, I
| see the sense in all our three variables (chomping, folding and escaping).a

Ok.  Let's just have chomp and escape then.

| That means that we can only apply these formats to !str. On the up side, it
| means that we would have just one "style" of multi-line scalars (block). For
| type families such as binary, it is easy enough to specify that all white
| space is ignored.

This reduces the power of the folded scalar so that it just
isn't worth it for me. 

| Possible solution: How about we allow !|<format> as a shorthand for
| !<implicit>|<format> (or at least as a shorthand for !str|<format>)?

This won't work since the parser needs to do the escaping due
to the possiblility of illegal characters.

Best,

Clark

-- 
Clark C. Evans                   Axista, Inc.
http://www.axista.com            800.926.5525
XCOLLA Collaborative Project Management Software

On Tue, May 21, 2002 at 10:13:08AM -0400, Oren Ben-Kiki wrote:
| this: !str|folded
|   Is horribly verbose!

I'd like to do !| as a short-hand for !str|

Best,

Clark

On 20/05/02 23:46 -0400, Oren Ben-Kiki wrote:
> Brian Ingerson wrote:
> > > >    one: __true__
> > >    two: _true_
> > > >    thr: _true
> > > 
> > > I'll go with _true - it doesn't look that bad.
> 
> Or maybe:
> 
>     four: .true

I like this one best so far.

> The word_char production is used in:
> 
> - Directive names;
> - Anchors;
> - Valid in "language" shorthand in transfer URI.
> 
> "Language" shorthand goes into the domain name part, where '_' is not
> allowed. So we'd have to continue to disallow '_' there.
> 
> As for anchors and directives, we could allow '_' there. Which variant do
> you mean, using '_' instead of '-' or in addition to '-'?

I'd like to use it *instead* of, since that matches Perl's and Python's and
Ruby's definition of word-char. What about Java?

My use cases are:

  ---
  foo: __BAR__
  _: too

Cheers, Brian

Brian Ingerson wrote:

> 
> I'd like to use it *instead* of, since that matches Perl's and Python's and
> Ruby's definition of word-char. What about Java?
> 

Same for Java and C. '-' is an operator.

Regards.
Rolf.

On 21/05/02 14:53 +0200, Rolf Veen wrote:
> Brian Ingerson wrote:
> 
> > 
> > I'd like to use it *instead* of, since that matches Perl's and Python's and
> > Ruby's definition of word-char. What about Java?
> > 
> 
> Same for Java and C. '-' is an operator.

This would simplify the regular expressions used in various processes. Most
engines define \w as [a-zA-Z0-9_]

Cheers, Brian

Brian Ingerson [mailto:ingy@...] wrote:
> > Or maybe:
> > 
> >     four: .true
> 
> I like this one best so far.

Well then! Clark?

> > As for anchors and directives, we could allow '_' there. 
> > Which variant do
> > you mean, using '_' instead of '-' or in addition to '-'?
> 
> I'd like to use it *instead* of, since that matches Perl's 
> and Python's and
> Ruby's definition of word-char. What about Java?
> 
> My use cases are:
> 
>   ---
>   foo: __BAR__
>   _: too

*Smack self on head*.

You mean you want values starting with '_' to be implicit strings? That is,
consider them as *letters*?

I thought you meant "word characters" in the sense of the "word_char"
production - this doesn't effect implicit typing, only what is allowed in
anchors etc.

At any rate, yes, making '_' be an honorary letter is the right thing to do,
as your examples demonstrate. I'll put it in.

Have fun,

	Oren Ben-Kiki

On Tue, May 21, 2002 at 09:45:32AM -0400, Oren Ben-Kiki wrote:
| > >     four: .true
| > I like this one best so far.

Let it be so.

| >   ---
| >   foo: __BAR__
| >   _: too
| 
| At any rate, yes, making '_' be an honorary letter is the right thing to do,
| as your examples demonstrate. I'll put it in.

+1

;) Clark