YAML Ain't Markup Language

On 01/12/02 10:53 +0200, Oren Ben-Kiki wrote:
> Brian Ingerson wrote:
> > The problem I was not seeing is that commented out lines must start with
> > '# ', not just '#'.
> 
> Right. Personally, I never leave out the space after a comment '#' (or, for
> that matter, after '//' or '/*'). Obviously that's not a universal style.
> 
> > Would someone please remind me why this is necessary for full
> > line comments?
> 
> We definitely wanted to allow:
> 
>    url: domain.tld:/some/path#fragment
> 
> This means that '#' would only start a comment if there were spaces involved
> somehow. We already have the notion that some characters are onlyindicators
> if followed by a space (e.g. ',', ':'). Then we considered things like
> Smalltalk #symbols. It made sense to make '#' a "space indicator", so the
> comment marker became '# '. That's actually not quite accurate because the
> '# ' must be either at the start of a line or be preceded by a space, so '#'
> still remained something of a special case.
> 
> > This is going to cause no end of grief, I'm sure. I would propose that
> > we allow non-trailing throwaways to not require the space after the '#'.
> 
> I think it would be confusing to have one set of rules for trailing comments
> and one for non-trailing ones. For example:
> 
> not comment: #symbol
>   #comment

Well I disagree. But I think I'm willing to drop my argument. We can
keep it as is. It would be nice if parsers could detect this kind of
thing and print a useful message.

Cheers, Brian

Brian Ingerson wrote:
> > I think it would be confusing to have one set of rules for trailing
> > comments
> > and one for non-trailing ones...

> Well I disagree. But I think I'm willing to drop my argument. We can
> keep it as is.

There's still the alternative of giving up on #symbols. What is more
important, allowing them or allowing #comments? Asking this differently,
given the following YAML:

---
a: foo
b: # null
c: #bar
d: baz
...

What would most people expect the value of 'c' to be - '#bar' or null? If
most people would read it as a null, then we probably should change the
rules accordingly.

> It would be nice if parsers could detect this kind of
> thing and print a useful message.

Helpful parser error messages are nice, but wherever possible its best to
"do the right thing" instead. Are we?

Have fun,

    Oren Ben-Kiki

On 02/12/02 11:33 +0200, Oren Ben-Kiki wrote:
> Brian Ingerson wrote:
> > > I think it would be confusing to have one set of rules for trailing
> > > comments
> > > and one for non-trailing ones...
> 
> > Well I disagree. But I think I'm willing to drop my argument. We can
> > keep it as is.
> 
> There's still the alternative of giving up on #symbols. What is more
> important, allowing them or allowing #comments? Asking this differently,
> given the following YAML:
> 
> ---
> a: foo
> b: # null
> c: #bar
> d: baz
> ...
> 
> What would most people expect the value of 'c' to be - '#bar' or null? If

They should expect whatever the spec says. Right? :)

> most people would read it as a null, then we probably should change the
> rules accordingly.
> 
> > It would be nice if parsers could detect this kind of
> > thing and print a useful message.
> 
> Helpful parser error messages are nice, but wherever possible its best to
> "do the right thing" instead. Are we?

I sense a role reversal here. So you're wearing the DWIM hat? I think we
have a spec to get out. 

I think we just need to pick a way here, and stick with it. Let's just
leave it asis if only because that's the way it is now.

Cheers, Brian

Brian Ingerson wrote:
> I sense a role reversal here. So you're wearing the DWIM hat?

Our first goal is to make YAML readable.

> I think we
> have a spec to get out. 

Right. I also want it to be a good spec.

> I think we just need to pick a way here, and stick with it. Let's just
> leave it asis if only because that's the way it is now.

Certainly I'm for preserving things as they are, especially if there's no
clear advantage to either alternative.

The '#' rules are rather new, however, so I'm not 100% comfortable in my
mind we have hit on the best solution. We do have a "last call" period for a
reason - trying things out and seeing if they work. Your experience with
actually using '#' indicated there's some cause for concern, and I think we
shouldn't automatically rule a minor improvement out - as opposed to when we
declare a release candidate. At that time, we should rule out any change
unless there's overwhelming evidence requiring a change.

I'd like to hear Clark's input on this... Clark?

Have fun,

    Oren Ben-Kiki

On Tue, Dec 03, 2002 at 05:02:28PM +0200, Oren Ben-Kiki wrote:
| The '#' rules are rather new, however, so I'm not 100% comfortable in my
| mind we have hit on the best solution. We do have a "last call" period for a
| reason - trying things out and seeing if they work. Your experience with
| actually using '#' indicated there's some cause for concern, and I think we
| shouldn't automatically rule a minor improvement out - as opposed to when we
| declare a release candidate. At that time, we should rule out any change
| unless there's overwhelming evidence requiring a change.

Historically, we've put out a straw man and then fixed it
as user feedback comes in.  As Oren said, the problem with
the pound(#) symbol is that it can occur within identifiers
that people may want to use:  34#wink

Thus we ended up making it a space indicator like other
items with the same problem (comma colon).   I suppose
that it is a rather odd case to have an identifer starting
with a colon, so we could make the comment require a space
before (but not necessarly a space after).    I'm ok with
that change.  This is the 3rd time that Brian (or perhaps
Steve) has brought this one up... so it seems a sore spot.

I'm ok with fixing it.

Clark

Clark C. Evans wrote:
> ... we could make the comment require a space
> before (but not necessarly a space after).    I'm ok with
> that change.

That's what I also thought.

> This is the 3rd time that Brian (or perhaps
> Steve) has brought this one up... so it seems a sore spot.

Yes, it does keeps re-surfacing every so often. Brian?

In an unrelated note - Clark, you mentioned we need "a lot of work" on the
spec before we make it a release candidate. I can see we need to put a more
work into the type definitions (especially the timestamp one). Do you see
that, aside from '#' and "polish" (spelling, wording, formatting, etc.), we
need to do any changes to the spec itself?

I do have some spare time now I can use for doing spec fixes... We really
should have a "Release Candidate" for January 1st.

Have fun,

    Oren Ben-Kiki

On Tue, Dec 03, 2002 at 09:47:46PM +0200, Oren Ben-Kiki wrote:
| In an unrelated note - Clark, you mentioned we need "a lot of work" on the
| spec before we make it a release candidate. I can see we need to put a more
| work into the type definitions (especially the timestamp one). Do you see
| that, aside from '#' and "polish" (spelling, wording, formatting, etc.), we
| need to do any changes to the spec itself?

There is quite a bit of English issues with the spec that I 
just don't have time yet to fix; I'll try to get some time
in the coming weeks.  On a related note, I'd also like to go
through the productions one by one again... I'm sure you've
done a fantastic job in the recent revisions but I've all
but lost track of which production does what these days.  
I'm thinking that getting on with a libyaml implementation
per the spec (where each function is given a production number)
is probably the best way to verify the productions.  

Clark

On Tue, 2002-12-03 at 20:04, Clark C. Evans wrote:
> There is quite a bit of English issues with the spec that I 
> just don't have time yet to fix; I'll try to get some time
> in the coming weeks.  On a related note, I'd also like to go
> through the productions one by one again... I'm sure you've
> done a fantastic job in the recent revisions but I've all
> but lost track of which production does what these days.  
> I'm thinking that getting on with a libyaml implementation
> per the spec (where each function is given a production number)
> is probably the best way to verify the productions.  

Speaking of productions, would it be possible to get a
machine-processable list of all the productions? Preferably in YAML
(just to show that it can be done)? I wrote an XSLT script to strip what
I could out of the spec but they still had to be manually tweaked due to
their inconsistent formatting. (I'm amazed that you guys maintain the
spec in HTML--wasn't there talk about moving to a presentation-agnostic
format a while ago? reStructuredText would be nice...)

Jason

On Tue, 2002-12-03 at 20:04, Clark C. Evans wrote:
> There is quite a bit of English issues with the spec that I 
> just don't have time yet to fix; I'll try to get some time
> in the coming weeks.  On a related note, I'd also like to go
> through the productions one by one again... I'm sure you've
> done a fantastic job in the recent revisions but I've all
> but lost track of which production does what these days.  
> I'm thinking that getting on with a libyaml implementation
> per the spec (where each function is given a production number)
> is probably the best way to verify the productions.  

Sounds like a good idea. I'll post production fixed for the '#' - these
should be minor.

Jason Diamond [SMTP:jason@...] wrote:
> Speaking of productions, would it be possible to get a
> machine-processable list of all the productions? Preferably in YAML
> (just to show that it can be done)? I wrote an XSLT script to strip what
> I could out of the spec but they still had to be manually tweaked due to
> their inconsistent formatting. (I'm amazed that you guys maintain the
> spec in HTML--wasn't there talk about moving to a presentation-agnostic
> format a while ago? reStructuredText would be nice...)

Using XSLT to extract productions from XHTML... brave man :-)

You are right that we should have picked some other basic format other than
HTML. We've looked at it a couple of times but nobody came up with something
that (1) would allow sufficient control over formatting, especially of the
productions; and at the same time (2) was simple and common enough so that
migration wouldn't be a major project.

As for making productions available in YAML, there's a spectrum between
making them available as:

# productions
---
- |
 printable_char ::= #x9
                  | #xA | #xD | #x85
                  | [#x20-#x7E]
                  | [#xA0-#xD7FF]
                  | [#xE000-#xFFFD]
                  | [#x10000-#x10FFFF]
...

And representing them as (for some definition of '!bnf/*'):

--- !bnf/grammar
- &line_feed           '#xA'    # ASCII line feed (LF)
- &carriage_return     '#xD'    # ASCII carriage return (CR)
- &next_line           '#x85'   # Unicode next line (NEL)
- &line_separator      '#x2028' # Unicode line separator (LS)
- &paragraph_separator '#x2029' # Unicode paragraph separator (PS)
- &line_break_char !bnf/oneof   # line break characters
  - *line_feed
  - *carriage_return
  - *next_line
  - *line_separator
  - *paragraph_separator
...

Which did you have in mind?

Have fun,

    Oren Ben-Kiki

> As for making productions available in YAML, there's a spectrum between
> making them available as:
> 
> ...
> 
> And representing them as (for some definition of '!bnf/*'):
> 
> ...
> 
> Which did you have in mind?

Obviously, the more detail that can be captured so that it's easily
processable, the better. I didn't have much more in mind than something
a tiny bit more detailed than your first example but I certainly
wouldn't complain if you gave me the second! The question is, could the
finely detailed productions be used to generate a complete parser?

Jason

Jason Diamond wrote:
> > Which did you have in mind?
> 
> Obviously, the more detail that can be captured so that it's easily
> processable, the better. I didn't have much more in mind than something
> a tiny bit more detailed than your first example but I certainly
> wouldn't complain if you gave me the second! The question is, could the
> finely detailed productions be used to generate a complete parser?

Obvipously this can be done, but it would require defining all the necessary
constructs in a rigorous way. While that has already been done in various
tools, it would still take time to YAML-ize all the details.

Being able to specify any BNF as YAML and generate parsers from it would be
useful... For example, it may offer an interesting basis for a YSCHEMA
language.

At any rate, I think we should focus on the spec itself first, and work out
machine-parsable BNF later as an application.

In the mean while, we'll have to do without. Extracting just the productions
in a human-readable way from the spec isn't very hard (just throw away
anything other than production tables).

Have fun,

    Oren Ben-Kiki

On 03/12/02 21:47 +0200, Oren Ben-Kiki wrote:
> Clark C. Evans wrote:
> > ... we could make the comment require a space
> > before (but not necessarly a space after).    I'm ok with
> > that change.
> 
> That's what I also thought.
> 
> > This is the 3rd time that Brian (or perhaps
> > Steve) has brought this one up... so it seems a sore spot.
> 
> Yes, it does keeps re-surfacing every so often. Brian?

What is the new proposal. And how are directives affected?

Cheers, Brian

Brian Ingerson wrote:
> What is the new proposal.

That '#' would start a common if and only if it is at the start of a line or
is _preceded_ by a space - of course, outside quotes, block scalars and so
on.

> And how are directives affected?

Oops. It makes them ambiguous. Good point. I don't suppose that changing the
directive syntax makes sense at this stage. On the other hand, this issue
does keep being raised...

Hmmm. Nice dilema we've got. I guess that this makes the issue a tie, so
we'll keep things as they are - unless someone comes up with a tie-breaker.
Clark?

Speaking of spec polish - I gave some thought to how to restructure the
plain scalar productions in case of changing the '#' behavior, and in the
process found I'm not too happy with their current state. So I'll re-work
them to something better regardless of '#'.

Have fun,

    Oren Ben-Kiki