YAML Ain't Markup Language

On Tue, May 29, 2001 at 07:06:42PM +0200, Oren Ben-Kiki wrote:
| Supposing the information model does make this distinction

Ok.  Let's make the binary/unicode distinction in 
the information model.

| === Top Level Production ===
| 
| I see we are back to "list of maps, separated by blank lines". Great. The
| API basics given in the YAML spec aren't explicit on how this translates to
| actuall calls. I'd expect that the "next()" calls at the top-level of the
| parser will return a map node, one per each "map block" in the file. As for
| the emitter, I'd expect that repeatedly calling "begin()" and "end()" on the
| top-level cursor will emit multiple map blocks, but the text doesn't seem to
| support this. I think that this should be made explicit.

Ok.  When I get time I'll make it more clear.

| === Classes and Color ===
| 
| Clark made a good case against comments - we can't use them because either
| they aren't part of the information model, and won't round-trip; or they
| round-trip and hence must be a part of the data model. He's right.
| 
| The same argument carries over to classes. The current spec states that if a
| class is not recognized, a warning is emitted and the class is ignored. This
| is unacceptable - consider a YAML pretty-printer, it will recognize a very
| small set of classes, if any. Yet it must preserve the class names.
| 
| The problem seems a classical place to use the color idiom. We have a piece
| of information - the class name - which we want to attach to any nodes in
| the YAML document, including scalar nodes. Some applications (the pretty
| printer) aren't interested in this information, but must preserve it. Other
| applications (e.g., a YAML-based application server) rely on this
| information.
| 
| There's no problem for attaching such information to map nodes. Use some
| special key for the class info - say, '#' - and you are done. For scalar and
| list nodes, the color idiom suggests you wrap them in a map node. Use the
| same key for specifying your "color", and another special key (say, '=') to
| specify the *value* of the node:
| 
|     delivery: %
|         = : 2000-JAN-10
|         # : date
| 
| The trick is that the random-access APIs should allow you to call
| "getValue()" on the 'delivery' node and obtain the date value. This is
| acceptable to an interface such as XML's DOM. It probably could work in
| languages dynamic enough to hack the type system into doing this implicitly:
| de-serialize 'delivery' into something which *behaves like* the string
| '2000-JAN-10' but is *at the same time* a map with two keys.
| 
| Complex? Sure, but Brain said, "don't worry about the implementation" :-) 
| 
| Seriously, I think there's cause to worry here. The color idiom has turned
| out to be the solution for many practical problems - not the least of which
| is schema evolution. Maybe you guys can come up with a better way to do it
| then wrapping scalar/list nodes in a map; I couldn't find any. But I
| strongly feel that YAML should address this somehow - and that classes are
| the perfect way to "eat our own dog food" in this regard.

I'd like to hear brian's perspective on this one.  I actually
kinda like it, although I'm partial since we had talked about it
for a long time on the sml-dev list.  We would need to push forward
on the API notion of "getValue()" as returning the node with 
a blank key.  Building this mechanism in is very cool since it
allows for "schema substitutability".  However, I'm not sure
how well it works for data serilization....

(on a technical note, I think "", a blank key, should be the
 default value for a node, and perhaps "__class__" could be
 the class name).

| Do we really need so many different formats? *4*? I see why we'd want base64
| - that's not really a text format, anyway. I can see why we'd want one
| format for cut-and-paste and one format for allowing escaping. That would be
| the block format and either the string or the simple scalar format. There's
| also the issue of multi-line values allowed in a map but not in a list. So
| we actually have 4.5 formats (3.5 for text, one for blobs).
| 
| Do we really need all of these? Can't we make due with just one block format
| and one stream format?
| 
| I also strongly dislike the fact that indentation rules are suspended in a
| string value. Why is that? I see the pain - one won't be able to, say,
| quickly skip a sub-tree by simply looking for a properly indented line;
| readability suffers, etc.
| 
| I don't see the gain: Cut&paste by itself isn't a good enough answer - if
| you are doing it by hand, any editor will easily indent the new block of
| text, and if you are doing it by a tool, well just add the indentation while
| you are at it.
| 
| In E-mail messages Clark said that base64 blocks are indented, but the spec
| says otherwise. Which is right? What's the harm in indenting blobs? We
| already allow long lines in blocks... in fact, there's no longer a way to
| break a long line in a block into two shorter ones.
| 
| I realize you had just about enough of this syntax issue, and that I'm fresh
| out of a 10 day vacation from it, but I think that either some strong
| rationale or some more work on this are due. I'll try to see if I can come
| up with any new ideas...

The primary rationale for the [ and " formats breaking
the indentaiton rules is to allow for easy cut and paste. 

| I'd rather you list me as oren@...

Fixed.


...

I'm not going to be able to put much more time into
this in the next few months, as I'm a key player
in a start-up, xgenda.com, which hopes to publicly
launch our product by September.

Best,

Clark

Clark C . Evans [mailto:cce@...] wrote:
> On Tue, May 29, 2001 at 07:06:42PM +0200, Oren Ben-Kiki wrote:
> | Supposing the information model does make this distinction
> 
> Ok.  Let's make the binary/unicode distinction in 
> the information model.

Brian? Can you say a word on how this would work in Perl?

> | === Classes and Color ===
> ...
> I'd like to hear brian's perspective on this one.  I actually
> kinda like it, although I'm partial since we had talked about it
> for a long time on the sml-dev list.  We would need to push forward
> on the API notion of "getValue()" as returning the node with 
> a blank key.  Building this mechanism in is very cool since it
> allows for "schema substitutability".  However, I'm not sure
> how well it works for data serilization....

Hmmm. If one is serializing a built-in map/list/scalar, you do it normally;
when serializing an "Object" it probably gets serialized into a map, with a
class attribute. It is only "typed scalars" which are a problem (e.g., a
number, a date, etc.). One problem about the proposed class-as-color syntax
is that it is rather cumbersome for something like a simple number...

> (on a technical note, I think "", a blank key, should be the
>  default value for a node, and perhaps "__class__" could be
>  the class name).

I rather like '=' for 'value' - it has the right intuitive semantics.
'__class__' for type is too verbose for my taste.

> The primary rationale for the [ and " formats breaking
> the indentaiton rules is to allow for easy cut and paste. 

I don't buy that. Cut&paste implies a tool - an editor or a program. Most
editors allow you to trivially indent a group of lines - certainly any
editor used for writing YAML had better support it. And a program doing
cut&paste can add indentation easily enough.

Besides, if "raw" cut&paste is that important, doesn't it apply to blocks as
well? we can simply use:

    block: |===arbitrary marker line===
block text,
raw, not indented at all,
no prefix for the lines,
cut & paste into it whichever way you want.
|===arbitrary marker line===
\===arbitrary marker line (if no trailing newline)===

Perl people know this as the '<<EOF' approach :-) This would leave simple
scalars as the only type of value which must be indented - and given that
most of these are single-line, that hardly matters. You'd end up with files
where every multi-line value is not indented, which I think will really ruin
readability. I'd rather we stick with strict indentation for everything,
including blocks, blobs and double-quoted streams.

Separate issue: having both double-quoted and simple scalar values. Here's a
proposal to use just one form:

- Block: as today, but use ` instead of \ to mark the end line.
- Scalar value: as today, allow escaping using \, always allow it to be
multi-line, but the continuation lines must be more indented then the first
line.
- List: as today (no special marker), but allow : as an optional prefix for
scalar values.

Example:

    @
        Ugly multi
            line text
        Pretty single line text with NL\n
        : Ugly single line text w/o NL
        : Pretty multi
          line text
        |multi line
        |block with NL
        `
        |multi line
        `block w/o NL\n
        [base64
        blob]

Everything is always strictly python-indented.

> I'm not going to be able to put much more time into
> this in the next few months, as I'm a key player
> in a start-up, xgenda.com, which hopes to publicly
> launch our product by September.

Hmmm. As I see it, we have at least one major issue to resolve (the class
issue) plus another where a "dictator hat" may be required (the text
syntax). Then we can all start doing development at our own pace...

Have fun,

    Oren Ben-Kiki

Oren Ben-Kiki wrote:
> 

Sorry guys. I'm a little busy this week. I started to reply to Oren's
message yesterday, but abandoned it because I couldn't give it the
required brain power. I'll try to comment quickly.

> Clark C . Evans [mailto:cce@...] wrote:
> > On Tue, May 29, 2001 at 07:06:42PM +0200, Oren Ben-Kiki wrote:
> > | Supposing the information model does make this distinction
> >
> > Ok.  Let's make the binary/unicode distinction in
> > the information model.
> 
> Brian? Can you say a word on how this would work in Perl?

Perl is a little up in the air on Unicode. I believe that recent
versions use UTF8 by default for *most* string operations. A byte mode
can be set to default back to the good ole days.

FWIW, I suggest using the single quote for Unicode data. (Get it? *Uni*
code :-). Using Clark's backtick for binary base-64 data. Using '|,\'
for blocks. Using double quotes for everything else (folded streams).
Unquoted streams are a convenience option for single line values only.

> 
> > | === Classes and Color ===
> > ...
> > I'd like to hear brian's perspective on this one.  I actually
> > kinda like it, although I'm partial since we had talked about it
> > for a long time on the sml-dev list.  We would need to push forward
> > on the API notion of "getValue()" as returning the node with
> > a blank key.  Building this mechanism in is very cool since it
> > allows for "schema substitutability".  However, I'm not sure
> > how well it works for data serilization....
> 
> Hmmm. If one is serializing a built-in map/list/scalar, you do it normally;
> when serializing an "Object" it probably gets serialized into a map, with a
> class attribute. It is only "typed scalars" which are a problem (e.g., a
> number, a date, etc.). One problem about the proposed class-as-color syntax
> is that it is rather cumbersome for something like a simple number...

Classes are a can of worms.

Although I'm not against the color idea, Perl will never embrace the
idea of having everything be an object. Using getValue() as the normal
method of retrieving a simple value is just too much work for Perl
people. A map should just become a hash, plain and simple. I can come up
with some creative ways of preserving YAML classes in Perl
transparently, but my plate's too full right now. 

(I'm trying to be a famous Perl guy in other realms, ya know ;)

> 
> > (on a technical note, I think "", a blank key, should be the
> >  default value for a node, and perhaps "__class__" could be
> >  the class name).
> 
> I rather like '=' for 'value' - it has the right intuitive semantics.
> '__class__' for type is too verbose for my taste.

Agree.

> > The primary rationale for the [ and " formats breaking
> > the indentaiton rules is to allow for easy cut and paste.
> 
> I don't buy that. Cut&paste implies a tool - an editor or a program. Most
> editors allow you to trivially indent a group of lines - certainly any
> editor used for writing YAML had better support it. And a program doing
> cut&paste can add indentation easily enough.
> 
> Besides, if "raw" cut&paste is that important, doesn't it apply to blocks as
> well? we can simply use:
> 
>     block: |===arbitrary marker line===
> block text,
> raw, not indented at all,
> no prefix for the lines,
> cut & paste into it whichever way you want.
> |===arbitrary marker line===
> \===arbitrary marker line (if no trailing newline)===
> 
> Perl people know this as the '<<EOF' approach :-) This would leave simple
> scalars as the only type of value which must be indented - and given that
> most of these are single-line, that hardly matters. You'd end up with files
> where every multi-line value is not indented, which I think will really ruin
> readability. I'd rather we stick with strict indentation for everything,
> including blocks, blobs and double-quoted streams.

I think it's nice to allow the cut-and-paste and let emitters always
reformat. But I'll support strict for now.

> 
> Separate issue: having both double-quoted and simple scalar values. Here's a
> proposal to use just one form:
> 
> - Block: as today, but use ` instead of \ to mark the end line.
> - Scalar value: as today, allow escaping using \, always allow it to be
> multi-line, but the continuation lines must be more indented then the first
> line.
> - List: as today (no special marker), but allow : as an optional prefix for
> scalar values.
> 
> Example:
> 
>     @
>         Ugly multi
>             line text
>         Pretty single line text with NL\n
>         : Ugly single line text w/o NL
>         : Pretty multi
>           line text
>         |multi line
>         |block with NL
>         `
>         |multi line
>         `block w/o NL\n
>         [base64
>         blob]
> 
> Everything is always strictly python-indented.

Whatever :(

If you are once again suggesting dropping double quotes, you're
needlessly going down a road of pain. Double quotes are simple,
intuitive and will always work for folded text (indented or not).

> 
> > I'm not going to be able to put much more time into
> > this in the next few months, as I'm a key player
> > in a start-up, xgenda.com, which hopes to publicly
> > launch our product by September.
> 
> Hmmm. As I see it, we have at least one major issue to resolve (the class
> issue) plus another where a "dictator hat" may be required (the text
> syntax). Then we can all start doing development at our own pace...

I'll volunteer as dictator :)

I want to start doing something on this after June 15th or so. I hope we
can resolve things by then. We were very close last week.

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

On Wed, May 30, 2001 at 02:56:00PM +0200, Oren Ben-Kiki wrote:
| Hmmm. If one is serializing a built-in map/list/scalar, you do 
| it normally; when serializing an "Object" it probably gets 
| serialized into a map, with a class attribute. It is only 
| "typed scalars" which are a problem (e.g., a number, a date, etc.).
| One problem about the proposed class-as-color syntax is that it 
| is rather cumbersome for something like a simple number...

Right... this is half the problem.  The other half of the 
problem is that the class should be known before the map's 
content should be loaded.  I think I like the class solution 
as it is... I think the round-tripping problems are 
problematic; but perhaps they are not that bad.

| Cut&paste implies a tool - an editor or a program. Most editors
| allow you to trivially indent a group of lines - certainly any
| editor used for writing YAML had better support it. And a program 
| doing cut&paste can add indentation easily enough.

Ok.  I'd rather have it always indented as well.

| I'd rather we stick with strict indentation for everything,
| including blocks, blobs and double-quoted streams.

Did you consider that an emitter can indent the blob
and double quoted streams?   Thus, indentation isn't
prevented.  And with machine generated YAML (99%),
the non-intendented case is minor.  Further, by running
YAML through a program which puts the file in canonical
form indenting will definately be the norm.  So... I didn't
see the harm in allowing this.  It doesn't really impact
the parsing complexity.

| Separate issue: having both double-quoted and simple scalar 
| values. Here's a proposal to use just one form:
| 
| - Block: as today, but use ` instead of \ to mark the end line.

I like the \, is there a reason why you wanted to change it?

| - Scalar value: as today, allow escaping using \, always allow 
|   it to be multi-line, but the continuation lines must be more
|   indented then the first line.

I understand, however,  I think the current division
of scalar types is rather nice balance of concerns.  
Most programmers will expect \ style escaping 
within quotes.

| - List: as today (no special marker), but allow : as an 
|   optional prefix for scalar values.

Well, if you leave the quote type in, then 
the : optional marker won't be needed:

  Example:
    @
        Single line scalar
        "Pretty multi line
         text without multiline."
        |
        |Block with leading and 
        |trailing new line.
        \
        [Base 64 blob]

| Hmmm. As I see it, we have at least one major issue to resolve (the class
| issue) plus another where a "dictator hat" may be required (the text
| syntax). Then we can all start doing development at our own pace...

I still don't know what to do about the class issue, but I 
think that there are solutions for this.  Let us take a 
language with only maps, lists, and scalars.  An external
"class map" could be put in place by the YAML load/save
mechanism.  Thus, as maps/lists/scalars are loaded, 
entries in this class map could be made.  Then when
the objects are serialized, the class could be written
back out.  Therefore, I think that a round-trip ability
is possible, it just may not be the most obvious solution.

As for the scalar types... I'm inclined to leave them 
as they are now.  It does seem like a nice balance of
concerns.  What I'd like to focus on (and what should
be added to the spec) is the canonoical form.  Here
are some base line suggestions:

  1. Indenting always occurs, i.e. quoted or binary
     scalars are indented.

  2. The tab setting for indents is 4 characters

  3. When possible the text is word-wrapped to 76
     characters; leaving for a minimum of 20 characters
     for scalar's content.  Thus, after 14 levels of
     indentation, text may go beyond 20 characters.

  4. If leading whitespace occurs on any line within
     a scalar, then the block format is used.

  5. If a character string is longer than 20 characters
     without having intermediate whitespace, then
     the quoted format is used.

     etc.

Best,

Clark

On Wed, May 30, 2001 at 01:23:41PM -0700, Brian Ingerson wrote:
| Perl is a little up in the air on Unicode. I believe that recent
| versions use UTF8 by default for *most* string operations. A byte mode
| can be set to default back to the good ole days.
| 
| FWIW, I suggest using the single quote for Unicode data. (Get it? *Uni*
| code :-). Using Clark's backtick for binary base-64 data. Using '|,\'
| for blocks. Using double quotes for everything else (folded stream).
| Unquoted streams are a convenience option for single line values only.

Hmm.  This is interesting, instead of marking which nodes are
"binary" we mark which nodes are "unicode".   Not a bad compromise,
as it is far easier with the language to know if something is
unicode or not.... I like this, it closely mirrors what Python 
and C does.  It is the unicode which is treated differently,
allowing regular "char" strings and binary to still be used
interchangeably.   This has one *big* impact, though.  A YAML 
document cannot be  encoded using UTF-16 (via the BOM), although
leaves can be encoded with UTF-8.

Also, I'm not sure that I like limiting unquoted streams to
a single line.  Being able to cut/paste in a HTML text without
having to escape the quotes and such is very valueable.  Further,
I thought we had agreed on [base64] instead of the back tick.

I *really* like the unicode idea though...  it solves alot
of problems rather cleanly.

| Classes are a can of worms.
| 
| Although I'm not against the color idea, Perl will never embrace the
| idea of having everything be an object. Using getValue() as the normal
| method of retrieving a simple value is just too much work for Perl
| people. A map should just become a hash, plain and simple. I can come up
| with some creative ways of preserving YAML classes in Perl
| transparently, but my plate's too full right now. 

A global "class map" could work.  It's not pretty... 
Alternatively, we can drop the class idea for now.

Along this same line of thought, I was asking myself
if class is equivalent to encoding for scalars...
just wondering.  Are they similar constructs?

| > readability. I'd rather we stick with strict indentation for everything,
| > including blocks, blobs and double-quoted streams.
| 
| I think it's nice to allow the cut-and-paste and let emitters always
| reformat. But I'll support strict for now.

...

| If you are once again suggesting dropping double quotes, you're
| needlessly going down a road of pain. Double quotes are simple,
| intuitive and will always work for folded text (indented or not).

I think the double quote mechanism stays in... just about
everyone knows how to use them.

| I want to start doing something on this after June 15th or so. I hope we
| can resolve things by then. We were very close last week.

I think we are still very close.  

Open Issues
~~~~~~~~~~~

A. Classes

   Problem: Round-tripping

   1. Drop classes for now
   2. Let the implemenation worry about them (class map)

B. Unicode vs Binary

   1. Introduce a isBinary scalar flag and keep all 
      scalars unicode.  This causes problems with
      YAR where we cannot know if a given node is
      binary or not, but would like it readable if
      it is ASCII.

   2. Introduce an isUnicode scalar flag.  Limit 
      encoding to ASCII for regular strings and to
      UTF-8 within single quoted strings.  


...

I think I'd pick A2 and B2 at this time.  


Clark


P.S.  One of the problems often sited with UTF-8
is that it is verbose.  We could clearly offer
the option that YAML texts are "gziped" and
require that a parser know how to gunzip.

"Clark C . Evans" wrote:
> 
> On Wed, May 30, 2001 at 02:56:00PM +0200, Oren Ben-Kiki wrote:
> | Hmmm. If one is serializing a built-in map/list/scalar, you do
> | it normally; when serializing an "Object" it probably gets
> | serialized into a map, with a class attribute. It is only
> | "typed scalars" which are a problem (e.g., a number, a date, etc.).
> | One problem about the proposed class-as-color syntax is that it
> | is rather cumbersome for something like a simple number...
> 
> Right... this is half the problem.  The other half of the
> problem is that the class should be known before the map's
> content should be loaded.  I think I like the class solution
> as it is... I think the round-tripping problems are
> problematic; but perhaps they are not that bad.

Since this is up in the air, I'd like a crack at implementing things as
they stand. I think I can deal with round tripping issues; from Perl's
side anyway.

> 
> | Cut&paste implies a tool - an editor or a program. Most editors
> | allow you to trivially indent a group of lines - certainly any
> | editor used for writing YAML had better support it. And a program
> | doing cut&paste can add indentation easily enough.
> 
> Ok.  I'd rather have it always indented as well.
> 
> | I'd rather we stick with strict indentation for everything,
> | including blocks, blobs and double-quoted streams.
> 
> Did you consider that an emitter can indent the blob
> and double quoted streams?   Thus, indentation isn't
> prevented.  And with machine generated YAML (99%),
> the non-intendented case is minor.  Further, by running
> YAML through a program which puts the file in canonical
> form indenting will definately be the norm.  So... I didn't
> see the harm in allowing this.  It doesn't really impact
> the parsing complexity.

Thanks for stating this clearly Clark. I assumed it was understood.

> 
> | Separate issue: having both double-quoted and simple scalar
> | values. Here's a proposal to use just one form:
> |
> | - Block: as today, but use ` instead of \ to mark the end line.
> 
> I like the \, is there a reason why you wanted to change it?

Agree

> 
> | - Scalar value: as today, allow escaping using \, always allow
> |   it to be multi-line, but the continuation lines must be more
> |   indented then the first line.
> 
> I understand, however,  I think the current division
> of scalar types is rather nice balance of concerns.
> Most programmers will expect \ style escaping
> within quotes.

I'm completely on board with the current proposal (except that I would
not allow the absence of double quotes in a multi-line folded stream.)

> 
> | - List: as today (no special marker), but allow : as an
> |   optional prefix for scalar values.
> 
> Well, if you leave the quote type in, then
> the : optional marker won't be needed:
> 
>   Example:
>     @
>         Single line scalar
>         "Pretty multi line
>          text without multiline."
>         |
>         |Block with leading and
>         |trailing new line.
>         \
>         [Base 64 blob]

Love this! 

Um is '[' or ']' used in mime-base64?

> 
> | Hmmm. As I see it, we have at least one major issue to resolve (the class
> | issue) plus another where a "dictator hat" may be required (the text
> | syntax). Then we can all start doing development at our own pace...
> 
> As for the scalar types... I'm inclined to leave them
> as they are now.  It does seem like a nice balance of
> concerns.  What I'd like to focus on (and what should
> be added to the spec) is the canonoical form.  Here
> are some base line suggestions:
> 
>   1. Indenting always occurs, i.e. quoted or binary
>      scalars are indented.

Fine.

> 
>   2. The tab setting for indents is 4 characters

Yup.

> 
>   3. When possible the text is word-wrapped to 76
>      characters; leaving for a minimum of 20 characters
>      for scalar's content.  Thus, after 14 levels of
>      indentation, text may go beyond 20 characters.

But never on blocks, right?

> 
>   4. If leading whitespace occurs on any line within
>      a scalar, then the block format is used.

Good.

> 
>   5. If a character string is longer than 20 characters
>      without having intermediate whitespace, then
>      the quoted format is used.

And also without having any newlines or YAML meta-chars.
20 - 30 seems fine.

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

"Clark C . Evans" wrote:
> Also, I'm not sure that I like limiting unquoted streams to
> a single line.  Being able to cut/paste in a HTML text without
> having to escape the quotes and such is very valueable.  Further,
> I thought we had agreed on [base64] instead of the back tick.

I see your point. But I think disallowing them in lists is confusing.
I'll go either way here.

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

On Wed, May 30, 2001 at 01:59:24PM -0700, Brian Ingerson wrote:
| > | - Scalar value: as today, allow escaping using \, always allow
| > |   it to be multi-line, but the continuation lines must be more
| > |   indented then the first line.
| > 
| > I understand, however,  I think the current division
| > of scalar types is rather nice balance of concerns.
| > Most programmers will expect \ style escaping
| > within quotes.
| 
| I'm completely on board with the current proposal (except that
| I would not allow the absence of double quotes in a multi-line
| folded stream.)

Ok.  This prevents easy way to enter folded text
that is not escaped (like HTML), but perhaps can 
make the whole proposal cleaner.  Oren, what do 
you think?  You seem to want to eliminate one of 
the forms as well -- would reducing the un-quoted 
form to a single-line variant work for you? 
Why not just eliminate all unquoted forms?

...

After some additional background thought on the unicode 
string proposal, I think I'm in favor.  Would it have the
similar escape sequences as the double quoted string?


| > What I'd like to focus on (and what should be added 
| > to the spec) is the canonoical form.  Here
| > are some base line suggestions:
| > 
| >   1. Indenting always occurs, i.e. quoted or binary
| >      scalars are indented.
| 
| Fine.
| 
| > 
| >   2. The tab setting for indents is 4 characters
| 
| Yup.
| 
| > 
| >   3. When possible the text is word-wrapped to 76
| >      characters; leaving for a minimum of 20 characters
| >      for scalar's content.  Thus, after 14 levels of
| >      indentation, text may go beyond 20 characters.
| 
| But never on blocks, right?

Right. 
 
| > 
| >   4. If leading whitespace occurs on any line within
| >      a scalar, then the block format is used.
| 
| Good.
| 
| > 
| >   5. If a character string is longer than 20 characters
| >      without having intermediate whitespace, then
| >      the quoted format is used.
| 
| And also without having any newlines or YAML meta-chars.
| 20 - 30 seems fine.

This one lost me a bit, I'm talking about:

   key: this-is-a-long-value-that-can't-be-word-wrapped

canonical form...

   key: "this-is-a-long-value-that-can't\
         be-word-wrapped"

...

I think the canonical form is where we need to 
spend a bit more time... but unforatunately, I'm
too busy to push this further... perhaps later 
next week.

Best,

Clark

On Wed, May 30, 2001 at 02:13:52PM -0700, Brian Ingerson wrote:
| "Clark C . Evans" wrote:
| > Also, I'm not sure that I like limiting unquoted streams to
| > a single line.  Being able to cut/paste in a HTML text without
| > having to escape the quotes and such is very valueable.
|
| I see your point. But I think disallowing them in lists is confusing.
| I'll go either way here.

At first, I was very weary of the "many ways to do it".

Given that we have different types of scalar's with
different types of constraints the current four
have very nice coverage:

 Unquoted:  Good for single or multi-line folded
            content lacking significant whitespace
            and having all printables.  Good for map
            scalars.  Good for non-escaped content
            (as long as the first character does not
            begin with an indicator)

 Quoted:    "Good for single or multi-line folded
             content where some of the whitespace is
             significant and non-printables may be
             escaped.  Good for list scalars.  Not good
             when alot of whitespace must be escaped
             or with content with frequent quote usage."

 Blocked: | Good where leading and intermediate
          |   w   h  i  t  e  s  p  a  c  e
          | is important to preserve and also good
          | where " $ and other special characters
          | need not be escaped.
          \

 Binary: [BASE-64-IS-GOOD-FOR-BINARY-DATA]

In short, I see each one as having a particular class
of data it is good at representing.  And I think the
normalization rules could nicely choose among them!
The quoted format is the most flexible, but it is
also, probably the least readable.

Is our focus here on "readability"?  If so, then I 
think all four of the above forms could be important.

Best,

Clark

"Clark C . Evans" wrote:
> On Wed, May 30, 2001 at 01:59:24PM -0700, Brian Ingerson wrote:
> | I'm completely on board with the current proposal (except that
> | I would not allow the absence of double quotes in a multi-line
> | folded stream.)
> 
> Ok.  This prevents easy way to enter folded text
> that is not escaped (like HTML), but perhaps can
> make the whole proposal cleaner.  Oren, what do
> you think?  You seem to want to eliminate one of
> the forms as well -- would reducing the un-quoted
> form to a single-line variant work for you?
> Why not just eliminate all unquoted forms?

Ya know... If we just want to make the rule "Double quotes are optional
unless ambiguity results", that's fine with me.

> After some additional background thought on the unicode
> string proposal, I think I'm in favor.  

Good.

> Would it have the
> similar escape sequences as the double quoted string?

Sure. Why not?
(read, "no strong opinion")

> I think the canonical form is where we need to
> spend a bit more time... but unforatunately, I'm
> too busy to push this further... perhaps later
> next week.

I disagree. *Forget* the canonical form for now, we don't really need
it. In fact, it's not even the appropriate time for us to make such
decisions. Let's get a couple reasonable implementations out and decide
canonical forms during a beta period. It's just not right for us to be
the ones determining what the proper ways to use YAML are. Let the users
decide. Power to the people!

Just relax :)

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

"Clark C . Evans" wrote:
> At first, I was very weary of the "many ways to do it".

YAML will give rest to your weary soul.

> Given that we have different types of scalar's with
> different types of constraints the current four

five :)

> have very nice coverage:
> 
>  Unquoted:  Good for single or multi-line folded
>             content lacking significant whitespace
>             and having all printables.  Good for map
>             scalars.  Good for non-escaped content
>             (as long as the first character does not
>             begin with an indicator)
> 
>  Quoted:    "Good for single or multi-line folded
>              content where some of the whitespace is
>              significant and non-printables may be
>              escaped.  Good for list scalars.  Not good
>              when alot of whitespace must be escaped
>              or with content with frequent quote usage."
> 
>  Blocked: | Good where leading and intermediate
>           |   w   h  i  t  e  s  p  a  c  e
>           | is important to preserve and also good
>           | where " $ and other special characters
>           | need not be escaped.
>           \
> 
>  Binary: [BASE-64-IS-GOOD-FOR-BINARY-DATA-THAT-CAN-
            OBVIOUSLY-SPAN-MORE-THAN-ONE-LINE]

  Unicode: 'Good for unicode data. It''s just cool
            to use the "single quote" for this!'


If this is YAML; I like it :)

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

Brian Ingerson [mailto:briani@...] wrote:
> > Given that we have different types of scalar's with
> > different types of constraints the current four
> 
> five :)

Sigh  :-)

> > have very nice coverage:
> > 
> >  Unquoted:  Good for single or multi-line folded
> >             content lacking significant whitespace
> >             and having all printables.  Good for map
> >             scalars.  Good for non-escaped content
> >             (as long as the first character does not
> >             begin with an indicator)

I see the use of "the simplest possible syntax" without any escaping but
with folding. But I find it rather arbitrary that you can't use it (the
multi-line version of it) in lists. Using " forces you to escape any \
characters in it... Is there a chance to allow ':' as an optional prefix for
scalars in lists? That would solve this issue.

Alternatively, it seems we had a notion to allow escaping in Unquoted text
as well. That would unify it with Quoted text - make the " optional, to be
used when there's potential ambiguity, such as multi line text in lists. I'd
rather do that than have a format which is only usable in maps and not in
lists.

Under this proposal, I suggested we switch the block terminating character
to ` instead of \ to remove one case of ambiguity:
  is this: \na single line block or an escaped newline?
Of course one could just use surrounding quotes to disambiguate it, I just
thought it would be nice to eliminate one more case of requiring them.

At any rate, I think we've trashed this to death. I'd rather use one of the
above two forms (optional ':' or unifying quoted and unquoted), but I'll go
with whatever you decide. I promise not to bug you about it again :-)

> >  Quoted:    "Good for single or multi-line folded
> >              content where some of the whitespace is
> >              significant and non-printables may be
> >              escaped.  Good for list scalars.  Not good
> >              when alot of whitespace must be escaped
> >              or with content with frequent quote usage."

OK.

> >  Blocked: | Good where leading and intermediate
> >           |   w   h  i  t  e  s  p  a  c  e
> >           | is important to preserve and also good
> >           | where " $ and other special characters
> >           | need not be escaped.
> >           \

OK. We're keeping it simple - so there's no way at all to break long lines
in a block.

> >  Binary: [BASE-64-IS-GOOD-FOR-BINARY-DATA-THAT-CAN-
>             OBVIOUSLY-SPAN-MORE-THAN-ONE-LINE]

OK.

>   Unicode: 'Good for unicode data. It''s just cool
>             to use the "single quote" for this!'

I'm confused. I suggested we make the distinction between "text" (Unicode
characters) and "binary" (byte array). Base64 indicates "binary", everything
else is text.

Are you suggesting that instead we make the distinction between "Unicode"
and "byte array which may be either binary blob or ASCII text"? Where the
only syntax for Unicode is 'single quoted text'?

I don't like it because:

- There's no way to write Unicode blocks.
- You are mixing up ASCII text and binary data in the same data type,
- And separating ASCII and Unicode text for no good reason. Languages are
evolving towards "text = Unicode", as they should, and UTF-8 makes it very
easy to deal with Unicode text even in languages such as C.

As someone working in Israel, and having worked a lot with European and
Japanese clients, I'm rather sensitive to this issue... It is a pain to use
second-quality language features just because you aren't an English speaker.

Have fun,

    Oren Ben-Kiki

Oren Ben-Kiki wrote:
> >   Unicode: 'Good for unicode data. It''s just cool
> >             to use the "single quote" for this!'
> 
> I'm confused. I suggested we make the distinction between "text" (Unicode
> characters) and "binary" (byte array). Base64 indicates "binary", everything
> else is text.
> 
> Are you suggesting that instead we make the distinction between "Unicode"
> and "byte array which may be either binary blob or ASCII text"? Where the
> only syntax for Unicode is 'single quoted text'?

To be honest, I lack any real savvy in unicode issues. I merely
suggested the 
single quote as a preferred syntax, if we were going to separate ascii
from 
unicode. I probably misunderstood the intent. If it doesn't make sense,
then
by all means let's consider all text to be utf8.

> 
> I don't like it because:
> 
> - There's no way to write Unicode blocks.
> - You are mixing up ASCII text and binary data in the same data type,
> - And separating ASCII and Unicode text for no good reason. Languages are
> evolving towards "text = Unicode", as they should, and UTF-8 makes it very
> easy to deal with Unicode text even in languages such as C.

Although Perl's unicode support isn't the best, I do believe that UTF8
is
the way strings are now assumed to be by default.

> As someone working in Israel, and having worked a lot with European and
> Japanese clients, I'm rather sensitive to this issue... It is a pain to use
> second-quality language features just because you aren't an English speaker.

That makes sense.

I willfully step aside on this issue.

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

On Thu, May 31, 2001 at 09:03:37AM +0200, Oren Ben-Kiki wrote:
| I'm confused. I suggested we make the distinction between 
| "text" (Unicode characters) and "binary" (byte array).
| Base64 indicates "binary", everything else is text.

Right.  Let's call this option "binary switch"

| Are you suggesting that instead we make the distinction 
| between "Unicode" and "byte array which may be either 
| binary blob or ASCII text"?

Yes.  This was Brian's suggestion, let's call this option
the "unicode switch".  It is clear that we need a switch.

Let's go over the use cases...

  Python   -  Has a separate ASCII string and Unicode
              string.   Unicode strings are specially
              marked and conversion from ASCII to Unicode
              is relatively easy.

  Java     -  Has byte and String, where string is Unicode.

   
  Perl     -  (no clue)

  C++      -  Separate ascii (char) and unicode (wchar_t) 
              strings.
  
Python and C/C++ fit more strongly with the "Unicode Switch"
where Java is better with "Binary Switch".  Perl?

...

On a related note, consider the YAR use case.  For the
Unicode switch things works pretty well...  ASCII files
are shown as ASCII and UTF-16 nodes are shown as unicode.
Every once and a while a binary file will show up as
a readable ASCII file.  But this will be rare.

However, for the Binary switch... YAR cannot operate 
without a list of "known text extensions" where all
other files (regardless about how ASCII they look)
must be treated as binary to avoid possible mangling!

Thus, from a "yar" perspective, I like the Unicode 
Switch much better...

| Where the only syntax for Unicode is 'single quoted text'?
| 
| I don't like it because:
| 
| - There's no way to write Unicode blocks.
| - You are mixing up ASCII text and binary data in the same data type,
| - And separating ASCII and Unicode text for no good reason. Languages are
| evolving towards "text = Unicode", as they should, and UTF-8 makes it very
| easy to deal with Unicode text even in languages such as C.

Yep.  I can see these problems.  However, then our YAR 
use case must have a ".yar" file which includes the list
of "text" extensions.  Furthermore, it will have to also
note if the file was stored as UTF-8 or UTF-16.  Hmmm.

| As someone working in Israel, and having worked a lot
| with European and Japanese clients, I'm rather sensitive 
| to this issue... It is a pain to use second-quality language
| features just because you aren't an English speaker.

I agree here...  but let's work through the YAR use
case a bit more; as it seems to be our first really
good application that is easily expressable and
self-contained.

Best,

Clark

On Thu, May 31, 2001 at 09:03:37AM +0200, Oren Ben-Kiki wrote:
| > >  Unquoted:  Good for single or multi-line folded
| > >             content lacking significant whitespace
| > >             and having all printables.  Good for map
| > >             scalars.  Good for non-escaped content
| > >             (as long as the first character does not
| > >             begin with an indicator)
| 
| I see the use of "the simplest possible syntax" without any escaping but
| with folding. But I find it rather arbitrary that you can't use it (the
| multi-line version of it) in lists.

I don't see why we need to forbid it's usage in lists...
although you may consider it ugly...

  list: @
    This is a multi line
     scalar value.
    This is the second entry.

| At any rate, I think we've trashed this to death. I'd rather use one of the
| above two forms (optional ':' or unifying quoted and unquoted), but I'll go
| with whatever you decide. I promise not to bug you about it again :-)

| > >  Blocked: | Good where leading and intermediate
| > >           |   w   h  i  t  e  s  p  a  c  e
| > >           | is important to preserve and also good
| > >           | where " $ and other special characters
| > >           | need not be escaped.
| > >           \
| 
| OK. We're keeping it simple - so there's no way at all to break long lines
| in a block.

Right --  4 simple "styles"  (mod unicode) rather than one 
          big complicated mechanism.

Best,

Clark

> I don't see why we need to forbid it's usage in lists...
> although you may consider it ugly...
> 
>   list: @
>     This is a multi line
>      scalar value.
>     This is the second entry.

First, it is specifically forbidden by the current spec.
Second, yes, it is ugly, a prefix ':' would do wonders here. I've promised
not to ask again for allowing it, so I won't :-)

> | I don't like it because:
> | 
> | - There's no way to write Unicode blocks.
> | - You are mixing up ASCII text and binary data in the same 
> | data type,
> | - And separating ASCII and Unicode text for no good reason. 
> | Languages are
> | evolving towards "text = Unicode", as they should, and 
> | UTF-8 makes it very
> | easy to deal with Unicode text even in languages such as C.
> 
> Yep.  I can see these problems.  However, then our YAR 
> use case must have a ".yar" file which includes the list
> of "text" extensions.  Furthermore, it will have to also
> note if the file was stored as UTF-8 or UTF-16.  Hmmm.

Let's see. YAR needs to satisfy the following requirements:

1. Always round-trip the file correctly, byte-to-byte identical.
2. Subject to (1), use a human-readable representation of a file.
3. Carry over any meta-data about the file which is relevant.

Solution:

- Represent each file as a map:
    file-name: %
        permissions: ...
        owner: ...
        group: ...
        character-set: ...
        content: ...

- The "character set" attribute explains how to convert the value of the
'content' entry to bytes to write. It should be one of
ascii/utf-8/utf-16be/utf-16le. If the content is a binary blob, either there
is no "character set" (the content isn't textual), or it could just say
"binary" (I know that's not a IANA recognized character set...)

- The yar file creator chooses which syntax format to give the content
according to the following heuristic. In each case, it ensures the encoding
will be such that the file will be re-created byte-to-byte identical to the
original:

If the file contains...
1. ... only 7-bit printable ASCII characters, plus newlines => syntax:
block, charset: ascii.
2. ... only utf-8 printable characters, plus newlines => syntax: block,
charset: utf-8.
3. ... like 1, with the odd 8-bit character (but not 2) => syntax: quoted
string, charset: ascii.
4. ... like 2, with the odd unprintable character => syntax: quoted string,
charset: utf-8.
5. ... what looks to be as a utf-16 file with only printable characters plus
newlines => syntax: block charset: utf-16be/utf-16le.
6. ... like 5 with the odd non-printable character => syntax: quoted string,
charset: utf-16be/utf-16le.
7. ... anything else => syntax: blob, charset: binary (or missing).

This works, is safe, is extensible, and doesn't require us to bend YAML out
of shape. The only things which bothers me is handling newlines when
transferring between DOS/Windows and the rest of the world. Question: in a
block, does YAML preserve whether a line ended with a \n or a \r\n? If not
we are OK.

And I'm still worried about the class marker... Can you say something about
what you mean by "class map"? I didn't get it.

Have fun,

    Oren Ben-Kiki

I meant to write:

> Question: in a
> block, does YAML preserve whether a line ended with a \n or a 
> \r\n? If not
^^^^^^^^^ yes
> we are OK.

Oren.

yOn Thu, May 31, 2001 at 01:46:01PM +0200, Oren Ben-Kiki wrote:
| > I don't see why we need to forbid it's usage in lists...
| > although you may consider it ugly...
| > 
| >   list: @
| >     This is a multi line
| >      scalar value.
| >     This is the second entry.
| 
| First, it is specifically forbidden by the current spec.

It is?  It was in a previous spec, but not the recent
one (the one dated the 26th) which was updated in a major
way on the 26th after it was released (my bad). 
Where explicitly, as I actually remember working this
through in my head so that it wasn't an exception.

| Second, yes, it is ugly, a prefix ':' would do wonders here.

Hmm.  Considering.


| Let's see. YAR needs to satisfy the following requirements:
| 
| 1. Always round-trip the file correctly, byte-to-byte identical.
| 2. Subject to (1), use a human-readable representation of a file.
| 3. Carry over any meta-data about the file which is relevant.

Good.


| - Represent each file as a map:
|     file-name: %
|         permissions: ...
|         owner: ...
|         group: ...
|         character-set: ...
|         content: ...
| 
| - The "character set" attribute explains how to convert the value of the
| 'content' entry to bytes to write. It should be one of
| ascii/utf-8/utf-16be/utf-16le. If the content is a binary blob, either there
| is no "character set" (the content isn't textual), or it could just say
| "binary" (I know that's not a IANA recognized character set...)

Ok.  So we've made an explicit node attribute 
called "character set".  Interesting.  Should
this be added using a special indicator?  ^utf-8

| This works, is safe, is extensible, and doesn't require us to bend YAML out
| of shape. The only things which bothers me is handling newlines when
| transferring between DOS/Windows and the rest of the world. Question: in a
| block, does YAML preserve whether a line ended with a \n or a \r\n? If not
| we are OK.

Yep.  I was worring about this one too.  I'm not
sure what the solution is.  Normalizing new lines
is a perfectly reasonable thing to do... except
in this case.

| And I'm still worried about the class marker... Can you 
| say something about what you mean by "class map"? I 
| didn't get it.

Assume you only have a object which is a map, list, or scalar.  
The YAML parser/emitter pair could keep a global "classmapping"
which associated all objects created via reading from the
serialized format with it's optional class map.  Then, when
writing the YAML file, the global variable could be used to
reconstruct the class map attribute.  Yes, this opens up
a small can of worms... garbage collection among them.
However, it does make round-tripping possible even when
class isn't supported by the system.

Best,

Clark

Oren Ben-Kiki wrote:
> 
> I meant to write:
> 
> > Question: in a
> > block, does YAML preserve whether a line ended with a \n or a
> > \r\n? If not
> ^^^^^^^^^ yes
> > we are OK.

Then by all means, preserve.

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

On Thu, May 31, 2001 at 01:46:01PM +0200, Oren Ben-Kiki wrote:
| - The "character set" attribute explains how to convert the value of the
| 'content' entry to bytes to write. It should be one of
| ascii/utf-8/utf-16be/utf-16le. If the content is a binary blob, either there
| is no "character set" (the content isn't textual), or it could just say
| "binary" (I know that's not a IANA recognized character set...)

I was thinking a bit differently.  Perhaps either
we should use "class" to indicate character set
or introduce another indicator.  Your thoughts.
Consider the current "built-in" classes,
"_int", "_real", etc.   Are not they encodings?

| The only things which bothers me is handling newlines when
| transferring between DOS/Windows and the rest of the world.

This was bugging me, but I like the YAML spec as it is.

| Question: in a block, does YAML preserve whether a line
| ended with a \n or a \r\n?

No.  This is one of the nice problems XML cleared up, let's
not roll back the clock.  I've yet to hear anyone complain
that XML's folding of \r\n and \r into just \n was anything
but helpful.

The only place it gets us into problems is for the YAR
use case... where the serializer doesn't know if a given
value is character or binary.   In this case, I think 
the behavior should be dependent upon the platform.
On unix platforms, files with \r\n are treated as binary
(to preserve the line endings), and on DOS boxes, files
ending with \n (without \r) are treated as binary.

This gives the expected behavior when moving files
between platforms.  By the way, with the current
treatment, any YAML saved on a Windows box will
have \r\n line endings.  And likewise, on a 
Unix box it will have \n line endings.  Since the
parser doesn't care, one can move the files back
and forth without affecting the canonical form.

This may not give exactly the same behavior 
as "tar", since the exact line endings won't
be preserved when switching platforms... but this
is always a sore spot anyway!  As for "diff",
there are (or should be) flags to not report
line ending differences.

Therefore, I'm pretty certain that I'd like to keep
the line ending folding as it is in the YAML spec.

Best,

Clark

Clark C . Evans [mailto:cce@...] wrote:
> Ok.  So we've made an explicit node attribute 
> called "character set".  Interesting.  Should
> this be added using a special indicator?  ^utf-8

Eeek. What next, a special notation for permissions (!), owner (~), and
group (?)?

I thought YAML was to use the minimal number of special properties - data
type, and reference handling, and (maybe!) classes. Anything else can and
should be done by using normal map keys.

> | And I'm still worried about the class marker... Can you 
> | say something about what you mean by "class map"? I 
> | didn't get it.
> 
> Assume you only have a object which is a map, list, or scalar.  
> The YAML parser/emitter pair could keep a global "classmapping"
> which associated all objects created via reading from the
> serialized format with it's optional class map.  Then, when
> writing the YAML file, the global variable could be used to
> reconstruct the class map attribute.  Yes, this opens up
> a small can of worms... garbage collection among them.
> However, it does make round-tripping possible even when
> class isn't supported by the system.

In essence you are proposing an alternate color mechanism - one based on a
separate store, "to the side of" the document, which contains the "color"
info for each node. In other words, admitting that YAML isn't good enough by
itself to handle this data and therefore a separate mechanism must be added
to it. I strongly dislike this.

Just like YAR is a great, concrete example of an application for resolving
syntax issues etc. I feel that classes are a great, concrete example of an
application for resolving the color issue. Let's "eat our own dog food"
here.

Have fun,

    Oren Ben-Kiki

Clark C . Evans [mailto:cce@...] wrote:
> On Thu, May 31, 2001 at 01:46:01PM +0200, Oren Ben-Kiki wrote:
> | - The "character set" attribute explains how to convert the 
> | value of the
> | 'content' entry to bytes to write. It should be one of
> | ascii/utf-8/utf-16be/utf-16le. If the content is a binary 
> | blob, either there
> | is no "character set" (the content isn't textual), or it 
> | could just say
> | "binary" (I know that's not a IANA recognized character set...)
> 
> I was thinking a bit differently.  Perhaps either
> we should use "class" to indicate character set
> or introduce another indicator.  Your thoughts.
> Consider the current "built-in" classes,
> "_int", "_real", etc.   Are not they encodings?

It seems I'm making a distinction here that you aren't.

Issue 1: How to convert YAML text to an in-memory object and vice-versa
(YAML issue).
Issue 2: How to convert YAR file content represented in a YAML scalar to a
file on the disk and vice versa (YAR issue).

Character set, as I described it, belongs to issue 2. It seems to me
completely outside the scope of YAML to include, as part of the object
model, the answer to the question "if this Unicode string was to be written
as the sole content of an on-disk file, which of the many possible encodings
to use for it".

Your questions about the built-in types, however, relate to issue 1: When
reading the YAML file into memory in, say, Java, should I create a String
in-memory object, a Float in-memory object (or maybe a Double?), or an
Integer in-memory object (or maybe a Long?). Great question, to which my
answer is: "determine how YAML will support color, and do it that way".

> | Question: in a block, does YAML preserve whether a line
> | ended with a \n or a \r\n?
> 
> No.  This is one of the nice problems XML cleared up, let's
> not roll back the clock.  I've yet to hear anyone complain
> that XML's folding of \r\n and \r into just \n was anything
> but helpful.
> 
> The only place it gets us into problems is for the YAR
> use case... where the serializer doesn't know if a given
> value is character or binary.   In this case, I think 
> the behavior should be dependent upon the platform.
> On unix platforms, files with \r\n are treated as binary
> (to preserve the line endings), and on DOS boxes, files
> ending with \n (without \r) are treated as binary.

A "binary" file will be represented as a base64 blob in the YAR file. You
don't have to go that far to handle an occasional \r\n or \n in a file; you
can just use a quoted string and escape them properly. That's still "mostly
text".

> This gives the expected behavior when moving files
> between platforms.  By the way, with the current
> treatment, any YAML saved on a Windows box will
> have \r\n line endings.  And likewise, on a 
> Unix box it will have \n line endings.  Since the
> parser doesn't care, one can move the files back
> and forth without affecting the canonical form.

That's neat.

> This may not give exactly the same behavior 
> as "tar", since the exact line endings won't
> be preserved when switching platforms... but this
> is always a sore spot anyway!  As for "diff",
> there are (or should be) flags to not report
> line ending differences.

So, YAR is more like "shar" then "tar". That makes a lot of sense.

> Therefore, I'm pretty certain that I'd like to keep
> the line ending folding as it is in the YAML spec.

OK.

Have fun,

    Oren Ben-Kiki

Clark C . Evans [mailto:cce@...] wrote:
> yOn Thu, May 31, 2001 at 01:46:01PM +0200, Oren Ben-Kiki wrote:
> | > I don't see why we need to forbid it's usage in lists...
> | > although you may consider it ugly...
> | > 
> | >   list: @
> | >     This is a multi line
> | >      scalar value.
> | >     This is the second entry.
> | 
> | First, it is specifically forbidden by the current spec.
> 
> It is?  It was in a previous spec, but not the recent
> one (the one dated the 26th) which was updated in a major
> way on the 26th after it was released (my bad). 
> Where explicitly, as I actually remember working this
> through in my head so that it wasn't an exception.

I just checked and you are right - I have a printed version of the "older"
26th draft. Sorry.

Things about the latest one which we agreed to fix:

- Indentation in quoted strings;
- Separate text from binary in the information model;
- Clearing up how the API handles "multi-map" files (with blank lines).

Other things which I noticed:

- The draft requires a blank line at the end of the document;
- There are some formatting issues;
- There's a special standing for ISO=8859-1 which I don't follow. Isn't
everything in Unicode? What does it mean to write an ISO-8859-1 escape
within a quoted string?

And, there's the class issue...

Have fun,

    Oren Ben-Kiki

> Things about the latest one which we agreed to fix:
> 
> - Indentation in quoted strings;
And blobs...
> - Separate text from binary in the information model;
> - Clearing up how the API handles "multi-map" files (with 
> blank lines).
> 
> Other things which I noticed:
> 
> - The draft requires a blank line at the end of the document;
> - There are some formatting issues;
> - There's a special standing for ISO=8859-1 which I don't 
> follow. Isn't
> everything in Unicode? What does it mean to write an ISO-8859-1 escape
> within a quoted string?
> 
> And, there's the class issue...

Have fun,

    Oren Ben-Kiki

On Sun, Jun 03, 2001 at 05:15:56PM +0200, Oren Ben-Kiki wrote:
| I had this notion about the syntactical form of class-as-color (or any
| color). What if one allows the first key pair to appear immediately after
| the '%':
| 
|     delivery: % : 10-JAN-2001
|         # : date
| 
| OK, the class has to appear on a separate line, but that's not *too* bad,
| textually. Especially after removing some whitespace:
| 
|     delivery:%: 10-JAN-2001
|         #: date

If we want to "abbreviate", why not just allow
the single line form...

  delivery: !date 10-JAN-2001

to be an abbreviation for ...

  delivery: %
     !  : date
     "" : 10-JAN-2001


Thoughts?  The above form is what would be
exposed through the information model.

| I'm using your "value is the empty key" syntax, to reduce the "line noise"
| effect. Speaking of line noise, how about saying that by convention, the '~'
| ("null") can be used as a key to denote comments:
| 
|     map: %
|         ~ : This map contains this and that...
|         key : value
|         ...
| 
|     delivery:%: 10-JAN-2001
|         #: date
|         ~: This is a comment about the delivery date
| 
| Comments are just another color! This solves all our problems with them
| (being in the info model on the one hand, but being ignored by applications
| on the other). I *love* the color idiom.

I do as well.  It may seem like extra noise if 
you are just worried about classes... however,
when you start to add other colors, it becomes
a very powerful mechanism... 

Hmm.  Just thinking.  The pattern above could work
for any indicator used as a key...

  delivery: !date #"Comment on delivery" 10-JAN-2001

| Now, if there's a way to de-serialize the above into something which behaves
| "like a string" in Perl/Python...

For Python, I think it works well.  All maps with
a recognized "class" attribute are converted to
a class on load (if they are recognized) otherwise
they are left as maps.

This works well.

Best,

Clark

P.S.  I think I'd like to use # for comment and ! for class,
      what do you think?

On Sun, Jun 03, 2001 at 04:57:10PM +0200, Oren Ben-Kiki wrote:
| > I guess we should add \n to the list of escape sequences?
| 
| It belongs together with \r, \t and the rest of them. No helping it. (\e for
| ESC would be nice - I often wished for it when writing VT100 escape codes

ok.  we add \r and \e
 
| > Hmm.  Assume a binary file, containing only 
| > "x\n" on a unix box.  When this gets moved
| > to a dos box, it will be "x\r\n".  Hmm.
| 
| The trap of "looking like text". Right.
| 
| > I guess YAR would have to do something
| > special here, like encode if new lines
| > are \n or \r\n as well as the character
| > encoding.  Icky.
| 
| If you realize it is a binary file, the only "special thing" you need to do
| is encode it as a base64 blob. It will then be written as "x\n" in DOS as
| well. The problem is, how do you realize "x\n" is a *binary* file
| containing, say, a short integer whose value unfortunately looks ASCII text
| when serialized?
| 
| I guess the short answer is "you don't". The longer one is "give YAR all
| sort of options, flags etc. to allow it to cope" - e.g., a '-b <path>'
| forcing the file <path> to be handled as binary, or '-x <bin-ext>' forcing
| all files ending with <bin-ext> to be handled as binary, etc.
| 
| Pretty it isn't, but that's what you get from trying to handle CP/M newlines
| rules, UNIX file systems, and not having BeOS mime type associations with
| files...

Well, it's either this or give up on the new line
equivalence.  I think the new line equivalence
is more important... YAR is a much smaller use case.

Clark

On Sun, Jun 03, 2001 at 09:44:47AM +0200, Oren Ben-Kiki wrote:
| Things about the latest one which we agreed to fix:
| 
| - Indentation in quoted strings;
| - Separate text from binary in the information model;
| - Clearing up how the API handles "multi-map" files (with blank lines).
| 
| Other things which I noticed:
| 
| - The draft requires a blank line at the end of the document;
| - There are some formatting issues;

Could you fix the above and send me the draft?  I'm 
totally swamped with my day job...

| - There's a special standing for ISO=8859-1 which I don't follow. Isn't
| everything in Unicode? What does it mean to write an ISO-8859-1 escape
| within a quoted string?

Ahh.  There are two escapes, \xXX and \uXXXX, where \x takes
a 8 bit value and \u takes a 16 bit value.  You need both
since \x20BABE Is a space followed by BABE, where \u20BABE
is an oriental character (20BA) followed by BE.

| And, there's the class issue...

A "clean" proposal would help.  I know you made
a few in the past, but perhaps if we put this
on the table we could come to a conclusion.

;) Clark

On Sun, Jun 03, 2001 at 09:12:11AM +0200, Oren Ben-Kiki wrote:
| It seems I'm making a distinction here that you aren't.
| 
| Issue 1: How to convert YAML text to an in-memory object 
| and vice-versa (YAML issue).
|
| Issue 2: How to convert YAR file content represented in a 
| YAML scalar to a file on the disk and vice versa (YAR issue).

Nice separation.

| A "binary" file will be represented as a base64 blob in 
| the YAR file. You don't have to go that far to handle an 
| occasional \r\n or \n in a file; you can just use a quoted 
| string and escape them properly. That's still "mostly text".

I guess we should add \n to the list of escape sequences?


| > This gives the expected behavior when moving files
| > between platforms.  By the way, with the current
| > treatment, any YAML saved on a Windows box will
| > have \r\n line endings.  And likewise, on a 
| > Unix box it will have \n line endings.  Since the
| > parser doesn't care, one can move the files back
| > and forth without affecting the canonical form.
| 
| That's neat.

It's just a strawman.  
 
| > This may not give exactly the same behavior 
| > as "tar", since the exact line endings won't
| > be preserved when switching platforms... but this
| > is always a sore spot anyway!  As for "diff",
| > there are (or should be) flags to not report
| > line ending differences.
| 
| So, YAR is more like "shar" then "tar".
| That makes a lot of sense.

Hmm.  Assume a binary file, containing only 
"x\n" on a unix box.  When this gets moved
to a dos box, it will be "x\r\n".  Hmm.
I guess YAR would have to do something
special here, like encode if new lines
are \n or \r\n as well as the character
encoding.  Icky.

Clark

----- End forwarded message -----

"Clark C . Evans" wrote:
> 
> On Sun, Jun 03, 2001 at 05:15:56PM +0200, Oren Ben-Kiki wrote:
> | I had this notion about the syntactical form of class-as-color (or any
> | color). What if one allows the first key pair to appear immediately after
> | the '%':
> |
> |     delivery: % : 10-JAN-2001
> |         # : date
> |
> | OK, the class has to appear on a separate line, but that's not *too* bad,
> | textually. Especially after removing some whitespace:
> |
> |     delivery:%: 10-JAN-2001
> |         #: date
> 
> If we want to "abbreviate", why not just allow
> the single line form...
> 
>   delivery: !date 10-JAN-2001
> 
> to be an abbreviation for ...
> 
>   delivery: %
>      !  : date
>      "" : 10-JAN-2001
> 

I wrote my last email suggesting "inline classes" without even seeing
this. We must be on a similar wave length. :)

However, be aware that we are suggesting two different things. 

You are suggessting an abbreviation of coloring.

I am suggesting a "lightweight" semantic for Inline classes. One that
would not round trip between Perl and Java, but would round trip between
Perl and Perl. It's a disposable class I guess, at least from the YAML
side. Something like this will be vital to allow scripting languages (or
at least Perl) to use objects in their normal, lightweight way.

FYI, In Perl an object is just any old data structure with a name
attached by the bless() function. It can access it's data in through
predefined methods or not. It actually turns out to be a very compelling
OO model. But that's a different discussion.

Now Perl can also benefit from a heavyweight colored YAML object. For
instance, to redefine the emitter sort order, or to ensure that the
emmitter uses the correct encoding. Or for round tripping data to Java.

But the bottom line is that if you want to keep Perl interested (and me
too) you *need* a lightweight semantic option. 

BTW, If the Perl implementation is done right, it will make YAML so much
more appealling tham XML.

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

On Sun, Jun 03, 2001 at 11:26:29AM -0700, Brian Ingerson wrote:
| >
| >   delivery: #date 10-JAN-2001
| > 
| > to be an abbreviation for ...
| > 
| >   delivery: %
| >      # : date
| >      = : 10-JAN-2001

(the above fixed so that # means class, and = means 
default value.)

| However, be aware that we are suggesting two different things. 
| You are suggessting an abbreviation of coloring.

Right.

| I am suggesting a "lightweight" semantic for Inline classes. One that
| would not round trip between Perl and Java, but would round trip between
| Perl and Perl. It's a disposable class I guess, at least from the YAML
| side. Something like this will be vital to allow scripting languages (or
| at least Perl) to use objects in their normal, lightweight way.

Ok.  Why wouldn't the abbreviation work for round tripping
between Perl and Perl?  Would you ever write out a "map" 
that has only a class (#) and a default value (=) ?

| BTW, If the Perl implementation is done right, it will make YAML 
| so much more appealling tham XML.

I hope so!

;) Clark

On Sun, Jun 03, 2001 at 07:18:32PM -0700, Brian Ingerson wrote:
| > On Sun, Jun 03, 2001 at 11:26:29AM -0700, Brian Ingerson wrote:
| > | >
| > | >   delivery: #date 10-JAN-2001
| > | >
| > | > to be an abbreviation for ...
| > | >
| > | >   delivery: %
| > | >      # : date
| > | >      = : 10-JAN-2001
| > 
| > (the above fixed so that # means class, and = means
| > default value.)
| > 
| > | However, be aware that we are suggesting two different things.
| > | You are suggessting an abbreviation of coloring.
| > 
| > Right.
| > 
| > | I am suggesting a "lightweight" semantic for Inline classes. One that
| > | would not round trip between Perl and Java, but would round trip between
| > | Perl and Perl. It's a disposable class I guess, at least from the YAML
| > | side. Something like this will be vital to allow scripting languages (or
| > | at least Perl) to use objects in their normal, lightweight way.
| > 
| > Ok.  Why wouldn't the abbreviation work for round tripping
| > between Perl and Perl?  Would you ever write out a "map"
| > that has only a class (#) and a default value (=) ?
| 
| Yes. Perl to Perl *would* round trip but it wouldn't be required to.
| Basically YAML allows the abbreviation but doesn't require it to round
| trip. It is used for implementation specific round tripping. Perl and
| Python *could* use it if they agreed upon an implementation.

Ok.  So the abbreviation would be acceptable, rather
than having a separate mechanism then? In other words, 
the abbreviated form and the non-abbreviated form would 
appear *identical* as far as the sequential "parser"
API is concerned.  Is this ok?

It would then be up to the "Perl Binding" to specify
that both of the two forms above turn into a 
delivery "scalar" having value 10-JAN-2001 
blessed as a "date" class. 

| The color scheme, on the other hand, represents the YAML object model,
| which every implementation is required to parse and preserve.

I'd like both forms to be YAML object which every implementation
must parse and preserve, one is just "syntax sugar" and is
not exposed in the API.

Yes?

Clark

Brian Ingerson [mailto:briani@...] wrote:
> Now Perl can also benefit from a heavyweight colored YAML object. For
> instance, to redefine the emitter sort order, or to ensure that the
> emmitter uses the correct encoding. Or for round tripping 
> data to Java.

Cool.

> But the bottom line is that if you want to keep Perl 
> interested (and me
> too) you *need* a lightweight semantic option. 

Granted. However, I don't see why Clark's "abbreviated" syntax can't serve
as a "lightweight semantic option".

I thought the idea is that:

    price: !real 10.5

And:

    price: %
        = : 10.5
        ! : real

Will both be de-serialized, in Perl, into a simple number "10.5", in Java
into a simple Float 10.5, etc. A reasonable emitter will use the first,
shorter form. That seems like a "lightweight semantic option" to me... What
is the additional requirement you have for such an option that the above
mechanism doesn't provide?

(I'm using '!' for class so we can use '#' for comments - I think that's a
more intuitive use).

> BTW, If the Perl implementation is done right, it will make 
> YAML so much
> more appealling tham XML.

Amen to that :-)

Have fun,

    Oren Ben-Kiki

Hi Oren,

After talking to Clark I agree that the abbreviation is now just an
abbreviation of color. Substituability mandates my change of mind. I
think it's of value but read on...

Oren Ben-Kiki wrote:
> > But the bottom line is that if you want to keep Perl
> > interested (and me
> > too) you *need* a lightweight semantic option.
> 
> Granted. However, I don't see why Clark's "abbreviated" syntax can't serve
> as a "lightweight semantic option".
> 
> I thought the idea is that:
> 
>     price: !real 10.5
> 
> And:
> 
>     price: %
>         = : 10.5
>         ! : real
> 
> Will both be de-serialized, in Perl, into a simple number "10.5", in Java
> into a simple Float 10.5, etc. A reasonable emitter will use the first,
> shorter form. That seems like a "lightweight semantic option" to me... What
> is the additional requirement you have for such an option that the above
> mechanism doesn't provide?

I think of 'real' as more of a special type. What about Foo?

If we have:

    key : !Foo a fooish string

That means I need to generate a Foo object which is a subclass of a YAML
object. All nodes will be objects which are subclasses of YAML. That's
because of this substituability thing. They pretty much have to be,
right? So my Perl might look like:

    $y = YAML->parse('doc1.yaml');
    print $y->getValue('key')->getValue;

prints:

    a fooish string

That's fine in Java, but it's way too ugly for Perl programmers to
accept. They'll want:

    print $y->{key};

Now this is just the simplest example. It gets really bad, really quick.

Even this isn't simpler:

    key : a fooish string

You would still need the exact same code as above, because of
substitutability it might turn into:

    key : @
        a fooish string
        a barrish string

Setting up Perl data for emission is just as problematic.

Now Perl has a wonderful "tie" facility that may allow me to expose the
simpler syntax. I'm not sure how well it will work out yet. But the
underlying implementation will be sloooow. C parser or not. In effect,
YAML no longer maps directly into scripting languages. It maps into OO
languages. A map is no longer a map. Its a YAML object. Same with
everything else.

So I'm willing to try this, but I'm skeptimistic about the usability
from Perl. This will never become a good alternative to Data::Denter.
(Long live Data::Denter :)

Perhaps a better alternative is to only provide substitutability within
classes. Therefore all classes will be first class YAML classes
(round-tripping, substitutable, etc), but regular data will never get
upgraded into a class. That way, maps are just maps like before. Hmm. I
really like that.

Maybe that's what you guys were already thinking? I'll have to go back
and check.

> 
> (I'm using '!' for class so we can use '#' for comments - I think that's a
> more intuitive use).

That makes sense.


Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

Brian Ingerson [mailto:briani@...] wrote:
> I think of 'real' as more of a special type. What about Foo?
> 
> If we have:
> 
>     key : !Foo a fooish string
> 
> That means I need to generate a Foo object which is a 
> subclass of a YAML
> object.

If 'Foo' is a recognized class, you just build a Foo object. 'Foo' doesn't
have to derive from any YAML class.

If 'Foo' is an *unrecognized* class, then you build a YAML-object object.
The 'YAML-object' class (or whatever we'll call it) behaves like both a
string an a map. In an "ideal" language (Sather, anyone? :-) all you'd need
to do is write:

class YAML-OBJECT: public MAP, public STRING { some tricks to make it work
};

Again, there's no new class created; the above class is used only once,
throughout the system.

> All nodes will be objects which are subclasses of YAML.
> That's because of this substitutability thing.

Nope. Let's enumerate:

map: %
    key: value

Is de-serialized to a normal, built-in hash.

list: @
   value
   value

Is de-serialized to a normal, built-in list.

key : value

Is de-serialized into a normal string.

map: %
    = : value

And:

key : !class value

Are de-serialized into a YAML-object. That is, *Only* if you are actually
*using* substitutability, you pay for it (The good old C++ notion of "don't
use, don't pay"). That's why I don't like Clark's idea of a list having a
default value of the first element; it forces one to always pay this price,
when for most lists it isn't required. I also don't think it is useful to
evolve a scalar into a list; that's not part of the "color" idiom, anyway.
Clark?

> If we have:
>    key : !Foo a fooish string
>
> ... my Perl might look like:
> 
>     $y = YAML->parse('doc1.yaml');
>     print $y->getValue('key')->getValue;

Ideally, if a proper YAML-object class can be written in Perl, it would look
like;

    # Access to "default" value of 'key' - 'fooish string'
    print $y->{key};
    # Access to value of 'key' as a map, to the class
    # member - 'Foo';
    print $y->{key}->{'!'};
    # Access to value of key as a map, to the value
    # member - 'fooish string'.
    print $y->{key}->{'='};

The question is: is this possible in Perl? I suspect it can...

> Even this isn't simpler:
> 
>     key : a fooish string
> 
> You would still need the exact same code as above
[i.e., the same code as accessing a YAML-object]

On that we agree; the question is, how simple can we make this code be, and
how efficient *in the normal case*. I'd rule out any implementation where
the above isn't de-serialized into a built-in Perl string.

> Now Perl has a wonderful "tie" facility that may allow me to 
> expose the
> simpler syntax. I'm not sure how well it will work out yet. But the
> underlying implementation will be sloooow.

I've no problem with an implementation where an *unknown*-class-annotated
value is working slower. Makes sense?

> So I'm willing to try this, but I'm skeptimistic about the usability
> from Perl. This will never become a good alternative to Data::Denter.
> (Long live Data::Denter :)

I think that YAML *must* be usable from Perl. Let's make it so!

> Perhaps a better alternative is to only provide 
> substitutability within
> classes. Therefore all classes will be first class YAML classes
> (round-tripping, substitutable, etc), but regular data will never get
> upgraded into a class. That way, maps are just maps like 
> before. Hmm. I really like that.

I'm not certain exactly what you mean...

> Maybe that's what you guys were already thinking? I'll have to go back
> and check.

It does sound a bit like what I described above...

The only problem with the above scheme is that given a normal Perl map,
suppose I "innocently" add the key '=' to it. Ideally, it should polymorph
into a YAML-object. Deleting that key should cause it to polymorph back into
a normal map. It seems to me that either of these operations is momentous
enough to warrant an explicit call, such as:

 $yaml_object = make_yaml_object_from_normal_map($normal_map, $value);
 ($value, $normal_map) = make_normal_map_from_yaml_object($yaml_object);

Acceptable?

> > (I'm using '!' for class so we can use '#' for comments - I 
> think that's a
> > more intuitive use).
> 
> That makes sense.

OK, I'm going to do an update to the draft (nothing about the
substitutability yet, just the syntactical issues we agreed on) - I'll throw
that in as well.

Have fun,

    Oren Ben-Kiki

Oren Ben-Kiki wrote:
> 
> Brian Ingerson [mailto:briani@...] wrote:
> > I think of 'real' as more of a special type. What about Foo?
> >
> > If we have:
> >
> >     key : !Foo a fooish string
> >
> > That means I need to generate a Foo object which is a
> > subclass of a YAML
> > object.
> 
> If 'Foo' is a recognized class, you just build a Foo object. 'Foo' doesn't
> have to derive from any YAML class.

But it probably should. That's where it gets it's default behavior. Like
sort_order, etc.

> If 'Foo' is an *unrecognized* class, then you build a YAML-object object.
> The 'YAML-object' class (or whatever we'll call it) behaves like both a
> string an a map. In an "ideal" language (Sather, anyone? :-) all you'd need
> to do is write:
> 
> class YAML-OBJECT: public MAP, public STRING { some tricks to make it work
> };
> 
> Again, there's no new class created; the above class is used only once,
> throughout the system.

OK. I think I follow.

> 
> > All nodes will be objects which are subclasses of YAML.
> > That's because of this substitutability thing.
> 
> Nope. Let's enumerate:
> 
> map: %
>     key: value
> 
> Is de-serialized to a normal, built-in hash.
> 
> list: @
>    value
>    value
> 
> Is de-serialized to a normal, built-in list.
> 
> key : value
> 
> Is de-serialized into a normal string.
> 
> map: %
>     = : value
> 
> And:
> 
> key : !class value
> 
> Are de-serialized into a YAML-object. That is, *Only* if you are actually
> *using* substitutability, you pay for it (The good old C++ notion of "don't
> use, don't pay"). That's why I don't like Clark's idea of a list having a
> default value of the first element; it forces one to always pay this price,
> when for most lists it isn't required. I also don't think it is useful to
> evolve a scalar into a list; that's not part of the "color" idiom, anyway.
> Clark?

Well, I *like* all of this. It's certainly more appealing than what I
had feared. If non-classed structures remain native, we're cooking with
gas!

Clark's idea of having a scalar change to a list, requires that it
actually be neither. It can only be an object.

Also, it doesn't seem to make sense that a !Foo could morph into a !Bar
does it. Maybe if !Bar were a subclass...??? Do we support subclasses?
[...]
> > Perhaps a better alternative is to only provide
> > substitutability within
> > classes. Therefore all classes will be first class YAML classes
> > (round-tripping, substitutable, etc), but regular data will never get
> > upgraded into a class. That way, maps are just maps like
> > before. Hmm. I really like that.
> 
> I'm not certain exactly what you mean...
> 
> > Maybe that's what you guys were already thinking? I'll have to go back
> > and check.
> 
> It does sound a bit like what I described above...

I think for once, Oren and I are on the same side of the fence, and
Clark might be on the other.

Clark?

Cheers, Brian

-- 
perl -le 'use Inline C=>q{SV*JAxH(char*x){return newSVpvf
("Just Another %s Hacker",x);}};print JAxH+Perl'

Brian Ingerson [mailto:briani@...] wrote:
> > If 'Foo' is a recognized class, you just build a Foo 
> > object. 'Foo' doesn't
> > have to derive from any YAML class.
> 
> But it probably should. That's where it gets it's default 
> behavior. Like
> sort_order, etc.

We must ensure that (de)serialization works for arbitrary objects. This
isn't difficult for languages such as Perl, Python and even Java. Reflection
capabilities which are built-in to the language allows you to do a
reasonable job, adequate for most objects.

Sure, we'll end up defining an interface which, if an object supports,
allows it to control the (de)serialization - what fields are printed first,
say, or not printing "derived" fields and re-computing them when the object
is read, etc. But that's luxury. The principle is "anything goes". The Java
serialization model is a good example of how such things can be done.

It may be worth it to consider subverting the Java mechanism for our
purposes somehow. Imagine every Java object in the world being YAML-aware
"out of the box". I know someone did something similar for XML - maybe it
can be used as a basis. The only problem is that it will litter the files
with too many Java-specific class names, and too many anchors... Hmmm...
This deserves some serious thought for which I don't really have the time.

> Clark's idea of having a scalar change to a list, requires that it
> actually be neither. It can only be an object.

Yup. That's why I dislike it.

> I think for once, Oren and I are on the same side of the fence, and
> Clark might be on the other.

It seems that way. Well, the (dis)advantage of a triumvirate is that two
people can always gang up on the third :-)

Have fun,

    Oren Ben-Kiki

On Tue, Jun 05, 2001 at 01:07:16AM -0700, Brian Ingerson wrote:
| > Are de-serialized into a YAML-object. That is, *Only* if you 
| > are actually *using* substitutability, you pay for it (The 
| > good old C++ notion of "don't use, don't pay").

I was only considering the iterator (parser) 
interface having this functionality built-in, 
since the additional overhead is an additional
(and rather simple) function that the user
of the library can just ignore if they so choose.

For language bindings, I was thinking that the
asScalar function would be more or less an *external*,
or "friend" function, not a method. I was using the 
method syntax for readability, I had not intended
that it actually be a method....

If, for some cases, it can be implemented as 
a method, well... more power to that language.
But I don't think this is at all a requirement,
an external function should be just fine, as ugly 
as it may be.

| > That's why I don't like Clark's idea of a list having a
| > default value of the first element; it forces one to always
| > pay this price, when for most lists it isn't required. I also
| > don't think it is useful to evolve a scalar into a list; 
| > that's not part of the "color" idiom, anyway.

I think it may be useful for times when something originally
thought to be single value (such as an address) turns out
to be multi-valued.  If you only allow "extensions" using
the map construct, this forces a particular way to extend
the scalar (use a map, and then a list) instead of just
converting the scalar into a list.  I've had this case
pop up before...

| > Clark?

Well, if it seems problematic... I'll compromise here. ;)

| Well, I *like* all of this. It's certainly more appealing than what I
| had feared. If non-classed structures remain native, we're cooking with
| gas!

I hope so.  Charcol may taste better, but it 
is very messy.  And who likes to chop their own wood?
For recreation, yes, but not for daily work!

| Clark's idea of having a scalar change to a list, requires that it
| actually be neither. It can only be an object.

Brian, thank you for working the proposal out.... I 
was not actually intending this.  Although I was 
thinking of an exteranal "asScalar()" function that 
returns a scalar when handed a list.  Is this possible?

| Also, it doesn't seem to make sense that a !Foo could morph
| into a !Bar does it. Maybe if !Bar were a subclass...??? Do 
| we support subclasses?

Hmm. Suppose you were expecting...

  tag:
    a: "hello"

and someone gave you,

  tag:
    a: "hello"
    b: "world"

Then the second case is substutable for tag.  Hmm.
How can this be codified?


| > > Perhaps a better alternative is to only provide
| > > substitutability within classes. Therefore all 
| > > classes will be first class YAML classes
| > > (round-tripping, substitutable, etc), but regular
| > > data will never get upgraded into a class. That way, 
| > > maps are just maps like before. Hmm. I really like that.
| > 
| > I'm not certain exactly what you mean...

I was thinking that the asScalar function is 
not a method... and thus completely separate
from both the native types and classes.  I hadn't
actually put serious mental energy into thinking
how it would work for a given class system.  If
you *have* a YAML class... then I can see where
asScalar() could be a nice method.

| I think for once, Oren and I are on the same side of the
| fence, and Clark might be on the other.

You will find my perspective leans towards
the iterator (parser) interface... which is 
quite a bit different from the native, 
internal representation.

I think we are all on the same side of the fence....

Best,

Clark

Clark C . Evans [mailto:cce@...] wrote:
> | > That's why I don't like Clark's idea of a list having a
> | > default value of the first element; it forces one to always
> | > pay this price, when for most lists it isn't required. I also
> | > don't think it is useful to evolve a scalar into a list; 
> | > that's not part of the "color" idiom, anyway.
> 
> I think it may be useful for times when something originally
> thought to be single value (such as an address) turns out
> to be multi-valued.  If you only allow "extensions" using
> the map construct, this forces a particular way to extend
> the scalar (use a map, and then a list) instead of just
> converting the scalar into a list.  I've had this case
> pop up before...

An example would be nice...

> | > Clark?
> 
> Well, if it seems problematic... I'll compromise here. ;)

So can we say its settled, subject to Brian confirming it is implementable?
Brian?

> | Also, it doesn't seem to make sense that a !Foo could morph
> | into a !Bar does it. Maybe if !Bar were a subclass...??? Do 
> | we support subclasses?
> 
> Hmm. Suppose you were expecting...
> 
>   tag:
>     a: "hello"
> 
> and someone gave you,
> 
>   tag:
>     a: "hello"
>     b: "world"
> 
> Then the second case is substutable for tag.  Hmm.
> How can this be codified?

Well, as long as you only add data members you are basically OK.

I wouldn't worry too much about it:

- If both classes aren't recognized, then as long as you just add members
you are OK;
- If both classes are recognized, then the language itself provides you with
compatibility. Presumably, the implementation of 'Foo' and 'Bar' will be
compatible if they are supposed to be. In C++, 'Bar' will derive from 'Foo'
and all will be well; in Perl, 'Bar' just has to provide all of 'Foo's
methods and members, etc.

So far, so good. The problems are:

- If 'Foo' is recognized and 'Bar' isn't, a twisted and heavyweight
mechanism may be able to help you somewhat;
- If 'Bar' is recognized and 'Foo' isn't, no force in the world can help
you.

I say the 80/20 solution here is just ignore these last two cases.

Have fun,

    Oren Ben-Kiki

On Tue, Jun 05, 2001 at 03:21:51PM +0200, Oren Ben-Kiki wrote:
| > I think it may be useful for times when something originally
| > thought to be single value (such as an address) turns out
| > to be multi-valued. 
|
| An example would be nice...

Examples:

  a) A phone number
  b) A contact person
  c) An address
  
In many of these cases, a business analyst will often
consider only one number, person, or address to be
what is needed.  But later on, it is discovered that
there are cases where more than one of the above
would be useful.  The contact person being one example
that has bit me before in a customer support database
that I was working on.   If you only allow a scalar to
become a map, then this will force an additional
(and unnecessary) intermediate map, as well as a 
duplication of the first value...

  contact: Jamie Hedward

Becomes...

  contact: %
      =: Jamie Hedward
      list: @
         Jamie Hedward
         Jenny Millsap

instead of...

  contact: @
     Jamie Hedward
     Jenny Millsap


| > Well, if it seems problematic... I'll compromise here. ;)
| 
| So can we say its settled, subject to Brian confirming
| it is implementable?

I'm willing to compromise here since this isn't a 
"common" thing to do... extending via map is far
more common (as maps are more common than lists).

Did you see my post on "asScalar" being an *external* 
function and not a member function?  In this case, it 
doesn't matter if the value is a map, scalar, or value.  
Also bindings my differ from language to language.  As 
long as the iterator interface supports the "asScalar", 
I really don't care if the "native representation" 
supports this construct (although it would be nice).

| > Hmm. Suppose you were expecting...
| > 
| >   tag:
| >     a: "hello"
| > 
| > and someone gave you,
| > 
| >   tag:
| >     a: "hello"
| >     b: "world"
| 
| Well, as long as you only add data members you are basically OK.

I like your breakdown, nice.

| I say the 80/20 solution here is just ignore these last two cases.

Good deal.

Clark


P.S.  Thank you for taking on the spec updates!

Brian Ingerson wrote:
> I've been thinking about a good format for 'yar', the YAML ARchive
> format that will be a readable and editable substitute for tar or zip.

Our trustworthy use case. It is a good app to demonstrate YAML, and is
useful besides. Thanks!

> My basic format idea is to have the first document in the yar stream be
> a directory containing all the necessary file system info. The directory
> entries will be an index pointing to other documents in the stream. Each
> following document will be the contents of a file. Plain text files can
> be stored as is. Binary docs can be encoded in base64 or something else.

You seem to assume that the YAR writer has random access to the data...
which seems reasonable. By placing the directory at the start you force the
writer to do a two-pass (at least) on the data, at the very least building a
full TOC in memory or re-accessing it when emitting the YAR file.

It does allow a reader application to quickly show you a TOC and given a
byte/character "start at offset" field, may allow very quick random access
for extracting one particular file.

However there may be a problem if the file system changes between the two
passes of the writer program (if a file size changes, if a file is
added/removed/renamed, etc.). You could work around it by emitting the TOC
to one file and the content to another in a single pass, then concatenating
both files into a single stream... But this prevents one from writing "yar
--create | whatever". Of course you could just treat changes between passes
as a warning/error... Thoughts?

What WinZip does is place the directory at the end of the stream. This
allows the writer to do a single-pass on the data, so there are no
consistency problems. But it prevents one from writing "whatever | yar
--extract". It still allows quick access to files if there's random access
to the YAR file, however (finding a trailing TOC is harder but still
possible).

The only true single-pass solution would be for YAR to directly reflect the
directory structure of the data. Something like:

!yar/dir /etc :
 perm: rw-r--r--
 files:
  !yar/file passwd :
   ...
  !yar/dir rc.d :
   ...

This means that there would be no quick way to extract a TOC out of a file,
however. Sigh. A tradeoff... what's more important, "yar --create | yar
--extract" or a very fast GUI-based YAR selective extraction utility? Is
"yar --create | whatever" less important than "whatever | yar --extract"?

BTW, you could make a YAR stream always consist of two consequtive
documents. The first would be a tree as above. The second (optional?) would
be a TOC. This would allow both single-pass read/write and the ability to do
TOC operations. The cost is replication of data, and added complexity. But
it may be worth it...

> [indenting top-level leaf values]
> This is a powerful notion. If a document conatins '^---', we no longer
> need to switch to a new separator. We can simply indent the top level
> scalar! This is good because the stream might already be started by the
> time we realize that there is a conflict.

Hmmm. Does this mean we can make '---' be THE separator, period, instead of
auto-detecting one? Clark?

> I'd like to know if anyone has ideas on yar in general.

I think we should take a look at the design of shar, tar and cpio before we
commit to the details... I remember reading a good document about the design
of cpio but I've no idea where to find it (it was years and years ago). A
quick search didn't show up anything useful. Also I know many people have
notions about what tar and cpio do wrong... We should take the time to
become familiar with this and "do better", just like we did with YAML vs.
XML :-)

I'll do some more digging for for relevant stuff the minute I finish (sigh)
updating the spec to cover option (E) and fix the productions bugs (yes,
there are some. Sorry about that).

Off the top of my head I think we should have (optional?) CRCs of some sort;
allow specifying a mime type per file (optional, I guess); in fact we may
want to use an "ignore unknown properties" policy to allow round-tripping
features within a platform with interoperability outside it (I think this is
a basic YAML idiom). A nice test case for this would be handling of Mac
resources so they round-trip on Macs and would look like a pair of files
elsewhere. I'd also consider round-tripping BeOS files as a gedanken use
case (BeOS being dead in the water means it probably won't ever become a
real one).

The interaction with compression is also interesting. Applying compression
to the YAR file as a whole removes many readability advantages of the
format... It seems if we compress each file content we may get most of the
size advantage while still having a readable file... except when there are
many small files... and when using the "tree" approach the indentation would
be a horrible waste... Hmmm.

We could just allow for it *and* allow an optional "transfer encoding"
property for each file anyway, so it would be the YAR writer choice which to
use. It seems best...

> Perhaps this
> discussion should be taken to yaml-dev or yaml-projects. Have thes
> (any) new mailing lists been started yet?

I'm not aware of any, and I think that until YAML is in a release candidate
state we can discuss such things here...

Have fun,

    Oren Ben-Kiki