RE: [Yaml-core] Canonical YAML?

[Robin G. <greenrd@pr...>]

On Fri, 17 Oct 2003, Oren Ben-Kiki wrote:
> Hmmm. While I realize why you'd want this to be weak - for your use
> cases - it is _so_ tempting to also define a strong canonical format!
> And so _easy_, too:
>
> The "strong" canonical format adds just two tiny rules:
>
> - No empty tags.
> - Use the canonical string representation of each scalar.
>
> See how easy it is? :-)

Good point! I like it!

> If you want, I'll send you the yod2html tool I'm using to convert the
> YAML sources of the spec to HTML. It would allow us to have a consistent
> look-and-feel for YAML specs.

OK, do so! Consistency is good. Or, I should say, needless inconsistency
is bad.

-- 
Robin

RE: [Yaml-core] Canonical YAML?

[Oren Ben-Kiki <oren@be...>]

Sean O'Dell wrote:
> Whatever it takes to solve the pending issues, keep in mind 
> the simple user 
> above all else.  Don't make it hard to learn, hard to 
> implement and don't 
> take away from its human-readability.  Of all the problems 
> you folks have to 
> solve in further refining YAML, I think keeping it easy to 
> learn, implement, 
> read and hand-generate/edit is of the utmost importance.  I 
> feel that if YAML 
> loses those features, there's very little reason to use it over XML.

We hear you :-)

Don't worry too much about the info model discussions having any effect
on the features you describe. As Clark said, we haven't changed the
syntax in about a year now (ditching formats doesn't really count,
because nobody used them anyway). If one of us even thinks of doing
something like that, the other two beat him back to his senses :-)

We've survived yet another flurry of info model issues, but for the
simple user that's mostly froth. Nothing has "really" changed. The
reason we insist on doing it is to make sure we don't leave hidden traps
that would bite the simple user later down the road, when he tries
anything more complicated. XML is full of those, so we might be a bit
over sensitive here... But I think/hope we are "real close" to
finalizing the spec anyway.

Have fun,

	Oren Ben-Kiki

RE: [Yaml-core] Canonical YAML?

[Oren Ben-Kiki <oren@be...>]

Robin G. wrote:
> So, logically, my criteria for Canonical YAML would be:
> 
> 1. Every legal Canonical YAML stream is a legal YAML stream.
>
> 2. The spec should in effect define an idempotent function f 
> from YAML to Canonical YAML

Obviously.

> 3. f(x)=f(y) iff x and y are weakly equivalent streams, as 
> defined above.

Hmmm. While I realize why you'd want this to be weak - for your use
cases - it is _so_ tempting to also define a strong canonical format!
And so _easy_, too:

Your "weak" format is defined as follows:

- Always use double quoted scalars. Quote all line breaks; otherwise,
just what you have to.
- Always specify a tag (use a standalone "!" if it is empty).
- Always use flow collections.
- Always specify a '--- %YAML:1.0' header at the start (this follows
from having to tag the top level node anyway).
- Specify anchor first, tag second (arbitrary choice, but one needs to
be made).
- Anchor all nodes (but see below).
- Anchors would be &0, &1, &2, etc. according to the order they are
encountered in the text.
- Order keys alphabetically (using "dumb" sort by Unicode character
value, nothing fancy).
- Order directives alphabetically (as above. I know, very theoretical).
- There's a subtle point about two ways of quoting characters in tags -
%XX and \XXX; use \XXX wherever possible as it is more general.
- One '\n' newline at the end of each document.
- No '...'.
- No comments.
- Minimal number of separation spaces (none at all where you can get
away with it).

The "strong" canonical format adds just two tiny rules:

- No empty tags.
- Use the canonical string representation of each scalar.

See how easy it is? :-)

> As a bonus, Canonical YAML might also be useful to create 
> ultra-lightweight parsers, for example for embedded systems.

Hmmm. Possibly. But subsetting YAML that strongly isn't really YAML
anymore. The only benefit you'll get is that the input for your embedded
device will also be parsable by YAML tools. Then again, if you wanted to
_look_ at it, you'd post-process it (using YAML-pretty-printer)
anyway... And you'd need to pre-process every external YAML document
before feeding it to your system as well... So it doesn't seem like much
of a benefit after all. You might be better off using something like
YAML byte codes (if/when we get around to formally defining a set of
those).

Note that each document (or entity or whatever we call it) would be one
line. A _big_ line. I'd be wary of using readline() or something like
that. Anchoring every node means that the loader would have to remember
_all_ the previous nodes, so this isn't a streaming format. Perhaps it
would be better to require that only aliased nodes are anchored?

All that said, a parser for the above _would_ be simpler.

> Any objections to this idea? If not, how should we proceed? 
> I'll just whip up a draft spec, or what?

If someone wanted to whip up a small specification along the above
lines, I don't see any problem with it. I don't see a burning need for
it either :-) It would have to mesh with the revised information model
section, though, so perhaps you should wait a bit with that until Clark
releases it.

If you want, I'll send you the yod2html tool I'm using to convert the
YAML sources of the spec to HTML. It would allow us to have a consistent
look-and-feel for YAML specs.

Have fun,

	Oren Ben-Kiki

RE: [Yaml-core] yaml ain't markup! a stream has entities (not documents)

[Oren Ben-Kiki <oren@be...>]

Hi "boots", you a Snark fan by any chance? :-)

boots wrote:
> First, I agree that getting the "document" out of YAML is a 
> great idea. The fact that YAML is a serialization is probably 
> the most important thing that should be emphasised, IMHO. I 
> also think that "entity" is a great choice to replace 
> "document" and that considerations of it relating to XML's 
> use are unwarranted. We have "entities" in data modelling, 
> object modelling and various other pursuits that are 
> unrelated to XML. YAML applying entity seems entirely fair 
> and unambiguous--and even obvious (now that you say it, of course).

+3/4. Losing the word "document" is a good idea. I'm a bit uncofortable
with the word "entity" though.

I'm concerned about standard data modeling terminology - things like
Entity-Relationship-Diagrams. An "entity" in data modeling is more like
a YAML node. The concept of a "document" is "a closed set of entities" -
closed in the sense that there are no references to entities outside the
set. Perhaps the word I'm looking for is "closure", but it doesn't feel
quite right and it is a horrible replacement to "document" anyway.

Some ideas off the top of my head:

Chunk/blob - too generic. Object - tempting... But suffers from the same
problem that entity has plus the baggage of the OO movement.

YAML data (== document)? This would make the phrases "YAML data" and
"YAML data stream" mean, well, exactly what they usually mean... But
we'd be forbidding the use of the term "YAML data" to describe a single
node. Besides, how would you say a stream contains multiple "data-s"?
Data is already plural.

Hmmm. Perhaps "entity" is the best we can do... I suppose people are
used to having the word "entity" be given a special meaning in different
contexts. For example, XML got away with a worse abuse of the word :-)

> I was chatting on a PHP related IRC the other night and was 
> discussing the use of YAML for a project and was asked to 
> explain why it should be used over XML. Unfortunately, it 
> became a debate of syntax and how one would do something as 
> "complicated" as soap with YAML. Perhaps a section in the FAQ 
> that delineates when/where YAML is a superior choice? I 
> realize that this group is probably biased on that point, but 
> a few good selling points would go a long way with certain 
> decision makers :)

The basic argument is that YAML is a better match for your application's
data model. Having a more readable syntax is an accidental bonus. Really
- the original motivation for YAML (and before it, SML) was not to
improve on XML syntax, but improve on its data model. It turned out to
be next to impossible to do this within the confines of the XML syntax
(so SML fizzled, after about two years of work on the problem). Clark
bit the bullet and started defining a new syntax instead. Of necessity,
not by choice; if we could have done it using XML syntax, we'd have a
much easier time gaining acceptance.

That said, PHP is a bit of a special case. Its basic data model is a map
with ordered keys (a YAML !omap). This doesn't map _that_ cleanly to
YAML. Granted most of the time the key order is not significant and a
normal YAML map suffices, but this means the application needs to
specify which is which.

I'd still claim that the impedance mismatch between PHP's native data
types and YAML is much lower than the mismatch between PHP and XML -
that is, the amount of work that needs to be done to serialize your data
to YAML should be much smaller. And, of course, if you handle the data
types of other languages as well - SQL, Python, Perl, Java, even C++ -
then there's no question that YAML is a much better match.

As for "complicated" SOAP stuff - YAML is, in general, at the same level
of abstraction as SOAP. So you don't need to worry about both YAML _and_
some additional complicated stuff; just plain YAML should do everything
you need SOAP for in XML land. Which goes to the previous point.

> One last thing: it appears that you good folk have totally 
> taken the ball and run with it--and are still running! I know 
> that you all want to make sure that you get it right and that 
> the spec is properly extensible. Still, is there a short-term 
> timeline to release a 1.0 spec? Personally, I have not been 
> able to justify to my clients/colleagues the use of a 
> beta/dev spec. The irony is that to me, the current revision 
> is far, far better than the first XML release. While all the 
> recent work on the models and emitters and what-have-you has 
> been fascinating, I lament the fact that those issues have 
> delayed the spec being released.

Well, since YAML was from day one more about the model than the syntax,
it was sort of inevitable. We're happy to have come up with a nice
syntax people feel is worthwhile on its own, but let's face it - the
last thing people need is an alternative _syntax_ for XML. We'd be
doomed before we even started.

That said, it _is_ about time we wrapped it up. Which Clark is now
doing. Hopefully we'll get a release candidate (at the very least) "real
soon now".

Have fun,

	Oren Ben-Kiki

Re: [Yaml-core] Canonical YAML?

[Sean O'Dell <sean@ce...>]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been following several of the discussions regarding extending basic YA=
ML=20
to handle native type conversions, namespaces and other various issues, and=
 I=20
would like to just sort of throw some advice out there.

The number one reason I started using YAML over XML was, aside from the=20
serialization aspect was its ease-of-use.  It was easy to learn and it's ve=
ry=20
easy to generate and edit YAML.  That it's also perfectly formatted for=20
parsing is genious.

Whatever it takes to solve the pending issues, keep in mind the simple user=
=20
above all else.  Don't make it hard to learn, hard to implement and don't=20
take away from its human-readability.  Of all the problems you folks have t=
o=20
solve in further refining YAML, I think keeping it easy to learn, implement=
,=20
read and hand-generate/edit is of the utmost importance.  I feel that if YA=
ML=20
loses those features, there's very little reason to use it over XML.

Thanks for hearing me out on this,

	Sean O'Dell
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/kDvs90m+cSzpQIERAuyQAJ9/COd8433YPo9l0Ta3abGYFM+T8gCfZVao
pZVpnX2O6XHXLC14XG1mJgo=3D
=3D2rIj
=2D----END PGP SIGNATURE-----

Re: [Yaml-core] yaml ain't markup! a stream has entities (not documents)

[Clark C. Evans <cce@cl...>]

On Fri, Oct 17, 2003 at 10:34:16AM -0700, boots wrote:
| One last thing: it appears that you good folk have totally taken the
| ball and run with it--and are still running! I know that you all want
| to make sure that you get it right and that the spec is properly
| extensible. Still, is there a short-term timeline to release a 1.0 spec?

My goal is to have it frozen by the end of the month.  All changes are
now primarly (after the recent relaxation of fragments) are purely
wording oriented.. in the model.   Really, I don't think we have had
a change in the last 10-12 months which has broken backward 
compatibility...   we have *added* things and relaxed previous
constraints (like the recently reported perl bug).

| Personally, I have not been able to justify to my
| clients/colleagues the use of a beta/dev spec. The irony is that to me,
| the current revision is far, far better than the first XML release.
| While all the recent work on the models and emitters and what-have-you
| has been fascinating, I lament the fact that those issues have delayed
| the spec being released.

Ok.  Point taken. 

Best,

Clark

Re: [Yaml-core] Canonical YAML?

[Clark C. Evans <cce@cl...>]

Ya.  This is a supreme idea Robin.   It can especially be used to drive
home the rationale for the information model.  A few more comments.

On Fri, Oct 17, 2003 at 05:18:39PM +0000, Clark C. Evans wrote:
| On Fri, Oct 17, 2003 at 12:50:40PM -0400, Robin G. wrote:
| | I'm thinking of writing a YAML parser (or two) and I'm interested in
| | defining a restricted, canonical subset of YAML with which to write the
| | output side of test cases with - so that the validation aspect of spec
| | compliance (as opposed to the "correct native representation" aspect) can
| | be tested with a bank of language-independent testcases.
| 
| Well, may I suggest a few items:
| 
|   1. You use double quoted for all strings, as this is the only
|      scalar style that can encode all content.
| 
|   2. All !tag thingies are always provided, even if it is the
|      empty tag, ! , which specifies implict typing
| 
|   3. Since double quoted is a 'flow', I also suggest using only
|      flow mappings and sequences.
| 
|   4. All mapping keys are sorted according to unicode code point.

    5. All nodes are given an anchor, starting with 0 and
       increasing numerically without a leading zero.

    6. No comments, or other styles, unnecessary spaces, or line 
       breaks are used  (all line breaks are escaped in the double
       quoted scalars).

    7. It is assumed that all nodes with a !tag are in canonical
       form, that is !int 10 and not !int 0xA

more below...

| | So, logically, my criteria for Canonical YAML would be:
| | 
| | 1. Every legal Canonical YAML stream is a legal YAML stream.
| | 
| | 2. The spec should in effect define an idempotent function f from YAML to
| | Canonical YAML
| | 
| | 3. f(x)=f(y) iff x and y are weakly equivalent streams, as defined above.
| 
| Exelent.

Yes, this makes the 'weak vs strong' graph model very concrete.  
This is *such* a good idea, it should drive home the point.

| | As a bonus, Canonical YAML might also be useful to create
| | ultra-lightweight parsers, for example for embedded systems.
| 
| By using only double quoted and flow collections, a minimal parser
| would be freed of having to keep track of whitespace, indenting,
| styles, and most of the complexity of YAML.

Heck, this would make it very easy to write language loaders
as the resulting BNF would be much simpler.  Each entity is
on its own line, thus working well with readline().  Each
port could call a "C" library to make a string canonical, 
or it could run a standalone yamlcanon program using a pipe.

Best,

Clark

Re: [Yaml-core] yaml ain't markup! a stream has entities (not documents)

[boots <jayboots@ya...>]

--- "Clark C. Evans" <cce@...> wrote:
> In recent press, some people are still categorizing us as a 
> 'document markup' language.  This kinda bugs me beacuse we are 
> definately not a document markup language, we do not have 
> mixed content nor do we want mixed content.  We are a data 
> serialization language.

Hi. I hope I am not intruding, but I've been in lurk mode and enjoying
everyone's discussions. Thanks! As an outsider, I have a couple of
comments that may be appropriate.

First, I agree that getting the "document" out of YAML is a great idea.
The fact that YAML is a serialization is probably the most important
thing that should be emphasised, IMHO. I also think that "entity" is a
great choice to replace "document" and that considerations of it
relating to XML's use are unwarranted. We have "entities" in data
modelling, object modelling and various other pursuits that are
unrelated to XML. YAML applying entity seems entirely fair and
unambiguous--and even obvious (now that you say it, of course).

I was chatting on a PHP related IRC the other night and was discussing
the use of YAML for a project and was asked to explain why it should be
used over XML. Unfortunately, it became a debate of syntax and how one
would do something as "complicated" as soap with YAML. Perhaps a
section in the FAQ that delineates when/where YAML is a superior
choice? I realize that this group is probably biased on that point, but
a few good selling points would go a long way with certain decision
makers :)

One last thing: it appears that you good folk have totally taken the
ball and run with it--and are still running! I know that you all want
to make sure that you get it right and that the spec is properly
extensible. Still, is there a short-term timeline to release a 1.0
spec? Personally, I have not been able to justify to my
clients/colleagues the use of a beta/dev spec. The irony is that to me,
the current revision is far, far better than the first XML release.
While all the recent work on the models and emitters and what-have-you
has been fascinating, I lament the fact that those issues have delayed
the spec being released.

Pip, pip and all that,
xo boots

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

Re: [Yaml-core] Canonical YAML?

[Clark C. Evans <cce@cl...>]

On Fri, Oct 17, 2003 at 12:50:40PM -0400, Robin G. wrote:
| I'm thinking of writing a YAML parser (or two) and I'm interested in
| defining a restricted, canonical subset of YAML with which to write the
| output side of test cases with - so that the validation aspect of spec
| compliance (as opposed to the "correct native representation" aspect) can
| be tested with a bank of language-independent testcases.

Well, may I suggest a few items:

  1. You use double quoted for all strings, as this is the only
     scalar style that can encode all content.

  2. All !tag thingies are always provided, even if it is the
     empty tag, ! , which specifies implict typing

  3. Since double quoted is a 'flow', I also suggest using only
     flow mappings and sequences.

  4. All mapping keys are sorted according to unicode code point.

|  "Although two XML documents are equivalent (aside from limitations given
| in this section) if their canonical forms are identical, it is not a goal
| of this work to establish a method such that two XML documents are
| equivalent if and only if their canonical forms are identical."

Absolutely.  The problems with canonical forms in XML (specifically 
namespaces, *ick*), are one of the many sour grapes that led Oren
and I to break off from the XML community. 

| So, logically, my criteria for Canonical YAML would be:
| 
| 1. Every legal Canonical YAML stream is a legal YAML stream.
| 
| 2. The spec should in effect define an idempotent function f from YAML to
| Canonical YAML
| 
| 3. f(x)=f(y) iff x and y are weakly equivalent streams, as defined above.

Exelent.

| As a bonus, Canonical YAML might also be useful to create
| ultra-lightweight parsers, for example for embedded systems.

By using only double quoted and flow collections, a minimal parser
would be freed of having to keep track of whitespace, indenting,
styles, and most of the complexity of YAML.

| Any objections to this idea? If not, how should we proceed? I'll just
| whip up a draft spec, or what?

No, this is great.  If we can come to a quick agreement, perhaps it
could go in the spec itself.

Best,

Clark

[Yaml-core] yaml ain't markup! a stream has entities (not documents)

[Clark C. Evans <cce@cl...>]

In recent press, some people are still categorizing us as a 
'document markup' language.  This kinda bugs me beacuse we are 
definately not a document markup language, we do not have 
mixed content nor do we want mixed content.  We are a data 
serialization language.

To this end, I was thinking of changing all references to 
'document' within our specification to 'entity'.  That is, 
a YAML stream contains one or more entities.  I like the word 
entity beacuse its definition 'something which exists as a 
discrete unit' better encapsulates each item in a YAML 
stream -- they are distinct items that stand-alone.   I think 
calling them a document may cause people to get the wrong 
idea about what we mean by this, in particular, people often
use document as a synonym to file.

I was also contemplating a tweak to the first paragraph of
the "Relation to XML" section:

    Newcomers to YAML often search for its correlation to the eXtensible
    Markup Language (XML).  Although the two languages may actually
    compete in several applications, there is no direct correlation 
    between them.  While XML may be used for data serialization, its 
    roots in the structured document community complicate this use
    with mixed content and the element vs attribute distinction.
    YAML, by contrast, is a data serialization language and by design 
    is not a document markup language.  

An obvious objection to 'entity' is XML's use of them for escaping
and for sub-document inclusion.  But I hardly think this should 
disqualify us for reusing the name in our own context; clearly 
document brings with it a whole raft of semantics as well.

Thoughts?

Clark

[Yaml-core] Canonical YAML?

[Robin G. <greenrd@pr...>]

Hi

I'm thinking of writing a YAML parser (or two) and I'm interested in
defining a restricted, canonical subset of YAML with which to write the
output side of test cases with - so that the validation aspect of spec
compliance (as opposed to the "correct native representation" aspect) can
be tested with a bank of language-independent testcases.

Obviously, such a Canonical YAML spec could not be expected to
canonicalise datatypes it knows nothing about. The introduction to the
Canonical XML spec[1], however, draws a misleading conclusion from the
same problem in XML-land, IMO - it says (sec. 1.3):

 "Although two XML documents are equivalent (aside from limitations given
in this section) if their canonical forms are identical, it is not a goal
of this work to establish a method such that two XML documents are
equivalent if and only if their canonical forms are identical."

But the analagous goal for YAML should be achievable, I think, if we use a
weak notion of equivalency such that nodes with tags not defined in the
spec are compared as if they were strings.

So, logically, my criteria for Canonical YAML would be:

1. Every legal Canonical YAML stream is a legal YAML stream.

2. The spec should in effect define an idempotent function f from YAML to
Canonical YAML

3. f(x)=f(y) iff x and y are weakly equivalent streams, as defined above.

As a bonus, Canonical YAML might also be useful to create
ultra-lightweight parsers, for example for embedded systems.

Any objections to this idea? If not, how should we proceed? I'll just
whip up a draft spec, or what?

-- 
Robin

[1] http://www.w3.org/TR/2001/REC-xml-c14n-20010315

P.S. What's up with the yaml.org nameserver?

[Yaml-core] bugs with scalars on next line

[Jonathan Swartz <swartz@po...>]

Using YAML 0.35, this program:

   #!/usr/bin/perl -w
   use YAML;

   my $item = YAML::Load(<<EOF);
   plain:
     This is on the next line.
   EOF

prints

   --- !perl/YAML::Error
   code: YAML_PARSE_ERR_NO_SEPARATOR
   msg: Expected separator '---'
   line: 2
   document: 2
   ...

If I change the space in front of "This is on the next line" to a tab, I get

    Died at /home/mhq/perl/lib/perl5/site_perl/5.6.1/YAML.pm line 1435.

Sorry if these are known bugs - is there a known bugs page?

Thanks
Jon

Re: [Yaml-core] "unknown types" / "weak equality" are out-of-scope

[Brian Ingerson <ingy@tt...>]

On 17/10/03 01:55 +0200, Oren Ben-Kiki wrote:
> Clark C. Evans wrote:
> > Right.   So, a YAML document can be valid, indeterminant, invalid.
> > If two unknown scalars are used within a mapping, and they 
> > have the same tag and value, then the mapping (and thus the document)
> > is invalid.   If a mapping contains two unknown scalars of the
> > same tag but different values, then the mapping (and thus the
> > document) is indeterminant.   Therefore, the possibility of unknown 
> > typing implies that a given YAML document in the context of a 
> > given YAML processor may have three states; if the state is 
> > invalid or valid, then this designation holds regardless of 
> > the processor used.
> 
> Right. Note that this depends on the weak model, since in effect you are
> performing weak equality tests on the keys.
> 
> > Ok.  So if YPATH uses the 'value' of a scalar, and the scalar 
> > is not known, then a warning should be given.
> 
> +1.
> 
> > | If on the other hand you accept that YSLT _by itself_ makes no 
> > | guarantees about the validity of input/output documents
> > 
> > This is the best approach... probably.
> 
> :-)
> 
> > | IMO the right solution is to distinguish between 
> > | schema-aware/specific 
> > | applications and schema-unaware/generic tools.
> > 
> > I'm not sure what you mean by schema aware/unaware.
> 
> I don't mean YSCHEMA. I mean "aware of all the types, and probably loads
> them to native types". That is, an application loading a YAML document
> to an Outlook calendar would be "schema aware" while YAML-Pretty-Print
> would be schema unaware.
> 
> > I see a 
> > schema as a transformation that changes !str typing into 
> > !whatever typing.
> 
> First, I try never to use the words "schema" and "transformation" in the
> same sentence. To me, a schema (YSCHEMA) is a Boolean predicate that may
> be applied to a document (valid/invalid). No transformations involved.
> 
> Of course, of necessity every loader computes such a Boolean
> (loadable/unloadable); hence every application that loads YAML to a
> native type (strong model, _not_ generic nodes) is by definition "schema
> aware".
> 
> > | The former always work at the strong model and pose no problem.
> > | The latter must fall back to the weak model when unknown tags are 
> > | encountered, and can only guarantee the validity of their output 
> > | given that their input was valid in the first place. 
> > 
> > I assume you mean this to be granular, ie, a loader can 
> > create both native nodes when it knows the !tag, and 'generic scalar' 
> > nodes when it doesn't.   As such, I see one model, with each
> > tag in the system having a flag if it is known or unknown.  I 
> > am not comfortable with two models (a weak vs a strong) as 
> > the problem is more complicated than this and for the model 
> > section to be valuable it must describe the interaction 
> > between nodes of a known and unknown types.
> 
> Not so. The weak model is exactly this mixture of known and unknown
> nodes you refer to, while in the strong model all nodes are known. At
> least that was the decision we reached after trashing this issue for an
> inch of its life :-)
> 
> I feel it is essential to define the pure/core/known "strong" model as
> this is the motivation for everything else. Defining it allows us to
> specify the true core of YAML, what applications actually need to
> consider when YAML-ing their data - how to cast their data in the form
> of map/seq/scalar, and nothing else.
> 
> We both agree that we also need a model where some tags are "known" and
> some are "unknown". That's the model you refer to when you say you see
> "one model". Its name is the "weak" model to distinguish it from the
> pure/core/known "strong" model. I feel it would be a mistake to neglect
> the "strong" model because that would mix together issues of weak
> equality and handling unknown types that are better separated from the
> core issue of map/seq/scalar modeling of data.
> 
> > Right.  So, nodes with an a tag which is unknown to a given 
> > processor will have a set of warnings to be used when the 
> > node is used in an equality setting (recursively defined) or 
> > when looking 'into' the
> > string value during a process.   This way literal copying does not
> > issue a warning, but just about every other use of scalars 
> > with an unknown tag does.
> 
> +1. This is all weak model stuff.
> 
> > On Thu, Oct 16, 2003 at 09:49:08AM -0700, Brian Ingerson wrote:
> > | Most everything boils down to:
> > |   * ALWAYS CONSIDER: Schema Blind vs Schema Aware
> > |   * Generic YAML processors should be as forgiving as possible
> > |   * Specific applications should be as rigid as they need to be
> 
> +1.
> 
> > Yes; although I'd also say that the default behavior for a 
> > YAML processor should be to notify the user of a potential 
> > problem due to an unknown type; and to let this notification 
> > be turned off.
> 
> +1.
> 
> > | I asserted to Clark that at the YPATH "level" you always 
> > | know the tag 
> > | by definition.
> > 
> > And this assertion is wrong.  As you may use YPath to select a 
> > node from a graph where the node selected has an "unknown tag".
> 
> +1.

There is a complete skew in semantics going on here. Specicically the
semantics of "know".

If a node is tagged '!boogle', you *know* what its tag is: 'boogle'.
Your loader may not have a specific native type adapter for it. In that
case, the loader has two choices:
  * reject the node
  * graph the node using a ynode

My assertions are:

* YPATH can be thought to only operate an a graph of pure ynodes.
* A ynode always has a tag
* The tag (and string value) should be considered canonical/strong to
  YPATH

If this does not produce the desired results, you need to rework your
loader schema.

Cheers, Brian
> 
> Have fun,
> 
> 	Oren Ben-Kiki
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core

[Yaml-core] Weak Equality

[Oren Ben-Kiki <oren@be...>]

Clark wrote:
> I'm convinced that we don't need weak equality, and that 
> untyped nodes should be 'selectable', aka round-trippable but 
> that any generic YAML tools should not use a untyped node's 
> string value.

+1 on that. But. This doesn't mean that we don't need the weak _model_.
The weak model is where you get a mixture of known-tag and unknown-tag
nodes, and describe what can or can not be done to them (e.g., select an
unknown-tag node, but not compare its value).

> I think it is sufficient to define "nodes in the graph model 
> which are untyped".  ie, to give each node in the graph model 
> a flag "typed or untyped"...

+1. This is the "weak" model. But I think it is also worthwhile to have
a core "strong" model where all tags are known. This alows a clean
separation of concerns. Data modeling issues (map/seq/str, identity vs.
equality) are covered in the "strong" model, and are all you need to
think about when modeling some data as YAML. Known vs. unknown,
indeterminate validity, and all that stuff is covered in the "weak"
model and you really only have to think about it if you are writing a
generic YAML tool (as opposed to an application that serializes/loads
data into/from YAML). Mixing both sets of issues together would be wrong
IMVHO.

I liked the outline you have presented for the info model section:

> My "outline" for the model section is now:
> 
>    4.1  Overview
>         - motivating why we do models (as a shared vocabulary)
>         - gives a pretty diagram (Brian's will work)
>         - very quickly outline of the models  (syntax -> 
> serial -> graph)
>         - discusses how there are actually two graph models: 
> strong and weak

Which are used by different people (application writers vs. generic tool
writers).

>         - perhaps gives a "stacked" diagram showing how each model
>           builds onto the previous model, making it more "syntaxish"
>
>    4.2  Challenges
>         - human presentation
>         - incremental delivery
>         - native typing
>         - getting information to flow nicely between stages
> 
>    4.3  Strong (Native) Graph

The focus is: the YAML information model for application writers.

>         - an idealized model where all types are known by the loader
>         - quick overview of what this models: native objects or
>           a generic model using canonical forms, or cave drawings
>         - defines a scalar node as an opaque value with a 
> mandatory tag
>         - defines each scalar tag as having a canonical form
>         - defines strong equality of scalars based on canonical form
>         - defines strong equality of collections recursively
>         - explain that this model represents how YAML views native
>           systems or canonical representations of YAML information.

Also:     - defines identity vs. equality;
          - describes the map/seq/scalar concepts for modeling data
          - for extra credit, mention the !value '=' tag and schema
evolution

>    4.3  Weak (Generic) Graph

The focus is: YAML information model for generic tool writers.

>         - a more realistic model where one or more types are not
>           known by the loader; so we have weak equality for some
>           subset of the nodes.

But we still have some known-tag nodes - this is the "mixed" graph you
are looking for, with all the details of the interactions between
known-tag and unknown-tag nodes (I'd rather not call them "typed" and
"untyped").

>         - quick overview discussing how not every native binding
>           will know about every scalar tag, and the problems caused

With a mention this mostly applies to generic tools and that
applications typically need not worry about it...

>         - defines a scalar node as a string value with an optional tag
>         - defines weak equality as yes, no, unknown terinary operator

Change to yes/no/error (incomparable).

>         - specifies how YAML texts which comply with the weak 
> graph may
>           fail to load into the strong graph:  (a) keys which are
>           unknown in a mapping that are equivalent under strong
>           equality, (b) scalar values which have a specified tag, but
>           where the string provided does not match the tag's 
> expectations,
>           for example  --- !int X

And that therefore generic YAML tools are inherently limited in their
verification of unknown types (valid, invalid, unknown).

>         - explain how this model represents what would be exposed
>           via a random access (DOM-like) viewer API.
>
>    4.4  Serial (Tree) Model

The focus is: incremental processing or arbitrarily large YAML streams.

>         - quick overview how this model adds only those elements
>           necessary to break a graph (either weak or strong) into
>           a sequential pipe
>         - introduces alias/anchor
>         - introduces key order
>         - introduces scalar "chunking"
>         - mentions that this model specifies the "minimum"
>           requirements for a push/pull parser or emitter API.
> 
>    4.5  Text (Syntax) Model

The focus is: YAML for users, or what the system _will_ ignore even if
it is "in the file".

>         - quick overview how this model adds all of the YAML goodies
>           that makes human presentation nice
>         - add styles
>         - add comments
>         - add directive
>         - add details (new line endings, indentation, etc.)
>         - mentions that this model specifies the requirements for
>           a lexer API that preserves all aspects of a YAML text.

I think this works rather well. We have reached a deeper understanding
of the issues involved in the weak vs. strong distinction, and I agree
that "weak equality" should not be a part of YPATH; Then again, it is
difficult to define the notion of valid/invalid/unknown YAML document
without referring to something very similar to "weak equality". OK,
comparing two unknown-tag nodes is "error" instead "unknown", so we have
equal/different/error instead of equal/different/unknown. You still have
to give this sort of comparison a name when it is used in definitions -
you might as well call it "weak equality" :-)

Have fun,

	Oren Ben-Kiki