YAML Ain't Markup Language

On Mon, Nov 12, 2001 at 02:00:59PM +0200, Oren Ben-Kiki wrote:
| Clark C . Evans [mailto:cce@...] wrote:
| > scalar: Is undefined; although it must have a 
| >  serialization pair (Method,Characters)
| And an identity, of course.

Right.  

What was important, is that a map and sequence
have identities, not nodes.

| > where
| > Characters is a sequence of unicode characters.
| 
| Yes. A scalar is anything you want it to be, as long as it
| has just a single identity. Period.
| 
| > Anchor = Identity in the serialization format.
| > Method = Class in the serialization format.
| 
| Right.
| 
| > In Beta, there are two node types Branch and Leaf.
| > 
| >   Leaf   = identical to scalar
| >   Branch = identical to map
| > 
| > In Beta, a sequence is just a Class, it uses a Branch
| > node type where all of the scalar values are integers.
| > 
| > Note: Alpha and Beta are slightly different.  I think 
| >       Beta is a bit simpler; as it moves the distinction
| >       between "map" and "list" into the realm of Class.
| 
| What does Beta gain us in practice?

Tons.  It simplifies the API.  Rather than having 
two sets of methods for iterating over branches.  
The map and list API are damn near identical, except 
the map uses a scalar for a key, and the list uses
an integer.  

So... there is nothing saying that a list could not 
just be using "scalar integers" for it's key.  This 
simplifies the sequential API by eliminating all of 
those duplicate methods for managing lists...

Further, it turns Brian's "sparse list" into 
something that makes sense.  If you wanted to 
separate Map and Sequence in the API; then you
are forced to make "sparse list" a Map.  But,
it has "scalar integer" keys... they just happen
to be non-sequential.  No big deal.  With Alpha
its quirky, with Beta it makes sense.

In this way, "map", "list" and "sparse list" become 
various "base classes" over a unified branch construct.

...

Taking this further, imagine what we do when we
have a "path" based processing mechanism.

  /3

Clearly this should return "two" in the
document below...

---
z: one
3: two

But... in the sequence below, should it
return "c" or should it not match anything?

---
- a
- b
- c
- d

If you distinguish between a map and a list at the
information model level, this expression shouldn't
return "c", it must not match, since the top level
object is not a map.  You must instead create a 
mechanism for talking about list indexes....

/[3]

Which would match "c" but not "two".  I see this
this as a big hair ball.  Now... let's assume that
you agreed here.  How would you like to index
a "sparse list"?  Clearly you have to use the 
map syntax, and not the list syntax.  That's
going to be easy to explain....

This ball of mess has been making me pull my
hair out lately.... and model Beta solves all
of these nasty issues without the jitters.

Frankly, I'm close to tears it's so pretty.
If you don't believe me... start to do an
XSLT like processor against YAML.  The path
example here is just the tip of the iceburg.

This is *not* pie-in-the-sky acedemic shit.

Best,

Clark

On Mon, Nov 12, 2001 at 08:54:16AM -0500, Clark C . Evans wrote:
| | > In Beta, there are two node types Branch and Leaf.
| | > 
| | >   Leaf   = identical to scalar
| | >   Branch = identical to map
| | > 
| | > In Beta, a sequence is just a Class, it uses a Branch
| | > node type where all of the scalar values are integers.
| | > 
| | > Note: Alpha and Beta are slightly different.  I think 
| | >       Beta is a bit simpler; as it moves the distinction
| | >       between "map" and "list" into the realm of Class.
| | 
| | What does Beta gain us in practice?
| 
| Tons.  It simplifies the API.  Rather than having 
| two sets of methods for iterating over branches.  
| The map and list API are damn near identical, except 
| the map uses a scalar for a key, and the list uses
| an integer.  

I forgot one more item here.  There is a good chance
that it will make the productions simpler.  The 
difference between "map" and "list" becomes a purely
"syntax" distinction; much like a block scalar vs
a flow scalar (with one difference, the form gives
a different "default" class).

I haven't played around, but I think it may 
help to make the productions more clear.   Further,
if you then think of "list" and "map" as *styles*
modules (much like block vs flow *style*) we could
introduce other styles for expressing the same
sort of information... aka a tabular structure,
I was recently thinking that it would be way cool
if CSV was a subset of YAML.  That'd rock.  A 
CSV style would create a list of lists of scalars.

Best,

Clark

I agree with everything you say about using the same YPATH syntax and same
API for maps and sequences. I don't see that forces us to merge them at the
information model level.

As for considering map and list to be mere "styles" I'm against it. It
basically says "everything is a map". It isn't. Sequences have different
semantics then maps. For example, a sequence always has entry 4 if it has
entry 5. A map doesn't. A sequence can only have integer keys. A map can
have, well, in theory any sort of key whatsoever (e.g. in Perl 6 or in
Java).

If we introduce tables, they will be a different *kind* of container. They
have their own semantics. For example, a table should allow me to refer to
its 3rd column as a single entity (I'd love to see you do that using a map).
Otherwise a table is just a sequence of sequences and doesn't justify any
special node kind. It is a different beast, period.

Styles are appropriate when thinking of *text* scalars - different way to
represent *the same thing*, which is a sequence of Unicode characters. Style
is a bad word to apply to differing concepts like sequence and map (and
definitely a table).

Again, I fully agree with the API and YPATH syntax points you raise. I just
don't see what they have to do with the information model. The spec
explicitly says you can have whatever API you want as long as it preserve
the information model, and a unified API with an 'isSeq' method certainly
does.

I think we are splitting hairs, and Brian is tearing his in despair by now
(or probably even just skipping these posts altogether :-). Let's let it go
and focus on the last remaining issue - scalar styles (I think that's the
last one, right?).

Have fun,

    Oren Ben-Kiki

On Mon, Nov 12, 2001 at 04:30:14PM +0200, Oren Ben-Kiki wrote:
| I agree with everything you say about using the same YPATH syntax and same
| API for maps and sequences. I don't see that forces us to merge them at the
| information model level.

If you merge them at the API level, then I argue
that you are using "Beta" information model since
you are treating them both in a similar manner with 
a lower level abstraction, aka a Branch.

| As for considering map and list to mere "styles" I'm against it.

No.  I consider them to be a combination of both:
(a) a Syntax, and 
(b) a Class.

The "native" representation of the "list" and "map" class
should be up to the language at hand.  In a PostgreSQL
relational database, for instance, I would choose to 
treat maps and lists identically... as Relations.

As for the syntax, I see these as our two core 
"styles".  Each style implying a different class
if it is not given otherwise.  For example, take
Brian's sparse list...

--- !list
0: one
5: five

| It basically says "everything is a map". It isn't. 

No it doesn't.  It's a "branch" as distinct from a "scalar".
That the "branch" needs to be "typed" (aka given a Class)
for it to work well in Python, Perl, etc. is a separable 
issue.  In PostgreSQL it will be implemented with a table
given particular domain constraints.

| Sequences have different semantics then maps. For example, 
| a sequence always has entry 4 if it has entry 5. A map doesn't.
| A sequence can only have integer keys.

Right.  These are constraints of the "org.yaml.Map" and
"org.yaml.List" classes, the *default* classes for each
of the map and list syntax.

| Styles are appropriate when thinking of *text* scalars - different 
| way to represent *the same thing*, which is a sequence of Unicode 
| characters. Style is a bad word to apply to differing concepts 
| like sequence and map (and definitely a table).

Ok.  Think "syntax with default class" instead of style.
For example, how do these two differ?

--- !list
0: one
1: two
--- !list
- one
- two

If we are to merge the APIs, as you said above, they 
must have an *identical* information model.  Right?

| Again, I fully agree with the API and YPATH syntax points you raise.

Let's focus on YPATH.  Either we must have a different
YPATH to match maps and lists... or we must have the
same information model for both.  You can't have it
both ways... it would be inconsistent.

| I just don't see what they have to do with the information model. 
| The spec explicitly says you can have whatever API you want as long 
| as it preserve the information model, and a unified API with an
| 'isSeq' method certainly does.

It's got everything to do with the information model.

| I think we are splitting hairs, and Brian is tearing his in despair by now
| (or probably even just skipping these posts altogether :-). Let's let it go
| and focus on the last remaining issue - scalar styles (I think that's the
| last one, right?).

Brian, do you want to make a distinction between the 
example map and list above?  This is a important question.
Are they *different* ... or are they the *same*    If they 
are different, we must treat them differently in the API
and the language embedding; or we violate the information
model.  If they are the same, then our informaiton model
which makes this distinction must be fixed.

It's not a trivial issue.  The information model says 
what things are important. 

Best,

Clark

On Mon, Nov 12, 2001 at 04:30:14PM +0200, Oren Ben-Kiki wrote:
| I agree with everything you say about using the same YPATH syntax and same
| API for maps and sequences. I don't see that forces us to merge them at the
| information model level.

Once again, consider these two examples.
 
--- !list
0: \
 one
1: two

--- !list
- \
 one
- two


Let's say I load both of these into a Python system, where !list
above is the default class for the sequence construct (using 
Alpha infomodel) or is the default class for the sequence 
syntax (using the Beta infomodel).  Thus, what I have in memory is...

['one','two']

| The spec explicitly says you can have whatever API you want as long 
| as it preserve the information model, and a unified API with an
| 'isSeq' method certainly does.

In the Alpha model, I must somewhere store that 
the first document is a "map" and the second 
is a "sequence".  Where do I do this?

In the Beta model, what *syntax* was used isn't important.
Just like it isn't important which synax was used to encode
"one" vs "two".  The sequence syntax (without the numbers) 
is just a short-hand for the second.   In the Beta model
the *class* determines what the internal representation is,
not the syntax.  In the Alpha model... things are less clear.

| I think we are splitting hairs, and Brian is tearing his 
| in despair by now (or probably even just skipping these
| posts altogether :-). 

The information model determines what is preserved and
what isn't preserved when one does a round-trip.  If
the syntax distinction isn't going to be preserved, then
it needs to come out of the information model, no?

Best,

Clark

On Mon, Nov 12, 2001 at 10:40:56AM -0500, Clark C . Evans wrote:
| --- !list
| 0: \
|  one
| 1: two
| 
| --- !list
| - \
|  one
| - two

...

Brian wrote:
| 3) I'm not sure that integers as map keys will round trip through
|    Perl. I think map keys get flattened to strings. Oh well.

If you explicitly label a map as a list, will you 
store it in a list... or put it in the map?  I thought
that you were going to allow this...

#sparce array
--- !list
0: one
4: two

Is this going to be stored in a map or a list?  Just curious.

Best,

Clark

On 12/11/01 12:37 -0500, Clark C . Evans wrote:
> On Mon, Nov 12, 2001 at 10:40:56AM -0500, Clark C . Evans wrote:
> | --- !list
> | 0: \
> |  one
> | 1: two
> | 
> | --- !list
> | - \
> |  one
> | - two
> 
> ...
> 
> Brian wrote:
> | 3) I'm not sure that integers as map keys will round trip through
> |    Perl. I think map keys get flattened to strings. Oh well.
> 
> If you explicitly label a map as a list, will you 
> store it in a list... or put it in the map?  I thought
> that you were going to allow this...

That's not what I'm talking about. I'm just saying that this:

foo: bar
8: days a week

might turn into:

foo: bar
"8": days a week

> 
> #sparce array
> --- !list
> 0: one
> 4: two
> 
> Is this going to be stored in a map or a list?  Just curious.

Neither. It will be stored in an array. What's a 'list' anyway? ;)

Cheers, Brian

Clark C . Evans wrote:
> Ok.  Think "syntax with default class" instead of style.
> For example, how do these two differ?
>
> --- !list
> 0: one
> 1: two
> --- !list
> - one
> - two
>
> If we are to merge the APIs, as you said above, they 
> must have an *identical* information model.  Right?

Sort of. In a streaming API, I'd expect to get

document-starts-here
'!list' transfer method
this is a map
key is '0'
value is 'one'
key is '1'
value is 'two'
document-starts-here
'!list' transfer method
entry is 'one'
entry is 'two'

I'd expect YAML->Load to produce identical in-memory constructs out of
these. Yes. However I would be *extremely* unhappy with a streaming API
which didn't report these two as different.

To answer the question of how these would round trip: Brian got that one
right. Serial->Memory->Serial isn't important (we have zillions of examples
of this). Memory->Serial->Memory is. The moment you specify a transfer
method you leave S->M->S round-trip behind. I don't care that I read the map
form and emit the seq one, just like I don't care if I read 12. and emit it
as 12.0 - it is exactly the same thing. 

> | Again, I fully agree with the API and YPATH syntax points you raise.
> Let's focus on YPATH.

Let's not. I want to get a spec out the door. We have some open issues
there: scalar styles, the references issue (which I'd love to hear Brian's
latest word on), and this info model stuff. YPATH is going to be a hole new
spec with problems of its own. I don't see how this discussion has anything
to do with it.

As a small example - suppose we do add 'table' as an additional 'branch
style'. How would YPATH deal with it? It seems to me that there's no way you
can have the same syntax for map/seq and for table. A map or a seq require
just *one* key to identify an entry. That's why I said your proposal makes
"everything look like a map" - it uses *one key*. A table needs *two*. To
make it worse table columns are (should be) addressible. I can just see us
having a general N-th dimentional branch as a result: /[*,2,*]/, anyone?
What is going to be the Perl6 slice syntax, BTW? :-)

"This brings us everywhere and nowehere". I'm a believer in XP's "You Ain't
Going To Need It" and "Use The Simplest Thing That Can Possibly Work". As
long as I can add anything I need later without breaking anything, I'm
happy. So...

Tell you what. How about you re-draft the information model section
according to your proposal, and post it? I'll go with it as long as
Joe-Average-Programmer (as Brian puts it) will find it as clear and
intuitive to use in Perl/Python/etc. as he would the current wording. I
think Brian will too. Fair?

Think of it as using proof-by-example, which is often much easier than
proof-by-logical-arguments. Witness the references issue as an example of
that :-)

BTW, as far as helping the productions, I'm not certain it helps that much,
but every bit would help there... We'd probably need to break the map/seq
sections into a "Branch" super-section, just like we have a scalar
super-section. That's all minor stuff. It is the wording of the information
model which make the difference.

Have fun,

    Oren Ben-Kiki

On Mon, Nov 12, 2001 at 10:31:37PM +0200, Oren Ben-Kiki wrote:
| I'd expect YAML->Load to produce identical in-memory constructs out of
| these. Yes. However I would be *extremely* unhappy with a streaming API
| which didn't report these two as different.

I view the reporting of the differnces between these two
constructs at the same level as reporting the various
"styles" of scalars.  IMHO, they are the same thingy.

| To answer the question of how these would round trip: Brian got that one
| right. Serial->Memory->Serial isn't important (we have zillions of examples
| of this). Memory->Serial->Memory is. The moment you specify a transfer
| method you leave S->M->S round-trip behind. I don't care that I read the map
| form and emit the seq one, just like I don't care if I read 12. and emit it
| as 12.0 - it is exactly the same thing. 

Ok.  This clarity needs to make it into the Spec...

| > Let's focus on YPATH.
| 
| Let's not.

This was in the context of the example; to 
shed light on what the information model requires.

| As a small example - suppose we do add 'table' as 
| an additional 'branch style'. How would YPATH deal
| with it? It seems to me that there's no way you
| can have the same syntax for map/seq and for table.

This is why there is a distinction between the
model and the syntax.  To manipulate a YPATH via
the 'table' construct, would be two branches.

  /row/col

Kinda simple.  A table isn't in the information
model; it is just syntax sugar; much like our
various scalar styles.  And like the scalar 
styles... we can't garuentee that it will be
round-tripped as a table.   Also, it means that 
one can't address a column separately... beacuse 
a table isn't in our information model.

Not that we're going to implement tables... I'm 
just using this as an example of possibly another
syntax that would work with the same information
model... 

| "This brings us everywhere and nowehere". I'm a believer
| in XP's "You Ain't Going To Need It" and "Use The Simplest
| Thing That Can Possibly Work".

Exactly.  Why have two constructs "map" and a "list" 
at the information model level when only one is 
needed.  

| Tell you what. How about you re-draft the information model section
| according to your proposal, and post it? I'll go with it as long as
| Joe-Average-Programmer (as Brian puts it) will find it as clear and
| intuitive to use in Perl/Python/etc. as he would the current wording. I
| think Brian will too. Fair?
|
| Think of it as using proof-by-example, which is often much easier than
| proof-by-logical-arguments. Witness the references issue as an example of
| that :-)


Definately fair.  This is how YAML got started anyway... right?
BTW, the names you've chosen for your productions is pretty...
very nice.  Thanks!

Best,

Clark

On 12/11/01 22:31 +0200, Oren Ben-Kiki wrote:
> Clark C . Evans wrote:
> > Ok.  Think "syntax with default class" instead of style.
> > For example, how do these two differ?
> >
> > --- !list
> > 0: one
> > 1: two
> > --- !list
> > - one
> > - two
> >
> > If we are to merge the APIs, as you said above, they 
> > must have an *identical* information model.  Right?
> 
> Sort of. In a streaming API, I'd expect to get

Just a quick point. Please keep in mind that when you ask my opinion on
stuff, I only tend to think of things from the Load/Store API. I really
don't ever think about the streaming API at all. I think Clark only
thinks about the streaming API. And Oren, thank God, is there to see
them both.

In other words, I would say, "Yes, Clark. Those two are exactly the
same." Oren would say, "It depends where you're standing."

So Oren is more correct, IMO. If I cared about a streaming API, I would
want all the information back from the parser. Not a dumbed down subset.

Why? I guess it's because we already have a Load/Store API for
dummies like me. So our alternate API should be full control.

---

Here's another thing that I find interesting. You can parse a YAML stream
into memory without knowing the entire serialization in advance. But you
can't emit a YAML serialization without having the entire structure in
memory, because of references.

Now I'm no expert, but I'll bet you could actually emit YAML without knowing
the entire structure, *if* you have access to the references from an earlier
serialization. So while alias/anchor tags have no place in the memory
structure, they probably should be returned from the parser API, so that an
emitter could start doing its work as early as possible. Am I making sense?

I see that we have the following use cases to look at. At least so we can
understand each other.

1a) serialized->(Parser)->intermediate->(Loader)->memory->(Application)
1b) (Application)->memory->(Storer)->intermediate->(Emitter)->serialized

2a) serialized->(Parser)->intermediate->(Filter)
2b) (Filter)->intermediate->(Emitter)->serialized

3a) serialized->(Parser/Loader)->memory->(Application)
3b) (Application)->memory->(Storer/Emitter)->serialized

A few points:

A) I *always* consider number 3 as being the only use case I care about.
   This is my fault, but that's just the way I think.

B) This spec should really just be concentrated on the *serialization*.

C) The needs of YAML stem from the *memory* models of various
   programming languages.

D) While interesting, the *intermediate* representation shouldn't be of
   concern right now. (Look at number '3'. No intermediate!)

So for instance, with anchors and aliases, I think they should exist at
the *intermediate* level but not at the *memory* level. But neither of
those have anything to do with the spec, which is concerned about the
*serialization*.

I guess it was more than "a quick point".

> Let's not. I want to get a spec out the door.

Amen!


> Think of it as using proof-by-example, which is often much easier than
> proof-by-logical-arguments. Witness the references issue as an example of
> that :-)

You go girl! ;)


Cheers, Brian

On 12/11/01 16:32 -0500, Clark C . Evans wrote:
> On Mon, Nov 12, 2001 at 10:31:37PM +0200, Oren Ben-Kiki wrote:
> | "This brings us everywhere and nowehere". I'm a believer
> | in XP's "You Ain't Going To Need It" and "Use The Simplest
> | Thing That Can Possibly Work".
> 
> Exactly.  Why have two constructs "map" and a "list" 
> at the information model level when only one is 
> needed.  

As Larry Wall says, "A little bit of redundancy is a good thing". I really
believe this. Why do all scripting languages support both hashes and arrays,
when hashes would suffice. It's not just for performance. It's to support
different ways of thinking.

Cheers, Brian

On 12/11/01 16:30 +0200, Oren Ben-Kiki wrote:
> I agree with everything you say about using the same YPATH syntax and same
> API for maps and sequences. I don't see that forces us to merge them at the
> information model level.
...
> Again, I fully agree with the API and YPATH syntax points you raise. I just
> don't see what they have to do with the information model. The spec
> explicitly says you can have whatever API you want as long as it preserve
> the information model, and a unified API with an 'isSeq' method certainly
> does.

Can someone please explain to a poor old JAPH like me the following terms:

- YPATH
- Information model
- Information model level

Then maybe I can get all up in arms too ;)

Cheers, Brian

Information Model

An information model is an abstraction of how
each processing system must treat YAML data.  
In particular the model describes which chunks
of data are stored and the relationship between
those chunks.  Implicitly, these relationships
are invariants which must be maintained for 
YAML data to be consistent (round-trip).

An information model is useful beacuse it codifies
expected behavior of processing systems.  Those
which do not preserve all of the datums mentioned 
by the information model or fail to preserve the
relationships between those datums are non-compliant.

For example, we have said that our "map" structure
is not ordered.  Thus, the YAML documents below are
the same in the information model.

---
a: one
b: two
---
b: two
a: one

What this means is that if a process is using YAML, it
cannot expect to encode semantic information which 
will be preserved by YAML in this ordering.  For another
example, in YAML the *style* of the scalar is not in
the information model.  Thus, the items in this list 
are all equivalent:
---
- This is a scalar
- |-
 This is a scalar
- \
 This is a scalar

What this means is that if a process is using YAML, it
should not expect to encode valueable inforamation by
using different styles.  After all, this document may
go into a visual editor (that is YAML aware) and be
written out as...

--- 
- "This is a scalar"
- "This is a scalar"
- "This is a scalar"

For another example, in the Alpha (our current) 
information model, there is a distinction drawn 
between a map and a sequence; even if the map and 
sequence constructs both have an explicit class 
(say an array).  Thus, the two documents below 
are different.

--- !array
0: one
1: two
--- !array
- one
- two

What this means is that a valid YAML process *must*
preserve this difference.  If an YAML implemetation 
reads both of these documents into memory, must 
somehow record the the first is a "map" and the 
second is an "sequence" in such a way that the user
can know which is which.  And finally, it must be
able to have the distinction when it serializes 
the structures above...

In the Beta information model, the sequence/map
distinction is not in the information model; instead
it is replaced by a "branch" construct.  In this way,
the sequence construct is a short-hand used when 
the map entries are 0,1,2,...  Thus, if we use the 
Beta information model, a processor doesn't have
to remember which syntax style was used, and could
output the two documents above as... (map version
dicarded).

--- !array
- one
- two
--- !array
- one
- two


Ok.  By now you're saying: "your're splitting hairs".
Well... kind of.  But when you don't split hairs,
you have to pull them instead.   *smile*  Seriously.  
The information model matters.  It describes what
a YAML processor must preserve and what is just "fluff".
This is important beacuse I may want to use YAML in
a system that doesn't obviously have "map" and "lists".
An example is a PostgreSQL database.  In the database,
one has "relations".  And, the Beta YAML information 
model works very well in this situtation.  


The informaiton model is related to the canonical form.  
The canonical form is a standard serialization of YAML
information, allowing streams to be compared character 
by character to determine equivalence.  A strong 
information model guides a canonical form.  However,
the information model must be abstract enough to allow 
various native implementations and processing techniques,
while remaining clear enough so that confusion among 
various implementations doesn't proceed.

Clark

P.S.  XML was written without an informaiton model, and
this has caused big time problems as vendors try to get
their tools to work together.   Just beacuse it's in
the syntax, doesn't mean that it's important and will
be preserved.

YAML Path (or YPATH) is a syntax used to identify 
particular nodes within a YAML tree.  Assume the
folowing document.

---
a: &001
 - &002 a
 - &003 \
  scalar value
b: &004
 x: &005 boo
 y: &006 
  p: &007
   - &008 yippe
   - &009 yappo
  q: &010
c: &011
 x: &012 moo
 y: *006
d: *004


A path is a query that returns a set of nodes (which I 
will write using their identity).

Path            | Set of nodes   | Notes
----------------+--------------------------------------
  /a/0          | *002           | Goes to "a" then to "0"
  /*/x          | *005 *012      | Matches anything, then "x"
  /d/x/p        | *007           |
  /b/x/p/*      | *008 *009      |
  //*[.='boo']  | *005           | Matches any node in the tree
                                 | with a scalar value 'boo'
  
YAML path is useful for extracting parts of a YAML graph, 
setting up various processing or notification systems:
"tell me when a document arrives such that this ypath
 expression returns at least one node".  Or choosing
which program is used to process an incoming YAML 
document on the basis of it's content.   Basically... it's 
horribly useful stuff.   By the way... by attaching an 
identity to each node we an *really* be clear how
YPATH works; explaining it will be easy (compared to xpath).

Anyway, this relates to the Information Model since 
these expressions will be queries on the information 
model... they will know *nothing* of the syntax.  So,
if we distinguish between a map (with integer keys)
and a sequence, then YPATH must also do this.  Thus,
we'd probably need a different way to express an
"index" of a node in a sequence, or some other way
to differentiate between the two of them (yuck).

The decisions about the Informaiton Model have a direct
affect on YPATH... unlike decisions about scalar styles
and other syntax sugar.  

Best,

Clark

On 13/11/01 11:17 -0500, Clark C . Evans wrote:
> Information Model
> 
> An information model is an abstraction of how
> each processing system must treat YAML data.  
> In particular the model describes which chunks
> of data are stored and the relationship between
> those chunks.  Implicitly, these relationships
> are invariants which must be maintained for 
> YAML data to be consistent (round-trip).

Clark, 

Thanks for your excellent (as usual :) explanation of an information model.

I now think I better understand our communications problems. (See below)

> For another example, in the Alpha (our current) 
> information model, there is a distinction drawn 
> between a map and a sequence; even if the map and 
> sequence constructs both have an explicit class 
> (say an array).  Thus, the two documents below 
> are different.
> 
> --- !array
> 0: one
> 1: two
> --- !array
> - one
> - two
> 
> What this means is that a valid YAML process *must*
> preserve this difference.  If an YAML implemetation 
> reads both of these documents into memory, must 
> somehow record the the first is a "map" and the 
> second is an "sequence" in such a way that the user
> can know which is which.  And finally, it must be
> able to have the distinction when it serializes 
> the structures above...

The problem we are having is that there are actually (at least) two different
processing layers with two different "information models" as it were. Let's
look at the flows that I suggested yesterday.

1a) serialized->(Parser)->intermediate->(Loader)->memory->(Application)
1b) (Application)->memory->(Storer)->intermediate->(Emitter)->serialized

2a) serialized->(Parser)->intermediate->(Filter)
2b) (Filter)->intermediate->(Emitter)->serialized

3a) serialized->(Parser/Loader)->memory->(Application)
3b) (Application)->memory->(Storer/Emitter)->serialized

As you can see there is a potential "information model" at both the
intermediate level and the memory level. Now I have up to this point
considered the "information model" (IM) to be only at the memory level.
The intermediate level (if one happens to be implemented) is merely an
API. You can use whatever temporary IM you want. Let me give you a real
life example.

Last night I started writing YAML.pm in pure Perl. This is what I call a
Parser/Loader (3a above). I parse a YAML stream and load it into memory.
There is no intermediate representation, or if there is, it is implicit
in my Perl code. The user of my code will be a Perl application that
wants to Load a YAML stream into memory.

Down the road, you will create libyaml.so, a C library for parsing YAML.
At that point I will rework YAML.pm to use your library/API. The YAML.pm
application API will remain the same. (YAML->load($stream);) The JAPH
will never know the difference, except that things will run faster. The
only user of your code in this case will be *me*.

So....

Given the above example, (both Oren and) I want to know the difference
between a !array coded as a map, and one coded as a seq. AT THE C API
LEVEL. I will then load them into identical Perl memory structures for
the application. The application (JAPH script) will not know how they
were encoded.

---

Clark, It seems to me that the "information model" we are defining
should just be the final, in-memory "graph". It also seems to me that
the C library (Parser) I mention above should have no knowledge of this
IM. It should simply define an API for accessing the full layout of the
serialization. The Loader is the one who uses the parser API to map the data
into memory according to the IM. 

Cheers, Brian

On Tue, Nov 13, 2001 at 10:48:49AM -0800, Brian Ingerson wrote:
|
| The problem we are having is that there are actually 
| (at least) two different processing layers with two 
| different "information models" as it were.

Correct.  Let's call the "in-memory" model the "information model"
and the other model the "serialization model".  The serialization
model has a 1-1 correspondence with the text and is what the
parser/emitter should provide; including the comments, scalar
style used, etc.


| Last night I started writing YAML.pm in pure Perl. This is what I call a
| Parser/Loader (3a above). I parse a YAML stream and load it into memory.
| There is no intermediate representation, or if there is, it is implicit
| in my Perl code. The user of my code will be a Perl application that
| wants to Load a YAML stream into memory.

Right.  You would be using the "information model"

| 
| Down the road, you will create libyaml.so, a C library for parsing YAML.
| At that point I will rework YAML.pm to use your library/API. The YAML.pm
| application API will remain the same. (YAML->load($stream);) The JAPH
| will never know the difference, except that things will run faster. The
| only user of your code in this case will be *me*.

Right.  And this "libyaml.so" will have two APIs.  Each one
of them will have a "pull/input" and a "push/output" variant.

cooked  This is the API that supports the information model
        directly.  It will not do comments, scalar styles, etc.

raw     This is the API that supports the serialization model
        directly; including comments, scalar styles, etc.

There will be a adapter ("cooker") from cooked pull/input 
api to the raw pull/input API.   Likewise, we will have
another adapter ("canonical") from the cooked push/output
api to the raw push/output API which does our default
serialization, the official canonical form.  

The cooked API will comply with the "informaition model"
and the raw API will comply with the "serialization model"


| Given the above example, (both Oren and) I want to know the difference
| between a !array coded as a map, and one coded as a seq. AT THE C API
| LEVEL. I will then load them into identical Perl memory structures for
| the application. The application (JAPH script) will not know how they
| were encoded.

Right.

| Clark, It seems to me that the "information model" we are defining
| should just be the final, in-memory "graph". It also seems to me that
| the C library (Parser) I mention above should have no knowledge of this
| IM. It should simply define an API for accessing the full layout of the
| serialization. The Loader is the one who uses the parser API to map 
| the data into memory according to the IM. 

Right, you are considering the "raw" API.  However, the "cooked"
sequential access API will also be available if you choose to
use it.  I'll publish both APIs later this week, or early next
week so that we can come to agreement here.

Perhaps we should talk a bit about the "canonical form", which
is important for "signatures" and raw byte-by-byte comparisons.

Best,

Clark