YAML Ain't Markup Language

I want to express explicit (and quite strong) agreement 
for Oren's writing below.   I do sympathize for those
which want to preserve key order or allow duplicate keys;
but there is only so far that I'm willing to bend.

My compromise is to have two layers (as Orr proposed).  One which
officially recongizes the ability to use the YAML *syntax* to write
duplicate keys and preserve key order -- but doesn't go much further.
This layer (the SERIAL model?) will formalize what a Parser produces,
maps having keys delivered in a particular order and the possibility
of duplicate keys.   If some application (or alternative set of 
higher level tools) wishes to build from that level... so be it.
However, the focus of YAML should remain on our own serialization
requirement w/o duplicate keys and key order.

On Wed, Sep 11, 2002 at 04:55:51PM +0300, Oren Ben-Kiki wrote:
| > but if we go below this threshold and only support the 3 that 
| > yaml currently has, then certain types of objects will simply 
| > not have a clean means to serialization --only hacks. i keep 
| > hearing the 80% deal in realtion to this, but honestly why 
| > deal with only 80% if 100% is just a stones throw away?
| 
| Moving from 3 types, with a solid theoretical and practical basis to 8 types
| is not "a stones throw away". I guess you mean ordered/unordered set/map
| with/out duplicates (not that I can see how can have an unordered set/map
| with duplicates, especially a map; how can you reliable identify a specific
| duplicate key? But anyway...). IMVHO it is unreasonable to force every YAML
| implementation to be able to represent all these.
| 
| Think about it this way. Maps and sequences (and scalars) can be represented
| in _every_ language we are dealing with, PHP included. OK, PHP is stronger;
| it can also handle ordered maps. And Perl is stronger because it has
| references (AKA pointers - which are different from YAML aliases). And
| Python is stronger because it has tuples. And Java is stronger because it
| has Integer vs. Float. And so on.
| 
| The core types we've chosen are "the least common denominator". It works
| everywhere. And luckily, you can represent anything using this common subset
| without too much trouble. Call it a hack if you will, but I think the extra
| '- ' at the start of each key/value pair isn't _that_ bad.
| 
| I believe in the spirit of "don't pay for it unless you use it". Requiring
| the least common denominator shifts the burden of supporting anything else
| just to the applications that actually use it.

On Wed, 2002-09-11 at 07:55, Oren Ben-Kiki wrote:
> OK. Obviously I haven't made myself clear. Consider that the above example
> may be piped through a YPretty tool. Anyone may take a notepad to this file
> and flip the order if the keys. Hence the above is _not_ the way to do it.
> 
> You want an omap, write it as:
> 
> Mymap: !omap
>  - one: A
>  - two: B
> 
> (In fact you may do without the !omap - see my latest post).
> 
> The '- ' tells everyone that the order is significant; changing it _will_
> change semantics. If there's no '- ', anyone is allowed to randomize the
> order at will, knowing for certain that it will _not_ change any semantics.

and i haven't made myself clear :) you don't get an ordered list with
this, you get a collection of collection.
 
> Well, what else can we do, really? Ruby doesn't have type A, PHP doesn't
> have type B. So if you want to do a Ruby<->PHP, either one has to emulate
> the types of the other. That's the "physics" of the thing; no way around it.
>
> > this is why i beleive omap! and the like need to be 
> > fundemental to yaml. this way the yaml.rb implementor can 
> > build an omap object for Ruby to be used under these situations.
> 
> This still doesn't solve anything, because if I write a simple !seq array in
> YAML, PHP will _still_ have to emulate it somehow.

so here's the danger: Ruby dumps and array:

Myarray:
  - one
  - two

PHP loads it, but when PHP dumps you get:

Myarray:
  - 0: one
  - 1: two

b/c phps arrays are also maps, i.e. what is yaml.php going to do? it
can't right decide a php array is a sequence or it is a mapping on its
own. and even if the user intervens, what if he used his php array in
both manners? you can't rightly ask a php'er to deal with this. they
simply won't and won't use yaml.

> Moving from 3 types, with a solid theoretical and practical basis to 8 types
> is not "a stones throw away". I guess you mean ordered/unordered set/map
> with/out duplicates (not that I can see how can have an unordered set/map
> with duplicates, especially a map; how can you reliable identify a specific
> duplicate key? But anyway...). IMVHO it is unreasonable to force every YAML
> implementation to be able to represent all these.
 
> Think about it this way. Maps and sequences (and scalars) can be represented
> in _every_ language we are dealing with, PHP included. OK, PHP is stronger;
> it can also handle ordered maps. And Perl is stronger because it has
> references (AKA pointers - which are different from YAML aliases). And
> Python is stronger because it has tuples. And Java is stronger because it
> has Integer vs. Float. And so on.

don't know about perls references, but int and float are distiguishable
in yaml, i don't see your point with that example.

> The core types we've chosen are "the least common denominator". It works
> everywhere. And luckily, you can represent anything using this common subset
> without too much trouble. Call it a hack if you will, but I think the extra
> '- ' at the start of each key/value pair isn't _that_ bad.
> 
> I believe in the spirit of "don't pay for it unless you use it". Requiring
> the least common denominator shifts the burden of supporting anything else
> just to the applications that actually use it.

yes, indeed your approach is LCD, mine is GCM (greatest common
multiples). so with LCD it becomes the domain of the user to deal with
these difficultes, in GCM is its up to the implementors, for instance to
provide ruby with an omap class.

there are up and downs with both. granted, GCM can't be 100% (i don't
think) but with the 8 i refer too it is very close, 99% say. LCD, as you
all have reported, is at about 80%.

> > because of these native=functional notions, clarks solution 
> > is a compromise that essentially makes two different yamls. a 
> > lite-xml yaml and the traditional one, seperated at the two 
> > different layers general and functional. this is apprent in 
> > the fact that going from genreal to functional can throw 
> > errors b/c the syntax dosen't conform to the traditional 
> > model.
> 
> It can also throw errors because my 'x' coordinate was given a string value
> instead of an integer; which would have worked in Perl but doesn't work in
> Java. It really is a language-specific thing. What YAML does is try to
> require the bare minimum, and allow you to build the rest on top of that
> "make the simple things simple and the complicated things possible".

yes that's interesting and curious. ruby map keys can be any object and
yaml supports this, but would bomb perl - LCD would toward only
supporting string key and make ruby people deal with it.

i understand where you coming from. and certainly there's no perfect
solution. the question is where to draw the line. 3 or 8? 80% or 99%.
yes, 99% is more work for us. most of you seem to think it's not worth
it. i tend to think it is.

certainly there is no wrong or right here, just the determination of
what yaml will be capable of and what it will not.

-- 
tom sawyer, aka transami
transami@...

Tom Sawyer [mailto:transami@...] wrote: 
> > The '- ' tells everyone that the order is significant; changing it 
> > _will_ change semantics. If there's no '- ', anyone is allowed to 
> > randomize the order at will, knowing for certain that it will _not_ 
> > change any semantics.
> 
> and i haven't made myself clear :) you don't get an ordered 
> list with this, you get a collection of collection.

You get whatever the loader produces. The PHP loader is free to convert this
directly into a PHP-thingie (what _is_ its name?).

> so here's the danger: Ruby dumps and array:
> 
> Myarray:
>   - one
>   - two
> 
> PHP loads it, but when PHP dumps you get:
> 
> Myarray:
>   - 0: one
>   - 1: two

That _is_ a problem if you do dump it this way.

> b/c phps arrays are also maps, i.e. what is yaml.php going to 
> do? it can't right decide a php array is a sequence or it is 
> a mapping on its own.

You are right, there should be some sort of logic operating here, plus
optional external instrictions (the latter is normal; every language has
some types for which such instructions are necessary). I'm guessing that a
default that says:

- If all integer that are exactly 0 .. N, ordered as expected, then dump as
a sequence

Would go a long way to eliminate the need for such instructions. As for
mappings vs. ordered-mappings, I have the suspicion that a default saying
that key order is _not_ significant and requiring ordered maps to be
declared explicitly would work most of the time.

I expect Yaml->dump() in PHP would have some flags that control this
behavior.

> and even if the user intervens, what if 
> he used his php array in both manners?

If he used his collection as an ordered map - e.g., he has "just one"
non-integer key and the key integer 7 appears as the first one, then
obviously this isn't a sequence. And forcing ruby to handle it as an ordered
mapping is exactly what you want, since otherwise it would lose significant
information.

> you can't rightly ask 
> a php'er to deal with this. they simply won't and won't use yaml.

First, note that PHP-ers will never have a problem loading back data that
was dumped from PHP. Even if that's all YAML will do, I think it is enough
to gain a reasonable foothold. As Brian often points out, load/dump from the
same application is the most common use case by _far_.

Second, before we twist YAML out of shape, I think it is best to look at
some PHP code, consider what flags would be needed for Yaml->dump() in PHP
to control this behavior, and then decide whether the extra effort is
reasonable or not.

I've asked this many times and I'll ask again - how many PHP-thingies are
used as anything other than a simple array or a simple hash table?

> don't know about perls references, but int and float are 
> distiguishable in yaml, i don't see your point with that example.

They are only distinguishable in the TYPED model. I was referring to the
core types trio - map/seq/str. Suppose I have an XyPoint class in Java that
expects Float coordinates. However I'm generating the input for it in Perl,
that doesn't care much about floats vs. ints, so it prints:

Point: { x : 1, y : 2.5 }.

My Java app barfs because X is an Integer instead of a Float. But a Perl
application would have no problem reading it. And there's no way to look at
a "1" and decide whether it is "1.0" or "1" without some knowledge of the
application... So you see the problem with omaps really isn't unique to them
or to PHP; it is a problem that is universal to writing cross-platform
applications.

A cross-platform application usually has a schema that is designed detached
from a particular language (you could say "by definition"). Hence when
dumping data that is expected to match such a schema, you _have_ the proper
instructions to give the loader (even if only in the form of a paper
document that needs transcribing).

It is the "semantics-aware" vs. "semantics-blind" issue again.
Cross-platform documents tend to get _loaded_ by semantics-aware code, not
by semantics-blind code (they may be _parsed_ by semantics-blind code,
certainly, but there's no problem at all there).

> > The core types we've chosen are "the least common denominator"...
> yes, indeed your approach is LCD, mine is GCM (greatest 
> common multiples)...

OK, it is good to clarify this. I don't think YAML should try for a GCM. It
would end up being SGML.

> yes that's interesting and curious. ruby map keys can be any 
> object and yaml supports this, but would bomb perl - LCD 
> would toward only supporting string key and make ruby people 
> deal with it.

Brian raised this point at the time, and we had drafts where keys were
limited to scalars. However, Parrot (Perl 6) will allow objects as keys, and
all the other languages already do, so we decided to include it in the LCD.
It was touch and go for a while even so :-)

> certainly there is no wrong or right here, just the 
> determination of what yaml will be capable of and what it will not.

Right. It is a case of diminishing returns... And we don't want to end up
with another SGML.

Have fun,

	Oren Ben-Kiki

On Wed, Sep 11, 2002 at 09:41:24AM -0600, Tom Sawyer wrote:
| so here's the danger: Ruby dumps and array:
| 
| Myarray:
|   - one
|   - two
| 
| PHP loads it, but when PHP dumps you get:
| 
| Myarray:
|   - 0: one
|   - 1: two

Well, I guess an alternative to just use the Parser interface
in PHP (just like many PHP users use expat for XML stuff)
and do your own conversion.  The code wouldn't be that bad,
if this is a real application, you know the schema and thus
would know what to expect and where you want to put it.

Yes, you don't get YAML's cool Load/Dump interface; but
sometimes it may be necessary to work at the lower level.
At this level you could even store your PHP omap as a 
map with the right key order... of course, another 
program or person may swap the keys on you; but as
my doctor says when I complain about my bumb shoulder
right after swimming: "Don't do that."  ;)

Best,

Clark

Oren Ben-Kiki (orenbk@...) wrote:
> A cross-platform application usually has a schema that is designed detached
> from a particular language (you could say "by definition"). Hence when
> dumping data that is expected to match such a schema, you _have_ the proper
> instructions to give the loader (even if only in the form of a paper
> document that needs transcribing).

In addition, I think the TestingSuite will really assist us in matters such as
these.  Because the primary implementations at the moment (.py, .rb, .pm) are
using it, we are forced to communicate when we add new tests, which communication
will help us more than anything written in the spec.  I can't keep a 100% track
of everything in the spec but between looking at the other implementors' assertions
and their reactions to new tests, we can usually get it right.

Personally, I think PHP is going to have to use some severe DWIM logic to get their
arrays to work at all.  The following tests I feel would be adequate for our
PHP<->Ruby concern:

  test: Sequence
  yaml: |
    - Mark McGwire
    - Sammy Sosa
    - Ken Griffey
  php: |
    array( 'Mark McGwire', 'Sammy Sosa', 'Ken Griffey' );
  ruby: |
    [ 'Mark McGwire', 'Sammy Sosa', 'Ken Griffey' ]
  ---
  test: Mapping
  yaml: |
    hr:  65
    avg: 0.278
    rbi: 147
  php: |
    # Perhaps mixup the keys predictably
    array( 'avg' => 0.278, 'hr' => 65, 'rbi' => 147 );
  ruby: |
    { 'hr' => 65, 'avg' => 0.278, 'rbi' => 147 }
    
  ---
  test: Sequence of Mappings
  yaml: |
    - hr: 65
    - avg: 0.278
    - rbi: 147
  php: |
    # This PHP function will convert an ordered array into a seq-map automatically!
    array_chunk( array( 'hr' => 65, 'avg' => 0.278, 'rbi' => 147 ), 1, TRUE );
  ruby: |
    h = { 'hr' => 65, 'avg' => 0.278, 'rbi' => 147 }
    # Ruby version of the array_chunk()
    h.collect { |x| { x[0] => x[1] } }
  ---
  test: Say we wanted to support an !omap
  yaml: |
    --- !omap
    - hr: 65
    - avg: 0.278
    - rbi: 147
  php: |
    # Perhaps the earlier mapping could be wrapped instead of this one
    yaml_omap( array( 'hr' => 65, 'avg' => 0.278, 'rbi' => 147 ) );
  ruby: |
    YAML::Omap[ 'hr', 65, 'avg', 0.278, 'rbi', 147 ]

If ordered keys are the exception to the mapping rule, it would be nice to see them
tagged as such don't ya think?  !omap is kinda cool in my eyes because it helps
identify those maps as.. well.. "special".  I don't know if users would see it
that way.

Anyways, point was: good tests in the TestingSuite should really help us from
spiraling out of control as far as interop.

_why

On 11/09/02 16:55 +0300, Oren Ben-Kiki wrote:
> Tom Sawyer [mailto:transami@...] wrote:
> OK. Obviously I haven't made myself clear. Consider that the above example
> may be piped through a YPretty tool. Anyone may take a notepad to this file
> and flip the order if the keys. Hence the above is _not_ the way to do it.

I know it's obvious to everyone but me. What is YPretty. What is a pretty
printing tool. I have *no* idea so please be desciptive. You have 500 words
to describe it.

Cheers, Brian

On 11/09/02 15:29 +0000, Clark C. Evans wrote:
> I want to express explicit (and quite strong) agreement 
> for Oren's writing below.   I do sympathize for those
> which want to preserve key order or allow duplicate keys;
> but there is only so far that I'm willing to bend.

+1

> My compromise is to have two layers (as Orr proposed).  One which
> officially recongizes the ability to use the YAML *syntax* to write
> duplicate keys and preserve key order -- but doesn't go much further.
> This layer (the SERIAL model?) will formalize what a Parser produces,
> maps having keys delivered in a particular order and the possibility
> of duplicate keys.   If some application (or alternative set of 
> higher level tools) wishes to build from that level... so be it.
> However, the focus of YAML should remain on our own serialization
> requirement w/o duplicate keys and key order.

I'll take Clark one step further. Let's not worry about defining the key
preserving model at all for now. It *will* appear as a representation if
people want it. And we *may* want to codify its properties into a model
at some point. Right now i feel that it's beyond our current scope. It's
impeding our progress.

This is all about timing. We need to freeze stable concepts to let
implementors catch up. THIS IS VITAL, PEOPLE! If we keep increasing
scope everytime we get close to resolution, then this is all merely a
mental exercise with no end result.

Think about the implementors as you are driving down the highway today.
GIVE EM A BRAKE.

Cheers, Brian

On 11/09/02 11:13 -0600, why the lucky stiff wrote:
> Oren Ben-Kiki (orenbk@...) wrote:
> > A cross-platform application usually has a schema that is designed detached
> > from a particular language (you could say "by definition"). Hence when
> > dumping data that is expected to match such a schema, you _have_ the proper
> > instructions to give the loader (even if only in the form of a paper
> > document that needs transcribing).
> 
> In addition, I think the TestingSuite will really assist us in matters
> such as these. Because the primary implementations at the moment (.py,
> .rb, .pm) are using it, we are forced to communicate when we add new
> tests, which communication will help us more than anything written in
> the spec. I can't keep a 100% track of everything in the spec but
> between looking at the other implementors' assertions and their
> reactions to new tests, we can usually get it right.

+10

I had this conversation with Steve on irc the other day. In many ways the
TestingSuite is such much more vital than written specifications. Words can
be twisted, but the code don't take no mess.

I'd be really pleased to see the people who are putting so much energy into
our spec, start codifying their ideas into the TestingSuite.

> Anyways, point was: good tests in the TestingSuite should really help us from
> spiraling out of control as far as interop.
> 
> _why
> 
> 
> 
> -------------------------------------------------------
> In remembrance
> http://www.osdn.com/911/
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core

Brian Ingerson [mailto:ingy@...] wrote: 
> This is all about timing. We need to freeze stable concepts 
> to let implementors catch up. THIS IS VITAL, PEOPLE! If we 
> keep increasing scope everytime we get close to resolution, 
> then this is all merely a mental exercise with no end result.

+1

> Think about the implementors as you are driving down the 
> highway today. GIVE EM A BRAKE.

I agree. Can I also mention the people who work on the spec document itself
:-)?
We want to freeze the thing. But we also want to freeze it right...

Also:
> I've been thinking of how to describe the concept of a native 
> representation which is somewhat inherently undefinable. 

I'm thinking along the same lines. Though we must place _some_ restrictions
on what information is reflected in it (never mind _how_ it is reflected).
For example, throwaway comments are called "throwaway" for a reason :-)

And:
> Does this mean that YAML can't be used in some languages? 
> Absolutely not. YAML should be able to be used in *any* 
> complete programming language. That's because any language 
> should be able to construct a "functional graph".

+1, but this is partial. You could say the same about the relational model -
every language can construct a table... But it doesn't map well to most data
types in use in code.

We also want the match to be "close" for commonly _used_ types in "most"
languages. I think !map/!seq/!str (or scalar, anyway) is a great choice
here, and that anything else can be made to pay some extra cost, one way or
the other.

Have fun,

	Oren Ben-Kiki

On 11/09/02 23:15 +0300, Oren Ben-Kiki wrote:
> Brian Ingerson [mailto:ingy@...] wrote: 
> > Does this mean that YAML can't be used in some languages? 
> > Absolutely not. YAML should be able to be used in *any* 
> > complete programming language. That's because any language 
> > should be able to construct a "functional graph".
> 
> +1, but this is partial. You could say the same about the relational model -
> every language can construct a table... But it doesn't map well to most data
> types in use in code.
> 
> We also want the match to be "close" for commonly _used_ types in "most"
> languages. I think !map/!seq/!str (or scalar, anyway) is a great choice
> here, and that anything else can be made to pay some extra cost, one way or
> the other.

Well resaid. The spirit of YAML is in the plain old map/seq/str model. We are
realizing the potential to do other stuff, which is great. But that's not
what we should formalize in the spec.

My current thought is to keep an information model in the spec, but have it
use layman wording and limit it to the basic models that we have today.
Basically it would just be describing what Perl/Python/Ruby programmers
implicitly know: how hashes/arrays/scalars work. The reason to do this is
that we can freeze the spec SOON.

Then, I would propose that we work on a separate "Useful YAML Information
Models" document which we can spend more time on. If you don't make a
separation between that which has been decided and that which is still very
much debatable, then you'll just stimey implementation.

Cheers, Brian

On Wed, Sep 11, 2002 at 02:36:10PM -0700, Brian Ingerson wrote:
| > We also want the match to be "close" for commonly _used_ types in "most"
| > languages. I think !map/!seq/!str (or scalar, anyway) is a great choice
| > here, and that anything else can be made to pay some extra cost, one way or
| > the other.
| 
| Well resaid. The spirit of YAML is in the plain old map/seq/str model. We are
| realizing the potential to do other stuff, which is great. But that's not
| what we should formalize in the spec.

I'm not quite sure.  But your posts resonate.  Esp. the one about
wasting time.  If I would have spent 1/2 the time on this last 
set of stuff on a YAML schema, we'd be quite far along.
I'm looking forward to getting back to libyaml implementation.

| My current thought is to keep an information model in the spec, but have it
| use layman wording and limit it to the basic models that we have today.
| Basically it would just be describing what Perl/Python/Ruby programmers
| implicitly know: how hashes/arrays/scalars work. The reason to do this is
| that we can freeze the spec SOON.

I was thinking of focusing on what I was calling the "generic" 
model in my last few posts (with duplicate keys and key order)
since this model is a very close model of what the syntax
allows.   This is, in essence, just removing the stuff about
the functional model and cleaning the rest up.

| Then, I would propose that we work on a separate "Useful YAML Information
| Models" document which we can spend more time on. If you don't make a
| separation between that which has been decided and that which is still very
| much debatable, then you'll just stimey implementation.

Yes.  I'm starting to see the wisdom of moving the "functional"
model out.  Perhaps that's where YAML tools work, but this
doesn't necessarly have to be in the YAML spec itself.  If people
then do stuff that supports the "functional" model, they can
use the tools; if not, so be it.   But this way the spec doesn't
necessarly tie people to the tool set that I had in mind.

For example, with a the current syntax, someone could easily
represent Minimal XML in YAML syntax, perhaps even building
a "native" loader into a subset of XML's DOM complete with
XPATH.  I'm not saying I like the option; but it is an alternative
which I have no right in saying "no" to.  

...

That said, I've enjoyed your last few posts and agree with
most of the stuff.   Give me another day or so to collect
my thoughts before I start a posting spree again.  This next
time I'll try to keep it shorter.

Best,

Clark

Debate has been fast and furious on the models question, and it may be
worth taking a breather for a second and consider why this is happening
and what it means.

Once upon a time, we were near a spec freeze.  Then people started
pressing their favorite missing features hard.  Why the paranoia?
I think because people felt that if their favorite feature didn't make
it into the 1.0 model scheme, it would be shut out of YAML forever.

But time has shown some interesting things.  (1) It's possible to
allow access to the YAML data at multiple levels, not just the highest
(graph/native/functional) level.  Some say this was the intention all
along, but somehow that message didn't get communicated.  (2) We can't
fit everything people want into the models.  (3) That doesn't mean
missing features are locked out.  They can be implemented as 
extensions to the language-specific modules or by the applications
themselves.  Language-model implementors and users are allowed to
make their own models and parsers.  (4) Our biggest theoretical
user-level tools (ygrep and yprint) don't even operate at one level
anyway; they operate at multiple levels, and can even operate without
direct model support.

This brings up a radical question: what do we need the models for
anyway?  I'm not saying eliminate them, but rather keep them in
perspective.  They are evolving from "the only way you may access
YAML data" to something like examples, ways you *can* access the
data.  Perhaps they should be introduced that way in the spec, to
encourage people to not see them as limiting, but rather as giving a
helping hand.

And now back to your regularly scheduled modules debate.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

looks like we're all feeling that way. i need a break too. clark and
steve have indicated the same. so i'll still be around, but i'm going to
mainly focus on some other things til monday.

talk you all later,
tom

p.s. mike, nice overview of some of the things we've covered. like the
spec idea too.


On Wed, 2002-09-11 at 17:20, Mike Orr wrote:
> Debate has been fast and furious on the models question, and it may be
> worth taking a breather for a second and consider why this is happening
> and what it means.
> 
> Once upon a time, we were near a spec freeze.  Then people started
> pressing their favorite missing features hard.  Why the paranoia?
> I think because people felt that if their favorite feature didn't make
> it into the 1.0 model scheme, it would be shut out of YAML forever.
> 
> But time has shown some interesting things.  (1) It's possible to
> allow access to the YAML data at multiple levels, not just the highest
> (graph/native/functional) level.  Some say this was the intention all
> along, but somehow that message didn't get communicated.  (2) We can't
> fit everything people want into the models.  (3) That doesn't mean
> missing features are locked out.  They can be implemented as 
> extensions to the language-specific modules or by the applications
> themselves.  Language-model implementors and users are allowed to
> make their own models and parsers.  (4) Our biggest theoretical
> user-level tools (ygrep and yprint) don't even operate at one level
> anyway; they operate at multiple levels, and can even operate without
> direct model support.
> 
> This brings up a radical question: what do we need the models for
> anyway?  I'm not saying eliminate them, but rather keep them in
> perspective.  They are evolving from "the only way you may access
> YAML data" to something like examples, ways you *can* access the
> data.  Perhaps they should be introduced that way in the spec, to
> encourage people to not see them as limiting, but rather as giving a
> helping hand.
> 
> And now back to your regularly scheduled modules debate.
> 
> -- 
> -Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
>    http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol
> 
> 
> -------------------------------------------------------
> In remembrance
> http://www.osdn.com/911/
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core
> 
-- 
tom sawyer, aka transami
transami@...

On 11/09/02 17:44 -0600, Tom Sawyer wrote:
> looks like we're all feeling that way. i need a break too. clark and
> steve have indicated the same. so i'll still be around, but i'm going to
> mainly focus on some other things til monday.
> 
> talk you all later,
> tom

Thank you all for the massive input!

Cheers, Brian

> 
> p.s. mike, nice overview of some of the things we've covered. like the
> spec idea too.
> 
> 
> On Wed, 2002-09-11 at 17:20, Mike Orr wrote:
> > Debate has been fast and furious on the models question, and it may be
> > worth taking a breather for a second and consider why this is happening
> > and what it means.
> > 
> > Once upon a time, we were near a spec freeze.  Then people started
> > pressing their favorite missing features hard.  Why the paranoia?
> > I think because people felt that if their favorite feature didn't make
> > it into the 1.0 model scheme, it would be shut out of YAML forever.
> > 
> > But time has shown some interesting things.  (1) It's possible to
> > allow access to the YAML data at multiple levels, not just the highest
> > (graph/native/functional) level.  Some say this was the intention all
> > along, but somehow that message didn't get communicated.  (2) We can't
> > fit everything people want into the models.  (3) That doesn't mean
> > missing features are locked out.  They can be implemented as 
> > extensions to the language-specific modules or by the applications
> > themselves.  Language-model implementors and users are allowed to
> > make their own models and parsers.  (4) Our biggest theoretical
> > user-level tools (ygrep and yprint) don't even operate at one level
> > anyway; they operate at multiple levels, and can even operate without
> > direct model support.
> > 
> > This brings up a radical question: what do we need the models for
> > anyway?  I'm not saying eliminate them, but rather keep them in
> > perspective.  They are evolving from "the only way you may access
> > YAML data" to something like examples, ways you *can* access the
> > data.  Perhaps they should be introduced that way in the spec, to
> > encourage people to not see them as limiting, but rather as giving a
> > helping hand.
> > 
> > And now back to your regularly scheduled modules debate.
> > 
> > -- 
> > -Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
> >    http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol
> > 
> > 
> > -------------------------------------------------------
> > In remembrance
> > http://www.osdn.com/911/
> > _______________________________________________
> > Yaml-core mailing list
> > Yaml-core@...
> > https://lists.sourceforge.net/lists/listinfo/yaml-core
> > 
> -- 
> tom sawyer, aka transami
> transami@...
> 
> 
> 
> -------------------------------------------------------
> In remembrance
> http://www.osdn.com/911/
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core

On Wed, Sep 11, 2002 at 07:54:50PM -0400, Steve Howell wrote:
> What are the chances that you could put some of your tutorial work in progress
> out on the Wiki?  Might be a good way to get feedback.

When I have a chance.  Right now it's just a little bit I put...
somewhere... a couple weeks ago, and a bit more in my head.

I'm not fond of wikis for editing long documents though.  It's fine for
adding short comments to a piece, but when you try to cut and paste
something that's bigger than a TEXTAREA, it pastes to some other part
of the document from where your cursor is, in the middle of a word even,
and then you have to go cut it out and try again.  It's the browsers'
fault, but grrr.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Wed, Sep 11, 2002 at 04:20:55PM -0700, Mike Orr wrote:
| This brings up a radical question: what do we need the models for
| anyway?  I'm not saying eliminate them, but rather keep them in
| perspective.  They are evolving from "the only way you may access
| YAML data" to something like examples, ways you *can* access the
| data.  Perhaps they should be introduced that way in the spec, to
| encourage people to not see them as limiting, but rather as giving a
| helping hand.

The purpose of an information model is to act as documentation
for implementers of the spec, informing the intent of the
productions which follow at a high level.  For example, the 
current model section illustrates a distinction between a serial 
and a graph model, as well as the role of parser vs loader.
This part of the model has both codified current pratice but
also helped to guide new work.

As for limiting, the original intent of the limits in the model
was to express what would round trip and what wouldn't -- to 
provide a guide to the implementation writers as to what is
expected of them.  Completing all of the productions is one
thing; packaging them up is another.
  
    It is important not to underestimate what happened
    this week.  YAML has made a significant shift.  Two
    weeks ago, it was a tool for serializing data in
    modern scripting languages.  With a focus of 
    interoperability so that a given text file would
    represent a typed structure in several languages
    at the same time.

    After this shift we _are_ different.   Two major
    things have happened:

      1.  We no longer are building in implicit types
          which are common to data seralization.

      2.  We are now going to allow for structures
          which we didn't in the past.

    In essence, we are now a "general purpose" syntax
    rather than a object serialization language.

That said, the old model (poorly expressed formalisms aside)
did a great job at expressing our "serialization language"
application.   A replacement model is going to be quite
different beacuse we are now different.  The intent of
YAML's usage is much more generalized... thus our model
will have to be more general, without serialization
limitations and more closely modeling the expressive
power of the raw syntax.

I'm not sure exactly what I'm going to do with my toolset
which is under construction.   I could either re-focus
it to work on the newer more "loose" model (which doesn't 
type things by default, allows for duplicates, and keeps
key order) or, I could just keep it focused on our older
"strict" functional model.   I'll probably play around
with each.   It might be that my toolset won't work on
"any" YAML document (as was planned) but would rather
be restricted to those which complied with a set of
strictions beyond the syntax requirements (restrictions
which most modern serialization langauge constructs meet).

I hope this explains.  In short, I think the "usefulness"
of the information model is being called into question 
beacuse you are looking at it through the eyes of a 
different YAML.  Once the model is patched up to reflect
this concensus which has emerged on the list, it should
once again be useful to those who have "bought" into
the concensus.   Or at least that's the hope.

Best,

Clark

----- Original Message -----
From: "Mike Orr" <iron@...>
> On Wed, Sep 11, 2002 at 07:54:50PM -0400, Steve Howell wrote:
> > What are the chances that you could put some of your tutorial work in
progress
> > out on the Wiki?  Might be a good way to get feedback.
>
> When I have a chance.  Right now it's just a little bit I put...
> somewhere... a couple weeks ago, and a bit more in my head.
>
> I'm not fond of wikis for editing long documents though.  It's fine for
> adding short comments to a piece, but when you try to cut and paste
> something that's bigger than a TEXTAREA, it pastes to some other part
> of the document from where your cursor is, in the middle of a word even,
> and then you have to go cut it out and try again.  It's the browsers'
> fault, but grrr.
>

Yep, that sounds irritating.

Perhaps a solution is to make lots of small documents, and then combine them
together as one page using MoinMoin's #include feature.

Then, people could theoretically create their own tutorials that steal pertinent
sections from yours.  Also, folks can help you edit small subsections, and if
you don't like their edits, just revert their changes or fork the page.

Modularity. :)

Do you know how MoinMoin's include syntax works?

Cheers,

Steve

Mike Orr (iron@...) wrote:
> I'm not fond of wikis for editing long documents though.  It's fine for
> adding short comments to a piece, but when you try to cut and paste

I don't know if you use Vi, Mike, but your mailer appears to be Mutt,
so I gather you use it so you can edit e-mail with $EDITOR.  When I
browse and edit Wiki, I like to use w3m [http://w3m.sf.net/] because
when you edit a TEXTAREA, you edit it in $EDITOR.  It's the best thing
since sliced bread (and its sequel Sliced Bread II: Electric Boogaloo).

_why

On Wed, Sep 11, 2002 at 11:51:34PM -0400, Steve Howell wrote:
> ----- Original Message -----
> > I'm not fond of wikis for editing long documents though.  It's fine for
> > adding short comments to a piece, but when you try to cut and paste
> > something that's bigger than a TEXTAREA, it pastes to some other part
> > of the document from where your cursor is, in the middle of a word even,
> > and then you have to go cut it out and try again.  It's the browsers'
> > fault, but grrr.

And when you PgUp/PgDn, the cursor doesn't move, so you PgDn several
times and then use the arrow key, and find you have to arrow again
through all the lines you PgDn'd.  Since I'm used to pressing 
PgUp/PgDn extensively, this bites me repeatedly.

> Perhaps a solution is to make lots of small documents, and then combine them
> together as one page using MoinMoin's #include feature.

Actually, I have another idea.  I'll start a wiki page with a
link to the tutorial on my web site (the tutorial that isn't
written yet).  Then I can edit it and people can post comments.  I'll
mark where I release a significant version so you can see how many
versions ago a certain comment was posted.

> Then, people could theoretically create their own tutorials that steal
> pertinent sections from yours.  

Anybody who wants to steal text can download the tutorial web page.

As for incorporating others' suggestions, I don't mind copying them
in myself, at least not at this stage.  And I'd rather have the
tutorial in standalone files rather than in MoinMoin, so that if
we decide to do something else with them, we won't have to 
(1) extract them from MoinMoin and (2) convert the wiki markup.

I'm thinking of a several-file tutorial.  One for general YAML,
one for PyYAML, and one each for the other language implementations.
Except, I won't  be writing the other-language files.
I'd like all the language files to express a common set of 
examples in each language, then list any language-specific or
implementation-specific features.  I've seen one or two existing PyYAML
tutorials somewhere or other (that Steve prob'ly wrote) that'll make a
good starting point.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Wed, Sep 11, 2002 at 11:15:37PM -0600, why the lucky stiff wrote:
> Mike Orr (iron@...) wrote:
> > I'm not fond of wikis for editing long documents though.  It's fine for
> > adding short comments to a piece, but when you try to cut and paste
> 
> I don't know if you use Vi, Mike, but your mailer appears to be Mutt,
> so I gather you use it so you can edit e-mail with $EDITOR.  When I
> browse and edit Wiki, I like to use w3m [http://w3m.sf.net/] because
> when you edit a TEXTAREA, you edit it in $EDITOR.  It's the best thing
> since sliced bread (and its sequel Sliced Bread II: Electric Boogaloo).

Yeah, I use mutt and vim.  My browser of choice is galeon, followed
by links, then lynx.  I've tried w3m but it doesn't have the
lynx-compatible key mappings out of the box like links does, and
that's important to me.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Thu, Sep 12, 2002 at 02:43:25AM +0000, Clark C. Evans wrote:
>     It is important not to underestimate what happened
>     this week.  YAML has made a significant shift.  Two
>     weeks ago, it was a tool for serializing data in
>     modern scripting languages.  With a focus of 
>     interoperability so that a given text file would
>     represent a typed structure in several languages
>     at the same time.
> 
>     After this shift we _are_ different.   Two major
>     things have happened:
> 
>       1.  We no longer are building in implicit types
>           which are common to data seralization.
> 
>       2.  We are now going to allow for structures
>           which we didn't in the past.
> 
>     In essence, we are now a "general purpose" syntax
>     rather than a object serialization language.

And this is a Very Good Thing.  We've doubled YAML's theoretical market
penetration, allowing it to flourish closer to its full potential.  And
also formally codified some things many people thought YAML had all
along, things that attracted them to YAML in the first place.

The first paragraph of the tutorial will mention YAML's twofold purpose,
both as a serialization format that's cross-platform and happens to be
human readable, and as a parser for a syntax that's *so* readable it's
popular for human-edited files (config files), files mainly for human
consumption (log files), and as a poor man's database editor (for
roundtripping things that are hard to edit in their native format,
I mean).

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

Mike,

What browser do you have?  Even Internet Explorer lets me edit with PgUp/PgDn.
What do you think of Why's suggestion to use the browser that allows you to edit
pages with vim?

MoinMoin already converts its wiki markup to HTML, so you don't have to.  You
can use "view source" to capture the HTML rendering, or write a script to get it
(either client-side or server-side).  The only thing you need to do is remove
the boilerplate, although some of the boilerplate lends a nice consistent
YAMLish branding to your work.

The only PyYaml tutorial that I recall writing myself is this one:

http://wiki.yaml.org/yamlwiki/YamlSlideShowIdea

The source for the presentation is pure YAML, which is in the spirit of Stefan's
suggestion to use yod, which is another great idea.   Of course, you are welcome
to steal from and/or extend the SlideShowell presentation.  The SlideShowell app
is a great example of cross-platform YAML, by the way.  It has versions in both
Ruby and Python.

Cheers,

Steve

P.S. Have you seen Why's yod documentation for the Ruby version of YAML?  It's
really impressive stuff, although it would be nice if he could somehow link up
the yod to some sort of web interface, so that people could edit it.

From: "Mike Orr" <iron@...>
> On Thu, Sep 12, 2002 at 02:43:25AM +0000, Clark C. Evans wrote:
 >
> >     In essence, we are now a "general purpose" syntax
> >     rather than a object serialization language.
>
> And this is a Very Good Thing.  We've doubled YAML's theoretical market
> penetration, allowing it to flourish closer to its full potential.  And
> also formally codified some things many people thought YAML had all
> along, things that attracted them to YAML in the first place.
>
> The first paragraph of the tutorial will mention YAML's twofold purpose,
> both as a serialization format that's cross-platform and happens to be
> human readable, and as a parser for a syntax that's *so* readable it's
> popular for human-edited files (config files), files mainly for human
> consumption (log files), and as a poor man's database editor (for
> roundtripping things that are hard to edit in their native format,
> I mean).
>

Don't forget about documentation formats like YOD and SlideShowell.   Also, Ingy
and I have made extensive use of YAML as a debugging tool.

We made a Wiki page describing Yaml Use Cases a while back:

http://wiki.yaml.org/yamlwiki/YamlUseCases

I am adding your examples now.

Cheers,

Steve

From: "Clark C. Evans" <cce@...>
>
>     It is important not to underestimate what happened
>     this week.  YAML has made a significant shift.  Two
>     weeks ago, it was a tool for serializing data in
>     modern scripting languages.  With a focus of
>     interoperability so that a given text file would
>     represent a typed structure in several languages
>     at the same time.
>

You may be exaggerating the significance of the shift.  Quoting directly from
the abstract in the 9/1/2002 version of the spec:

"YAML(tm) (rhymes with "camel") is a straightforward machine parsable data
serialization format designed for human readability and interaction with
scripting languages such as Perl and Python. YAML is optimized for data
serialization, formatted dumping, configuration files, log files, Internet
messaging and filtering."

So, clearly multiple use cases for YAML were planned all along.  It could be
argued that data serialization only ever represented 17% of the usage.

Cheers,

Steve

On Thu, 2002-09-12 at 05:55, Steve Howell wrote:
> From: "Clark C. Evans" <cce@...>
> >
> >     It is important not to underestimate what happened
> >     this week.  YAML has made a significant shift.  Two
> >     weeks ago, it was a tool for serializing data in
> >     modern scripting languages.  With a focus of
> >     interoperability so that a given text file would
> >     represent a typed structure in several languages
> >     at the same time.
> >
> 
> You may be exaggerating the significance of the shift.  Quoting directly from
> the abstract in the 9/1/2002 version of the spec:
> 
> "YAML(tm) (rhymes with "camel") is a straightforward machine parsable data
> serialization format designed for human readability and interaction with
> scripting languages such as Perl and Python. YAML is optimized for data
> serialization, formatted dumping, configuration files, log files, Internet
> messaging and filtering."
> 
> So, clearly multiple use cases for YAML were planned all along.  It could be
> argued that data serialization only ever represented 17% of the usage.

i'll come in here on the middle line real quick. my bet is that data
serialization was 50% of the goal. indeed there was the other 50%
including all the other possible use cases. that's why we ened up with
such a wonderful syntax. (if data serialization were the only original
goal then the syntax could have been garbage, who would have really
cared?)

-tom

p.s. i want to look in to this yod thing and remote editing. someone
built a remote editor tool for using vim to edit ruby scripts off a web
server so maybe that could be useful? plus if we use yaml, instead of
the markup, the syntax highlighter will be a nice touch. i'm willing to
help with this, but someone needs to guide me or at least get me started
down the right path.

cheers!


> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core
> 
-- 
tom sawyer, aka transami
transami@...

Brian Ingerson [mailto:ingy@...] wrote:
> My current thought is to keep an information model in the 
> spec, but have it use layman wording and limit it to the 
> basic models that we have today. Basically it would just be 
> describing what Perl/Python/Ruby programmers implicitly know: 
> how hashes/arrays/scalars work. The reason to do this is that 
> we can freeze the spec SOON.

+10. This means that the GRAPH model should not preserve key order IMVHO.

> Then, I would propose that we work on a separate "Useful YAML 
> Information Models" document which we can spend more time on. 
> If you don't make a separation between that which has been 
> decided and that which is still very much debatable, then 
> you'll just stimey implementation.

The issue of XML-in-YAML definitely deserves its own spec. It is when you
try to cast XML in YAML that you really realize how complex it is and what
is the core of the trouble people have with it.

Have fun,

	Oren Ben-Kiki

On Thu, 2002-09-12 at 10:39, Oren Ben-Kiki wrote:
> Brian Ingerson [mailto:ingy@...] wrote:
> > My current thought is to keep an information model in the 
> > spec, but have it use layman wording and limit it to the 
> > basic models that we have today. Basically it would just be 
> > describing what Perl/Python/Ruby programmers implicitly know: 
> > how hashes/arrays/scalars work. The reason to do this is that 
> > we can freeze the spec SOON.
> 
> +10. This means that the GRAPH model should not preserve key order IMVHO.

please see my new models page. i think it effective offers a way in
which we can move foward with both possibilites. to not take the
"logical" model into account will cause many unneccessary changes down
the road should it ever became an acceptable alternative.

and honestly its not that hard. i gues that's what gets me. on the one
hand i am saying, here have it just like you want it. nothing's
different, only that now it's quite precisely layed out. over here i'm
going to offer a "fork in the road" at the final model level so that you
can use order and duplication restraints too. but for all extensible
purposes you can jsut ignore that entirely if you want.

in my proposal/spec i call this version 1 (withou logical) and. version
2 (with logical), for lack of better names.

> > Then, I would propose that we work on a separate "Useful YAML 
> > Information Models" document which we can spend more time on. 
> > If you don't make a separation between that which has been 
> > decided and that which is still very much debatable, then 
> > you'll just stimey implementation.
> 
> The issue of XML-in-YAML definitely deserves its own spec. It is when you
> try to cast XML in YAML that you really realize how complex it is and what
> is the core of the trouble people have with it.

i wonder how much easier it becomes with a logical model. the only real
difficulty i see is attribute vs child-tag.

-tom


> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core
> 
-- 
tom sawyer, aka transami
transami@...

Tom Sawyer [mailto:transami@...]  wrote:
> > +10. This means that the GRAPH model should not preserve key order 
> > +IMVHO.
> 
> please see my new models page. i think it effective offers a 
> way in which we can move foward with both possibilites. to 
> not take the "logical" model into account will cause many 
> unneccessary changes down the road should it ever became an 
> acceptable alternative.

I'll rephrase. OI don't really care about models. I care about
implementations, tools, and allowed operations on YAML files.

I want that everywhere where order is significant in a YAML file there
should be a '- '. I want tools and people to be free to re-order anything
else.

This simplifies tools and implementations _a lot_. And it costs very little.

Have fun,

	Oren Ben-Kiki

On Thu, 2002-09-12 at 11:18, Oren Ben-Kiki wrote:
> Tom Sawyer [mailto:transami@...]  wrote:
> > > +10. This means that the GRAPH model should not preserve key order 
> > > +IMVHO.
> > 
> > please see my new models page. i think it effective offers a 
> > way in which we can move foward with both possibilites. to 
> > not take the "logical" model into account will cause many 
> > unneccessary changes down the road should it ever became an 
> > acceptable alternative.
> 
> I'll rephrase. OI don't really care about models. I care about
> implementations, tools, and allowed operations on YAML files.

models define implementations. implementations are used to create tools.

if you don't care abaout the model's then why bother with yaml at all?
just make up tools as you go along.

> I want that everywhere where order is significant in a YAML file there
> should be a '- '. I want tools and people to be free to re-order anything
> else.

'-' everywhere there is order? cool with me. that's excatly why it's
important to think about how the syntax integrates with the ideas of
order and duplication, should they be applicable, BEFORE it becomes a
big pain in the arse to do so.

> This simplifies tools and implementations _a lot_. And it costs very little.

i'll gladly pay you tuesday for a hamburger today ;)

-- 
tom sawyer, aka transami
transami@...

On 12/09/02 20:18 +0300, Oren Ben-Kiki wrote:
> Tom Sawyer [mailto:transami@...]  wrote:
> > > +10. This means that the GRAPH model should not preserve key order 
> > > +IMVHO.
> > 
> > please see my new models page. i think it effective offers a 
> > way in which we can move foward with both possibilites. to 
> > not take the "logical" model into account will cause many 
> > unneccessary changes down the road should it ever became an 
> > acceptable alternative.
> 
> I'll rephrase. OI don't really care about models. I care about
> implementations, tools, and allowed operations on YAML files.
> 
> I want that everywhere where order is significant in a YAML file there
> should be a '- '. I want tools and people to be free to re-order anything
> else.

Also, an ordering-schema could go a long way towards filling in the gaps. You
could retain order without having to code it in the document or make it part
of the model. Right Steve?

Cheers, Brian

PS Steve and I worked on a top-secret XML/YAML project that did the above.

On 12/09/02 11:53 -0600, Tom Sawyer wrote:
> On Thu, 2002-09-12 at 11:18, Oren Ben-Kiki wrote:
> > Tom Sawyer [mailto:transami@...]  wrote:
> > > > +10. This means that the GRAPH model should not preserve key order 
> > > > +IMVHO.
> > > 
> > > please see my new models page. i think it effective offers a 
> > > way in which we can move foward with both possibilites. to 
> > > not take the "logical" model into account will cause many 
> > > unneccessary changes down the road should it ever became an 
> > > acceptable alternative.
> > 
> > I'll rephrase. OI don't really care about models. I care about
> > implementations, tools, and allowed operations on YAML files.
> 
> models define implementations. implementations are used to create tools.

I disagree. Models should summarize common implementation patterns. Why
define all the possibilities up front? We need to be very flexible
here. 

The best approach IMO is to define only the clearest models we
see so far. And only for tools that really care about interoberability. 90%
don't. For instance, to me, YAML.pm is simply Perl's best Data Dumper. And
that's what I use it for 90% of the time. I certainly didn't need a model to
implement it. 

On the other hand, ypath needs YAML.pm to provide a representation that will
work with it. So we define a model and try to get as much mileage out of it
as possible. No need to define more than that at this point.

> if you don't care abaout the model's then why bother with yaml at all?
> just make up tools as you go along.

Better stated: why bother with YAML models at all? just make up tools as you
go along. and when there are enough tools that look alike, you can
standardize on a model.

> 
> > I want that everywhere where order is significant in a YAML file there
> > should be a '- '. I want tools and people to be free to re-order anything
> > else.
> 
> '-' everywhere there is order? cool with me. that's excatly why it's
> important to think about how the syntax integrates with the ideas of
> order and duplication, should they be applicable, BEFORE it becomes a
> big pain in the arse to do so.

This is one reasonable solution. It's standalone, and schema-unaware. But it
also strays from native usage. An array of single key maps is just not very
normal. 

Alternatively, if a formatting-schema (for lack of better term) was
associated with the document it could specify the order of maps.

With this combination of possibilities, YAML is definitely a useful tool in a
slew of domains, without having to be so complex as to serve every single one
in a uniform manner.

> 
> > This simplifies tools and implementations _a lot_. And it costs very little.
> 
> i'll gladly pay you tuesday for a hamburger today ;)

You've successfully made me hungry. Time for lunch.

Cheers, Brian

On Thu, 2002-09-12 at 12:00, Brian Ingerson wrote:

> Also, an ordering-schema could go a long way towards filling in the gaps. You
> could retain order without having to code it in the document or make it part
> of the model. Right Steve?

oren's post got me thinking about that too. it might be good to flush
this out a bit.

i think there's is an important difference in the order that a schema
can define and that of an order collection. a schema says "order these
things in such-and-such a manner" and that can be very precisely defined
or loosly defined. e.g. alphabetically, or just that a certain element
must always appear prior another but within that contraint anything
goes. schemas are geared for use in the human side of things --the xml
type uses for instance. on the other hand, an ordered map simply means
that the order is significant and so needs to stay constant from one
transfomation or proccess to the next. so a pretty-printer would know
not to alter the order of such a collection. it's a simple, and
resonably common, binary switch: "order significant/order not
significant." which would be overly complexified if relegated to a
schema.

-tom

> Cheers, Brian
> 
> PS Steve and I worked on a top-secret XML/YAML project that did the above.
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Yaml-core mailing list
> Yaml-core@...
> https://lists.sourceforge.net/lists/listinfo/yaml-core
> 
-- 
tom sawyer, aka transami
transami@...

On Wed, Sep 11, 2002 at 11:00:08PM -0700, Mike Orr wrote:
> The first paragraph of the tutorial will mention YAML's twofold purpose,
> both as a serialization format that's cross-platform and happens to be
> human readable, and as a parser for a syntax that's *so* readable it's
> popular for human-edited files (config files), files mainly for human
> consumption (log files), and as a poor man's database editor (for
> roundtripping things that are hard to edit in their native format,
> I mean).

Brian convinced me that the second set of uses really are
"serialization" after all.  So I guess the real difference between the
two is that whether the human is expected to read the .yml file.  In
the first case (as a replacement for pickle, Storable, Serializable and
XML-RPC), the YAML data is rarely read by a human unless he's debugging
the program, whereas in the second case (config files, log files,
database editor) the primary purpose of the .yml file is for human
reading/editing.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Thu, 2002-09-12 at 12:26, Brian Ingerson wrote:
> I disagree. Models should summarize common implementation patterns. Why
> define all the possibilities up front? We need to be very flexible
> here. 

two words: scientific method

> The best approach IMO is to define only the clearest models we
> see so far. And only for tools that really care about interoberability. 90%
> don't. For instance, to me, YAML.pm is simply Perl's best Data Dumper. And
> that's what I use it for 90% of the time. I certainly didn't need a model to
> implement it. 

perhaps that's the problem. is the logical model not clear to you?

concering interop: why is SOAP such a big deal? you see, since you are
going to tell me what YOU care about, guess i'll reiterate what I care
about: a YAML replacement for SOAP. it's called ONI. the more
interoperatible yaml is the better.

> On the other hand, ypath needs YAML.pm to provide a representation that will
> work with it. So we define a model and try to get as much mileage out of it
> as possible. No need to define more than that at this point.

okay, go ahead drive until you run out of gas. ZAYML will come along i
guess soon enough.

is ypath anywhere near complete? i've been hearing so much about what
ypath needs i'm wondering if i missed the stone it's set in somewhere?

> > if you don't care abaout the model's then why bother with yaml at all?
> > just make up tools as you go along.
> 
> Better stated: why bother with YAML models at all? just make up tools as you
> go along. and when there are enough tools that look alike, you can
> standardize on a model.

the reality is even if your model's not written on paper, it is still in
your head, and is reflected in the implementation. you can't escape
models. they're how we think.
 
> > > I want that everywhere where order is significant in a YAML file there
> > > should be a '- '. I want tools and people to be free to re-order anything
> > > else.
> > 
> > '-' everywhere there is order? cool with me. that's excatly why it's
> > important to think about how the syntax integrates with the ideas of
> > order and duplication, should they be applicable, BEFORE it becomes a
> > big pain in the arse to do so.
> 
> This is one reasonable solution. It's standalone, and schema-unaware. But it
> also strays from native usage. An array of single key maps is just not very
> normal. 
> 
> Alternatively, if a formatting-schema (for lack of better term) was
> associated with the document it could specify the order of maps.
> 
> With this combination of possibilities, YAML is definitely a useful tool in a
> slew of domains, without having to be so complex as to serve every single one
> in a uniform manner.

talked about this in last post.
 
> You've successfully made me hungry. Time for lunch.

what'chya have? i just ate a ham and swiss tortillia. never had that
before. pretty good. :)

-tom


-- 
tom sawyer, aka transami
transami@...

On Thu, Sep 12, 2002 at 07:28:13AM -0400, Steve Howell wrote:
> P.S. Have you seen Why's yod documentation for the Ruby version of YAML?  

No, I haven't.  Where is it?

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Thu, 2002-09-12 at 12:53, Brian Ingerson wrote:
> I totally agree with your pretty printing use case. But that can be
> accomplished in code without requiring a formal model to be worked out with
> all its ramifications and deep seated philosophical issues.

all the ramifications? deep seated philosphical issues? this isn't
Hegel! :)
 
> YAML.pm has a loader option to load a YAML document into a shadowed native
> graph. The shadowing preserves the key order. So when you dump the graph, key
> order is preserved. 
> 
> YAML.pm also has an option to load a YAML document into non-native graph that
> supports key order among other things like style and type uri. This would be
> useful to support a YPATH perhaps.

you already have most of the functionailty built into the tool already!
for heaven sakes, man! what is the problem?

> We only need models when a particular cross-platform application with
> widespread impact (like ypath) requires it. Frankly, ypath as a specification
> hasn't even been started in earnest.

perhaps it a good time to ride that pony backwards and see how the
implementation effects the model. i'll bite.

-- 
tom sawyer, aka transami
transami@...

On Thu, Sep 12, 2002 at 11:26:13AM -0700, Brian Ingerson wrote:
> > > I want that everywhere where order is significant in a YAML file there
> > > should be a '- '. I want tools and people to be free to re-order anything
> > > else.
> > 
> > '-' everywhere there is order? cool with me. that's excatly why it's
> > important to think about how the syntax integrates with the ideas of
> > order and duplication, should they be applicable, BEFORE it becomes a
> > big pain in the arse to do so.
> 
> This is one reasonable solution. It's standalone, and schema-unaware. 

"If you want your values to stay in a certain order, put a '- ' before
them.  Then YAML tools and the native data structures they create will
preserve the order.  Otherwise, the tools are free to reorder the values
or put them into an unordered data structure if they wish."
  RATIONALE: many times the order is insignificant, many programming
  languages do not have an ordered hash type built in but only an
  unorderd hash type, and using a third-party ordered hash type may
  cause inefficiency if it's used when not strictly needed.

It's an easy-to-remember rule you can frame, and at the syntax level
it's just one more "- ".  It allows people to selectively preserve
order without a schema, and when a schema alternative becomes available
they can switch to it instead.  At that point, it won't be a lame-duck
syntax because it consists only of syntaxes we have to support anyway.
Meanwhile, it lets us concentrate on other things.

> But it also strays from native usage. An array of single key maps is
> just not very normal. 

True, but if we try to think of it as a single unit rather than a
sequence of 1-element mappings, it's not so bad.  Can we somehow get
the API to recognize it as a single unit?

At the native level, it has to load into:

1) A list of one-element dictionaries  (ugly)
2) A list of pairs  (also ugly)
3) A dictionary  (loses order on some platforms, always loses dupes)
4) A dictionary of lists (loses order on some platforms, preserves dupes) 
5) A third-party ordered dictionary (with or without dupe preservation)

There's no other choice, and (1) is as good a default as any.  And it
ensures the same structure will be written back to the YAML file.  It
violates the "one unit" principe, but that's because of the limitations
of the programming languages we've chosen to support.

And hopefully the user-level loader/importer/reader will have an option
to load it into a dictionary in one step so you don't have to.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

From: "Tom Sawyer" <transami@...>
> [...] on the other hand, an ordered map simply means
> that the order is significant and so needs to stay constant from one
> transfomation or proccess to the next. so a pretty-printer would know
> not to alter the order of such a collection. it's a simple, and
> resonably common, binary switch: "order significant/order not
> significant." which would be overly complexified if relegated to a
> schema.
>

I am glad to see another person recognizing pretty-printers (and cousins) as a
compelling use case for ordered-key toggles.

Schema solutions are too complex, and often impossible, for generic pretty
printers.  Also, pretty printers often go hand-in-hand with hand-edited files,
where putting all the dashes is a particular nuisance.

This is mostly YAGNI now, so don't hold up the spec, just glad to see that some
one is on the same page as me.

Cheers,

Steve

iron@... [mailto:iron@...] wrote:
> "If you want your values to stay in a certain order, put a '- 
> ' before them.  Then YAML tools and the native data 
> structures they create will preserve the order.  Otherwise, 
> the tools are free to reorder the values or put them into an 
> unordered data structure if they wish."
>   RATIONALE: many times the order is insignificant, many programming
>   languages do not have an ordered hash type built in but only an
>   unorderd hash type, and using a third-party ordered hash type may
>   cause inefficiency if it's used when not strictly needed.

+10
 
> True, but if we try to think of it as a single unit rather 
> than a sequence of 1-element mappings, it's not so bad.  Can 
> we somehow get the API to recognize it as a single unit?

Sure we could. Remember the loader is free to choose any native data type it
wants _for any node_, not only scalars.

> At the native level, it has to load into:
> 
> 1) A list of one-element dictionaries  (ugly)
> 2) A list of pairs  (also ugly)
> 3) A dictionary  (loses order on some platforms, always loses dupes)
> 4) A dictionary of lists (loses order on some platforms, 
>    preserves dupes) 
> 5) A third-party ordered dictionary (with or without dupe 
>    preservation)

Or, in PHP, into whatever-they-call-it that gave raise to this thread in the
first place.

> And hopefully the user-level loader/importer/reader will have 
> an option to load it into a dictionary in one step so you 
> don't have to.

Which is what I'd expect a reasonable PHP implementation to do.

Have fun,

	Oren Ben-Kiki

Brian Ingerson wrote:
> > Also, an ordering-schema could go a long way towards filling in the 
> > gaps. You could retain order without having to code it in 
> > the document 
> > or make it part of the model. Right Steve?

And Tom Sawyer [mailto:transami@...] wrote:
> i think there's is an important difference in the order that 
> a schema can define and that of an order collection.

I agree with Tom. Examples:

A schema may say that 'x' comes before 'y' in every Point object (ok,
contrived).

A schema can't say that a specific list of "employee: salary" pairs in a
specific document should be sorted by the seating order in the last meeting,
except for the CEO that appears first (again, contrived).

IMVHO a schema describes a class of documents rather than completes the
information of a single document. Using an explicit '- ' notation for the
latter solves the problem.

Have fun,

	Oren Ben-Kiki

From: "Brian Ingerson" <ingy@...>
>
> Also, an ordering-schema could go a long way towards filling in the gaps. You
> could retain order without having to code it in the document or make it part
> of the model. Right Steve?
>

So, if I weren't worried about 100% YAML compliance, I'd be able to take these
documents...

---
Name: Steve Howell
Hometown: Columbia, MD
Languages: [Python, Ruby, Java]
---
First name: Brian
Last name: Ingerson
Language: Perl
---
Last Name: Ganapathy
First Name: Vijay
Birthplace: India
Languages:
 - English
 - French

...and I'd be able to put them through a slightly extended YAML loader and feed
them into a pretty printer and the keys would be preserved.  No schema would
help there.  The key order was determined by the users who made their documents.

Why do we need an extension to a YAML loader, when theoretically I could write
my own custom loader on top of a lower exposed layer?  Well, because I'm lazy.

So, to answer your question, yes, an ordering schema would go a long way to
filling in the gaps, but it wouldn't go far enough.  This doesn't mean the YAML
spec has to address this now, but this use case may come up for YAML
implementers who want to give a little help to application developers.

Cheers,

Steve

On 12/09/02 20:34 -0400, Steve Howell wrote:
> From: "Brian Ingerson" <ingy@...>
> >
> > Also, an ordering-schema could go a long way towards filling in the
> > gaps. You could retain order without having to code it in the
> > document or make it part of the model. Right Steve?
> 
> So, if I weren't worried about 100% YAML compliance, I'd be able to take these
> documents...
> 
> ---
> Name: Steve Howell
> Hometown: Columbia, MD
> Languages: [Python, Ruby, Java]
> ---
> First name: Brian
> Last name: Ingerson
> Language: Perl
> ---
> Last Name: Ganapathy
> First Name: Vijay
> Birthplace: India
> Languages:
>  - English
>  - French
> 
> ...and I'd be able to put them through a slightly extended YAML loader
> and feed them into a pretty printer and the keys would be preserved.
> No schema would help there. The key order was determined by the users
> who made their documents.
> 
> Why do we need an extension to a YAML loader, when theoretically I
> could write my own custom loader on top of a lower exposed layer?
> Well, because I'm lazy.
> 
> So, to answer your question, yes, an ordering schema would go a long
> way to filling in the gaps, but it wouldn't go far enough. This
> doesn't mean the YAML spec has to address this now, but this use case
> may come up for YAML implementers who want to give a little help to
> application developers.

It's been a long day, and I don't remember the context of the above
statement. But I think I can clarify. Let me restate it:

    An ordering-schema could go a long way towards filling in the gaps for
    people who want pedantic, iron-clad, spec endorsed compatibility.

I'm with you. I'd just use YAML.pm to load documents into a custom,
key-ordered representation, that didn't have any model.

And who knows? If Why and others did the exact same thing, then maybe someday
when we are all really bored, we could codify this into an official model.

Anyway seems like we're in violent agreement here.

Cheers, Brian

From: "Brian Ingerson" <ingy@...>
> > 
> > So, to answer your question, yes, an ordering schema would go a long
> > way to filling in the gaps, but it wouldn't go far enough. This
> > doesn't mean the YAML spec has to address this now, but this use case
> > may come up for YAML implementers who want to give a little help to
> > application developers.
> 
> It's been a long day, and I don't remember the context of the above
> statement. But I think I can clarify. Let me restate it:
> 
>     An ordering-schema could go a long way towards filling in the gaps for
>     people who want pedantic, iron-clad, spec endorsed compatibility.
> 
> [...]
>
> Anyway seems like we're in violent agreement here.
> 

Definitely!

Sorry I didn't pick up on the point that you were making before, but now I do.

Cheers,

Steve