RE: [Yaml-core] Summary of the IRC session

[Oren Ben-K. <or...@ri...>]

It is the holiday season here (Hebrew new year's) so I didn't log in for a
measly two days. As usual when I do that, it takes me two days to catch up
:-) So I won't be posting my promised summary of how the
optional-type-family spec would look like; instead I'll try to address the
issues Clark has raised, with the occasional quote from other people.

First, timestamps in the core spec, which I think is the more minor issue:

Clark C . Evans [mailto:cc...@cl...] wrote:
> | - As for timestamps...
> | They aren't different in any way from the use cases for 
> | using URLs in 
> | YAML, or IP addresses, or E-mail addresses, etc.
> 
> Time is different.  URLs and such can be compared for 
> equality directly by string comparison and various operators 
> arn't defined. For timestamp it is more complicated.

Wrong. URLs can *NOT* be compared for equality using string comparisons, due
to %XX encoding. Or, they both can be compared with strcmp, as long as the
author sticks to a reasonable convention.

> I've yet to hear one timezone issue...

I think that time without proper timezone is like currency without currency
code. Sure, "most" people deal only with one currency/time zone, but that's
like saying "most" strings don't contain non-printables (or non-ASCII) so we
don't need escaping and UTF. The full model for strings is Unicode; the full
model for time requires time zone and for currency requires currency code.
Even your own application has several app-specific date/time type handling,
since current YAML timestamps do not suffice.

> I've been very up front over the last year about 
> compatibility with most SQL databases, hence my support for 
> NULL, BOOLEAN, INTEGER data types.  FLOAT isn't perfect since 
> many databases don't support float well (or do so via fixed 
> point).  As part of this support a general TIMESTAMP is very 
> common and indeed core.

AFAIK most databases also have DATE and TIME data types, which are used as
commonly (if not more).

I'm not against us defining a standard time type (or for that matter
currency); I just want it to be in a separate spec from the core -
_properly_. The issue is complex enough to warrant this. If some subset of
that separate spec would be "more useful", fine. But we'd better consider
the whole issue before we define this subset.

> So, for the last call we have a serious question:
> 
>    Do we wait for a revisit of the type system for 1.0?  If so, then 
>    I think we should recall the "last call" status as this will
>    be a fundamental change and one that may take further revisions
>    to get right. 

I'm not certain it is that complicated... How about you give me a breather
to actually draft a proposal so we'll be able to evaluate how much of an
impact it actually has? I suspect it would be rather minor (and likewise to
the implementation).

>    Else, I'd like to make the two changes to timestamp (seconds
>    optional and using midnight) to make it more useable.   But
>    not make any other serious changes.

I think it is much better to start a separate spec for date/time/stamp/span
types.

> In any case, we can continue our discussion of a new implicit
> type mechanism...   

Right:

> In my opinion, for greatest compatibility, it
["implicit typing"]
> should not be 
> done in the loader as this would cause each language to have 
> their own implicit typing mechanism; giving code duplication 
> and thus implying interoperability concerns.  Indeed, this
> step could be done in a shared "C" libyaml which all native 
> bindings leverage.

No. The loader has language-specific code anyway in deciding the native data
type to assign to each node. Keep in mind it is explicitly allowed to do so
on a per-node basis, i.e. not all nodes with the same type family
necessarily have the same native type. I don't see how the logic to do that
in Java or Python is shared, and how it can be in libyaml.

The whole idea is to consider the type family with the name "" as a normal
type family (almost) and let the loader do whatever it wants for it (as it
does for every other type family). If it wants to represent "2002-08-25"
using one native type and "xyzzy" using another that's no different from
using one native type for !int 12345678901234567890 and another for !int 12.

> |     1) I want 2002-10-12 to be implicitly a string, because it's 
> | simpler and more portable.
> 
> Why not have 34.3 be implicitly a string beacuse its simpler 
> and more portable.

That's *exactly* what the proposal is about :-) It *would* be a string all
the way up to the loader. If you want to load it into an integer native data
type, fine. If you want to load it into a string (e.g., Perl, TCL, Korn
Shell), also fine. As long as you preserve the fact that this had no
transfer method and its string representation (which comes for free in
Perl), that's a valid round-tripping tactic.

> I think there are alot of issues to work out with the 
> pluggable implicit type mechanism.  For instance, how do you specify 
> the implicit type rules so that the document is portable?

First, you hang a framed reminder saying "I shall consider schema-blind and
schema-aware code separately" in some prominent place :-) You then re-read
it before and after considering any of the points below. It is the key to
the whole thing.

Then, you switch from the false dichotomy:

> I see handling implicits as two complementary processes:
> 
>   1) finding a !transfer|method given a regular expression
>      or perhaps even a path expression.
> 
>   2) binding the !transfer|method to a native data type.

To a unified process:

"The loader uses the transfer method as a hint as to what native data type
to use for the particular node."

For round-trip the dumper should be able to reconstruct the type family part
(not the format!) of the transfer method, or the lack of it if there wasn't
any, and to keep the same value ("same" under the type family definition; no
type family => "same" == !strcmp).

You may protest that changing 010 to 8 fails this, and I'd say that as far
as generic YAML is concerned the document was changed but as far as the
application is concerned it wasn't (another reason why the native
model/representation is required). This is exactly the same as loading a URL
with %XX encoding and dumping it differently. BTW, this implied by the
current spec...

So, how is the document portable? Because schema-blind generic YAML tools
treat these scalars as strings and preserve them character-for-character.
That's as portable as you can get - I don't know of anything that is more
portable. And if your application is schema aware, there's no problem. No
need for #IMPLICIT:OFF (shudder).

> Certainly you can keep the concerns "tied together" here, but 
> this would then limit the interoperability of yaml texts which
> use implicit typing.

It certainly does not! What is more interoperable than a string? Unless you
imply reduced interoperability between different schemas using schema-aware
applications. Well... different schemas are not interoperable!

> Also, I'm concerned that this pluggable type mechanism will 
> become something similar to DTDs where you can't really use 
> the given document unless you know all of the implicit rules.

It is self-evident that a schema blind tool can't *create* content. It also
can't really "understand" the document semantics. It is restricted to a much
lower level of functionality. The YPATH/CYATL tools are no exception,
because schema-awareness comes through the scripts they accept rather from
the tool itself.

It is also self evident that a schema aware tool has no such problem. For
example, a YPATH/CYATL tool given proper schema-aware scripts, _can_ achieve
the proper semantics. In this view, implicit typing is just a small part of
the "schema awareness" that such tools inevitably require in order to
provide the required level of functionality.

You seem to want the impossible; to want that all schema-blind tools to
provide semantically correct operations on a document without knowing the
schema or being given any external information about it. Trying for this
impossible goal will only twist YAML out of shape, and you'd still not get
it.

Instead we should focus on the question of "how should generic tools be
given the required schema awareness?". For YPATH, this may mean plug-in
comparison operators for scalars _and_ containers. For example, any GIS data
would require "A near B" operator working on coordinates. "A same-URL B"
would require %XX handling. A "same-URI" comparison would need a table of
special characters per URI scheme. And so on.

CYATL would need the same for the input side; it gets the schema-awareness
of the output side for free, since that's exactly what a CYATL script is.

This approach keeps everything clean and achieves maximal portability when
the schema is not known.

> > I'm very worried about this layer
[Implicit Typing]
> > if it requires access a third 
> > document in order to give information about what nodes have what 
> > types. This approach is very good for human readability, but costs 
> > quite a bit as far as administration goes (the need for a local 
> > registry, etc.) especially if the process works with more than one 
> > language.

Well, sometimes it is free (e.g., CYATL). Also, as Mike Orr
[mailto:ir...@ms...] wrote:

> Does the schema have to be a "document"?

Right on. A schema-aware application can be schema-aware in any way it
wants. What Clark seems to have in mind is a "universal schema-aware tool",
something that is flat out impossible without some external document. But
there's just no helping this anyway.

> This stuff is non-trivial if done right and will take us 
> quite some time to work out.  I'd like to finish our built-in 
> implicit type system and then work out how this 
> alternate-typing can be added later (after we have YPATH and 
> SCHEMA) with a directive or other mechanism.

Ick. The right way is to eat our own dog food - define a proper "plug-in"
typing mechanism and use it for our own data types, so we'll _know_ it
works. Doing our types one way and telling everyone else to do it using
another is wrong.

> DTDs caused XML to have 3 states: (a) DTD unaware, (b)
> DTD aware with default attributes, (c) DTD aware and 
> validated.    My fear is that this implict system will
> create a two level system, one where the implied type 
> information is not used (cuz its not available) and one where 
> implied types are used.

What you fear is what I accept as inevitable. Schema-aware and schema-blind
again. It isn't just the implicit types. It is the whole semantics of the
document. You _must_ have a two-level system.

As Steve Howell [mailto:sh...@zi...] wrote:

> 1) One layer of code only understands maps, sequences, 
> aliases, and untyped scalars.  These concepts make up the 
> very heart of YAML, in my humble opinion.
> ...
> 2) Another layer of code will deal with type conversions.  
[From the above to whatever native type].
> How does layer #2 decide how all these types get mapped?  
> Well, it depends.  But I'm gonna do a lot work to make the 
> APIs as simple as possible, but flexible enough for 
> real-world demands.

That's exactly what I'm suggesting. So:

> If someone wants to write a custom loader and do custom 
> type detection via paths/regex then we are not obligated to 
> make such type detection work with generic YAML tools.

Exactly! That's what makes them generic! For example, a YAML-pretty-print
has no need whatsoever for such detection. The generic tool works in layer
(1) above.

OK, YPATH will need to be provided with appropriate comparison operators to
be used _at that level_. But even it will _not_ load a !point {x: 1, y: 2}
map to a Point structure, even if has loaded an "A near B" operator. And
besides, the paths given to YPATH are anything but schema-blind - it is the
YPATH _tool_ that is schema-blind.

> However, if we provide a pluggable implicit type detection mechanism, 
> then we must think hard about how this will impact the 
> generic model.

Right - "Make the type family optional". I was going to go over the spec
today doing this in detail, but instead I had to read >100 messages and
write this looong reply :-) I hope it addresses most of the issues, but I'll
still need to write a proper proposal.

> It is completely unacceptable to me that implementations can
> pick/choose how they want to do implicit typing.  Either we have
> it or we don't.

I fail to see why. It is like saying that you find it unacceptable that some
applications load some values to BigInt or Long instead of to Integer (or
whatever). Who cares? I agree with Mike Orr:

> | That's what I've been thinking.  Whether to use implicit 
> | types is not 
> | a function of the document, it's a function of the application.

Only I'd change the word "application" to "schema" (which should address
your worry about this being code-driven). "Schema" not in the sense of a
particular document, but in the sense of the abstract definition of the
semantics of the format of the file. Its incarnation as a concrete schema
document (or code, or any mixture thereof) will usually evolve almost as
fast as the application code - but the _intent_ will remain fixed (more or
less). It is there that the definition of implicit types resides, together
with many other things (like the fact that 'Point' has 'x' and 'y' in UTM-36
coordinates, or whatever).

Maybe we need two words. How about "semantics" for the above abstract term
and "schema" for a concrete schema document?

> | For instance, maybe YAML is only completely interoperable under the 
> | guidance of a schema.
>
> I'd rather not...

It makes perfect sense if you replace the word "schema" with the word
"semantics". Which is, I suspect, what Brian had in mind. It is
_inevitable_. It makes much less sense in the "concrete document" definition
of a schema.

> but requiring a schema for particular types of
> tasks may well be required.

Schema-blind and schema-aware again. Exactly. IMVHO implicit types should
not harm schema-blind tools, since they would have to allow for loadable
operators _anyway_ (e.g., the "Point-near-Point" example). And implicit
types obviously don't harm any schema-aware tools. So where's the harm?

> I'd like to view "schema" as a "validation" concept not as
> one that provides additional information.  So, something providing
> type information is more of a transform. 

I agree with "concrete schema document is for validation". But think, does
that really exclude implicit types? What is the big difference from saying
"points have 'x' and 'y' coordinates" and saying "'x's value must match the
following regexp"?

Sticking with schema-is-validation, it becomes obvious that such a "schema"
is just a _part_ of the "semantics". For example, a (validation) schema will
not tell you that the 'x' value is a UTM-36 coordinate, or that the 'center'
data member means "the geometric center of the shape when drawn in this
coordinate system", and so on. Like the schema defines the keys of a
mapping, for validation, but not their semantics (and hence not the exact
mapping to native data types such as 'map point'), the schema defines the
regexps for allowed values for certain fields, but not their semantics (the
exact mapping to native data types such as 'int').

Each schema-aware (I should say, semantics-aware) application decides how to
map the nodes to native data types to preserve the _semantics_, not just the
_schema_. Deciding on the mapping of implicit types is just a part of this
decision making. And none of these decisions are going to be provided inside
the validation-schema document - it is out of its scope. Hence
application-specific implicit types are not the problem you fear they are -
or maybe that the problem you fear is worse than you originally thought
since it applies to explicit types as well...

Think about it this way. Is it OK for me to write two Perl applications, one
which maps !perl/Foo to one particular Perl class and another that maps it
to a different one? It seems this is OK as long as I round trip the data
correctly and obey the intended semantics of the document. So, what's the
problem with me doing the same to implicit types?

> Just for musing, what if we made all implicit typing
> be explicit.  Is there some nice syntaxish way to do
> this without causing too much distress?
[e.g. !, (), etc.]
> Icky.

You said it :-)

> Ok.  What if we had a directive at the top which "enabled" 
> particular implicits?

Then "Icky" would be an understatement :-)

I won't quote the various horrid proposals (!, (), #IMPLICIT, #TYPE,
#CONVERT and so on). Don't you see that #CONVERT is just XML's DTD in sheep
clothing (when embedded inside the document)? Not to mention the backward
compatibility you've tossed out the window.

Arrrggghhhh!!! (Brian warned me I'll have kittens. I'm having _dinosaurs_).

The whole question only arises because of the artificial separation of
implicit typing from conversion to native typing, and is a great example of
how far out of shape you'd have to twist YAML and still fail to gain the
impossible goal of having generic tools be semantics-aware.

OK, so far this post was about disagreeing with most of what you guys agreed
on in the last few days. I'll note that there were quite a few people who
didn't like the approach, besides me... The following isn't the proper
proposal I promised (since it has been 4 hours already since I started
working on this). Consider it an executive summary :-)

- Syntax model stays the same.

- Serial model stays the same, type family is optional, format allowed for
collections.

- Generic model stays the same, type family is optional, format allowed for
collections.

- Native model is renamed to "native representation"; wording changes to
explain this refers to the way the YAML generic model is realized in native
data structures and that some code is required to "extract" the Generic
model information from this native representation ("Viewer" or "Dumper").

- The diagram stays the way it is.

- The Loader makes a decision on the native data type used to store the node
based on (1) the serial model path to it; (2) its content; (3) the transfer
method, if any.

- For round tripping the Dumper/Viewer should reconstruct the same transfer
method (up to format, which may be different); the value (again, up to
format). If format changes the value needs stay the same under the document
semantics.

- Semantics-blind YAML tools work at the generic model level. They can't do
anything with values other than strcmp-ing them in the hope that the format
is the same (even for a simple thing like integers this fails more often
than not).

- To construct semantics-aware tools, one must provide some information
about the semantics, either as code or as declarations.

- For example, YPATH should allow plug-in operators - for either implicit or
explicit types. Hence a YPATH/YQUERY could allow a plug-in called
"Point-near-Point" as well as "integer-greater-than"; both are useful and
neither have any special status. Call such plug-ins the "comparison-schema".

- As another example, a generic loader would allow the loading of
instructions on how to assign a native type to each node. Call such
instructions the "loading-schema".

- *This is not a transformation*. In particular, typing schema specifies how
to load explicitly typed nodes as well as implicitly typed ones.

- Finally, a generic validator would interpret a document containing a
predicate that pronounces every YAML document as "valid" or "invalid" under
some validation-schema.

- A typing-schema is a separate animal from a validation-schema or a
comparison-schema. All these are but a part of the total semantics.

- All the above schemas imply running code. Some are easier than others to
express in a platform-neutral way (requiring an interpreter written for all
relevant platforms), but this is possible in principle for all of them.

- There's no way to provide the "full" document semantics as a collection of
documents, unless you include all the application code that works with the
document.

- Since each application can load any node (with either implicit or explicit
semantics) to whatever native type it feels like (as long as it preserves
semantics), there's no need to distinguish implicit types with !, () or any
other special notation (note this is true in the current spec as well).

- In a word: this proposal makes YAML DWIM. If you want it to be. If you
want to be strict about it, add an explicit transfer method to each node,
and/or provide validation/typing/comparison schemas, and so on. If you don't
want to be strict, just treat everything as a string (but always preserve
the transfer method).

- There are no interoperability problems that I can see (if you see one,
please describe a scenario). This is compatible with existing YAML
documents. There are no new directives. There's no ugly syntax for
implicits. New implicits can be added later on without breaking
compatibility or harming existing applications.

- I'm dead set against (), #CONVERT, #IMPLICIT and so on - these are DTDs in
disguise and I'll have none of that.

- I want date/time in a separate spec, which would cover it properly; it is
likely one of the types listed there would be a UTC-based timestamp and
would prove useful for more applications than, say,
time-period-in-fortnights, just like some ISO date formats are more useful
than others. Likewise for currency (with currency code).

I hope that clarifies things a bit. I'm firing this off and will re-check
E-mail later this evening (otherwise I know I'll have 100 E-mails to read
tomorrow :-).

Have fun,

	Oren Ben-Kiki