YAML Ain't Markup Language

Good news thanks.  Any chance $YAML::ForceBlock = 1, or ForceBlock can 
make it back in.  I cannot update past 0.35, because I rely on this.  I 
have an application that screen scrapes a web page and dumps it as yaml; 
However, the data scrapped must be blocks within the yaml.  I cannot 
have a 1 be interpreted by anything that reads the yaml file as an 
integer 1.  It must be read as "1".  Is there another way to achieve 
this with YAML.pm, or consumer of the yaml file.

Thanks
-- 
Windows
Start Here
Frustrating Hanging Crashing
Blue Screen of Death
Reboot


Ingy dot Net wrote:
> Dear Yamlistas of the Perly variety,
> 
> I recently released a new YAML.pm that fixes most of the bugs reported
> over the last two years. I also got the code and test suite into a much
> more organized state, so future releases should not be so painful.
> 
> In a parallel universe, Audrey Tang of Perl6/Pugs renown, released
> the YAML::Syck module which is a fully functional Perl 5 binding to
> Why's wonderful libsyck. (I believe Perl6 has had Syck YAML support
> for some time now)
> 
> Also, in yet another abstract tangential astral plane of auspicious
> hackery, the extraordinary Miyagawa released JSON::Syck.
> 
> So finally Perl has the rock solid YAML/JSON support that Ruby has been
> enjoying for some time. Yay.
> 
> ---
> 
> Note that unless you are using any of the fancier features of YAML.pm
> outside of the typical Load/Dump interface, (or you don't have a
> compiler on your machine), please consider switching from YAML to
> YAML::Syck. The basic interface is exactly the same so all you'll need
> to do is change `use YAML;` to `use YAML::Syck`. YAML::Syck is 10 times
> faster and is far more stable than YAML.pm.
> 
> Future plans for YAML.pm include grafting the fancier features including a
> possible streaming interface to YAML::Syck.
> 
> Cheers, Ingy
>

The implicit values is totally discontinued in YAML.pm. Eveything is treated
as a string now.

This is also true for YAML::Syck.

Try it. I don't think you'll need ForceBlock. If you still do, let me know.

Cheers, Ingy

On 25/01/06 16:10 -0800, John Pywtorak wrote:
> Good news thanks.  Any chance $YAML::ForceBlock = 1, or ForceBlock can 
> make it back in.  I cannot update past 0.35, because I rely on this.  I 
> have an application that screen scrapes a web page and dumps it as yaml; 
> However, the data scrapped must be blocks within the yaml.  I cannot 
> have a 1 be interpreted by anything that reads the yaml file as an 
> integer 1.  It must be read as "1".  Is there another way to achieve 
> this with YAML.pm, or consumer of the yaml file.
> 
> Thanks
> -- 
> Windows
> Start Here
> Frustrating Hanging Crashing
> Blue Screen of Death
> Reboot
> 
> 
> Ingy dot Net wrote:
> >Dear Yamlistas of the Perly variety,
> >
> >I recently released a new YAML.pm that fixes most of the bugs reported
> >over the last two years. I also got the code and test suite into a much
> >more organized state, so future releases should not be so painful.
> >
> >In a parallel universe, Audrey Tang of Perl6/Pugs renown, released
> >the YAML::Syck module which is a fully functional Perl 5 binding to
> >Why's wonderful libsyck. (I believe Perl6 has had Syck YAML support
> >for some time now)
> >
> >Also, in yet another abstract tangential astral plane of auspicious
> >hackery, the extraordinary Miyagawa released JSON::Syck.
> >
> >So finally Perl has the rock solid YAML/JSON support that Ruby has been
> >enjoying for some time. Yay.
> >
> >---
> >
> >Note that unless you are using any of the fancier features of YAML.pm
> >outside of the typical Load/Dump interface, (or you don't have a
> >compiler on your machine), please consider switching from YAML to
> >YAML::Syck. The basic interface is exactly the same so all you'll need
> >to do is change `use YAML;` to `use YAML::Syck`. YAML::Syck is 10 times
> >faster and is far more stable than YAML.pm.
> >
> >Future plans for YAML.pm include grafting the fancier features including a
> >possible streaming interface to YAML::Syck.
> >
> >Cheers, Ingy
> >

The example is

simple key : {
  also simple : value,
  ? not a
  simple key : any
  value
}

We can also consider a minimal example:

simple key : {
}

The problem is that "simple key :" sets indent level to 0, so the content
must be indented with at least one space. Thus to make it correct, I
should write

simple key : {
 }

I'm reluctant to force this syntax in my parser because I found it
non-intuitive. What do you think?


-- 
xi

On 17/02/06 22:23 +0200, Kirill Simonov wrote:
> The example is
> 
> simple key : {
>   also simple : value,
>   ? not a
>   simple key : any
>   value
> }
> 
> We can also consider a minimal example:
> 
> simple key : {
> }
> 
> The problem is that "simple key :" sets indent level to 0, so the content
> must be indented with at least one space. Thus to make it correct, I
> should write
> 
> simple key : {
>  }
> 
> I'm reluctant to force this syntax in my parser because I found it
> non-intuitive. What do you think?

What do you think about this invalid yaml:

    ---
    foo:
        bar: [1, 2,
    3, 4]

I want to make sure I understand what you dislike.

Cheers, Ingy

On Fri, Feb 17, 2006 at 07:38:12PM -0800, Ingy dot Net wrote:
> On 17/02/06 22:23 +0200, Kirill Simonov wrote:
> > We can also consider a minimal example:
> > 
> > simple key : {
> > }
> > 
> > The problem is that "simple key :" sets indent level to 0, so the content
> > must be indented with at least one space. Thus to make it correct, I
> > should write
> > 
> > simple key : {
> >  }
> > 
> > I'm reluctant to force this syntax in my parser because I found it
> > non-intuitive. What do you think?
> 
> What do you think about this invalid yaml:
> 
>     ---
>     foo:
>         bar: [1, 2,
>     3, 4]
> 
> I want to make sure I understand what you dislike.

Not that I particularly like this syntax, but I believe it should
become correct, for two reasons:

- In Python, it's valid. Python syntax is very similar to YAML and has
  the same "block" and "flow" contexts. Compare it with:

    if foo:
        bar = [1, 2,
    3, 4]

  In Python, line breaks are ignored when the parser is between '[' and
  ']', so that

    [1,
    2,
    3,
    4]

  is counted as a single line.

- This restriction does not help the parser. '[]' and '{}' completely
  define the structure of a document in the "flow" context and has the
  same function as indentation in the "block" context.


-- 
xi

Example 9.26:

---
|
.
..
..literal
.
..text

=9E# Comment
...

is equivalent to

---
!!str "\nliteral\n\ntext\n"
...

But why is there only one leading line break in "\nliteral..."?
Shouldn't it be "\n\nliteral..."?


--=20
xi

On Sat, Feb 18, 2006 at 01:19:12PM +0200, Kirill Simonov wrote:
| Example 9.26:
          4.77 and 4.78?

| But why is there only one leading line break in "\nliteral..."?
| Shouldn't it be "\n\nliteral..."?

I believe you're correct.  Oren?

Clark

On Sat, Feb 18, 2006 at 03:13:26PM -0500, Clark C. Evans wrote:
> On Sat, Feb 18, 2006 at 01:19:12PM +0200, Kirill Simonov wrote:
> | Example 9.26:
>           4.77 and 4.78?
> 
> | But why is there only one leading line break in "\nliteral..."?
> | Shouldn't it be "\n\nliteral..."?
> 
> I believe you're correct.  Oren?

Thanks. Another problematic example is
Example 9.30. Folded Lines:

>
.folded
.line

.next
.line

...* bullet
...* list

.last
.line

# Comment

It is interpreted as

%YAML 1.1
---
!!seq [
    !!str "folded line\n\
    next line\n\
    \  * bullet\n\
    \  * list\n\
    last line\n"
]

So that a single line break between the lines.

    .line

and
    ...* bullet

It means that the line break after ".line" is folded. But the spec says

    ...empty lines separating lines of a different type are never
    folded...

By different types it seems to mean lines starting with a space and
lines starting with a non-space character.

As far as I understand the productions, folding may happen only between
lines starting with non-space characters.

In this case, the correct interpretation should be

%YAML 1.1
---
!!seq [
    !!str "folded line\n\
    next line\n\n\
    \  * bullet\n\
    \  * list\n\n\
    last line\n"
]

Is there an easy rule that describes how folding should be done?

-- 
xi

Kirill,

Folding is somewhat difficult concept (and entirely due to my
requirements, Oren and Brian are humoring me with this one).  The idea
is to allow paragraph text wrapping in a way that does not introduce
spurious carriage returns needed only for text "readability".   Hence,
each paragraph is seen as a sequence of characters -- without a carriage
return in them.  There are two complications:  (a) an actual carriage
return is indicated with two adjacent carriage returns, (b) to allow for
bulleted lists, carriage returns *between* indented content is
preserved.

The rules for this, unfortunately, are kinda ugly:

  1. Move the entire folded block into a string buffer, stripping
     all "block" indentation.

  2. Replace single occurances of a carraige return that is not
     immediately followed by a whitespace with a single space.

  3. Replace runs of N adjacent carriage returns with N-1
     carriage returns.

  4. Stripping trailing whitespace; adding a terminating carriage
     return.

I'm not sure if this is exactly what is in the specification; but
it is how I had implemented this style a long while back.

On Sat, Feb 18, 2006 at 10:33:51PM +0200, Kirill Simonov wrote:
| >
| .folded
| .line
| 
| .next
| .line
| 
| ...* bullet
| ...* list
| 
| .last
| .line

So, in my buffer after stripping block indentation we have:

   "folded\nline\n\nnext\nline\n\n"
   "  * bullet\n  * list\n\n"
   "last\nline\n"

After replacing single occurances of a carriage returns not
immediately followed by a tab or space:

   "folded line\n\nnext line\n\n"
   "  * bullet\n  * list\n\n"
   "last line "

Then replacing *runs* of N carriage returns with N-1:

   "folded line\nnext line\n"
   "  * bullet\n  * list\n"
   "last line "

After the final step, giving:

| %YAML 1.1
| ---
| !!seq [
|     !!str "folded line\n\
|     next line\n\
|     \  * bullet\n\
|     \  * list\n\
|     last line\n"
| ]

which, seems correct to the interpretation we have.

| It means that the line break after ".line" is folded. But the spec says
| 
|     ...empty lines separating lines of a different type are never
|     folded...

I think that this was an *attempt* to explain the folding rule for
indented items, but it doesn't reflect the intent.  So, I think in this
case the example is correct, but the descriptive rule is wrong.  The
rule above, btw, also means that an extra carriage return between
bullets is ignored -- which I think is the intended effect.


| In this case, the correct interpretation should be
| 
| %YAML 1.1
| ---
| !!seq [
|     !!str "folded line\n\
|     next line\n\n\
|     \  * bullet\n\
|     \  * list\n\n\
|     last line\n"
| ]
| 
| Is there an easy rule that describes how folding should be done?

Yea, you're correct that the example is wrong given the spec's rule;
however, I think it is the rule that needs to be fixed.  I hope the
recipe above helps.

Kind Regards,

Clark

On Sat, Feb 18, 2006 at 04:33:31PM -0500, Clark C. Evans wrote:
> The rules for this, unfortunately, are kinda ugly:
> 
>   1. Move the entire folded block into a string buffer, stripping
>      all "block" indentation.
> 
>   2. Replace single occurances of a carraige return that is not
>      immediately followed by a whitespace with a single space.
> 
>   3. Replace runs of N adjacent carriage returns with N-1
>      carriage returns.
> 
>   4. Stripping trailing whitespace; adding a terminating carriage
>      return.
> 
> I'm not sure if this is exactly what is in the specification; but
> it is how I had implemented this style a long while back.

Well, it make sense, but it seems that most implementations use a
different algorithm.

[...]
>    "folded line\nnext line\n"
>    "  * bullet\n  * list\n"
>    "last line "

I've tested three implementations, the old PyYAML (I'm not sure I've
taken the correct version), PySyck, and YAML.pm

I've got:

=================================================
>>> # The original document
>>> print "--- >\n folded\n line\n\n next\n line\n\n   * bullet\n   * list\n\n last\n line\n"
--- >
 folded
 line

 next
 line

   * bullet
   * list

 last
 line
=================================================
>>> # PyYAML - strangely enough, it didn't remove the extra line break
>>> # between "folded line" and "next line":
>>> print yaml.l("--- >\n folded\n line\n\n next\n line\n\n   * bullet\n   * list\n\n last\n line\n")
folded line

next line

  * bullet
  * list

last line
=================================================
>>> # PySyck
>>> print syck.load("--- >\n folded\n line\n\n next\n line\n\n   * bullet\n   * list\n\n last\n line\n")
folded line
next line

  * bullet
  * list

last line
=================================================
# YAML.pm
$ perl -e 'use YAML; print Load("--- >\n folded\n line\n\n next\n line\n\n   * bullet\n   * list\n\n last\n line\n")'
folded line
next line

  * bullet
  * list

last line
=================================================
    

> After the final step, giving:
> 
> | %YAML 1.1
> | ---
> | !!seq [
> |     !!str "folded line\n\
> |     next line\n\
> |     \  * bullet\n\
> |     \  * list\n\
> |     last line\n"
> | ]

What's even funnier, neither of the implementations can parse this
document correctly, for instance, Syck produces:

>>> print syck.load(r"""---
...  "folded line\n\
...    next line\n\
...    \  * bullet\n\
...    \  * list\n\
...    last line\n"
... """)
folded line
next line
\  * bullet
\  * list
last line


> Yea, you're correct that the example is wrong given the spec's rule;
> however, I think it is the rule that needs to be fixed.  I hope the
> recipe above helps.

Hm... well, ok, I'll do it. It will even make the block scalar scanner
easier.


-- 
xi

Clark C. Evans wrote:
> Kirill,
> 
> Folding is somewhat difficult concept (and entirely due to my
> requirements, Oren and Brian are humoring me with this one).  The idea
> is to allow paragraph text wrapping in a way that does not introduce
> spurious carriage returns needed only for text "readability".   Hence,
> each paragraph is seen as a sequence of characters -- without a carriage
> return in them.  There are two complications:  (a) an actual carriage
> return is indicated with two adjacent carriage returns, (b) to allow for
> bulleted lists, carriage returns *between* indented content is
> preserved.

Isn't this type of special casing the sort of thing that really hurts
people when it happens unexpectedly? It seems like the aim is to produce
an inline, readable document format but should yaml really be doing this
(I know it's only one special case, but each one of these is one more
thing someone has to learn in order to be able to 'know' yaml).

again I'm not criticising the reasoning behind it or the intentions,
merely the possible side affects that could (and have) negatively
affected the perception of yaml.

Tim Parkin

On Sat, Feb 18, 2006 at 10:56:38PM +0000, Tim Parkin wrote:
> Clark C. Evans wrote:
> > Kirill,
> > 
> > Folding is somewhat difficult concept (and entirely due to my
> > requirements, Oren and Brian are humoring me with this one).  The idea
> > is to allow paragraph text wrapping in a way that does not introduce
> > spurious carriage returns needed only for text "readability".   Hence,
> > each paragraph is seen as a sequence of characters -- without a carriage
> > return in them.  There are two complications:  (a) an actual carriage
> > return is indicated with two adjacent carriage returns, (b) to allow for
> > bulleted lists, carriage returns *between* indented content is
> > preserved.
> 
> Isn't this type of special casing the sort of thing that really hurts
> people when it happens unexpectedly? It seems like the aim is to produce
> an inline, readable document format but should yaml really be doing this
> (I know it's only one special case, but each one of these is one more
> thing someone has to learn in order to be able to 'know' yaml).

I agree. The same function can be expressed with an application-specific
tag, for example

--- !folded |
 folded
 line

 next
 line

   * bullet
   * list

 last
 line

# Comment

Folded scalars looks too specific for a meta-format like YAML.

On the other hand, quoted and plain scalars have very similar folding
rules...

-- 
xi

On Sun, Feb 19, 2006 at 01:09:41AM +0200, Kirill Simonov wrote:
| On Sat, Feb 18, 2006 at 10:56:38PM +0000, Tim Parkin wrote:
| > Clark C. Evans wrote:
| > > Kirill,
| > > 
| > > Folding is somewhat difficult concept (and entirely due to my
| > > requirements, Oren and Brian are humoring me with this one).
| >
| > Isn't this type of special casing the sort of thing that really hurts
| > people when it happens unexpectedly? It seems like the aim is to produce
| > an inline, readable document format but should yaml really be doing this

Well, you have to use a special character, > , in order to get this
sort of behavior.  I doubt it would happen to someone by accident; 
it has very large use cases when serializing business document
information.   That said, this style was made _before_ we allowed
un-adorned scalars to continue on more than one line.

| > (I know it's only one special case, but each one of these is one more
| > thing someone has to learn in order to be able to 'know' yaml).
| 
| I agree. The same function can be expressed with an application-specific
| tag, for example

I think `tags` are meant to play a completely different role;
this guides *formatting* of data in the text stream.  A tag
specifies the data _type_ of the content.  I see them as 
completley othogonal concerns.  

| Folded scalars looks too specific for a meta-format like YAML.

You could argue that, if there is enough concensus we can remove it 
from the next version of the specification I guess; especially given 
the inconsistency in their parsing in existing implementations.

| On the other hand, quoted and plain scalars have very similar folding
| rules...

Yes; and these rules were derived somewhat from the folded-style in
an attempt to consolidate the patterns.  With the plain style as it
currently is, there is less of a need for the folded style.

Best,

Clark

On Sun, Feb 19, 2006 at 12:43:54AM +0200, Kirill Simonov wrote:
| Well, it make sense, but it seems that most implementations use a
| different algorithm.

Yea; that's a problem.  Let's use the same algorithem as used in Syck
and fix the specification to best match what it currently does. 

Thank you so much!

Best,

Clark

Clark C. Evans wrote:
> On Sun, Feb 19, 2006 at 12:43:54AM +0200, Kirill Simonov wrote:
> | Well, it make sense, but it seems that most implementations use a
> | different algorithm.
> 
> Yea; that's a problem.  Let's use the same algorithem as used in Syck
> and fix the specification to best match what it currently does. 
> 
> Thank you so much!
>
Sounds OK to me... After thinking about it overnight I didn't see any
obvious use cases where it may cause problems. I may just be panicking
about something that isn't important. Sorry.

Tim