docutils-develop

Re: [Docutils-develop] reST Includes

[Dethe E. <de...@ma...>]

On Fri, 2002-08-16 at 13:04, David Goodger wrote:
> Thanks for your comments.  How about sharing them with docutils-develop?

Oops!  I meant to do that, but forgot that this list doesn't reply to
list by default.  Here's my earlier response:

> > 1. Is a directive the right way to go for this?
> 
> I think so.  The only other way would be to construct a custom front
> end which knows how to join the subdocuments together, but that's a
> lot of work for the simple case.

OK, cool.

> > 2. Any comments on the following syntax?
> > 
> > .. include:: path/subdocument.rst
> >    :parse: true | false (default: true)
> 
> Looks fine.  What does the "parse" attribute mean?

parse = true
  Treat the included text as reST and parse as part of the document

parse = false
  Treat the included text as literal text (maybe it's already HTML or
whatever) and just insert it.

> Here's my thinking to date, from the to-do list
> (http://docutils.sf.net/spec/notes.html#misc-include):
> 
>     _`misc.include`: ``#include`` one file in another.  But how to
>     parse wrt sections, reference names, conflicts?  Parse it in the
>     current document's context (C-preprocessor semantics), or
>     separately and then merge?

Treat it all as one document unless notified otherwise?  I think that's
reasonable default behaviour.  There could be another flag on the
include directive, as there is in the recursive parse within reST I
believe, telling it whether to process the included text within the same
context or not, but I think the default should be to assume the same
context.

So now the directive would look like:

.. include:: path/subdocument.rst
   :parse: true | false (default: true)
   :same-context: true | false (default: true)


>     Use C-preprocessor semantics for locating include files?  E.g.,
>     ``.. include:: file.txt`` will read another file into the current
>     one, relative to the current file's directory, and ``.. include::
>     <standard>`` will read a standard include file from
>     ``docutils/include/``.  (Should "quotes" be required around
>     non-standard include files?)

Umm.  I don't see a lot of value in using "standard" include locations. 
URI are better: Include off the net, include from a file, include from a
relative path...

--Dethe

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Dethe Elza wrote:
<digression>
> On Fri, 2002-08-16 at 13:04, David Goodger wrote:
>> Thanks for your comments.  How about sharing them with
>> docutils-develop?
> 
> Oops!  I meant to do that, but forgot that this list doesn't reply
> to list by default.

That's Mailman's default and *strongly recommended* behavior (their
emphasis).  Seems like "reply to list by default" is aberrant.  See
http://www.unicom.com/pw/reply-to-harmful.html.  Short form: always
use your mailer's "Reply-All" feature for mailing lists, and
"Reply-Originator-Only" when you want private email.  Anyhow, back to
the point.
</digression>

>>> 2. Any comments on the following syntax?
>>> 
>>> .. include:: path/subdocument.rst
>>>    :parse: true | false (default: true)
>> 
>> Looks fine.  What does the "parse" attribute mean?
> 
> parse = true
>   Treat the included text as reST and parse as part of the document
> 
> parse = false
>   Treat the included text as literal text (maybe it's already HTML or
> whatever) and just insert it.

I think a ":raw:" flag (attribute with no value) would be better.
When I saw "parse" I thought of the parsing context: parse separately
or together.  A bit confusing perhaps.

>> Here's my thinking to date, from the to-do list
>> (http://docutils.sf.net/spec/notes.html#misc-include):
>> 
>>     _`misc.include`: ``#include`` one file in another.  But how to
>>     parse wrt sections, reference names, conflicts?  Parse it in the
>>     current document's context (C-preprocessor semantics), or
>>     separately and then merge?
> 
> Treat it all as one document unless notified otherwise?  I think
> that's reasonable default behaviour.

Agreed.

> There could be another flag on the include directive, as there is in
> the recursive parse within reST I believe, telling it whether to
> process the included text within the same context or not, but I
> think the default should be to assume the same context.

What "recursive parse within reST" are you talking about?

In any case, that route opens up a big can of worms.  So it can remain
on the to-do list until truly needed.

> So now the directive would look like:
> 
> .. include:: path/subdocument.rst
>    :parse: true | false (default: true)
>    :same-context: true | false (default: true)

Factoring in the above, we get::

    .. include:: path/subdocument.rst
       :raw: (true if set)

That's feasible.  I wonder how to actually do the parse, though.  I
don't think it's a good idea to alter the already-read data, so a
separate, nested parse would probably be in order.  Also, the error
reporting mechanism would have to be revamped to include the file
which is the source of system messages.

>>     Use C-preprocessor semantics for locating include files?  E.g.,
>>     ``.. include:: file.txt`` will read another file into the current
>>     one, relative to the current file's directory, and ``.. include::
>>     <standard>`` will read a standard include file from
>>     ``docutils/include/``.  (Should "quotes" be required around
>>     non-standard include files?)
> 
> Umm.  I don't see a lot of value in using "standard" include
> locations. URI are better: Include off the net, include from a file,
> include from a relative path...

Yes, it's probably YAGNI (you ain't gonna need it).  Although I
wouldn't want to deal with generic URI yet (probably also YAGNI, at
least for now), just simple filesystem paths.

So, care to give it a try?

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[Dethe E. <de...@ma...>]

> <digression>
> > Oops!  I meant to do that, but forgot that this list doesn't reply
> > to list by default.
> 
> That's Mailman's default and *strongly recommended* behavior (their
> emphasis).  Seems like "reply to list by default" is aberrant.  See
> http://www.unicom.com/pw/reply-to-harmful.html.  Short form: always
> use your mailer's "Reply-All" feature for mailing lists, and
> "Reply-Originator-Only" when you want private email.  Anyhow, back to
> the point.
Yes, and when I get my main system back up and running it will be
easier.  I'm using a temporary box right now and not only are things
unfamiliar, but there doesn't appear to be a hotkey for "reply all." 
<grumble/>
> </digression>


> I think a ":raw:" flag (attribute with no value) would be better.
> When I saw "parse" I thought of the parsing context: parse separately
> or together.  A bit confusing perhaps.

:raw: is OK with me.

> > There could be another flag on the include directive, as there is in
> > the recursive parse within reST I believe, telling it whether to
> > process the included text within the same context or not, but I
> > think the default should be to assume the same context.
> 
> What "recursive parse within reST" are you talking about?

Ummm.  There was something somewhere in the code.  I can't remember
exactly where and I don't have a chance to dig it out right now.  I
could be wrong.  Your mileage may vary.  Void where prohibited.

> In any case, that route opens up a big can of worms.  So it can remain
> on the to-do list until truly needed.

Yes, or the or-not-to-do list...

> Factoring in the above, we get::
> 
>     .. include:: path/subdocument.rst
>        :raw: (true if set)

Looks good.

> That's feasible.  I wonder how to actually do the parse, though.  I
> don't think it's a good idea to alter the already-read data, so a
> separate, nested parse would probably be in order.  Also, the error
> reporting mechanism would have to be revamped to include the file
> which is the source of system messages.

Um.  I'm not sure if I'm up to revamping the error reporting mechanism. 
I'm trying to find something that's *easier* than 

  cat file1.rst file2.rst file3.rst | html.py > finished.html

I think I can handle creating a directive and even writing up how to
create a directive, but I'm not up for a revamp of the core system.

> > Umm.  I don't see a lot of value in using "standard" include
> > locations. URI are better: Include off the net, include from a file,
> > include from a relative path...
> 
> Yes, it's probably YAGNI (you ain't gonna need it).  Although I
> wouldn't want to deal with generic URI yet (probably also YAGNI, at
> least for now), just simple filesystem paths.

Agreed.  Although Python makes it pretty darn easy to have full URIs if
we want them...

> So, care to give it a try?

Yup, I'll take a look.  Is image still the best directive to get started
with?

--Dethe

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Dethe Elza wrote:
>>> There could be another flag on the include directive, as there is
>>> in the recursive parse within reST I believe, telling it whether
>>> to process the included text within the same context or not, but I
>>> think the default should be to assume the same context.
>> 
>> What "recursive parse within reST" are you talking about?
> 
> Ummm.  There was something somewhere in the code.  I can't remember
> exactly where and I don't have a chance to dig it out right now.  I
> could be wrong.  Your mileage may vary.  Void where prohibited.

There *is* plenty of recursive parsing going on.  I just wasn't sure
which one you meant.  Not to worry.

>> Factoring in the above, we get::
>> 
>>     .. include:: path/subdocument.rst
>>        :raw: (true if set)
> 
> Looks good.
> 
>> That's feasible.  I wonder how to actually do the parse, though.  I
>> don't think it's a good idea to alter the already-read data, so a
>> separate, nested parse would probably be in order.  Also, the error
>> reporting mechanism would have to be revamped to include the file
>> which is the source of system messages.
> 
> Um.  I'm not sure if I'm up to revamping the error reporting
> mechanism. I'm trying to find something that's *easier* than
> 
>   cat file1.rst file2.rst file3.rst | html.py> finished.html
> 
> I think I can handle creating a directive and even writing up how to
> create a directive, but I'm not up for a revamp of the core system.

Those two would be great!  I'll be happy to take care of revamping
``utils.Reporter``.

> Is image still the best directive to get started with?

Yes, it does all the parsing you'll need.  And also look at the
directives/admonitions.py module to see how to use
``state.nested_parse()``.  If the files you're including contain
section titles (which the general case would require), you'll need to
pass in ``match_titles=1``.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

I was updating the to-do list from the recent discussion, when I got to
thinking about the ":raw:" parameter:

    A "raw" flag (attribute with no value) would inform us that the
    imported data is raw (already in native format, like HTML).
    Perhaps there should be an attribute value, saying what the native
    format is.  Instead, perhaps the "raw" directive should take an
    "include" parameter (explicit attribute or second parameter in its
    directive data).

So, should it be::

    .. raw:: html fragment.html

or::

    .. include:: fragment.html
       :raw: html

?

A description of the "raw" directive is at
http://docutils.sf.net/spec/rst/directives.html#raw-data-pass-through.
The latest notes about the "include" directive are at
http://docutils.sf.net/spec/notes.html#misc-include.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

RE: [Docutils-develop] reST Includes

[Garth K. <ga...@de...>]

> So, should it be::
> 
>     .. raw:: html fragment.html
> 
> or::
> 
>     .. include:: fragment.html
>        :raw: html

+1 to having both both available (i.e. ``raw::`` mapping directly to
``include::`` with ``:raw:`` set, or vice versa). 

Now, anyone want to explain why it's ``html`` and not a full MIME type?
:) 

Regards,
Garth.

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Garth Kidd wrote:

>> So, should it be::
>> 
>>   .. raw:: html fragment.html
>> 
>> or::
>> 
>>   .. include:: fragment.html
>>      :raw: html
> 
> +1 to having both both available (i.e. ``raw::`` mapping directly to
> ``include::`` with ``:raw:`` set, or vice versa).

I didn't mention that the "raw" directive's main function is::

    .. raw:: html

       <p>Arbitrary raw HTML here, will be inserted as-is
       into HTML output.</p>

Normally, the "raw" directive's content will be supplied within the document
body.  So the question becomes, does the "raw" directive grow an "include"
attribute (or optional argument), or does the "include" directive grow a
"raw" attribute?  I'm not sure that "both" is a good answer here.

> Now, anyone want to explain why it's ``html`` and not a full MIME type?
> :) 

The "text/" part is implicit? ;-)  But seriously, MIME types *have* been
adopted for reStructuredText-format PEPs (official announcement any day
now), with "text/plain" as the default and "text/x-rst" as the new
alternative.  As for using MIME types with the "raw" directive here, I guess
I'd have to see how they'd be useful.  In other words, would such a MIME
type ever begin with anything *other* than "text/"?

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

RE: [Docutils-develop] reST Includes

[Garth K. <ga...@de...>]

> I didn't mention that the "raw" directive's main function is::
> 
>     .. raw:: html
> 
>        <p>Arbitrary raw HTML here, will be inserted as-is
>        into HTML output.</p>

Oh, okay. Whups. 

So, does raw also work for raw XML to insert into XML output, raw PDF to
insert into PDF output, raw PostScript to insert into PostScript output,
and so on, or are we assuming that it's only really HTML output that
matters? Getting back to the implicit ``text/``, do the MIME content
types for all of the suspect reST output types all begin with ``text/``?


> Normally, the "raw" directive's content will be supplied 
> within the document body.  So the question becomes, does the 
> "raw" directive grow an "include" attribute (or optional 
> argument), or does the "include" directive grow a "raw" 
> attribute?  I'm not sure that "both" is a good answer here.

Aah, so that's the distinction. My apologies; I've been tuned out for a
little while, and was too lazy to read back on this thread. Oops. 

I'd say that raw should have a ``:source:`` attribute. To me,
``include::`` would be used to pull in reST content from another
document. 

> > Now, anyone want to explain why it's ``html`` and not a full MIME 
> > type? :) 
> 
> The "text/" part is implicit? ;-)  

That's enough for me. 

> But seriously, MIME types 
> *have* been adopted for reStructuredText-format PEPs 

Cool! 

Regards,
Garth.

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Garth Kidd wrote:
> So, does raw also work for raw XML to insert into XML output, raw PDF to
> insert into PDF output, raw PostScript to insert into PostScript output,
> and so on,

Yes.

> or are we assuming that it's only really HTML output that matters?

Never!

> Getting back to the implicit ``text/``, do the MIME content
> types for all of the suspect reST output types all begin with ``text/``?

I don't know about PDF, but I suspect this is a red herring.  I don't think
we have a big enough set of formats to justify using MIME types.  We've
already got a set of identifiers that are perfectly suitable: the Writer
format names.

OTOH, MIME types *are* future-proof.  (This was the reasoning behind the PEP
decision.)  But they feel like over-engineering, a classic case of "don't
add functionality before it's needed".  I'd rather wait until significant
use cases present themselves, if ever.

> I'd say that raw should have a ``:source:`` attribute. To me,
> ``include::`` would be used to pull in reST content from another
> document. 

Noted.  And I tend to agree.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[fantasai <fan...@es...>]

David Goodger wrote:
> 
> Garth Kidd wrote:
> 
> >> So, should it be::
> >>
> >>   .. raw:: html fragment.html
> >>
> >> or::
> >>
> >>   .. include:: fragment.html
> >>      :raw: html
> >
> > +1 to having both both available (i.e. ``raw::`` mapping directly to
> >``include::`` with ``:raw:`` set, or vice versa). 
>
> Normally, the "raw" directive's content will be supplied within the document
> body.  So the question becomes, does the "raw" directive grow an "include"
> attribute (or optional argument), or does the "include" directive grow a
> "raw" attribute?  I'm not sure that "both" is a good answer here.

My first thought after seeing the two syntaxes was that the first
is a shortcut for the second. But, if
  .. raw:: format path
is the same as
  .. include:: path
     :raw: format
why have the raw attribute at all? You can simply have two separate
directives: 'include', which parses as reST, and 'raw', which doesn't.

> > Now, anyone want to explain why it's ``html`` and not a full MIME type?
> > :)
> 
> The "text/" part is implicit? ;-)

I have a feeling a lot of SGML/XML markup formats don't have their own
MIME type--that they're just sent as text/sgml or text/xml. Perhaps
:raw:'s value shouldn't be half a MIME type, but a token used as a
language identifier. One might want to skip printing raw content that
doesn't match the output format, and if there are several different
XML languages, a MIME type won't label the difference.

~fantasai

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

fantasai wrote:
> I have a feeling a lot of SGML/XML markup formats don't have their
> own MIME type--that they're just sent as text/sgml or text/xml.
> Perhaps :raw:'s value shouldn't be half a MIME type, but a token
> used as a language identifier. One might want to skip printing raw
> content that doesn't match the output format, and if there are
> several different XML languages, a MIME type won't label the
> difference.

Good point.  The intended usage of the "format" argument is exactly
that: a token matching the output format (Writer name or alias).  I
don't think MIME types are suitable for our purposes.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[Dethe E. <de...@ma...>]

On Fri, 2002-08-16 at 17:30, David Goodger wrote:
> Dethe Elza wrote:
> > I think I can handle creating a directive and even writing up how to
> > create a directive, but I'm not up for a revamp of the core system.
> 
> Those two would be great!  I'll be happy to take care of revamping
> ``utils.Reporter``.

Excellent.

> > Is image still the best directive to get started with?
> 
> Yes, it does all the parsing you'll need.  And also look at the
> directives/admonitions.py module to see how to use
> ``state.nested_parse()``.  If the files you're including contain
> section titles (which the general case would require), you'll need to
> pass in ``match_titles=1``.

Ah, that's the "recursive parse" I was thinking of!  Thanks.

--Dethe

Re: [Docutils-develop] reST Includes

[Aahz <aa...@py...>]

In all the discussion of include files, it's still not clear to me how
best to accomplish what I want: including a .py file as text.  I don't
want the text in the reST file because I'm going to use a *second*
include directive to get the output from the .py file.
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Aahz wrote:
> In all the discussion of include files, it's still not clear to me how
> best to accomplish what I want: including a .py file as text.  I don't
> want the text in the reST file because I'm going to use a *second*
> include directive to get the output from the .py file.

I'm not sure I understand what you're saying.  By "including a .py file as
text", do you mean that you want to insert a .py file into the document as a
**literal block**?  I don't think that's part of the proposal Dethe is
working on, but it seems useful and simple enough to do.  Something like::

    .. include:: example.py
       :literal:

If that's not what you meant, please explain.

And I don't follow the second sentence at all.  Perhaps an illustrative
example?

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[Aahz <aa...@py...>]

On Thu, Sep 12, 2002, David Goodger wrote:
> Aahz wrote:
>>
>> In all the discussion of include files, it's still not clear to me how
>> best to accomplish what I want: including a .py file as text.  I don't
>> want the text in the reST file because I'm going to use a *second*
>> include directive to get the output from the .py file.
> 
> I'm not sure I understand what you're saying.  By "including a .py file as
> text", do you mean that you want to insert a .py file into the document as a
> **literal block**?  I don't think that's part of the proposal Dethe is
> working on, but it seems useful and simple enough to do.  Something like::
> 
>     .. include:: example.py
>        :literal:
> 
> If that's not what you meant, please explain.

Yeah, I think that's what I meant, but I don't know reST well enough to
be certain.  ;-)

> And I don't follow the second sentence at all.  Perhaps an illustrative
> example?

Well, suppose I've got the following Python code::

    print "Hello, world!"

I'm going to want to include both::

    print "Hello, world!"

and::

    Hello, world!

Thus, there needs to be a directive to *run* the Python code and capture
stdout (and/or stderr), probably as some kind of literal block.
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/

Re: [Docutils-develop] reST Includes

[David G. <go...@us...>]

Aahz wrote:
> Well, suppose I've got the following Python code::
> 
>     print "Hello, world!"
> 
> I'm going to want to include both::
> 
>     print "Hello, world!"
> 
> and::
> 
>     Hello, world!
> 
> Thus, there needs to be a directive to *run* the Python code and
> capture stdout (and/or stderr), probably as some kind of literal
> block.

I see.  You want a directive that does the equivalent of
``os.system()``, and inserts the output into the document.  Hmmm.
Seems dangerous to me, to embed system calls in a document.  Perhaps
that should be left outside of the document, in the user's production
system (a makefile or a script or whatever)?  Or, the directive could
be disabled by default and only enabled with an explicit command-line
option or config file setting.  Even then, an interactive prompt may
be useful, such as:

    The file.txt document you are processing contains a "system"
    directive requesting that the ``sudo rm -rf /`` command be
    executed.  Allow it to execute?  (y/N)

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] reST Includes

[Richard J. <rj...@ek...>]

On Sat, 14 Sep 2002 8:28 am, David Goodger wrote:
> I see.  You want a directive that does the equivalent of
> ``os.system()``, and inserts the output into the document.  Hmmm.
> Seems dangerous to me, to embed system calls in a document.  Perhaps
> that should be left outside of the document, in the user's production
> system (a makefile or a script or whatever)?  Or, the directive could
> be disabled by default and only enabled with an explicit command-line
> option or config file setting.  Even then, an interactive prompt may
> be useful, such as:
>
>     The file.txt document you are processing contains a "system"
>     directive requesting that the ``sudo rm -rf /`` command be
>     executed.  Allow it to execute?  (y/N)

I think that's a reasonable level of precaution.


    Richard

Re: [Docutils-develop] reST Includes

[Aahz <aa...@py...>]

On Fri, Sep 13, 2002, David Goodger wrote:
> Aahz wrote:
>>
>> Well, suppose I've got the following Python code::
>> 
>>     print "Hello, world!"
>> 
>> I'm going to want to include both::
>> 
>>     print "Hello, world!"
>> 
>> and::
>> 
>>     Hello, world!
>> 
>> Thus, there needs to be a directive to *run* the Python code and
>> capture stdout (and/or stderr), probably as some kind of literal
>> block.
> 
> I see.  You want a directive that does the equivalent of
> ``os.system()``, and inserts the output into the document.  Hmmm.
> Seems dangerous to me, to embed system calls in a document.  Perhaps
> that should be left outside of the document, in the user's production
> system (a makefile or a script or whatever)?  Or, the directive could
> be disabled by default and only enabled with an explicit command-line
> option or config file setting.  Even then, an interactive prompt may
> be useful, such as:
> 
>     The file.txt document you are processing contains a "system"
>     directive requesting that the ``sudo rm -rf /`` command be
>     executed.  Allow it to execute?  (y/N)

Well, I wasn't expecting this to be a standard directive, I was just
explaining the context for wanting the Python code in a separate file.
If you want to add this as a standard directive, you've probably got the
correct approach.
-- 
Aahz (aa...@py...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/