#91 Include directive path argument should support a configurable root.
I'm building a static website using restructuredtext and I'm really annoyed by the lack of support for root-relative paths.
I have already implemented multiple directive plugins for my website build process, some of which use external files in a different format. They use the convention that if a path begins with /, they are relative to the root directory of the project.
IMO the requirement for relative paths is very limiting - I could have a directory with files intended for inclusion but:
- I would need to use
../../which is very ugly - relative paths are different depending on the current file path - this is very fragile and limits
An additional argument is that C and C++ have been using #include for like 40 years and relative paths (while sometimes useful) are only for specific cases. The leading convention is to use root-relative paths.
I could implement another plugin for it but ... copy-pasting docutils own code only to change few lines is definitely smelly.
/in a (posix) filesystem denotes the actual root, surely? Using it as a marker for "project root" seems odd.Perhaps you could look at
tools/buildhtml.py, or atSphinx? Both allow for consistency in the CWD for when the documentation is rendered, perhaps allieviating your concern about fragility.I don't undertand how the relative path requirements are limiting, please could you provide some more context?
A
(P.S. this should be moved to "Feature Requests" -- @milde are you able to do that? I don't think I can move it, or at least there doesn't seem to be a button.)
Why do you think so? Absolute paths make no sense for any such project.
tools/buildhtml.pyand Sphinx - I'm using a different framework for building static websites. In addition, Sphinx is intended for documentation only and what I write resembles a virtual book.What I need is a format that supports somewhat rich text that is able to be transformed into HTML. reST with its extensibility (directive plugins) is the best thing I could find. I want support for root-relative paths because it would allow me to write many articles referring to common parts that have fixed location within the repository. I can then change/reorder pages and not worry that changing their paths breaks include directive.
Ticket moved from /p/docutils/patches/193/
Moving this to "feature-requests", as there is no patch attached.
The heading is IMO misleading: The "include" directive supports "root-relative" (i.e. absolute) paths following the conventions of Python's
open()standard function. (It passes the path toopen()after resolving the special syntax for standard “include” data files.)Mind that for most Docutils front-end tools (rst2html, rst2latex, ...), there is no notation of a "project root".
From the description, I guess the you may want a way to specify a "root directory" that differs from the file system's root. This would require a new configuration setting.
Alternatives:
open()to open paths starting with '\' relative to a "project root" in the custom build system.Fine. I have no knowledge on the configuration, majority is probably supplied by the framework I use, not docutils.
How? First time I'm hearing something like this.
No. This is definitely beyond the project. It should not require external system tools like this to support the build.
To illustrate my problem:
Suppose that
book_bar/subsection_1/article_1wants to includeinclude/common_1. Currently I need to use something like.. include:: ../../include/common_1. This results in very fragile article code - during writing I move them quite often. If at any point I would like to reorganize articles, include paths will break. What I want is.. include:: /include/common_1that will work regardless where the current article is placed.The solution does not necessarily have to be a new feature. If there is a way I can achieve such behavior e.g. through reST plugin, I'm fine wih it too.
There is no natural choice for a "project directory" in stock Docutils.
Adding a configuration setting for an "include path root" will be
considered, if there is a consensus from the frameworks adding "project"
support that this is helpful.
Open questions remain, e.g. whether to use an configurable "include path
root" only for included reStructuredText files or also for code, literal
include, and parsed includes.
In the first case, it might be better to support a configurable list of
standard include directories.
Several workarounds exist with current Docutils
(none is fully clean and simple):
.. include:: /path-to/include/common_1.Con: long path, requires change if the complete project moves.
Maybe a symlink can help to provide a short absolute path.
<docutils package root>/parsers/rst/include/Con: this is an internal directory, changes with every Docutils update.
Use symlinks in your project's sub-directories:
project/include
project/book1/
project/book1/include -> ../include
Convince your framework to re-write absolute include path arguments
Convince your framework to use
os.chroot()or pychroot
Con: requires Linux,
os.chroot()also requires "root" privilegesMay lead to problems with imports and other file access.
website generators
supports an "include path root".
Transfer a working solution to your framework or switch the framework
(Sphinx, while complex and geared towards documentation, should be
well-suited for a book project, too.)
All of workarounds you have listed seem dirty for me (placing various constraints on the repository, file system or something else) so I think I will have to write my own plugin for it.
Not really sure, this is beyond my knowledge of Docutils. I can only share my experience with C and C++ build systems which do a lot of similar stuff, though the situation there is simpler from this point of view because everything there is code. Overall, I think that there is a consensus from the frameworks adding "project" support that this is helpful because majority of frameworks I have seen (if not all) do have a mechanism where some paths are root-relative and the framework expects certain files in certain places.
You've mentioned a framework a couple of times, what framework is it please? Is it public?
Have you seen Sphinx-Book-Theme[1]? That project aims to create online book type things.
[1] https://sphinx-book-theme.readthedocs.io/en/stable/
A
Last edit: Adam Turner 2022-05-04
Yes, it's public. I'm using Nikola (https://getnikola.com/).
Regarding Sphinx, I don't know much about it. Looking at it seems like it's very close to Nikola (both use conf.py, Jinja, reST and have similar build process). I don't really know if Sphinx would be better tool for my project. I choose Nikola because vast majority of static site generations (an online list I found had ~500 entries) are either exclusively for blogs or documentation and Nikola was a python-based barebone generator that offer ability to be heavily customized.
Would you be able to share a document structure demonstrating the problem?
From a brief glance,
nikola buildseems to build everything from a constant CWD, so there is no problem with changing paths.What might be possible is an option to the
includedirective (either as a direct option or a configuration setting) that sets the CWD, perhapsWith a configurable root location for
include, it might alleviate your concerns about style?A
Yes, this sounds like a reasonable solution - an option that tells the directive to use a path relative to something configurable. Obviously the path should be specified elsewhere - otherwise the directive would be just pointless.
For "elsewhere", I propose the configuration settings.
Before attempting a patch, we should check with the various frameworks
(Sphinx, Nicola, Pelican, ...) how useful they consider this to be
and whether we should
<angle-backeted-files>standard "include" data files.I would argue the first option (include root) is the simpler and more explicit / less surprising choice.
Sphinx currently uses the current working directory as the root for all includes. If a setting was added Sphinx would support it. I can't speak for Nicola or Pelican.
A
This depends on what you are using as comparison:
/as "project root" is similar to handling of absolute URLs by http servers.However, we want to use it for files, not URLS.
Open issue: support only
/or also\?<...>is already used for standard “include” data files (cf. directives.html).C and C++ use this syntax with a configurable search path for the
#includepreprocessor directive.I'm fine with both. Both
<>and/make sense and are a convention in some technologies. And btw, paths and URLs are both part of URIs. I would not support\, C and C++ do not support it either and I haven't seen\as a path separator in any web tech too.A first attempt. Configurable root directory for "include" and "raw" directives.
Still lacking:
test cases,
handling of other included files (csv-table, images),
documentation.
Last edit: Günter Milde 2022-08-04
First, sorry for a late response and overall inactivity but yeah, this is what I want. Thanks. Will write to Nikola maintainers about the feature.
A file with unit tests.
TODO:
New patch including the unit tests and support for "csv-table" directive. Based on [r9293] .
TODO:
Related
Commit: [r9293]
The updated patch now provides a configurable root directory for file paths in the "include", "raw", and "csv-table" directives + documentation and tests.
Before it can be applied to master, we need a decision on the behaviour of image URLs when interpreted as file path (i.e. whenever Docutils needs to access locally saved images: embedded images, LaTeX writer, reading meta-data from the image with PIL).
IMV, it would make sense to apply the "include-root" setting also in this context.
I have not read and understood the entire thread, so apologies
if my comment here is not helpful.
When I think of a "root directory" for Sphinx I think
in terms of a webserver's "Document Root" -- the uppermost
part of the filesystem hierarchy to which the webserver
(symlinks, etc., aside) has access.
The sphinx-quickstart tool seems to make "document root" be
the directory containing the RST source directory.
The "images" and "_static" directories used by Sphinx
are placed in the same directory as the RST source directory.
(If I'm right, it's been a while since I used sphinx-build.)
So the directory containing the RST source plays the role
of "topmost part of the filesystem hierarchy" that is of
interest to Sphinx.
So, to me, it makes sense for the Sphinx "document root" to default
to the directory containing the "sourcedir" argument that is
supplied to the sphinx-build command. This gives consistency
to the directory structures used whether it is sphinx-build
or sphinx-quickstart (and it's Makefile) that is used to
build the docs.
The question of defaults is orthogonal to the question of whether
there should be configuration directives controlling what
the "root directory" should be for various kinds of documentation
elements, images, static files, etc. But I think it does
impact how the various root directories are specified.
It seems to me that the specification of the root directory
in the configuration file should be relative to some default
location. To be specific, without having put in a lot of thought,
it seems that being relative to the "sourcedir" argument to
"sphinx-build" would be a good choice. That way a single configuration
file can be re-used in different "sphinx-build" invocations
each of which has a different "sourcedir" argument.
tldr; you can stop reading now.
As an aside, I've never been comfortable with the directory
layout created by sphinx-quickstart. I don't like the way
it mixes sphinx-quickstart created content with content
supplied by the sphinx user. (Or that's how it feels to me.)
Of course, everything setup by sphinx-quickstart can be changed
by frobbing config.py, but I'm talking about defaults -- which
is likely the directory structure most people use. I would love
to see a "better" default directory layout. (And I seem to recall
that there's some single knob/argument that does something
to better separate sphinx-quickstart supplied files from
user supplied files from build directories, etc. I think.)
What you get when you type just "sphinx-quickstart", I'm guessing
from what I've seen, seems to result in a lot of projects
putting the stuff sphinx-quickstart creates under revision
control. Because it's simpler to put the whole directory
created by sphinx-quickstart under revision control than
to tease out what files are sphinx's and which are user-created.
Or to take the time to figure out how to use sphinx-build
or tweak the sphinx-quickstart invocation. Especially when first
using sphinx. Putting sphinx-supplied stuff under revision
control makes it harder to use newer versions of sphinx/newer
sphinx features as they are released. (I bit the bullet
and figured out how to use sphinx-build and am using that
instead of sphinx-quickstart and its Makefile, in my
Makefiles/whatever.) I wanted to write this paragraph to inject
some off-topic-but-related-to-directory-layout
feedback and don't expect anything to change. Thanks for
listening and feel free to ignore this digression.
Hoping at least some of the above is useful,
Karl
Last edit: Günter Milde 2023-11-20
I guess the full development of my thoughts leads to this
observation (which might not be true because I'm not steeped
in Sphinx):
If "root path" declarations take absolute paths in config.py
then a single config.py is not (in the general
case) reusable in sphinx-build invocations which
use different "sourcedir" paths, because if the sourcedir
is different there's likely different image dirs/etc.
There seem to be multiple solutions.
1) Allow absolute file system
paths, relative to the overall root of the system's filesystem
hierarchy, as values for the various "root path" declarations,
and have sphinx-build take new arguments, each of which
would set a "root path" value similar to the way "sourcedir"
works. ("imageroot" "staticroot", etc.)
2) Have the values of the various "root path" declarations
be relative to a well-known filesystem path, like the directory
containing the config.py file.
3) Do nothing. Different invocations of sphinx-build may
need to use different configuration files, each of which
would correspond with a different "sourcedir" value.
Perhaps there would be a common config file "included"
by each top-level config file, the top-level config file
being the one that sets the various "root paths". But
this would be up to the Sphinx user to construct.
4) Do something else. Magic that completely reworks the
expected sphinx directory structure, to not only accommodate
multiple sorts of root directories but also to naturally
and simply separate sphinx supplied files from user-supplied
files. There would then magically be zero level of effort
involved in starting to use Sphinx in a way that cleanly
works with revision control and with multiple RST source
directories (each of which may be expected to produce
output formats which differ from each other, given that
various output formats sometimes require different image
file formats, etc.). Huzzah!
Regards,
Karl kop@karlpinc.com
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
We are here at the Docutils (not Sphinx) where there is no
conf.pynor a "project directory". So the "natural" and backwards compatible default for the new "include_root" setting is the filesystem root.In docutils there is only one primary input file (which may include other files via rST directives). Users are free to include files from anywhere in the file system.
The new setting should help users to organize the to-be-included files and use simpler paths inside the rST source.