.. Copyright ÂŠ 2009 Bruce Frederiksen
.. Permission is hereby granted, free of charge, to any person obtaining a copy
.. of this software and associated documentation files (the "Software"), to deal
.. in the Software without restriction, including without limitation the rights
.. to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
.. copies of the Software, and to permit persons to whom the Software is
.. furnished to do so, subject to the following conditions:
.. The above copyright notice and this permission notice shall be included in
.. all copies or substantial portions of the Software.
.. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
.. IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
.. FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
.. AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
.. LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
.. OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
.. THE SOFTWARE.
crumb: Modifying Pyke
Which source code repository to use. And the other tools that you'll
need run the units tests, and rebuild the html documentation.
Index to This Page
* `Mercurial Repositories`_
* `Mercurial Keyword Extension`_
* `Which Repository Do I Use?`_
* `Compiling PLY Tables Files`_
* `Compiling the Compiler.krb File`_
* `Running Unit Tests`_
* `Rebuilding the HTML Documentation`_
With Mercurial_, you clone the entire repository locally on your computer.
Then you can make changes and commit those changes to your local repository.
If you think those changes might be interesting to everybody, make your local
repository (or a clone of it) publicly available (either on your own server,
or on one of the `Mercurial Hosting Sites`_) and send me an email. I will
pull your changes, examine them, and push them to the master repository on
Mercurial Keyword Extension
The Pyke sources use the Mercurial `Keyword Extension`_ as a holdover from
when the repository used Subversion rather than Mercurial.
The ``hgrc_keywords`` file has been provided to enable and configure this
extension for Pyke use. You can append this file to either your personal
.hgrc configuration file (which would then apply to all of your Mercurial
projects) or the project .hg/hgrc file (see `hgrc`_ in the Mercurial wiki).
If you use a ``post-clone`` `Mercurial hook`_, or append ``hgrc_keywords``
manually after cloning, the keywords won't be expanded properly when the
project is first cloned. But they will be expanded properly if the clone is
done with the -U option and then an ``hg update`` done in the newly cloned
repository (after the changes to .hg/hgrc have been made).
The keyword expansions are only used by the tools that generate the html
documentation (see `Rebuilding the HTML Documentation`_, below).
Which Repository Do I Use?
Normally, you will clone one of the following four repositories locally to
make a master copy of what's on sourceforge. Then you would clone your master
copy (which is very fast) to make separate clones for each development task
that you are working on for Pyke.
So it is best to keep all of these clones together in a common directory.
There are four repositories on sourceforge that you can start with:
Use this for bug fixes, code and documentation cleanup, and anything else
that would go into a point release for release 1. I merge the changes made
here into all of the other repositories. So this code goes into both the
Python2.x and Python3.x versions of Pyke.
Use this for major new features. I merge the changes made in release_1 into
the pyke repository (but not the other way around). And I merge the changes
made in the pyke repository into the pre_2to3 repository. So the code here
also goes into both the Python2.x and Python3.x versions of Pyke.
Use this for bug fixes, code and documentation cleanup, and anything else
that would go into a point release for release 1, but only apply to the
Python3.x version of Pyke. I merge the changes made in release_1 into the
pre_2to3_r1 repository (but not the other way around). And I merge the
changes made in the pre_2to3_r1 repository into the pre_2to3 repository.
So changes here only go into the Python3.x version of Pyke.
This code is maintained in a state just prior to running Python's
2to3_ tool on it. So you can't just run the code here directly.
The ``run_2to3`` script runs 2to3 on the current copy of the sources. Do
**not** run this in a repository clone that you still want to use to do
commits! Instead, commit all of your changes, then clone the repository
and do ``run_2to3`` in the clone. If anything doesn't work, go back to
the first repository to fix it, delete the clone, and repeat the whole
process. This was done to minimize merge conflicts caused by the 2to3
The ``run_pre_test`` script will:
* clone the current repository
* then in the clone do:
* ``testpyke`` -3.1
* python setup.py -q sdist --formats zip
* insert '3.1' after 'pyke' in the name of the source distribution zip
``Run_pre_test`` assumes that you either have the keywording options set
in your personal .hgrc file, or have clone hooks in place to copy these
into the .hg/hgrc file of all clones within your pyke work area. See
`Mercurial Keyword Extension`_, above.
Normally I merge changes from the pyke repository and the pre_2to3_r1
repository into pre_2to3 so that nothing needs to be done in this repository.
Most major new features would be developed in the ``pyke`` repository and
merged into pre_2to3. Making changes to pre_2to3 directly would only be
done when those changes are for major new features that only apply to the
Python3.x version of Pyke.
So, for example, if you wanted to work on the ``release_1`` repository, you
$ mkdir pyke_repos
$ cd pyke_repos
$ hg clone -U http://pyke.hg.sourceforge.net:8000/hgroot/pyke/release_1 master
$ hg clone master task_1
$ cd task_1
This assumes that you've added the `hgrc_keywords`_ file to your ~/.hgrc
file. See `Mercurial Keyword Extension`_, above.
Compiling PLY Tables Files
Pyke uses PLY_ (Python Lex and Yacc) as it's parser generator. PLY compiles
the Pyke grammars into a set of three tables files:
- kfbparser_tables.py (from kfbparser.py)
- krbparser_tables.py (from krbparser.py)
- scanner_tables.py (from scanner.py)
A copy of PLY is included in the source directory (pyke/krb_compiler/ply) so
that there there can be no version mismatch between the version of PLY used to
compile these tables files and the version of PLY installed on your machine.
To regenerate these tables files, at the top-level source directory::
>>> from pyke.krb_compiler import kfbparser, krbparser, scanner
>>> scanner.init(scanner, 0, True)
>>> krbparser.init(krbparser, True)
>>> kfbparser.init(kfbparser, True)
or just run the "testall.py" program from the doctest-tools package::
$ cd pyke/krb_compiler
Compiling the Compiler.krb File
Pyke uses itself to compile your `knowledge base`_ sources (`.kfb`_, `.kqb`_
and `.krb`_ files) into Python source files.
The knowledge base file that Pyke uses for this is
pyke/krb_compiler/compiler.krb. This gets compiled into compiler_bc.py, which
is stored in the source code repository.
.. this code is hidden and will create the pyke/krb_compiler/compiled_krb
directory, if needed, for the code section following:
>>> import os, os.path
>>> dir=root + '/compiled_krb'
>>> if not os.path.isdir(dir): os.mkdir(dir)
To recompile the compiler_bc.py file, from the top-level source directory::
$ mkdir pyke/krb_compiler/compiled_krb
>>> from pyke import krb_compiler
>>> krb_compiler.compile_krb('compiler', 'pyke.krb_compiler.compiled_krb',
$ mv pyke/krb_compiler/compiled_krb/compiler_bc.py pyke/krb_compiler
.. this code is also hidden and deletes the
pyke/krb_compiler/compiled_krb/compiler_bc.py file and
pyke/krb_compiler/compiled_krb directory created above.
>>> os.remove(dir + '/compiler_bc.py')
Running Unit Tests
The `doctest-tools`_ package is required to run the unit tests (see
`Other Required Packages`_ for more details).
The ``testall.py`` and ``testdoc.py`` scripts from ``doctest-tools`` can be run
In addition, the top-level directory contains a ``testpyke`` script that will
delete all of the compiled_krb directories, then run ``testall.py`` twice. The
first run must recompile all of the `knowledge base`_ sources (`.krb`_,
`.kfb`_ and `.kqb`_ files) into the compiled_krb directories in order to run
the tests. The second run reuses the files compiled in the first run. This
makes sure that all of the tests run properly whether they have to compile the
knowledge base sources or not.
Rebuilding the HTML Documentation
The ``doc/html`` directory contains all of the documents that you are reading
now. These are ready to browse directly from your hard drive if you'd like.
The documentation is generated using the rest2web_ package, which uses
docutils_ (see `Other Required Packages`_ for more details).
The sources for the documentation are in ``doc/source``. Each .txt file there
is converted into an .html file in the doc/html directory by running::
This takes about 9 seconds. It:
#. Temporarily appends hyperlink references onto all of the \*.txt files.
#. Runs ``r2w`` to regenerate the files in ``doc/html``
- except for those in ``doc/html/stylesheets`` and ``doc/html/images``.
#. Strips all of the hyperlink references from the \*.txt files.
#. Creates a new sitemap.xml file with all of the dates that the files were
This process uses the date information expanded by the Mercurial `Keyword
Extension`_. See `Mercurial Keyword Extension`_, above.
I've gone ahead and placed the generated html files in the source repository
so that you can browse the documentation locally without having to run
``bin/gen_html``. So you only need these procedures if you change the
documentation (i.e., change the .txt files in doc/source).
To test all of the code examples in the documents, use the ``testall.py``
command from the `doctest-tools`_ package::
$ cd doc