[70f7f9]: doc / source / about_pyke / modifying_pyke.txt Maximize Restore History

Download this file

modifying_pyke.txt    301 lines (230 with data), 11.6 kB

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
.. $Id$
..
.. Copyright Š 2009 Bruce Frederiksen
..
.. Permission is hereby granted, free of charge, to any person obtaining a copy
.. of this software and associated documentation files (the "Software"), to deal
.. in the Software without restriction, including without limitation the rights
.. to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
.. copies of the Software, and to permit persons to whom the Software is
.. furnished to do so, subject to the following conditions:
..
.. The above copyright notice and this permission notice shall be included in
.. all copies or substantial portions of the Software.
..
.. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
.. IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
.. FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
.. AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
.. LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
.. OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
.. THE SOFTWARE.
restindex
crumb: Modifying Pyke
page-description:
Which source code repository to use. And the other tools that you'll
need run the units tests, and rebuild the html documentation.
/description
format: rest
encoding: utf8
output-encoding: utf8
include: yes
initialheaderlevel: 2
/restindex
uservalues
filedate: $Id$
/uservalues
===================================
Modifying Pyke
===================================
Index to This Page
=======================
* `Mercurial Repositories`_
* `Mercurial Keyword Extension`_
* `Which Repository Do I Use?`_
* `Compiling PLY Tables Files`_
* `Compiling the Compiler.krb File`_
* `Running Unit Tests`_
* `Rebuilding the HTML Documentation`_
Mercurial Repositories
======================
With Mercurial_, you clone the entire repository locally on your computer.
Then you can make changes and commit those changes to your local repository.
If you think those changes might be interesting to everybody, make your local
repository (or a clone of it) publicly available (either on your own server,
or on one of the `Mercurial Hosting Sites`_) and send me an email. I will
pull your changes, examine them, and push them to the master repository on
sourceforge.
Mercurial Keyword Extension
---------------------------
The Pyke sources use the Mercurial `Keyword Extension`_ as a holdover from
when the repository used Subversion rather than Mercurial.
The ``hgrc_keywords`` file has been provided to enable and configure this
extension for Pyke use. You can append this file to either your personal
.hgrc configuration file (which would then apply to all of your Mercurial
projects) or the project .hg/hgrc file (see `hgrc`_ in the Mercurial wiki).
If you use a ``post-clone`` `Mercurial hook`_, or append ``hgrc_keywords``
manually after cloning, the keywords won't be expanded properly when the
project is first cloned. But they will be expanded properly if the clone is
done with the -U option and then an ``hg update`` done in the newly cloned
repository (after the changes to .hg/hgrc have been made).
The keyword expansions are only used by the tools that generate the html
documentation (see `Rebuilding the HTML Documentation`_, below).
Which Repository Do I Use?
--------------------------
Normally, you will clone one of the following four repositories locally to
make a master copy of what's on sourceforge. Then you would clone your master
copy (which is very fast) to make separate clones for each development task
that you are working on for Pyke.
So it is best to keep all of these clones together in a common directory.
There are four repositories on sourceforge that you can start with:
release_1
Use this for bug fixes, code and documentation cleanup, and anything else
that would go into a point release for release 1. I merge the changes made
here into all of the other repositories. So this code goes into both the
Python2.x and Python3.x versions of Pyke.
pyke
Use this for major new features that would result in a major new release
(e.g., release 1.2). I merge the changes made in release_1 into the pyke
repository (but maybe not the other way around). And I merge the changes
made in the pyke repository into the pre_2to3 repository. So the code here
goes into both the Python2.x and Python3.x future versions of Pyke.
pre_2to3_r1
Use this for bug fixes, code and documentation cleanup, and anything else
that would go into a point release for release 1, but only apply to the
Python3.x version of Pyke. I merge the changes made in release_1 into the
pre_2to3_r1 repository (but not the other way around). And I merge the
changes made in the pre_2to3_r1 repository into the pre_2to3 repository.
So changes here only go into the next point release of the Python3.x version
of Pyke.
.. warning::
This code is maintained in a state just prior to running Python's
2to3_ tool on it. So you can't just run the code here directly.
The ``run_2to3`` script runs 2to3 on the current copy of the sources. Do
**not** run this in a repository clone that you still want to use to do
commits! Instead, commit all of your changes, then clone the repository
and do ``run_2to3`` in the clone. If anything doesn't work, go back to
the first repository to fix it, delete the clone, and repeat the whole
process. This was done to minimize merge conflicts caused by the 2to3
changes.
The ``run_pre_test`` script will:
* clone the current repository
* then in the clone do:
* ``run_2to3``
* ``testpyke`` -3.1
* python setup.py -q sdist --formats zip
* insert '3' after 'pyke' in the name of the source distribution zip
file.
``Run_pre_test`` assumes that you either have the keywording options set
in your personal .hgrc file, or have clone hooks in place to copy these
into the .hg/hgrc file of all clones within your pyke work area. See
`Mercurial Keyword Extension`_, above.
pre_2to3
Normally I merge changes from the pyke repository and the pre_2to3_r1
repository into pre_2to3 so that nothing needs to be done in this repository.
Most major new features would be developed in the ``pyke`` repository and
merged into pre_2to3. Making changes to pre_2to3 directly would only be
done when those changes are for major new features that only apply to the
Python3.x version of Pyke.
So, for example, if you wanted to work on the ``release_1`` repository, you
would::
$ mkdir pyke_repos
$ cd pyke_repos
$ hg clone -U http://pyke.hg.sourceforge.net:8000/hgroot/pyke/release_1 master
$ hg clone master task_1
$ cd task_1
.. note::
This assumes that you've added the `hgrc_keywords`_ file to your ~/.hgrc
file. See `Mercurial Keyword Extension`_, above.
Compiling PLY Tables Files
==========================
Pyke uses PLY_ (Python Lex and Yacc) as it's parser generator. PLY compiles
the Pyke grammars into a set of three tables files:
- kfbparser_tables.py (from kfbparser.py)
- krbparser_tables.py (from krbparser.py)
- scanner_tables.py (from scanner.py)
A copy of PLY is included in the source directory (pyke/krb_compiler/ply) so
that there there can be no version mismatch between the version of PLY used to
compile these tables files and the version of PLY installed on your machine.
To regenerate these tables files, at the top-level source directory::
$ python
>>> from pyke.krb_compiler import kfbparser, krbparser, scanner
>>> scanner.init(scanner, 0, True)
>>> krbparser.init(krbparser, True)
>>> kfbparser.init(kfbparser, True)
or just run the "testall.py" program from the doctest-tools package::
$ cd pyke/krb_compiler
$ testall.py
Compiling the Compiler.krb File
===============================
Pyke uses itself to compile your `rule base`_ sources (`.krb`_ files) into
Python source (``.py``) files.
The knowledge base file that Pyke uses for this is
pyke/krb_compiler/compiler.krb. This gets compiled into compiler_bc.py, which
is stored in the source code repository.
.. this code is hidden and will create the pyke/krb_compiler/compiled_krb
directory, if needed, for the code section following:
>>> import os, os.path
>>> os.chdir('../../..')
>>> root='pyke/krb_compiler'
>>> dir=root + '/compiled_krb'
>>> os.path.isdir(root)
True
>>> if not os.path.isdir(dir): os.mkdir(dir)
To recompile the compiler_bc.py file, from the top-level source directory::
$ mkdir pyke/krb_compiler/compiled_krb
$ python
>>> from pyke import krb_compiler
>>> krb_compiler.compile_krb('compiler', 'pyke.krb_compiler.compiled_krb',
... 'pyke/krb_compiler/compiled_krb',
... 'pyke/krb_compiler/compiler.krb')
['compiler_bc.py']
$ mv pyke/krb_compiler/compiled_krb/compiler_bc.py pyke/krb_compiler
.. this code is also hidden and deletes the
pyke/krb_compiler/compiled_krb/compiler_bc.py file and
pyke/krb_compiler/compiled_krb directory created above.
>>> os.path.isdir(root)
True
>>> os.remove(dir + '/compiler_bc.py')
>>> os.rmdir(dir)
Running Unit Tests
==================
The `doctest-tools`_ package is required to run the unit tests (see
`Other Required Packages`_ for more details).
The ``testall.py`` and ``testdoc.py`` scripts from ``doctest-tools`` can be run
anywhere.
In addition, the top-level directory contains a ``testpyke`` script that will
delete all of the compiled_krb directories, then run ``testall.py`` twice. The
first run must recompile all of the `knowledge base`_ sources (`.krb`_,
`.kfb`_ and `.kqb`_ files) into the compiled_krb directories in order to run
the tests. The second run reuses the files compiled in the first run. This
makes sure that all of the tests run properly whether they have to compile the
knowledge base sources or not.
Rebuilding the HTML Documentation
=================================
The ``doc/html`` directory contains all of the documents that you are reading
now. These are ready to browse directly from your hard drive if you'd like.
The documentation is generated using the rest2web_ package, which uses
docutils_ (see `Other Required Packages`_ for more details).
The sources for the documentation are in ``doc/source``. Each .txt file there
is converted into an .html file in the doc/html directory by running::
$ cd doc/source
$ bin/gen_html
This takes about 9 seconds. It:
#. Temporarily appends hyperlink references onto all of the \*.txt files.
#. Runs ``r2w`` to regenerate the files in ``doc/html``
- except for those in ``doc/html/stylesheets`` and ``doc/html/images``.
#. Strips all of the hyperlink references from the \*.txt files.
#. Creates a new sitemap.xml file with all of the dates that the files were
last modified.
.. note::
This process uses the date information expanded by the Mercurial `Keyword
Extension`_. See `Mercurial Keyword Extension`_, above.
I've gone ahead and placed the generated html files in the source repository
so that you can browse the documentation locally without having to run
``bin/gen_html``. So you only need these procedures if you change the
documentation (i.e., change the .txt files in doc/source).
To test all of the code examples in the documents, use the ``testall.py``
command from the `doctest-tools`_ package::
$ cd doc/source
$ testall.py