I have personally hit the sre stack limit only once, and that was a long,
long time ago, before I completely changed the tokenizer code. Since then
I have not hit the limit. The issue is not so much the total length of the
Spyce code, nor the number of files, but with how the given Spyce string
affects the stack of the (stack-based) regular expression matcher. The sre
module is used to quickly search through the Spyce code for delimeters
that signify the beginning and end of chunks, statements, comments,
lambdas, etc. So, I would not worry about the stack limit. It is a
shortcoming of the Python module, and will eventually be fixed. If you
encounter a problem, please let me know and we'll deal with it then. If
necessary, we can make it an option in the configuration file. In general,
it's better to use the new sre module, over the pre version, since it
supports Unicode strings.
Now, with respect to your question about the long recompile times... I
don't think I follow your suggestion. It's possible to 'compile' a
statically included file, but is it really needed? Think of it as a
#define in C. What happens currently is that the included file is read in
when it is needed. As in C, it's a pre-processor step that happens before
compilation. While it is possible to cache some information to speed up
the pre-processing, I don't think that it's worth the effort.
You can shorten your compile times in the following way. I imagine that
you are including lots of code for you bulletin board system, with
constants, functions, etc... Is there some reason that it needs to be
Spyce code, and not simply a Python module or a Spyce module? Namely, you
can simply [[ import foo ]] your Python module or [[.import name=foo ]]
your Spyce module. These are straight Python files; they don't need to be
converted into Python. The tokenizer and the parser for the Spyce to
Python conversion step are written in Python, so they are slow. And, if
you can avoid them, that should solve your problem. i.e. You still get to
split up your code into pieces, but only index.spy needs to go through the
relatively slow Spyce-to-Python transformation machinery.
>Therefore, is it possible to make [[.include file=""]] cache the compiled
>version of the file being included, and only recompiles when the file
The whole point of the debug option is to turn off caching, so I don't
think that would be consistent with the intended design.
I hope that helps.
All the best,
On Tue, 14 Jan 2003, Tiberius Teng wrote:
> First I would like to thank Rimon's help about the SF issue. Because
>I used a lot of +='s in my code and not willing to modify it, I think
>I'm either (1) Wait SF supports Python 2.2.1 or (2) Look for another
>place to host my demo forum.
> I've just upgraded my test platform to Apache 2.0.43 + mod_python
>3.0.1, along with Spyce 1.3.4. When I'm reading the CHANGES file, I've
>noticed the stack size issue caused by pre->sre switch. Because I'm
>including all files from my SpyceBB's index file, and total code size
>has exceeded 75KB these days (fortunately nothing happened yet), I want
>to know where the 'bondary' will be ... Should I take care of my "total"
>file size ? Or the limit is per-file basis, not related with total
>codesize included ?
> Speak of SpyceBB ... I avoided namespace/parameter passing headache
>by including all files from the index.spy, and use it as a frontend
>dispatcher. However, on a development environment, because it recompiles
>all included files every time, compile time becomes longer and longer
>when the file size grows ... Therefore, is it possible to make
>[[.include file=""]] cache the compiled version of the file being
>included, and only recompiles when the file changes ?