#71 Spurious "invalid syntax" errors for modules

[Allan Crooks]

This was recreated in Windows, after I made a change to
the source code so we wouldn't get an ImportError if
the curses module couldn't be imported.

I get strange errors in 3.0 alpha when trying to
process some modules. I've picked a simple example out
for you.

Module:
http://cvs.sourceforge.net/viewcvs.py/\*checkout*/dopal/dopal/src/dopal/__init__.py?content-type=text%2Fplain

Renamed it to dopal_init.py and tried to run epydoc on it.

Output:

------
[
Error: Error during parsing: invalid syntax
(C:\Temp\dopal_init.py,
line 21)
[................................
Warning: Empty group u'Front-end modules' in dopal_init
Warning: Empty group u'Core-level modules' in dopal_init
Warning: Empty group u'Object-level modules' in dopal_init

Traceback (most recent call last):
File "c:\python24\lib\site-packages\epydoc\cli.py",
line 636, in ?
cli()
File "c:\python24\lib\site-packages\epydoc\cli.py",
line 315, in cli
main(options, names)
File "c:\python24\lib\site-packages\epydoc\cli.py",
line 271, in main
add_submodules=(options.action!='text'))
File
"C:\Python24\Lib\site-packages\epydoc\docbuilder.py",
line 203, in build_doc_index
val_doc.init_groups()
File
"C:\Python24\Lib\site-packages\epydoc\apidoc.py", line
689, in init_groups
del self.group_names[i], self.groups[group]
IndexError: list assignment index out of range

[Edward Loper]

Logged In: YES
user_id=195958

It looks like this actually shows two bugs: one, a problem
in deleting empty groups, which I just fixed in
subversion; and another, which causes the parse error,
which I was unable to replicate.

I don't see anything very unusual on line 21 of the file
that you sent me, so I'm not sure why it would be
generating a parse error. When I run it, I don't get
one. Try running epydoc with "-vvv", and let me know what
the 3-4 lines of output before the parse error are?

[Allan Crooks]

Logged In: YES
user_id=39733

First of all, I forgot to mention version of Python I'm
using - it's 2.4.1.

Secondly, I forgot to thank you for writing the software -
it's really great stuff. :)

The module itself imports fine, so I don't really know why
it's failing either. Anyway I can get whatever error is
causing the problem escape to the console? I can modify the
source code locally to figure it out.

Output with -vvv switch:

Anyway, with -vvv flag:

-----

Building documentation
[ 0%] dopal_init (dopal_init.py)
Info: Parsing C:\Temp\dopal_init.py
Error: Error during parsing: invalid syntax
(C:\Temp\dopal_init.py,
line 21)
Merging parsed & introspected information
Linking imported variables
[ 0%] dopal_init
Indexing documentation
[ 0%] dopal_init
Parsing docstrings
[ 0%] dopal_init
Inheriting documentation
Sorting & Grouping
[ 0%] dopal_init
Warning: Empty group u'Front-end modules' in dopal_init
Warning: Empty group u'Core-level modules' in dopal_init
Warning: Empty group u'Object-level modules' in dopal_init

Traceback (most recent call last):
File "c:\python24\lib\site-packages\epydoc\cli.py", line
636, in ?
cli()
File "c:\python24\lib\site-packages\epydoc\cli.py", line
315, in cli
main(options, names)
File "c:\python24\lib\site-packages\epydoc\cli.py", line
271, in main
add_submodules=(options.action!='text'))
File "C:\Python24\Lib\site-packages\epydoc\docbuilder.py",
line 203, in build_doc_index
val_doc.init_groups()
File "C:\Python24\Lib\site-packages\epydoc\apidoc.py",
line 689, in init_groups
del self.group_names[i], self.groups[group]
IndexError: list assignment index out of range

----

[Edward Loper]

Logged In: YES
user_id=195958

Could you try making the following change to
epydoc/docbuilder.py, on around line 373:

try:
parse_doc = parse_docs(
filename=filename, context=parent_docs[1])
except ParseError, e:
- log.error(e)
+ raise

(i.e., replace "log.error(e)" with "raise), and try
running it again? That should at least let me know what
part of the epydoc parsing code is having trouble. (I'd
do this myself, but as I said earlier, I can't replicate
the problem when I run epydoc on that file.)

[Allan Crooks]

[Allan Crooks]

Logged In: YES
user_id=39733

Figured out what it was.

First of all, my local copy of the file is different to what
you've been using - mine seems to have a combination of
different file endings for some reason. It seems the
tokenize module doesn't like this, and that's what the
syntax error is related to.

After hacking around, I figured out how you can modify the
code to fix it.

docparser.py, line 517
- module_file = codecs.open(module_doc.filename, 'r',
encoding)
+ module_file = codecs.open(module_doc.filename, 'rU',
encoding)

You'll need to check whether that version of Python supports
universal newlines or not though (it was implemented in 2.3
- I'm not sure what the minimum version of Python is going
to be for Epydoc 3).

[Edward Loper]

Logged In: YES
user_id=195958

Would you mind attaching a copy of the file with the mixed
line endings, so I can play with it?

[Allan Crooks]

Logged In: YES
user_id=39733

I had already attached the file with the last post - guess I
should have mentioned it. :)

[Edward Loper]

Logged In: YES
user_id=195958

Fixed in subversion (rev. 987)

I fixed it by adding "U" to the mode, as you suggested. I
also made the parse error messages a little more detailed,
which will hopefully make it easier to tell where the
problem is.

"U" does require python 2.3, although it appears to be
silently ignored before python 2.3. Presumably having
mixed line-termination characters should be fairly
uncommon, so I'm not too worried about it.