Re: [Doxygen-develop] Support for other languages

[Dimitri v. H. <di...@st...>]

On Thu, Dec 23, 2004 at 01:49:37PM -0800, Vicki Brown wrote:
> I've been examining and comparing several "code documentation tools".
> Having looked at several now, I really like Doxygen. In fact, I would
> love to use Doxygen except for one show-stopper difficulty. The Company
> where I am currently employed (where I would most likely deploy Doxygen)
> uses Python.
> 
> But... Doxygen allows the use of input filters which convert snippets
> of code (e.g., data structure declarations, method calls) from non-C++
> languages into C++ syntax.  Doxygen is then able to create web pages,
> diagrams, etc. This looked promising.
> 
> I found pydoxy and was quite pleased ... at first. Then I discovered
> that the code listings in the resulting HTML pages _still look like
> C++_.  This is not so promising.
> 
> While I can certainly understand why the filter creates "pseudo" C++
> code out of the Python code, that isn't what the  ultimate web pages
> should show to developers who want to browse that code. The developers
> wrote Python; they want to browse Python.
> 
> The Doxygen Wish List lists "support for other languages" as a 10 in
> difficulty. I would dearly love to see this item fulfilled.
> 
> However, I do not think that all languages should be equally difficult to
> support. Languages that already have a filter (e.g. Python) _could_ be much
> easier to support if the filter mechanism can be leveraged.
> 
> If the filters passed along both the translated code and the original
> code, Doxygen could use the former for its analysis and the latter for
> its code displays.  Ideally, this would involve an extension of the
> current interface, so that existing filters would still work as they
> do now.
> 
> How difficult would this be?

You can already show the original python sources by setting 
FILTER_SOURCE_FILES to NO. The main problem is that doxygen does not
understand python so it will not do syntax highlighting and cross-referencing
properly. 

To understand this better you should know that doxygen has two independent 
"parsers" for source code:

- scanner.l: which analyses the structure of code and parses any 
  supported language to generate plain documentation. It basically parses
  only the outline of the source code (not the body of functions). 
- code.l: which filters the source code and uses the information found
  by scanner.l to keep track where functions/classes start and end.
  In this step the cross-references are computed. The filter is single pass,
  so the correct result is directly written to the output without creating
  an intermediate representation first (like a real compiler would do).

So if you transform a piece of code with a filter (and not just substitutes
some words) you should also use the same filter to get syntax-highlighting 
and cross-referencing correct (i.e. FILTER_SOURCE_FILES = YES). Since 
there is no intermediate representation for function bodies, it is very 
hard to keep a mapping of where things in the filtered source file are 
located in the original source, which is what you are asking for.

Conclusion is that it is hard to *properly* support additional languages.
In general the more a language resembles one of the supported languages the 
easier it is to add support for it. So adding support for say JavaScript is 
easier than adding support for Python, which is again easier than adding
support for VHDL.

I've been thinking about making parsers more "pluggable" but for now this
is still long term stuff.

Regards,
  Dimitri

P.S. For Python you could also look at: pydoc, happydoc, or epydoc.