pasdoc-main

[Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Michalis K. <mic...@gm...>]

1. The recent French translation patch incompatibility clearly indicate
that our last release was done too long ago :) Let's change it. I want
to make PasDoc 0.12.0 release at some point next week.

We don't have *much* new stuff since last 0.11.0 release, but we have
something (see ChangeLog file). And the real goals are to
- Make an official release with new PasDoc_Languages.pas layout (as this
is heavily used by translators) and
- Show that We're Not Dead Yet :)

2. This brings me to the topic of the PasDoc2 branch. Stuff there is
still mostly not applied. The PasDoc_Languages.pas is one of the few
exceptions that actually got applied to trunk from PasDoc2 branch...

Quick test showed that the version from the PasDoc2 branch still fails
on a lot of testcases. This was mentioned e.g. here
https://sourceforge.net/mailarchive/message.php?msg_id=4883125E.2050809%40gmail.com
, and many issues are still not fixed. Some bugs that I've seen still
present: on error_introduction_twice_anchors (not detcted dupl anchors),
ok_abstract_sealed (esp. Range check error, and maybe "Warning:
unhandled section..." is also wrong), ok_back_comment (Range check
error), ok_vorbisfile. See the above mail for details, and run the test
suite (see tests/README) for more.

Even the autodoc fails (cd source/autodoc; make), with
- Such content, "
", is not allowed directly within the tag @orderedlist
- some warnings about unresolved links
- Fatal Error: bad Add mode (PasDoc_Items.pas, line 4169)

So a quick merge PasDoc2->trunk is out of the question.

So I'm trying to merge gradually, changing something to trunk/ and
changing something in branches/PasDoc2/, to eventually make these two
branches equal at the end, and bug-free. As of now, some small changes
were applied, and some were rejected (reverted to trunk state in
branches/PasDoc2/), see SVN commit logs for reasoning.

The really large architectural stuff is not really touched yet, so the
real and big decisions are still ahead.

Help in fixing PasDoc2 branch would be helpful. Please let me do the
final merging to trunk/, but everyone feel welcome to make fixes to
PasDoc2 branch to fix the issues. (Just make sure you do "svn up" often,
to synch with my own changes, as I plan to work intensively on this
within few next days.)

Ideally, autodoc and tests in tests/ subdirectory should proceed
perfectly, and then the incentive to apply the PasDoc2 branch will be
much larger.

Also, a documentation of user-visible improvements in PasDoc2 branch
would be helpful (I mean, a nice wiki page with examples, something
ready to show to users). The PasDoc2 branch supposedly includes
@groupbegin/end support, but I can't really test it since even autodoc
fails. And looking at code --- it's different than the planned usage on
http://pasdoc.sipsolutions.net/MichalisKamburelis#Support_for_groups_of_items
. So I'm not even sure if I want @groupbegin/end implemented this way,
and I don't know of other user-visible improvements from PasDoc2 branch.

I definitely want to decide before the 0.12.0 release what to do with
PasDoc2 branch. Merge what (if any) should be still merged, and
semi-officially "close" the PasDoc2 branch as a finished work. The issue
with applying PasDoc2 branch is hanging over us for ~2 years, so some
decision needs to be made, and then the new development should focus
again on trunk branch.

Michalis

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Hans-Peter D. <DrD...@ao...>]

Michalis Kamburelis schrieb:

> So a quick merge PasDoc2->trunk is out of the question.

Unfortunately I'm busy with other projects, so that I cannot help much :-(

AFAIR handling of new Delphi syntax deserves some general changes in the 
PasDoc data model, to allow e.g. for local types. That's why I didn't 
implement newer (>D7) language constructs in PasDoc2.

DoDi

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Hans-Peter D. <DrD...@ao...>]

Michalis Kamburelis schrieb:

> Quick test showed that the version from the PasDoc2 branch still fails
> on a lot of testcases. This was mentioned e.g. here
> https://sourceforge.net/mailarchive/message.php?msg_id=4883125E.2050809%40gmail.com
> , and many issues are still not fixed.

Some "bugs" can not be fixed without a deep change to the PasDoc data 
structures, so that e.g. nested declaration sections (in class 
declarations) can be handled at all. I've been waiting for according 
specifications or actions from your part, but nothing happened. So I 
only could do some more "cosmetic" changes to the existing model and 
generators, as far as I could figure out how the existing model was 
intended to work.

Another observation was the source code pollution, caused by 
documentation comments in e.g. class declarations. Such comments IMO 
should be moved into the implementation part, what would require to 
parse that part as well. That's why I concentrated on external 
description files, that are and remain applicable even after changes to 
the source code structure, and do not rely so much on extensions to the 
various language versions (Delphi, FPC...). Then parsers can be 
implemented quite independently from the internal PasDoc model, e.g. can 
be borrowed from FPC or Lazarus. Their output would provide an 
declaration tree, or a flat list of qualified identifiers, that can be 
matched with any (applicable!) PasDoc tree model, and/or with external 
topic files. The existing PasDoc model was inapplicable, up to the time 
when I left the project.


> The really large architectural stuff is not really touched yet, so the
> real and big decisions are still ahead.
> 
> Help in fixing PasDoc2 branch would be helpful. Please let me do the
> final merging to trunk/, but everyone feel welcome to make fixes to
> PasDoc2 branch to fix the issues. (Just make sure you do "svn up" often,
> to synch with my own changes, as I plan to work intensively on this
> within few next days.)

I just received a note from another contributor, who wanted to provide 
Unicode-related improvements. He seems to be blocked by your many recent 
changes to the trunk. Can you provide a roadmap, about the intended 
changes, and how other activities could fit into the revised project?


> Also, a documentation of user-visible improvements in PasDoc2 branch
> would be helpful (I mean, a nice wiki page with examples, something
> ready to show to users). The PasDoc2 branch supposedly includes
> @groupbegin/end support, but I can't really test it since even autodoc
> fails. And looking at code --- it's different than the planned usage on
> http://pasdoc.sipsolutions.net/MichalisKamburelis#Support_for_groups_of_items
> . So I'm not even sure if I want @groupbegin/end implemented this way,
> and I don't know of other user-visible improvements from PasDoc2 branch.

We differ in many ideas, how things should be made working. I've 
implemented and tested my approach(es) for feasability, but only on the 
small scale, i.e. the implementation may not work yet in certain cases. 
But unless decisions are made, how things *shall* work in a future 
PasDoc version, I consider even minor bugfixes to the existing code 
nothing but a waste of time.


> I definitely want to decide before the 0.12.0 release what to do with
> PasDoc2 branch. Merge what (if any) should be still merged, and
> semi-officially "close" the PasDoc2 branch as a finished work. The issue
> with applying PasDoc2 branch is hanging over us for ~2 years, so some
> decision needs to be made, and then the new development should focus
> again on trunk branch.

Does there exist any documentation of the (current/intended) tree 
structure, in general and in implementation details? I definitely 
deserve some concrete material, that could convince me to resume working 
on PasDoc.

DoDi

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Michalis K. <mic...@gm...>]

Hans-Peter Diettrich wrote:
> Michalis Kamburelis schrieb:
> 
>> Quick test showed that the version from the PasDoc2 branch still fails
>> on a lot of testcases. This was mentioned e.g. here
>> https://sourceforge.net/mailarchive/message.php?msg_id=4883125E.2050809%40gmail.com
>> , and many issues are still not fixed.
> 
> Some "bugs" can not be fixed without a deep change to the PasDoc data 
> structures, so that e.g. nested declaration sections (in class 
> declarations) can be handled at all. I've been waiting for according 
> specifications or actions from your part, but nothing happened. So I 
> only could do some more "cosmetic" changes to the existing model and 
> generators, as far as I could figure out how the existing model was 
> intended to work.

My objections were about bugs in your PasDoc2 branch code, bugs that
didn't exist in original trunk code. I honestly don't know what
"specifications" or "actions" on my part you were waiting for. These
bugs didn't exist in the trunk code in the first place. Some of them are
range check errors when parsing valid code, and I can't imagine how any
"specification" or (more) architectural changes of pasdoc would help
here. Bugs need to be fixed, and when you introduce them --- it's
usually your job to fix them.

I understand that you didn't have time to do it. It's Ok. Neither did I.
Just don't blame it on waiting for some action on my part, something
that I had no idea about.

> Another observation was the source code pollution, caused by 
> documentation comments in e.g. class declarations. Such comments IMO 
> should be moved into the implementation part, what would require to 
> parse that part as well. That's why I concentrated on external 
> description files, that are and remain applicable even after changes to 
> the source code structure, and do not rely so much on extensions to the 
> various language versions (Delphi, FPC...). 

This is an old question, whether documentation should go to the
interface, and thus "pollute" the interface. In my opinion yes, because
this makes interface more useful. But I agree that it's a matter of
taste, I wrote my opinion about this on
http://pasdoc.sipsolutions.net/MichalisKamburelis#Should_documentation_be_placed_inside_units_interface.2C_or_in_separate_files_.3F

>Then parsers can be 
> implemented quite independently from the internal PasDoc model, e.g. can 
> be borrowed from FPC or Lazarus. Their output would provide an 
> declaration tree, or a flat list of qualified identifiers, that can be 
> matched with any (applicable!) PasDoc tree model, and/or with external 
> topic files. The existing PasDoc model was inapplicable, up to the time 
> when I left the project.

So you have to parse the code anyway, in any approach (comments in the
interface, or implementation, or external files). The reasons for using
our own parser were, and still are, that

1. At least fcl-passrc still doesn't offer all the features. There were
many improvements recently, but still e.g. FPC macros are not supported.
Not to mention that latest Delphi's features (which are the main lack of
pasdoc according to bugreports) aren't probably implemented there
either, as FPC doesn't handle many of them.

2. Noone implemented it. Using fcl-passrc, or any other open parser (I
remember at least JEDI parser mentioned in the past) is always a welcome
option. But you have to prove that it works (on testsuite, on
autodoc). Right now, since we know that fcl-passrc is not perfect,
implementing it as an alternative parser (that can be used instead of
the current one, activated e.g. by command-line option) would probably
be the right way to go.

Whether PasDoc_Items model should be upgraded for new Delphi features is
IMO orthogonal to the fact that parser should be upgraded / replaced.
I'm all after upgrading the PasDoc_Items model, as long as it's done in
a way that can be applied and doesn't break stuff that is already
working correctly.

> I just received a note from another contributor, who wanted to provide 
> Unicode-related improvements. He seems to be blocked by your many recent 
> changes to the trunk. Can you provide a roadmap, about the intended 
> changes, and how other activities could fit into the revised project?

Looks like I also received a mail from him, looks like he's talking to
us in parallel. And his work is completely applied to trunk now, see my
mail "Support for Delphi Unicode compilers" a second ago.

Although I didn't hear from him about being blocked by my changes. And
it would be strange, as we had 2 years of inactivity (except applying
small patches and translations), followed by changes mostly to
pasdoc_gui in recent days. I honestly can't see how I could do *less*
work, so I really can't imagine how I'm blocking someone. The pasdoc
architecture stayed virtually untouched for 2 years, plenty of time if
someone wants to provide patches. And many people did, and I tried to
apply them quickly --- as long as I could see that the change is good
and code compiles and works.

> 
> We differ in many ideas, how things should be made working. I've 
> implemented and tested my approach(es) for feasability, but only on the 
> small scale, i.e. the implementation may not work yet in certain cases. 

Your implementation doesn't work. And instead of fixing the bugs, you
only respond with emails talking how "the model is inapplicable" or
such. I don't know how you tested your code, but it *doesn't work*.
Autodoc fails, testsuite fails. These both tests were available to you
in pasdoc SVN, and mentioned to you many times. See previous mails for
details.

> But unless decisions are made, how things *shall* work in a future 
> PasDoc version, I consider even minor bugfixes to the existing code 
> nothing but a waste of time.

You seem to demand from me some big decision, where I don't see a need
to make one. The "plan" is to keep PasDoc working, parsing Pascal
sources with latest Delphi and FPC features, and make output useful.
Whoa, that was deep.

Everything may be changed (alternative parser, PasDoc_Items tree, etc.)
when it's needed. Just do it as gradually as possible, and don't
introduce bugs (or at least fix them afterwards).

> 
> Does there exist any documentation of the (current/intended) tree 
> structure, in general and in implementation details? I definitely 
> deserve some concrete material, that could convince me to resume working 
> on PasDoc.
> 

The autodoc http://pasdoc.sipsolutions.net/PasDocAutoDoc is our
documentation.

Michalis

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Hans-Peter D. <DrD...@ao...>]

Michalis Kamburelis schrieb:

> My objections were about bugs in your PasDoc2 branch code, bugs that
> didn't exist in original trunk code.

Let me try to separate the modifications:

> I honestly don't know what
> "specifications" or "actions" on my part you were waiting for. These
> bugs didn't exist in the trunk code in the first place.

Some hacks have been removed, that had been introduced in trunk for 
parsing new Delphi syntax, but without according handling of the 
syntactic constructs. The consequential errors indicate the incapability 
of handling such post-D7 constructs, due to limitations in the PasDoc 
tree model. Also a specification of the new Delphi grammar is required, 
not available to me at that time.

> Some of them are
> range check errors when parsing valid code,

I didn't come across such errors, and wasn't informed about such situations.

> and I can't imagine how any
> "specification" or (more) architectural changes of pasdoc would help
> here.

The PasDoc model does not include nested declarations, according to new 
Delph syntax; e.g. type, var and const declarations inside class 
declarations.

> Bugs need to be fixed, and when you introduce them --- it's
> usually your job to fix them.

When a program says "done" without doing anything, then I consider this 
fake behaviour a bug.


> I understand that you didn't have time to do it. It's Ok. Neither did I.
> Just don't blame it on waiting for some action on my part, something
> that I had no idea about.

Sorry for the lack of communication. You should have understood the 
implications of the new Delphi syntax, and their impact on the PasDoc 
tree model, in both parser and generator code. The inappropriate tree 
structure also prevents handling of local (nested) procedures, that's 
why I could not extend the parser to also process implementation sections.



>> We differ in many ideas, how things should be made working. I've 
>> implemented and tested my approach(es) for feasability, but only on the 
>> small scale, i.e. the implementation may not work yet in certain cases. 
> 
> Your implementation doesn't work.

It works with Delphi syntax up to D7.

> And instead of fixing the bugs, you
> only respond with emails talking how "the model is inapplicable" or
> such.

Right. Please provide a tree model that can deal with nested 
declarations, or let somebody else provide it.


> You seem to demand from me some big decision, where I don't see a need
> to make one.

Sorry that I could not make it more clear to you :-(

 > The "plan" is to keep PasDoc working, parsing Pascal
> sources with latest Delphi and FPC features, and make output useful.
> Whoa, that was deep.

Then add wings to PasDoc first, before you ask anybody else to make it fly.

> Everything may be changed (alternative parser, PasDoc_Items tree, etc.)
> when it's needed. Just do it as gradually as possible, and don't
> introduce bugs (or at least fix them afterwards).

It was not my task to modify the tree model.

>> Does there exist any documentation of the (current/intended) tree 
>> structure, in general and in implementation details? I definitely 
>> deserve some concrete material, that could convince me to resume working 
>> on PasDoc.
>>
> 
> The autodoc http://pasdoc.sipsolutions.net/PasDocAutoDoc is our
> documentation.

You are kidding. That "documentation" says nothing about the existing 
tree structure, nor about the handling of nested declarations, nor about 
the language syntax that is or should be implemented.

DoDi

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Michalis K. <mic...@gm...>]

Hans-Peter Diettrich wrote:
>> Some of them are
>> range check errors when parsing valid code,
> 
> I didn't come across such errors, and wasn't informed about such situations.

See the mail at the beginning of this thread, where I start with
mentioning which tests fail. And see the referenced mails two years ago.

https://sourceforge.net/mailarchive/message.php?msg_name=4CC36F84.9060207%40gmail.com

https://sourceforge.net/mailarchive/message.php?msg_id=4883125E.2050809%40gmail.com

https://sourceforge.net/mailarchive/message.php?msg_name=492059DE.6060607%40gmail.com

Michalis

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Hans-Peter D. <DrD...@ao...>]

Michalis Kamburelis schrieb:
> Hans-Peter Diettrich wrote:
>>> Some of them are
>>> range check errors when parsing valid code,
>> I didn't come across such errors, and wasn't informed about such situations.
> 
> See the mail at the beginning of this thread, where I start with
> mentioning which tests fail.

All tests with post-D7 syntax will fail, of course, because I removed 
all new syntax elements from the parser. I don't consider such files 
valid code, for now.

Once provisions for nested declarations have been added to PasDoc, the 
parser has to be reorganized accordingly, and *after all that* the new 
syntax elements can be added, and possibly also parsing of the 
implementation part.

DoDi

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Michalis K. <mic...@gm...>]

Hans-Peter Diettrich wrote:
>> See the mail at the beginning of this thread, where I start with
>> mentioning which tests fail.
> 
> All tests with post-D7 syntax will fail, of course, because I removed 
> all new syntax elements from the parser. I don't consider such files 
> valid code, for now.

A lot of stuff broken in PasDoc2 branch doesn't have any relation to
post-Delphi7 features. Just read the mails where I point out which tests
fail, and stop covering behind non-existing issues. Some examples (once
again): pasdoc parsing its own code (while pasdoc source doesn't use
post-D7 features, yet they fail around @orderedlist and "bad Add mode"),
ok_back_comment (//< comments), ok_vorbisfile (fpc macros).

And, a second note: you removed support for some post-D7 features,
because you considered the code hacky, and then you say you wait for
some specifications what to do next. Well, here's how things are
supposed to work: if you remove some code, then be prepared to know how
to replace it with something better. You are responsible for doing a
better design and proving that it works. You may get some help from
others, but not until I can see that you bring some advantages --- in
this case, you made some changes that I consider ugly (mixing generators
and scanners knowledge) and you introduced bugs (*not* only at post-D7
features).

Right now, the end result of PasDoc2 branch:
1. You broke some post-D7 features (which I can accept, but only as an
intermediate step to implementing them even better),
2. You broke some stuff not related to post-D7 features (which is a
verifiable fact by simply running the testsuite, yet you somehow manage
to ignore it in your emails).

Sorry if it sounds harsh, but that's the way I see it. A single commit
from you to PasDoc2 branch, attempting to fix at least a single problem
there, may change my mind. Another mail, where you repeat that you wait
for someone else to design a better model, and ignore actual bugs that
are pointed out to you, will not help in advancing the PasDoc2 branch.

Michalis

Re: [Pasdoc-main] PasDoc 0.12.0 release next week, and decisions for PasDoc2 branch

[Michalis K. <mic...@gm...>]

Michalis Kamburelis wrote:
> Right now, the end result of PasDoc2 branch:

Reading it back, maybe it was too harsh. After all, some parts of
PasDoc2 branch were applied.

My main point was that I see many bugs in PasDoc2 branch (breaking not
only post-D7 features), which I mentioned many times, and I don't see
them getting fixed. I'm all after blaming it on the lack of time of
everyone, and closing this discussion.

Michalis