srcml-discussion — General discussions about the SrcML framework

Re: [SrcML] static source code analyzis

[Frank R. <fra...@in...>]

On Wed, May 04, 2005 at 01:36:31AM +0200, Leif Bladt wrote:
> My main goal is a tool which supports the programmer with static 
> analyses (like possible errors, unused code, will be defined later in 
> detail). Therefore a plug-in into an existing IDE (like Eclipse), 
> instead of a single GUI-application, would be fine. Because I don't 
> have a clue how to write an Eclipse plug-in, I thought about an Ant 
> plug-in (this is probably just a normal java program). 

Ant plugin? Ant is a build platform which is comparable to GNU make.
I don't think it makes sense to plug into that.

As for an Eclipse plugin this has been planned for a while now. We can
offer you a book on the subject (Gamma et.al: Contributing to Eclipse)
in which you can read up on how plugins for Eclipse are created. Basically
it's a very simple process of writing a special XML file for your plugin.

The statical analyses per se don't really require Eclipse, so the Eclipse
integration would rather mean making the complete SrcML Framework available
within Eclipse (i.e. allowing the current file/project to be parsed). We'll
have to think of how to get the plugins done easily and still be able to
run the rest without Eclipse as well.

> So for 
> this tool I will need the "analyze platform" which handles the analyze 
> plug-ins, and the plug-ins themselves. I'm still thinking about a 
> reasonable set of plug-ins. be patient :-)

The platform will be neccessary of course. What I'm wondering (without having
thought about it in lengths yet, so I guess you're ahead of me there) is
whether we should have different types of plugins for 1) performing the
analysis and 2) presenting the result. The Eclipse plugin should then be a
plugin of the second type. And of course one to invoke the platform and the
plugins of type 1.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

Not all comments are bad. But they are generally deodorant;
they cover up mistakes in the code. (Christian Sepulveda)

[SrcML] static source code analyzis

[Leif B. <lei...@in...>]

I explored the field of "static source code analyzis" a bit, and wanted 
to ask you what you think about it...

My main goal is a tool which supports the programmer with static 
analyses (like possible errors, unused code, will be defined later in 
detail). Therefore a plug-in into an existing IDE (like Eclipse), 
instead of a single GUI-application, would be fine. Because I don't 
have a clue how to write an Eclipse plug-in, I thought about an Ant 
plug-in (this is probably just a normal java program). This way you can 
also include this in your build process and could be useful. So for 
this tool I will need the "analyze platform" which handles the analyze 
plug-ins, and the plug-ins themselves. I'm still thinking about a 
reasonable set of plug-ins. be patient :-)

Now feel free to criticize me and tell me what you think about.

Leif Bladt

[SrcML] new plugin system

[Frank R. <fra...@in...>]

Just a quick update on the new plugin system:

All modules in CVS have been adjusted for the new system by now and all
plugins are now placed in their own directories. I've also modified the
build.xml scripts in order to reduce the amount of manual labor required
to get things going again. (Make sure you update with "cvs update -d" to
also get the new directories)

Documentation for the new plugin system is already available in the SrcML
Wiki on srcml.de and of course you can see how it works by checking out
config/PluginManager.java . Right now we don't yet have an implementation
for the dependency graph and the automatic loading of the dependency
plugins. But Simon should be able to add that within the next week.

Note that this is not related to the previous API discussions (about whether
or not the API will be pluggable), but was needed simply because the old
plugin system was unsatisfying. With the new system a plugin is really
independent from the applications (unless of course you develop your own
application and need to access specific methods in the plugin class which
are not available for all plugins of that type). Plugins can now be published
separately as .jar files and the installation consists of copying that file
into ~/.srcml/plugins/ . That's all there is to it. The PluginManager will
take care of making the plugin available then. 

For more details and usage examples take a look at the wiki documentation
or reply to the mailinglist.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

You can be discouraged by failure, or you can learn from it. So go ahead and 
make mistakes, make all you can. Because, remember that's where you'll find
success - on the far side of failure. (Thomas J Watson Sr)

[SrcML] API redesign

[Dennis W. <den...@in...>]

So,

we have been talking about what changes to make to the API. After today,
the most recent stat would be this I guess:

Replacing elements of the DOM tree still is the main part of the "API".
API elements will implement interfaces to provide functionality, so
languages with different syntax and equal semantics can be handlen
without much hassle.
For example: SrcML currently has a loop tag which stands for all kinds
of loops (while, for etc. language-independent ) - if I'm wrong here
correct me please. So if to language share a similar "for" concept which
contains a init part, a condition part and a "action" part, both will
replace the DOMElements with their implementations (if neccessary) -
with both implementing a special "for" interface which allows access to
those 3 parts. 
An application therefore needn't care too much of what underlying
language is used, as long as the semantics are equal.

That means every supported language will have a "helper" class which
provides the means to transform the DOM tree. 
(As a sidenote to Frank: thinking about this will also provide
us the means to add things like refactoring - as we discussed
yesterday, and this still seems to be able to support a pluggable
structure as those interfaces are provided with SrcMl while the
language plugins might be loaded).

I think Frank is as eager as I am to know what other people think about
this ?

-
Dennis

Re: [SrcML] new plugin handling and API structure

[Dennis W. <den...@in...>]

On Thu, 21 Apr 2005 19:39:58 +0200
Frank Raiser <fra...@in...> wrote:

> Now what is the 'order' of API plugins?
> Let's say we have a simple dependency on one other plugin. Then this
> dependency plugin will be the first to be loaded. Therefore its
> dependencies will recursively be loaded before the dependency plugin
> itself is loaded. The last one to load is the actual plugin we wanted
> to load. Now if we consider the possibility of influencing the order
> by the order in which the dependency plugins are listed in plugin.xml
> (by adding the corresponding edges to the dependency tree), the plugin
> developer basically  has free choice over the first depth level of the
> tree. I don't know whether you really plan on giving a plugin
> developer the ability to define the order in which dependencies of
> dependencies are resolved? If not I think this approach should be
> pretty good for us already.
Uhm... I actually was trying to say that this approach looks good. And
the order by using this method is clear. A programer wouldn't care for
order of the "super" plugin dependencies. Never thought that this would
work out after all.
 
> Ok. So we have been talking about two different things here. I never
> intended to enforce a program restart in order to load another
> document with another API. My idea was to prepare the match table just
> before the SrcML class is instantiated for a document. Then for
> another document there'd be a method to reset the match table and
> prepare a new one for the new document. So for me this match table is
> only important while instantiating the SrcML class and modifications
> to it do not affect existing DOM trees (which should remain unchanged
> to avoid confusion). So to sum it up from my point of view:
> - new document takes a new match table
> - only one match table per document
That is actually what I meant. And if you check my old mail you see
while I called a method after creation of SrcML the basic idea is the
same no matter if you do it when instancing or setting it up after it is
created.

-
Dennis

Re: [SrcML] new plugin handling and API structure

[Frank R. <fra...@in...>]

On Thu, Apr 21, 2005 at 07:09:46PM +0200, Dennis Waldherr wrote:
> I'm not worrying of runtime speed, I just want to make sure that it will
> be clear that the topologic sort will not change order of API plugins.

Now what is the 'order' of API plugins?
Let's say we have a simple dependency on one other plugin. Then this
dependency plugin will be the first to be loaded. Therefore its dependencies
will recursively be loaded before the dependency plugin itself is loaded. The
last one to load is the actual plugin we wanted to load.
Now if we consider the possibility of influencing the order by the order in
which the dependency plugins are listed in plugin.xml (by adding the
corresponding edges to the dependency tree), the plugin developer basically 
has free choice over the first depth level of the tree.
I don't know whether you really plan on giving a plugin developer the ability
to define the order in which dependencies of dependencies are resolved? If not
I think this approach should be pretty good for us already.

> Of course it would be confusion to iterate a transformation upon a tree.
> My intend was to allow for different APIs to be used for the first
> transformation from xml to API. If the API was fixed after loading one
> could not write a application which handles different languages - unless
> he restarts his own program or uses multiple java instances.

Ok. So we have been talking about two different things here. I never intended
to enforce a program restart in order to load another document with another
API. My idea was to prepare the match table just before the SrcML class is
instantiated for a document. Then for another document there'd be a method to
reset the match table and prepare a new one for the new document.
So for me this match table is only important while instantiating the SrcML
class and modifications to it do not affect existing DOM trees (which should
remain unchanged to avoid confusion).
So to sum it up from my point of view:
- new document takes a new match table
- only one match table per document

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

Ever tried. Ever failed. 
No matter. Try again. Fail again. Fail better. (Samuel Beckett)

Re: [SrcML] new plugin handling and API structure

[Dennis W. <den...@in...>]

On Thu, 21 Apr 2005 17:43:23 +0200
Frank Raiser <fra...@in...> wrote:

[Plugin structure]
Looks ok to me.

> 
> > I meant especially an API plugin. But while normal plugins will
> > usually have  more dependencies I'm not sure that there will be any
> > topologic reorder at all.
> 
> While it might be true that for API plugins a topological sorting is
> unneccessary I still think they should be treated just like other
> plugins. Doing no topological reordering for API plugins creates a
> special case that developers will have to be aware of again. I'd
> sacrifice this little bit of runtime to sorting the dependency tree
> before I require developers to always remember these plugins will be
> an exception to the rule.
I'm not worrying of runtime speed, I just want to make sure that it will
be clear that the topologic sort will not change order of API plugins.
 
> > Since you save language type within a SrcML file you can load a
> > default match  table given by that name. There is no inconsistency
> > because loading a match table would not change the already loaded
> > tree.
> 
> It's not getting inconsistent at runtime. I think the problem is that
> from my point of view it is preferrable to keep a once loaded tree
> fixed. I believe it will create too much confusion if your program
> decides at some point to simply replace the tree with new nodes which
> essentially means that at this point in the program the underlying API
> will be changed.
Of course it would be confusion to iterate a transformation upon a tree.
My intend was to allow for different APIs to be used for the first
transformation from xml to API. If the API was fixed after loading one
could not write a application which handles different languages - unless
he restarts his own program or uses multiple java instances.
 
> > But I guess there's no need to further discuss this matter as we
> > already did that  personally.
> Just to try summing it up for the other readers on this list it comes
> down to a matter of personal taste. The question right now is which
> way will we go for the actual implementation? I especially would like
> to hear other people's opinion about this too.
That would be interesting.
 
> There's also the option to mix both ideas. Let's say a plugin
> specifies two plugins it depends on. Both plugins provide a similar
> methodology which could cause ambiguity. While we could simply throw
> both of them into the dependency graph and basically ignore this
> ambiguity letting either plugin win we could honor the order these
> dependencies are listed in the plugin.xml by adding additional edges
> to the dependency graph. In this case it would mean adding an edge
> from the first plugin to the second to ensure the provided
> functionality of the second will get overwritten by whatever the first
> plugin provides. Therefore the plugin developer can specify the
> loading order dynamically.
That should do the trick too of course.
 
-
Dennis

Re: [SrcML] new plugin handling and API structure

[Frank R. <fra...@in...>]

On Thu, Apr 21, 2005 at 02:10:03PM +0200, Dennis Waldherr wrote:
> On Wed, 20 Apr 2005 23:30:03 +0200
> Frank Raiser <fra...@in...> wrote:
> > What kind of information are you thinking of there? I'd like to have the
> > meta-data part of the config.xml to strictly adhere to a XML schema to have a
> > common ground for plugins.
> Information about author and homepage for example are not required for usual 
> operation of a plugin because it is content meant to be read by humans. 
> Therefore, instead of placing it at the root header element of the xml file
> a subelement would fit better.
> But this - I guess - is just matter of taste.

This makes sense of course. So let's try to keep the operation specific
meta-data within the plugin attributes and the parts most interesting to
humans separated from that.

So here's my current suggestion for the plugin.xml format which will be shared
for all kinds of plugins:

<!-- operational meta-data to create a unique identifier for this plugin,
to describe the type and provide a Class as entry point -->
<plugin
    name="some interesting name"
    version="x.y.z"
    type="parser"
    classname="foo.bar.Baz"
    >

    <!-- authors of the plugin -->
    <authorgroup>
        <author>
        <!-- 
        author content as described by docbook DTD:
        http://www.docbook.org/tdg5/en/html/author.html
        -->
        </author>
        <author>
        ...
        </author>
    </authorgroup>
    
    <!-- accompanying webpage - optional -->
    <link linkend="http://get.my.plugin/here"/>

    <!-- dependencies on other plugins - optional -->
    <dependencies>
        <depends name="other plugin" version="x.y.z"/>
        <depends name="another plugin" version="u.v.w"/>
    </dependencies>

    <!-- plugin specific content -->
</plugin>

> I meant especially an API plugin. But while normal plugins will usually have 
> more dependencies I'm not sure that there will be any topologic reorder at all.

While it might be true that for API plugins a topological sorting is
unneccessary I still think they should be treated just like other plugins.
Doing no topological reordering for API plugins creates a special case that
developers will have to be aware of again. I'd sacrifice this little bit of
runtime to sorting the dependency tree before I require developers to always
remember these plugins will be an exception to the rule.

> > Just a remark here: srcML is a project from kent.edu. We agreed on using SrcML
> > for our project to avoid too much confusion.
> I doubt it's really possible to avoid confusion much.

Indeed. It's a rather pathetic try to at least reduce the amount of confusion
:)

> > What I really dislike about this approach though is the need to explicitly
> > call a loadMatchTable _after_ loading the file. So either you load the file
> > and there is no DOM tree created yet, then I consider this bad, as I only ever
> > want to write one line to get the whole tree set up from the file. And if the
> > tree is already created it would consist of dom4j DOM nodes and thus we would
> > have different kinds of trees before and after the call to loadMatchTable().
> > This is a design inconsistency I'd like to avoid.
> Since you save language type within a SrcML file you can load a default match 
> table given by that name. There is no inconsistency because loading a match table
> would not change the already loaded tree.

It's not getting inconsistent at runtime. I think the problem is that from my
point of view it is preferrable to keep a once loaded tree fixed. I believe it
will create too much confusion if your program decides at some point to simply
replace the tree with new nodes which essentially means that at this point in
the program the underlying API will be changed.

> But I guess there's no need to further discuss this matter as we already did that 
> personally.

Just to try summing it up for the other readers on this list it comes down to
a matter of personal taste. The question right now is which way will we go for
the actual implementation? I especially would like to hear other people's
opinion about this too.

> Yes and therefore no ambiguity because the import order of the xml file would
> not have bee changed whereas the loading order of plugins is dynamic.

Here we are at the point of either topologically sorting by plugin
dependencies or demanding the developer to specify the plugin load order. Once
more we have two different opinions here and it'd be interesting to get to
hear from a few more people.
There's also the option to mix both ideas. Let's say a plugin specifies two
plugins it depends on. Both plugins provide a similar methodology which could
cause ambiguity. While we could simply throw both of them into the dependency
graph and basically ignore this ambiguity letting either plugin win we could
honor the order these dependencies are listed in the plugin.xml by adding
additional edges to the dependency graph. In this case it would mean adding an
edge from the first plugin to the second to ensure the provided functionality
of the second will get overwritten by whatever the first plugin provides.
Therefore the plugin developer can specify the loading order dynamically.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

When you have eliminated the impossible, whatever remains,
however improbable, must be the truth. ("Sherlock Holmes")

Re: [SrcML] new plugin handling and API structure

[Dennis W. <den...@in...>]

On Wed, 20 Apr 2005 23:30:03 +0200
Frank Raiser <fra...@in...> wrote:

> What kind of information are you thinking of there? I'd like to have the
> meta-data part of the config.xml to strictly adhere to a XML schema to have a
> common ground for plugins.
Information about author and homepage for example are not required for usual 
operation of a plugin because it is content meant to be read by humans. 
Therefore, instead of placing it at the root header element of the xml file
a subelement would fit better.
But this - I guess - is just matter of taste.

 
> Yes a plugin will have more than one dependency. We already have that right
> now. Consider a view plugin which is supposed to output plain source code. Now
> we also need the PrintSelectionPlugin and the WhitespacePlugin for it to work.
> And of course we want to use our own Input and Output Plugins too. So there's
> already a whole bunch of possible plugin dependencies.
> It shouldn't be much of a problem though to treat this more generally and 
> support a complete tree (and including a cycle test of course). So I don't
> really see a reason to restrict plugins to strictly depend on one other 
> plugin only.
> 
> Actually I think just about every bigger plugin will need several other
> plugins. Most plugins will make use of input or output plugins, they will
> require parts of the API and in case of extending another plugin they will
> also depend on that plugin.
I meant especially an API plugin. But while normal plugins will usually have 
more dependencies I'm not sure that there will be any topologic reorder at all.
 
> Just a remark here: srcML is a project from kent.edu. We agreed on using SrcML
> for our project to avoid too much confusion.
I doubt it's really possible to avoid confusion much.
 
> What I really dislike about this approach though is the need to explicitly
> call a loadMatchTable _after_ loading the file. So either you load the file
> and there is no DOM tree created yet, then I consider this bad, as I only ever
> want to write one line to get the whole tree set up from the file. And if the
> tree is already created it would consist of dom4j DOM nodes and thus we would
> have different kinds of trees before and after the call to loadMatchTable().
> This is a design inconsistency I'd like to avoid.
Since you save language type within a SrcML file you can load a default match 
table given by that name. There is no inconsistency because loading a match table
would not change the already loaded tree.
But I guess there's no need to further discuss this matter as we already did that 
personally.

 
> I also don't see how the ambiguity is solved any different to the earlier
> proposal. If I understood you correctly the extra file will contain the import
> statements from several plugins in the same order as we would have gotten the
> plugin instantiations through the dependency tree. Or what happens in case of
> a plugin wanting to match 'if' to a class and another plugin wanting to match
> it to another class? Won't it end up being the same with the more specialized
> plugin overwriting the matching of the earlier plugin?
Yes and therefore no ambiguity because the import order of the xml file would
not have bee changed whereas the loading order of plugins is dynamic.
 
> As for the custom view I don't think I really want to have that dynamically
> available through a loadMatchTable() call. I'm not really fond of the idea
> that the DOM nodes are exchanged, as this also exchanges the semantics and
> available methods behind the nodes I can see that getting way too confusing.
> If you want a custom view you are getting just that by specifying the exact
> plugins you need for your view anyways.
Yes, thinking of that a plugin would make more sense for it.
 
> >  > Now we can take full advantage of the new PluginManager. First the
> > > dependency tree will be built. The first plugin will then be
> > > instantiated first and this means that the API will from now on match
> > > every 'if' tag with the my.own.If class. After that the new plugin
> > > will be instantiated replacing this match so that every 'if' tag is
> > > now represented in the DOM tree through my.own.improved.If .
> > I think "plugins" differ a bit from the matching which seems to me more
> > like a class bases structure (therefore the text above).
> 
> Indeed the matching part is a speciality of API plugins, but I try to think of
> this plugin instantiation as a general mean for plugins to set up their
> working space. If you think of something like Eclipse you could have a plugin
> do simple things like adding its icons or menu entries to the program at the
> time it gets loaded for example. Setting up the matching table is just the
> equivalent of what an API plugin is doing to prepare its use.
My perspective doesn't require to change that. As we discussed those match
tags will be placed in the plugin's xml file. The plugin would be still the
same in both cases, it just would access it's configuration differently.

> > > Once this is realized applications can use whichever version of the
> > > API they require. All that is needed is a single line in the beginning
> > > telling the SrcML framework what the application expects:
> > > 
> > > Plugin p = PluginManager.loadPlugin(
> > >     "my own improved API plugin", "0.0.3", PluginManager.TYPE_API);
> > Hmm I'm not even sure if the thing we were discussing about actually is
> > a plugin. Of course a dynamic plugin structure might still be a good
> > idea.
> 
> In case of developing a new plugin you don't need such a loadPlugin()
> statement of course, as it will already be loaded through the dependency
> mechanism. 
As before I was especially talking about API plugins - because that was actually
the whole instigator of this discussion.
 
> But once you are developing a normal application there is no automatic
> dependency checking and loading available. In that case the application itself
> will simply try to load the neccessary plugins once they're needed. This would
> even cut down on the size of published .class/.jar applications, as you don't
> even have to include the complete API. Correct me if I'm wrong there, but it
> should be possible to 'link' the application's .class files to the API
> java/class files and then run the application without those other class files
> being available (in the CLASSPATH). If this works loading the plugins through
> the application will at least provide us with proper error messages. If it
> doesn't we might have to think of another way of ensuring the enduser knows
> what plugin is missing. A simple traceback that class ThisOrThat wasn't found
> will not be sufficient of course.
Yes, the ClassLoader will throw an error/exception when attempting to load the
class.

 
-
Dennis

Re: [SrcML] new plugin handling and API structure

[Frank R. <fra...@in...>]

On Wed, Apr 20, 2005 at 10:55:01PM +0200, Dennis Waldherr wrote:
> > <plugin 
> >     name="plugin name" 
> >     version="x.y.z"
> >     homepage="http://find.the.plugin.here"
> Instead of this attribute I'd rather have a new tag for custom
> information and include a "homepage" tag in there. This can also include
> author information etc. and make it more extensible.
> 
> >     type="parser"
> >     classname="de.srcml.foo.Bar"
> >     >
> >   <authors>
> >     <!-- typical authors markup. might be best if similar to docbook
> >     -->
> >   </authors>
> See above

What kind of information are you thinking of there? I'd like to have the
meta-data part of the config.xml to strictly adhere to a XML schema to have a
common ground for plugins.

> > 2) Once the correct plugin is found the dependencies will be checked
> > recursively. To this end a dependency tree is built and topologically
> > sorted.
> I'm still not sure if a plugin will have more than one dependency to
> other plugins. This is especially problematic because the only plugins I
> can imagine working parallel are filter plugins which do not require a
> stream if any kind and won't modify the underlying content.

Yes a plugin will have more than one dependency. We already have that right
now. Consider a view plugin which is supposed to output plain source code. Now
we also need the PrintSelectionPlugin and the WhitespacePlugin for it to work.
And of course we want to use our own Input and Output Plugins too. So there's
already a whole bunch of possible plugin dependencies.
It shouldn't be much of a problem though to treat this more generally and 
support a complete tree (and including a cycle test of course). So I don't
really see a reason to restrict plugins to strictly depend on one other 
plugin only.

Actually I think just about every bigger plugin will need several other
plugins. Most plugins will make use of input or output plugins, they will
require parts of the API and in case of extending another plugin they will
also depend on that plugin.

> I've got a proposal:
> (I'll currently ignore the plugins)
> Why not make the matching table an extra file ?
> This would - besides the "match" tag - include one or several import
> tags: 
> <import name="myBaseMatchTableFile" version="x.y.z">
> 
> Imports overwrite matching defined above them. This would remove the
> ambiguity problem, which plugins have, for match tables.
> Additionally developers can use this to - kind of - create a custom
> "view" of the underlying srcMl tree. You'd have to call
> SrcML.loadMatchTable(...) after loading the srcML file and before
> actually letting the system replace the nodes.

Just a remark here: srcML is a project from kent.edu. We agreed on using SrcML
for our project to avoid too much confusion.

What I really dislike about this approach though is the need to explicitly
call a loadMatchTable _after_ loading the file. So either you load the file
and there is no DOM tree created yet, then I consider this bad, as I only ever
want to write one line to get the whole tree set up from the file. And if the
tree is already created it would consist of dom4j DOM nodes and thus we would
have different kinds of trees before and after the call to loadMatchTable().
This is a design inconsistency I'd like to avoid.

I also don't see how the ambiguity is solved any different to the earlier
proposal. If I understood you correctly the extra file will contain the import
statements from several plugins in the same order as we would have gotten the
plugin instantiations through the dependency tree. Or what happens in case of
a plugin wanting to match 'if' to a class and another plugin wanting to match
it to another class? Won't it end up being the same with the more specialized
plugin overwriting the matching of the earlier plugin?

As for the custom view I don't think I really want to have that dynamically
available through a loadMatchTable() call. I'm not really fond of the idea
that the DOM nodes are exchanged, as this also exchanges the semantics and
available methods behind the nodes I can see that getting way too confusing.
If you want a custom view you are getting just that by specifying the exact
plugins you need for your view anyways.

>  > Now we can take full advantage of the new PluginManager. First the
> > dependency tree will be built. The first plugin will then be
> > instantiated first and this means that the API will from now on match
> > every 'if' tag with the my.own.If class. After that the new plugin
> > will be instantiated replacing this match so that every 'if' tag is
> > now represented in the DOM tree through my.own.improved.If .
> I think "plugins" differ a bit from the matching which seems to me more
> like a class bases structure (therefore the text above).

Indeed the matching part is a speciality of API plugins, but I try to think of
this plugin instantiation as a general mean for plugins to set up their
working space. If you think of something like Eclipse you could have a plugin
do simple things like adding its icons or menu entries to the program at the
time it gets loaded for example. Setting up the matching table is just the
equivalent of what an API plugin is doing to prepare its use.

> > Once this is realized applications can use whichever version of the
> > API they require. All that is needed is a single line in the beginning
> > telling the SrcML framework what the application expects:
> > 
> > Plugin p = PluginManager.loadPlugin(
> >     "my own improved API plugin", "0.0.3", PluginManager.TYPE_API);
> Hmm I'm not even sure if the thing we were discussing about actually is
> a plugin. Of course a dynamic plugin structure might still be a good
> idea.

In case of developing a new plugin you don't need such a loadPlugin()
statement of course, as it will already be loaded through the dependency
mechanism. 

But once you are developing a normal application there is no automatic
dependency checking and loading available. In that case the application itself
will simply try to load the neccessary plugins once they're needed. This would
even cut down on the size of published .class/.jar applications, as you don't
even have to include the complete API. Correct me if I'm wrong there, but it
should be possible to 'link' the application's .class files to the API
java/class files and then run the application without those other class files
being available (in the CLASSPATH). If this works loading the plugins through
the application will at least provide us with proper error messages. If it
doesn't we might have to think of another way of ensuring the enduser knows
what plugin is missing. A simple traceback that class ThisOrThat wasn't found
will not be sufficient of course.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

When you can flatten entire cities at a whim, a tendency towards
quiet reflection and seeing-things-from-the-other-fellow's-point-
of-view is seldom necessary.
        -- (Terry Pratchett, Small Gods)

Re: [SrcML] new plugin handling and API structure

[Dennis W. <den...@in...>]

On Wed, 20 Apr 2005 19:47:29 +0200
Frank Raiser <fra...@in...> wrote:

> - New plugin architecture
 
> We want plugins to integrate as easily as possible, and therefore the
> current .srcml/plugin_type.xml files in which every plugin had to be
> manually registered for now should be replaced by a .srcml/plugins
> directory. The installation of a new plugin is then reduced to copying
> the plugin's .jar file into this directory.
Sounds like a good idea to me.

 
> The content of a plugin's .jar file should be a plugin.xml file and
> additional .class files for the plugin's implemented methods.
> Optionally source files or other helpful files may be included of
> course. The responsibility of the rewritten config handler will be to
> check the .srcml/plugins directory for any .jar files and read out the
> plugin.xml file from each. Now these plugin.xml files will contain a
> fixed meta-data portion as well as additional tags important for the
> individual plugin. Here's the common parts every plugin will have to
> include in it's plugin.xml definition:
> 
> <plugin 
>     name="plugin name" 
>     version="x.y.z"
>     homepage="http://find.the.plugin.here"
Instead of this attribute I'd rather have a new tag for custom
information and include a "homepage" tag in there. This can also include
author information etc. and make it more extensible.

>     type="parser"
>     classname="de.srcml.foo.Bar"
>     >
>   <authors>
>     <!-- typical authors markup. might be best if similar to docbook
>     -->
>   </authors>
See above

>   <dependencies>
>     <depends name="other plugin's name" version="minimum version of
>     plugin"/> <depends name="yet another plugin" version="u.v.w"/> 
>   </dependencies>
>   <!-- plugin specific information -->
> </plugin>
> 
> Plugins for all platforms (for now this includes API, Parser and View
> platforms) will then be loaded through a single method in the 
> de.srcml.config.PluginManager class. Here's a sample call to load the
> java parser plugin for the ParserPlatform:
> 
> Plugin p = PluginManager.loadPlugin(
>     "java", "0.1.0", PluginManager.TYPE_PARSER);
> 
> And here's what the PluginManager class will do with such a call:
> 
> 2) Once the correct plugin is found the dependencies will be checked
> recursively. To this end a dependency tree is built and topologically
> sorted.
I'm still not sure if a plugin will have more than one dependency to
other plugins. This is especially problematic because the only plugins I
can imagine working parallel are filter plugins which do not require a
stream if any kind and won't modify the underlying content.

> - new API structure
[...]
> So the idea is to get away from a monolithic API towards a pure plugin
> based API. A parser developer can then provide the neccessary API
> classes through an additional API plugin without having to sync with
> the SrcML team first. The actual API code will then be reduced to that
> of an APIPlatform only responsible for loading the neccessary plugins.
> Right now when a SrcML document is loaded with the SAXParser the nodes
> of the resulting DOM tree are replaced by nodes which are instances of
> the API classes. This substitution process will be described as part
> of an API plugin. so apart from the above mentioned meta information
> such a plugin will have to provide the API part of a plugin.xml also
> contains a matching table like this:
> 
>   <match name="if" class="my.own.If"/>
>   <match name="loop" class="my.own.Loop"/>
> 
> Now with the dependencies it is also easy to extend the API
> functionality by means  of a plugin. Suppose you have written an
> enhanced version of the If class providing more functionalities you
> want to use throughout several projects in your company you can simply
> extend your class from the existing one like this:
> 
> package my.own.improved;
> public class If extends my.own.If { ... }
> 
> And package it into a new plugin which apart from the mandatory
> meta-data only needs:
> 
>   <depends name="the previous plugin" version="x.y.z"/>
>   [..]
>   <match name="if" class="my.own.improved.If"/>
I've got a proposal:
(I'll currently ignore the plugins)
Why not make the matching table an extra file ?
This would - besides the "match" tag - include one or several import
tags: 
<import name="myBaseMatchTableFile" version="x.y.z">

Imports overwrite matching defined above them. This would remove the
ambiguity problem, which plugins have, for match tables.
Additionally developers can use this to - kind of - create a custom
"view" of the underlying srcMl tree. You'd have to call
SrcML.loadMatchTable(...) after loading the srcML file and before
actually letting the system replace the nodes.

 > Now we can take full advantage of the new PluginManager. First the
> dependency tree will be built. The first plugin will then be
> instantiated first and this means that the API will from now on match
> every 'if' tag with the my.own.If class. After that the new plugin
> will be instantiated replacing this match so that every 'if' tag is
> now represented in the DOM tree through my.own.improved.If .
I think "plugins" differ a bit from the matching which seems to me more
like a class bases structure (therefore the text above).
 
> Once this is realized applications can use whichever version of the
> API they require. All that is needed is a single line in the beginning
> telling the SrcML framework what the application expects:
> 
> Plugin p = PluginManager.loadPlugin(
>     "my own improved API plugin", "0.0.3", PluginManager.TYPE_API);
Hmm I'm not even sure if the thing we were discussing about actually is
a plugin. Of course a dynamic plugin structure might still be a good
idea.

-
Dennis

[SrcML] new plugin handling and API structure

[Frank R. <fra...@in...>]

This is a summary of the discussion we had about restructuring the API and
turn it into a plugin architecture. Everyone please feel free to comment on
these points and add your own ideas where they differ from the proposed
changes.

As these changes will heavily affect the applications it will be best to get
them into CVS as soon as possible to reduce the affection for the other
projects. It is also not viable to wait with the restructuring until the
semester is over, as this would mean all projects would have to be adjusted
again.
The actual changes to applications should however not be too drastic. With the
currently planned design the plugin loading code should be significantly
shorter and as the current API will be the default API plugin there won't be
any changes neccessary for the actual usage.

- New plugin architecture

In order for the planned API restructuring we need a more powerful plugin
structure behind it. Therefore the config module will have to be extended and
the existing plugins will have to be adjusted to the new plugin scheme.

We want plugins to integrate as easily as possible, and therefore the current
.srcml/plugin_type.xml files in which every plugin had to be manually
registered for now should be replaced by a .srcml/plugins directory. The
installation of a new plugin is then reduced to copying the plugin's .jar file
into this directory.

The content of a plugin's .jar file should be a plugin.xml file and additional
.class files for the plugin's implemented methods. Optionally source files or
other helpful files may be included of course. The responsibility of the
rewritten config handler will be to check the .srcml/plugins directory for any
.jar files and read out the plugin.xml file from each. Now these plugin.xml
files will contain a fixed meta-data portion as well as additional tags
important for the individual plugin. Here's the common parts every plugin will
have to include in it's plugin.xml definition:

<plugin 
    name="plugin name" 
    version="x.y.z"
    homepage="http://find.the.plugin.here"
    type="parser"
    classname="de.srcml.foo.Bar"
    >
  <authors>
    <!-- typical authors markup. might be best if similar to docbook -->
  </authors>
  <dependencies>
    <depends name="other plugin's name" version="minimum version of plugin"/>
    <depends name="yet another plugin" version="u.v.w"/> 
  </dependencies>
  <!-- plugin specific information -->
</plugin>

Plugins for all platforms (for now this includes API, Parser and View
platforms) will then be loaded through a single method in the 
de.srcml.config.PluginManager class. Here's a sample call to load the java
parser plugin for the ParserPlatform:

Plugin p = PluginManager.loadPlugin(
    "java", "0.1.0", PluginManager.TYPE_PARSER);

And here's what the PluginManager class will do with such a call:

1) It checks its list of plugins (gained earlier from reading in all the
plugin.xml files.. or maybe reading them now on demand to save some startup
time) for a plugin of this type and name. It will then check if there is a
plugin matching exactly the requested version number, otherwise the latest
version is used. 

2) Once the correct plugin is found the dependencies will be checked
recursively. To this end a dependency tree is built and topologically sorted.

3) Walk the dependency tree from the plugin requiring no dependencies to the
actually requested plugin. For each plugin the class given by the classname
attribute in plugin.xml is instantiated. Although this instantiation is often
unneccessary we will see later when discussing the API changes how this can be
quite comfortable. It'll also behave the same all the time then, so it'll be
easier to understand for other developers.

Of course if anything fails (plugin not found, cyclic dependencies, etc) an
exception will be thrown detailing the problem.

- new API structure

The monolithic API we have right now has a big flaw we have overseen so far.
Assume someone writes a parser for a new language. This developer will provide
the parser as a plugin to the ParserPlatform. However every new language will
most probably require new tags. These tags should must be supported by the
API. With a monolithic API this means that the developer has to contact us
first, so we can introduce these tags into the API and they're available with
the next release then. Obviously this is not going to work out.

So the idea is to get away from a monolithic API towards a pure plugin based
API. A parser developer can then provide the neccessary API classes through an
additional API plugin without having to sync with the SrcML team first. The
actual API code will then be reduced to that of an APIPlatform only
responsible for loading the neccessary plugins.
Right now when a SrcML document is loaded with the SAXParser the nodes of the
resulting DOM tree are replaced by nodes which are instances of the API
classes. This substitution process will be described as part of an API plugin.
so apart from the above mentioned meta information such a plugin will have to
provide the API part of a plugin.xml also contains a matching table like this:

  <match name="if" class="my.own.If"/>
  <match name="loop" class="my.own.Loop"/>

Now with the dependencies it is also easy to extend the API functionality by
means  of a plugin. Suppose you have written an enhanced version of the If
class providing more functionalities you want to use throughout several
projects in your company you can simply extend your class from the existing
one like this:

package my.own.improved;
public class If extends my.own.If { ... }

And package it into a new plugin which apart from the mandatory meta-data only
needs:

  <depends name="the previous plugin" version="x.y.z"/>
  [..]
  <match name="if" class="my.own.improved.If"/>

Now we can take full advantage of the new PluginManager. First the dependency
tree will be built. The first plugin will then be instantiated first and this
means that the API will from now on match every 'if' tag with the my.own.If
class. After that the new plugin will be instantiated replacing this match so
that every 'if' tag is now represented in the DOM tree through
my.own.improved.If .

Once this is realized applications can use whichever version of the API they
require. All that is needed is a single line in the beginning telling the
SrcML framework what the application expects:

Plugin p = PluginManager.loadPlugin(
    "my own improved API plugin", "0.0.3", PluginManager.TYPE_API);







Ok.. thanks for reading that far :p
Now how do we implement this causing the least troubles? I suggest first
implementing the new PluginManager and switching the existing plugins over.
That should be realizeable pretty fast, although it requires several changes
to some of the existing apps. But as mentioned earlier these changes are
better done now than in a time of even more apps existing.

A second step would be switching the API into the APIPlatform then and
refactoring the existing API into a plugin.

And now feel free to comment on the above points. If nothing essentially new
is added we will probably start implementing the PluginManager and present the
above on our meeting next week.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

Flon's axiom:
There does not now, nor will there ever,
exist a programming language in which it is the least bit hard
to write bad programs. (Lawrence Flon)

[SrcML] XML Schema + Validation

[Frank R. <fra...@in...>]

The DTD we have been using so far has not yet been used for actual
validation of the SrcML documents. I have however decided that once
we go for actual validation we might as well drop the DTD in favor of
a proper XML Schema.

Right now you can find a 1:1 translation of the DTD as XML Schema in
api/srcml.xsd . Note that there have been several corrections to the
DTD as it hasn't been updated along with the rest of the project. The
new XML Schema however is also used in the api/tests/SchemaTest.java
unit test which parses all java files in the SrcML API module and
validates them against the schema.

Note that the srcml.xsd still can be enhanced further to make up for
the deficiencies we had in the DTD. However I suggest waiting with that
until we start developing a parser for the next language. It will be
easier and probably less error-prone to adjust the Schema to both 
parsers/languages at that point.

A side effect of the XML Schema support and need of validation is
however the need for a new dependency: Xerces-J. Luckily its SAXReader
also works better than the dom4j internal SAXReader we've been using
so far. This means we get faster parsing of .srcml files and as a bonus
we can now also read out the encoding information from the .srcml file.

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

When debugging, novices insert corrective code; 
experts remove defective code. (Richard Pattis)

[SrcML] some more papers

[Frank R. <fra...@in...>]

I've found a few more papers that might come in handy in regard
to the planned subprojects:

W Yang., "Identifying Syntactic Differences Between Two Programs"
http://citeseer.ist.psu.edu/yang91identifying.html

And this page contains a whole list of linked articles about
program analysis and slicing:
http://www.cs.wisc.edu/wpis/html/

-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

The most damaging phrase in the language is,
"It's always been done that way." (Rear Admiral Grace Hopper)

[SrcML] CFG support

[Frank R. <fra...@in...>]

As we will be spreading development to hopefully several students in about
a month I decided to move our discussions and progress reports to this 
list for later reference. This list will also be used to coordinate design
decisions and for general exchange of ideas.

I'm currently working on a Control Flow Graph (CFG) support for the API and 
it's coming along quite nice. There will be three levels of detail: full, expr
and block. 
Full detail might only be suitable in case we add an actual compiler backend,
as I don't see many other reasons to dissect the control flow within an
addition or similar expressions. 
The expression detail level might become the most used version as it creates a
new node for every expression, but abstracts from the details of the
expression.
For even further abstraction the block level will combine several expressions
following each other without any branching into one node.

The first application which will make use of these CFGs will probably be the
source code presenter. It will be a great aid in teaching programming when you
can easily get a visualization of the control flow for a section of your
program (f.ex. to talk about the possible flows in a for-loop including break,
continue or return statements). However there is no guarantee that the CFG
support will be fully available at the start of the semester.

As for the presentation of the CFG this will not be part of the API. The API
will only support creation of the graph and will provide several helper
methods in order to work with the created graph. Right now a proof of concept
visualization is done using a jython script which outputs a dot-compatible
graph representation. Using the dot tool we can get a simple .ps or what might
proof to be more useful in the future a .svg file for the graph.
Using the ViewPlatform we can get a highly customizable visualization of the
graph, as it allows us to easily render the expression a node is representing
as plain source code. An example of a graph visualized like this can be found
in the attached .ps file.

Now for the remaining problems:

- We need a more formal way of defining what exactly will be a new node in
  the graph and which parts of the DOM tree can share a node (actually three
  definitions for the different levels are needed).
- The CFG can easily be determined for a method on its own, but what about a
  whole class? The current implementation is only dissecting the class into
  its methods and creates a forest of rooted graphs for all those methods.
  We could take a closer look at a main method though and create an actual
  flow with links between those methods (also see the next point for this).
- How are method calls treated within an expression? Obviously this is a form
  of branching, however for the block and expression level detail the
  recursion would normally just treat the whole expression as one node. Of
  course that expression may consist of any number of method calls. Will it
  be enough to just find those calls and add edges to the respective methods'
  CFG trees (and back again)?
- how can we create useful unit tests for the CFG creation? I can only think
  of preselected sample source files with known should-be CFGs with which to
  compare the results. That would at least catch some problems when the CFG
  creation process is modified and returning different results.

Please feel free to join in on the discussion. Especially comments and ideas 
on the open problems are very much appreciated.
-- 
Raiser, Frank
Student @ University of Ulm (www.uni-ulm.de)

The difficult is what takes a little time;
the impossible is what takes a little longer. (Fridtjof Nansen)