webmacro-user

[WebMacro-user] concurrency problem with #include as template

[Carl H. S. <ca...@da...>]

Hello,

We've encountered a concurrency problem with the #include as template 
directive. It appears that when multiple threads of execution are 
running, webmacro can intermittently select a different template than 
the one requested. We have created a test program that demonstrates the 
problem. If anyone can provide insight into the cause of the problem, 
(or even better, a fix), we would be very appreciative.

I have posted wmtest.jar at 
http://www.datarepresentations.com/wmtest/wmtest.jar
The jar contains the compiled test classes and templates, as well as the 
source code.

In the test, we have a Page object (which extends ArrayList) that 
contains an arbitrary number of PageItems. Each PageItem has a 
getTemplate() method that returns the name of the template file for that 
object. In the Page template, we do the following:

#foreach $pageItem in $page {
    #include as template "$pageItem.Template"
}

There are several  subclasses of PageItem ( PageItemA, PageItemB, and 
PageItemC ). Each of these subclasses has a unique method getA, getB, or 
getC. In the template file that corresponds to each PageItem, we call 
these methods using $pageItem.A, $pageItem.B, or $pageItem.C. As long as 
the correct template is being used, this works. But if PageItemA's 
template were to call $pageItem.B, a ContextException would be thrown. 
If things are working correctly, this should never happen.

We add 1000 of each (PageItems A,B,C) to the Page. When we run a single 
test, everything is fine. This confirms that our objects and templates 
work as expected. But when we run multiple concurrent tests, we 
occasionally see the following error in the log:

1:10:18 PM      engine  ERROR   No public property C on variable 
$pageItem of class wmtest.PageItemB at 
C:\webmacroTest\src\templates\PageItemC.wmt:1.10

This says that WebMacro attempted to render PageItemB using the template 
for PageItemC !!!

To run the test, put the wmtest.jar file (along with webmacro.jar and 
concurrent.jar) in the classpath and run (using j2sdk 4 or 5):

java wmtest.WMTest 1 0

The first number is the number of tests to run. The second is the number 
of milliseconds to sleep in between starting each test. (Each test runs 
in a separate thread. So "wmtest.WMTest 2 1000" starts two threads 1 
second apart).

The output for one test should look like:
Starting test #0
1:47:38 PM      LogFile NOTICE  --- Log Started ---
1:47:38 PM      broker  NOTICE  starting org.webmacro.Broker: 
WebMacro.properties
1:47:38 PM      broker  NOTICE  Loaded settings from WebMacro.defaults, 
WebMacro.properties, (System Properties)
1:47:38 PM      wm      NOTICE  new WebMacro(WebMacro.properties) v2.0
Finished test #0

With only one test, there is never an error. But running many 
simultaneous test often produces the error.  The problem is intermittent 
and very timing sensitive. On a 3GHz P4 WinXPSP2, "wmtest.WMTest 2 1000" 
almost always produces the error. On slower machines, you need to 
increase the number of tests and decrease the time interval to see the 
error. On a 1.0GHz P3 Win2000SP4, we found "wmtest.WMTest 8 700" usually 
produced the error. We have seen the error using both J2sdk 4 and 5 on 
Windows and Linux.

With 2 tests, on the P4 we get:
Starting test #0
1:48:47 PM      LogFile NOTICE  --- Log Started ---
1:48:47 PM      broker  NOTICE  starting org.webmacro.Broker: 
WebMacro.properties
1:48:47 PM      broker  NOTICE  Loaded settings from WebMacro.defaults, 
WebMacro.properties, (System Properties)
1:48:47 PM      wm      NOTICE  new WebMacro(WebMacro.properties) v2.0
Starting test #1
1:48:47 PM      wm      NOTICE  new WebMacro(WebMacro.properties) v2.0
1:48:47 PM      engine  ERROR   No public property B on variable 
$pageItem of class wmtest.PageItemA at 
C:\webmacroTest\src\templates\PageItemB.wmt:1.10
Failed test #0
Finished test #1

The error is very timing sensitive, so you may need to try different 
numbers until you see it on your machine. Also the exact error is 
different each time. We have seen every erroneous combination of 
PageItem[ABC] and the template for [ABC].

Re: [WebMacro-user] concurrency problem with #include as template

[Marc P. <ma...@an...>]

On 4 Feb 2006, at 18:56, Carl H. Sayres wrote:

> Hello,
>
> We've encountered a concurrency problem with the #include as  
> template directive. It appears that when multiple threads of  
> execution are running, webmacro can intermittently select a  
> different template than the one requested. We have created a test  
> program that demonstrates the problem. If anyone can provide  
> insight into the cause of the problem, (or even better, a fix), we  
> would be very appreciative.

Thanks for this Carl. I haven't had time to run the test but I've  
looked at the source of it.

It definitely looks like you've found some obscure problem, but it's  
possible it is either in the include mechanism, template cache  
implementation, or the introspection code.

Anybody got time to look at this?

cheers

Re: [WebMacro-user] concurrency problem with #include as template

[Eric B. R. <eb...@tc...>]

My theory is that SimpleCacheManager.java is broken (http:// 
cvs.sourceforge.net/viewcvs.py/webmacro/webmacro/src/org/webmacro/ 
resource/SimpleCacheManager.java?rev=1.2&view=markup)

It seems the get(query, helper) function would allow concurrent  
readers to load and put the same resource in the _cache at the same  
time.  Obviously, in the end we'd only have one entry in the map, but  
we'd be loading the template twice.

But I don't see how this explains getting the *wrong* template back.   
I would think that concurrent.jar's ConcurrentHashMap would have to  
be broken in order for this to happen.

Just a theory tho.  I've never run across this myself.  Maybe Brian  
can chime in?

Also, a note to Carl.  I realize your wmtest.jar file is designed to  
expose this bug, but if your real code is creating new WM instances  
for *every* thread, you should consider not doing that.  Instead,  
keep one global WM instance.  Just make sure to give each thread  
their own Context object via _wm.getContext().  When WM is working  
properly, a separate Context per thread is all that's needed to  
guarantee thread-safety.

eric

On Feb 4, 2006, at 4:09 PM, Marc Palmer wrote:

>
> On 4 Feb 2006, at 18:56, Carl H. Sayres wrote:
>
>> Hello,
>>
>> We've encountered a concurrency problem with the #include as  
>> template directive. It appears that when multiple threads of  
>> execution are running, webmacro can intermittently select a  
>> different template than the one requested. We have created a test  
>> program that demonstrates the problem. If anyone can provide  
>> insight into the cause of the problem, (or even better, a fix), we  
>> would be very appreciative.
>
> Thanks for this Carl. I haven't had time to run the test but I've  
> looked at the source of it.
>
> It definitely looks like you've found some obscure problem, but  
> it's possible it is either in the include mechanism, template cache  
> implementation, or the introspection code.
>
> Anybody got time to look at this?
>
> cheers
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Webmacro-user mailing list
> Web...@li...
> https://lists.sourceforge.net/lists/listinfo/webmacro-user

Re: [WebMacro-user] concurrency problem with #include as template

[Carl H. S. <ca...@da...>]

Thanks Eric, I appreciate the help. Our application follows the 
WMServlet model of one WM per Servlet. In trying to track down the 
problem, we tried a single WM shared among servlets, one WM per servlet, 
and a new WM per HttpSerlvet.doPost. All three cases exhibit the same 
problem. Similarly, in WMTest.java, moving the WM instance out of the 
Test constructor and making it static has no effect on the test results. 
The problem remains.

Eric B. Ridge wrote:

> Also, a note to Carl.  I realize your wmtest.jar file is designed to  
> expose this bug, but if your real code is creating new WM instances  
> for *every* thread, you should consider not doing that.  Instead,  
> keep one global WM instance.  Just make sure to give each thread  
> their own Context object via _wm.getContext().  When WM is working  
> properly, a separate Context per thread is all that's needed to  
> guarantee thread-safety.

Re: [WebMacro-user] concurrency problem with #include as template

[Carl H. S. <ca...@da...>]

I tried adding the following to WebMacro.properties:

PropertyOperator.CacheManager=org.webmacro.resource.TrivialCacheManager

The error still occurs.

Carl


Eric B. Ridge wrote:

> My theory is that SimpleCacheManager.java is broken (http:// 
> cvs.sourceforge.net/viewcvs.py/webmacro/webmacro/src/org/webmacro/ 
> resource/SimpleCacheManager.java?rev=1.2&view=markup)
>
> It seems the get(query, helper) function would allow concurrent  
> readers to load and put the same resource in the _cache at the same  
> time.  Obviously, in the end we'd only have one entry in the map, but  
> we'd be loading the template twice.
>
> But I don't see how this explains getting the *wrong* template back.   
> I would think that concurrent.jar's ConcurrentHashMap would have to  
> be broken in order for this to happen.
>

Re: [WebMacro-user] concurrency problem with #include as template

[Carl H. S. <ca...@da...>]

I think I solved it! But I'm new to the inner workings of WebMacro, so 
please let me know if I've overlooked something.

The problem appears to be in the IncludeDirective.write() method. The 
IncludeDirective class has two instance fields:
    protected Macro _macFilename;
    protected String _strFilename;

In the case of an included file, the filename is stored in _strFilename. 
But if the included filename comes from a macro, then the macro name is 
stored in _macFilename.

The write() method has no problem with the single file case, using 
_strFilename which never changes. But in the case of a macro, write() 
stores the name of the file it is trying to include in _strFilename.  If 
two threads do this simultaneously, one will clober the other by 
overwriting _strFilename.

My first solution was to synchronize the write() method. This does solve 
the problem, but it has negatve performance implications.

My second solution is to use a local variable in write() to store the 
filename that we are including, passing through anything previously 
stored in _strFilename, to preserve the simple file include case.

Here is my new write() method:

    public void write (FastWriter out, Context context) throws 
PropertyException, IOException
    {
       String localFilename = _strFilename;
       Broker broker = context.getBroker();

        // the filename arg passed to us was a Macro, so
        // evaluate and check it now
        if (_macFilename != null)
        {
            localFilename = _macFilename.evaluate(context).toString();
            if (localFilename == null || localFilename.length() == 0)
            {
                throw makePropertyException("Filename cannot be null or 
empty");
            }
        }

        if (_log.loggingDebug() && 
context.getCurrentLocation().indexOf(localFilename) > -1)
        {
            // when in debug mode, output a warning if a template tries 
to include itself
            // there are situtations where this is desired, but it's 
good to make
            // the user aware of what they're doing
            _log.warning(context.getCurrentLocation() + " includes 
itself.");
        }

        // this should only be true if StrictCompatibility is set to false
        // and "as <something>" wasn't specified in the arg list
        if (_type == TYPE_DYNAMIC)
            _type = guessType(broker, localFilename);

        if (_log.loggingDebug())
        {
            _log.debug("Including '" + localFilename + "' as "
                    + ((_type == TYPE_MACRO) ? "MACRO"
                    : (_type == TYPE_TEMPLATE) ? "TEMPLATE"
                    : (_type == TYPE_TEXT) ? "TEXT"
                    : "UNKNOWN.  Throwing exception"));
        }

        Object toInclude = getThingToInclude(broker, _type, localFilename);
        switch (_type)
        {
            case TYPE_MACRO:
                // during runtime evaluation of a template,
                // a TYPE_MACRO doesn't really work as expected.
                // we logged a warning above in build(), but
                // we still need to write it out as a template
                // so just fall through
            case TYPE_TEMPLATE:
                out.write(((Template) 
toInclude).evaluateAsBytes(out.getEncoding(), context));
                break;

            case TYPE_TEXT:
                // static types are strings
                out.write(toInclude.toString());
                break;

            default:
                // should never happen
                throw makePropertyException("Unrecognized file type: " + 
_type);
        }
    }

The most important change is the third line:
       String localFilename = _strFilename;
Then every other place where _strFilename was referenced has been 
changed to localFilename.
This has no effect for the single included file case. In the case of a 
macro:
        if (_macFilename != null)
        {
            localFilename = _macFilename.evaluate(context).toString();
            if (localFilename == null || localFilename.length() == 0)
            {
                throw makePropertyException("Filename cannot be null or 
empty");
            }
        }
then localFilename is set to the corret value  within  each thread . And 
the threads can't  overwrite  each other's data.

Oh, yes, and my test now does not fail!

Let me know if you agree with my analysis.

Carl

Re: [WebMacro-user] concurrency problem with #include as template

[Keats K. <ke...@xa...>]

This is a good catch.  It's important to understand how directives work
in WM, so we can avoid this kind of problem in the future.  (I know I
need to review the directives I have worked on in light of this.)
Here's my understanding of the process:

At build time an instance of the XXXDirective class is created for each
reference of the directive in the template. The compiled template is
cached and may be accessed by multiple threads, so any state information
in a directive should generally not be modified during the expansion
(evaluation) phase.  You could do this using synchronized access, but in
general you're better off keeping this state information in a separate
context (Context, Session, BrokerLocal, etc.).

Somebody please correct me if I've gotten it wrong.

Keats

Carl H. Sayres wrote:

> I think I solved it! But I'm new to the inner workings of WebMacro, so 
> please let me know if I've overlooked something.
>
> The problem appears to be in the IncludeDirective.write() method. The 
> IncludeDirective class has two instance fields:
>    protected Macro _macFilename;
>    protected String _strFilename;
>
> In the case of an included file, the filename is stored in 
> _strFilename. But if the included filename comes from a macro, then 
> the macro name is stored in _macFilename.
>
> The write() method has no problem with the single file case, using 
> _strFilename which never changes. But in the case of a macro, write() 
> stores the name of the file it is trying to include in _strFilename.  
> If two threads do this simultaneously, one will clober the other by 
> overwriting _strFilename.
>
> My first solution was to synchronize the write() method. This does 
> solve the problem, but it has negatve performance implications.
>
> My second solution is to use a local variable in write() to store the 
> filename that we are including, passing through anything previously 
> stored in _strFilename, to preserve the simple file include case.
>
> Here is my new write() method:
>
>    public void write (FastWriter out, Context context) throws 
> PropertyException, IOException
>    {
>       String localFilename = _strFilename;
>       Broker broker = context.getBroker();
>
>        // the filename arg passed to us was a Macro, so
>        // evaluate and check it now
>        if (_macFilename != null)
>        {
>            localFilename = _macFilename.evaluate(context).toString();
>            if (localFilename == null || localFilename.length() == 0)
>            {
>                throw makePropertyException("Filename cannot be null or 
> empty");
>            }
>        }
>
>        if (_log.loggingDebug() && 
> context.getCurrentLocation().indexOf(localFilename) > -1)
>        {
>            // when in debug mode, output a warning if a template tries 
> to include itself
>            // there are situtations where this is desired, but it's 
> good to make
>            // the user aware of what they're doing
>            _log.warning(context.getCurrentLocation() + " includes 
> itself.");
>        }
>
>        // this should only be true if StrictCompatibility is set to false
>        // and "as <something>" wasn't specified in the arg list
>        if (_type == TYPE_DYNAMIC)
>            _type = guessType(broker, localFilename);
>
>        if (_log.loggingDebug())
>        {
>            _log.debug("Including '" + localFilename + "' as "
>                    + ((_type == TYPE_MACRO) ? "MACRO"
>                    : (_type == TYPE_TEMPLATE) ? "TEMPLATE"
>                    : (_type == TYPE_TEXT) ? "TEXT"
>                    : "UNKNOWN.  Throwing exception"));
>        }
>
>        Object toInclude = getThingToInclude(broker, _type, 
> localFilename);
>        switch (_type)
>        {
>            case TYPE_MACRO:
>                // during runtime evaluation of a template,
>                // a TYPE_MACRO doesn't really work as expected.
>                // we logged a warning above in build(), but
>                // we still need to write it out as a template
>                // so just fall through
>            case TYPE_TEMPLATE:
>                out.write(((Template) 
> toInclude).evaluateAsBytes(out.getEncoding(), context));
>                break;
>
>            case TYPE_TEXT:
>                // static types are strings
>                out.write(toInclude.toString());
>                break;
>
>            default:
>                // should never happen
>                throw makePropertyException("Unrecognized file type: " 
> + _type);
>        }
>    }
>
> The most important change is the third line:
>       String localFilename = _strFilename;
> Then every other place where _strFilename was referenced has been 
> changed to localFilename.
> This has no effect for the single included file case. In the case of a 
> macro:
>        if (_macFilename != null)
>        {
>            localFilename = _macFilename.evaluate(context).toString();
>            if (localFilename == null || localFilename.length() == 0)
>            {
>                throw makePropertyException("Filename cannot be null or 
> empty");
>            }
>        }
> then localFilename is set to the corret value  within  each thread . 
> And the threads can't  overwrite  each other's data.
>
> Oh, yes, and my test now does not fail!
>
> Let me know if you agree with my analysis.
>
> Carl

Re: [WebMacro-user] concurrency problem with #include as template

[Eric B. R. <eb...@tc...>]

Yeah, you're absolutely right Keats.  So is Carl.  I'm headed out of  
town, but will work on a fix and commit it next week when I return.

I'm also considering taking my laptop and the WM sources with me.  I  
want to play with the parser.  I hate our whitespace handing rules  
and have some ideas on making it better.  The long plane flights will  
probably give me enough time to work it out.

I've also discovered a few classes that should probably implement the  
Visitable interface that don't.  I've been working on a little static  
analysis tool to find usages of variables and such in templates and  
link them back up to known object types... as a way to find and head  
off template-level bugs.  I had to revert to some reflection trickery  
while visiting a few Template node types.

eric

On Feb 6, 2006, at 2:12 PM, Keats Kirsch wrote:

> This is a good catch.  It's important to understand how directives  
> work
> in WM, so we can avoid this kind of problem in the future.  (I know I
> need to review the directives I have worked on in light of this.)
> Here's my understanding of the process:
>
> At build time an instance of the XXXDirective class is created for  
> each
> reference of the directive in the template. The compiled template is
> cached and may be accessed by multiple threads, so any state  
> information
> in a directive should generally not be modified during the expansion
> (evaluation) phase.  You could do this using synchronized access,  
> but in
> general you're better off keeping this state information in a separate
> context (Context, Session, BrokerLocal, etc.).
>
> Somebody please correct me if I've gotten it wrong.
>
> Keats
>
> Carl H. Sayres wrote:
>
>> I think I solved it! But I'm new to the inner workings of  
>> WebMacro, so please let me know if I've overlooked something.
>>
>> The problem appears to be in the IncludeDirective.write() method.  
>> The IncludeDirective class has two instance fields:
>>    protected Macro _macFilename;
>>    protected String _strFilename;
>>
>> In the case of an included file, the filename is stored in  
>> _strFilename. But if the included filename comes from a macro,  
>> then the macro name is stored in _macFilename.
>>
>> The write() method has no problem with the single file case, using  
>> _strFilename which never changes. But in the case of a macro, write 
>> () stores the name of the file it is trying to include in  
>> _strFilename.  If two threads do this simultaneously, one will  
>> clober the other by overwriting _strFilename.
>>
>> My first solution was to synchronize the write() method. This does  
>> solve the problem, but it has negatve performance implications.
>>
>> My second solution is to use a local variable in write() to store  
>> the filename that we are including, passing through anything  
>> previously stored in _strFilename, to preserve the simple file  
>> include case.
>>
>> Here is my new write() method:
>>
>>    public void write (FastWriter out, Context context) throws  
>> PropertyException, IOException
>>    {
>>       String localFilename = _strFilename;
>>       Broker broker = context.getBroker();
>>
>>        // the filename arg passed to us was a Macro, so
>>        // evaluate and check it now
>>        if (_macFilename != null)
>>        {
>>            localFilename = _macFilename.evaluate(context).toString();
>>            if (localFilename == null || localFilename.length() == 0)
>>            {
>>                throw makePropertyException("Filename cannot be  
>> null or empty");
>>            }
>>        }
>>
>>        if (_log.loggingDebug() && context.getCurrentLocation 
>> ().indexOf(localFilename) > -1)
>>        {
>>            // when in debug mode, output a warning if a template  
>> tries to include itself
>>            // there are situtations where this is desired, but  
>> it's good to make
>>            // the user aware of what they're doing
>>            _log.warning(context.getCurrentLocation() + " includes  
>> itself.");
>>        }
>>
>>        // this should only be true if StrictCompatibility is set  
>> to false
>>        // and "as <something>" wasn't specified in the arg list
>>        if (_type == TYPE_DYNAMIC)
>>            _type = guessType(broker, localFilename);
>>
>>        if (_log.loggingDebug())
>>        {
>>            _log.debug("Including '" + localFilename + "' as "
>>                    + ((_type == TYPE_MACRO) ? "MACRO"
>>                    : (_type == TYPE_TEMPLATE) ? "TEMPLATE"
>>                    : (_type == TYPE_TEXT) ? "TEXT"
>>                    : "UNKNOWN.  Throwing exception"));
>>        }
>>
>>        Object toInclude = getThingToInclude(broker, _type,  
>> localFilename);
>>        switch (_type)
>>        {
>>            case TYPE_MACRO:
>>                // during runtime evaluation of a template,
>>                // a TYPE_MACRO doesn't really work as expected.
>>                // we logged a warning above in build(), but
>>                // we still need to write it out as a template
>>                // so just fall through
>>            case TYPE_TEMPLATE:
>>                out.write(((Template) toInclude).evaluateAsBytes 
>> (out.getEncoding(), context));
>>                break;
>>
>>            case TYPE_TEXT:
>>                // static types are strings
>>                out.write(toInclude.toString());
>>                break;
>>
>>            default:
>>                // should never happen
>>                throw makePropertyException("Unrecognized file  
>> type: " + _type);
>>        }
>>    }
>>
>> The most important change is the third line:
>>       String localFilename = _strFilename;
>> Then every other place where _strFilename was referenced has been  
>> changed to localFilename.
>> This has no effect for the single included file case. In the case  
>> of a macro:
>>        if (_macFilename != null)
>>        {
>>            localFilename = _macFilename.evaluate(context).toString();
>>            if (localFilename == null || localFilename.length() == 0)
>>            {
>>                throw makePropertyException("Filename cannot be  
>> null or empty");
>>            }
>>        }
>> then localFilename is set to the corret value  within  each  
>> thread . And the threads can't  overwrite  each other's data.
>>
>> Oh, yes, and my test now does not fail!
>>
>> Let me know if you agree with my analysis.
>>
>> Carl
>
>
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Webmacro-user mailing list
> Web...@li...
> https://lists.sourceforge.net/lists/listinfo/webmacro-user

Re: [WebMacro-user] concurrency problem with #include as template

[<Web...@St...>]

On Mon, 6 Feb 2006, Eric B. Ridge wrote:

| I'm also considering taking my laptop and the WM sources with me.  I want to
| play with the parser.  I hate our whitespace handing rules and have some ideas
| on making it better. 

I think changing the whitespace rules now is a very bad idea. They do 
work, and changes to such fundamental parts of the system is _not_ welcome 
by pretty much anyone, I believe.

| The long plane flights will probably give me enough time
| to work it out.

It will be interesting to se what you come up with - but I believe you at 
least should have _full_ configurable fallbacks to "as they are now" 
situation.

Really, everyone's output will be changed!! That just ain't no good for 
adoption.

| 
| I've also discovered a few classes that should probably implement the
| Visitable interface that don't.  I've been working on a little static analysis
| tool to find usages of variables and such in templates and link them back up
| to known object types... as a way to find and head off template-level bugs.  I
| had to revert to some reflection trickery while visiting a few Template node
| types.

For my own part, the thing about WebMacro is that there is _Way_ to much 
cruft. It should be _entirely_ ripped apart, and then about 90% of it 
should be deleted, and then the remaining parts should be put together 
again as a tight little API. The big idea with the Broker and all those 
little annoying caching elements and whatnot is just plain annoying. Then 
one have logging.
  And the license should be changed. GPL is bad for business when combined 
with Java, and "advertising clause" is just too "last century".

Regards,
Endre.

Re: [WebMacro-user] concurrency problem with #include as template

[Marc P. <ma...@an...>]

On 7 Feb 2006, at 14:26, Endre St=F8lsvik wrote:

> On Mon, 6 Feb 2006, Eric B. Ridge wrote:
>
> | I'm also considering taking my laptop and the WM sources with =20
> me.  I want to
> | play with the parser.  I hate our whitespace handing rules and =20
> have some ideas
> | on making it better.
>
> I think changing the whitespace rules now is a very bad idea. They do
> work, and changes to such fundamental parts of the system is _not_ =20
> welcome
> by pretty much anyone, I believe.

Actually not true. WM's whitespace handling is REALLY REALLY BAD. =20
I've used it in some propery production environments now with split =20
teams and it is hard to justify some of the behaviour, and producing =20
nicely formatted plain text is painful.

I would be very happy for the whitespace to be improved BUT it =20
depends what Eric calls improvement :)

> | The long plane flights will probably give me enough time
> | to work it out.
>
> It will be interesting to se what you come up with - but I believe =20
> you at
> least should have _full_ configurable fallbacks to "as they are now"
> situation.
>

Definitely. Perhaps this means branching the parser source into two =20
packages... o.w.parser and o.w.parser2 so that we can just specify in =20=

the WM properties which to use :)

> Really, everyone's output will be changed!! That just ain't no good =20=

> for
> adoption.
>

I don't imagine Eric would propose such wreckage... although if you =20
make it "WM 3" then you might mitigate this to a degree... especially =20=

if we could provide a script that would 100% reliably alter existing =20
templates to produce the same results with the new parser.

> | I've also discovered a few classes that should probably implement =20=

> the
> | Visitable interface that don't.  I've been working on a little =20
> static analysis
> | tool to find usages of variables and such in templates and link =20
> them back up
> | to known object types... as a way to find and head off template-=20
> level bugs.  I
> | had to revert to some reflection trickery while visiting a few =20
> Template node
> | types.
>
> For my own part, the thing about WebMacro is that there is _Way_ to =20=

> much
> cruft. It should be _entirely_ ripped apart, and then about 90% of it
> should be deleted, and then the remaining parts should be put together
> again as a tight little API. The big idea with the Broker and all =20
> those
> little annoying caching elements and whatnot is just plain =20
> annoying. Then
> one have logging.

You know you have my agreement there :)

WM's strengths are:

* The introspection engine
* The parser
* The template rendering

All the rest is basically bloat... existing template providers, =20
caching providers and tools separated out from the core and =20
reorganised a bit would make it a much more attractive package.

The thing is - are there enough good men to do the work? Is there =20
enough agreement and will to make these tough changes for WM 3? What =20
value can we add with these changes in WM 3?

For the latter, as I've stated before, I think we can add quite a bit:

* JSP integration
* Spring MVC integration
* Bean-style configuration WM instances
* Run-time (mutable) access to all directives, tools, providers and =20
other config, so that you can change properties at runtime, on a new =20
WM instance, and swap that for your current instance (hot swapping =20
the references to avoid threading/sync issues within WM itself)... so =20=

that applications can -conditionally- add custom directives etc as =20
beans created by the application not WM, without manipulating =20
property lists etc

>   And the license should be changed. GPL is bad for business when =20
> combined
> with Java, and "advertising clause" is just too "last century".

Last time I checked WM was not just GPL, you can chose the license =20
you want.

Re: [WebMacro-user] concurrency problem with #include as template

[Eric B. R. <eb...@tc...>]

On Feb 7, 2006, at 11:05 AM, Marc Palmer wrote:

> Actually not true. WM's whitespace handling is REALLY REALLY BAD.  
> I've used it in some propery production environments now with split  
> teams and it is hard to justify some of the behaviour, and  
> producing nicely formatted plain text is painful.

yeah, it's terrible.  We use WM for Java-code generation, SQL schema  
generation, and for HTML.  Ya gotta process the output with something  
like Jalopy or HTMLTidy to make readable output.

> I would be very happy for the whitespace to be improved BUT it  
> depends what Eric calls improvement :)

we'll see.  First I've gotta (re)familiarize myself with WM's  
parser.  It's pretty complex.

> Definitely. Perhaps this means branching the parser source into two  
> packages... o.w.parser and o.w.parser2 so that we can just specify  
> in the WM properties which to use :)

Yeah, of course.  we'd have o.w.parser and  
o.w.NewButSlightlyLessSuckyParser.

> I don't imagine Eric would propose such wreckage... although if you  
> make it "WM 3" then you might mitigate this to a degree...  
> especially if we could provide a script that would 100% reliably  
> alter existing templates to produce the same results with the new  
> parser.

Such a script would be nuts!  If I do come up with an improvement we  
can make it configurable.

> You know you have my agreement there :)
>
> WM's strengths are:
>
> * The introspection engine
> * The parser
> * The template rendering
>
> All the rest is basically bloat... existing template providers,  
> caching providers and tools separated out from the core and  
> reorganised a bit would make it a much more attractive package.

In order for this to happen someone actually has to do it.  Endre has  
been complaining about this for as far back as I can remember.  I'm  
not personally motivated to mess with this.  It might be bloat, but  
it doesn't get in my way.  The parser, on the other hand, does.

eric

Re: [WebMacro-user] concurrency problem with #include as template

[<Web...@St...>]

| 
| >  And the license should be changed. GPL is bad for business when combined
| > with Java, and "advertising clause" is just too "last century".
| 
| Last time I checked WM was not just GPL, you can chose the license you want.

Yes, between one which is "viral" (GPL), and another with advertising 
clause.

Regards,
Endre

[WebMacro-user] WM license

[Keats K. <ke...@xa...>]

I agree that it would be nice to see the license simplified and more 
relaxed; like LGPL or MPL or whatever.  I wonder if Justin is willing to 
release his proprietary interest?

Keats

Endre Stølsvik wrote:

>| 
>| >  And the license should be changed. GPL is bad for business when combined
>| > with Java, and "advertising clause" is just too "last century".
>| 
>| Last time I checked WM was not just GPL, you can chose the license you want.
>
>Yes, between one which is "viral" (GPL), and another with advertising 
>clause.
>
>Regards,
>Endre
>
>  
>

Re: [WebMacro-user] WM license

[<Web...@St...>]

On Wed, 8 Feb 2006, Keats Kirsch wrote:

| I agree that it would be nice to see the license simplified and more relaxed;
| like LGPL or MPL or whatever.  I wonder if Justin is willing to release his
| proprietary interest?

"New BSD-style" (or maybe ASL?) would be best for such a "deep down" yet 
small library. LGPL is slightly bad for java - RMS just won't clarify the 
wording about "linking" etc and how it regards to java, as I understand 
the controversy. MPL is about two million difficult lines, and made for 
Mozilla.

Regards,
Endre.

Re: [WebMacro-user] WM license

[Eric B. R. <eb...@tc...>]

I would vote for the "New BSD-style" license as well.  Or just put it =20
in the Public Domain.

If we were able to make such a change, is Justin the only required to =20
give permission?  What about the other contributors?

eric

On Feb 9, 2006, at 9:02 AM, Endre St=F8lsvik wrote:

> On Wed, 8 Feb 2006, Keats Kirsch wrote:
>
> | I agree that it would be nice to see the license simplified and =20
> more relaxed;
> | like LGPL or MPL or whatever.  I wonder if Justin is willing to =20
> release his
> | proprietary interest?
>
> "New BSD-style" (or maybe ASL?) would be best for such a "deep =20
> down" yet
> small library. LGPL is slightly bad for java - RMS just won't =20
> clarify the
> wording about "linking" etc and how it regards to java, as I =20
> understand
> the controversy. MPL is about two million difficult lines, and made =20
> for
> Mozilla.
>
> Regards,
> Endre.
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through =20
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD =20
> SPLUNK!
> http://sel.as-us.falkag.net/sel?=20
> cmd=3Dlnk&kid=3D103432&bid=3D230486&dat=3D121642
> _______________________________________________
> Webmacro-user mailing list
> Web...@li...
> https://lists.sourceforge.net/lists/listinfo/webmacro-user

Re: [WebMacro-user] WM license

[<Web...@St...>]

On Sat, 11 Feb 2006, Eric B. Ridge wrote:

| I would vote for the "New BSD-style" license as well.  Or just put it in the
| Public Domain.

I was also thinking PD: The problem with WebMacro as I see it now is that 
it is _dangerously_ close to abandonware. PDing it shouldn't hurt anyone 
very much, I'd assume, and would hopefully just make it more palatable 
than the current licenses do. But BSD (w/o the "advertising clause", aka 
"New") is also good.
  (http://www.gnu.org/philosophy/bsd.html)

To be frank, I wouldn't have touched WebMacro as it stands now with my 
current knowledge of licenses, and given the now-existent competition with 
their simpler license.

| 
| If we were able to make such a change, is Justin the only required to give
| permission?  What about the other contributors?

How many are there? "CVS-dump" all committers.

Regards,
Endre.