Re: [Freemarker-devel] In 2.3.19, sharing cache storage

[Jaime G. <jai...@or...>]

I think the my: prefix (or whatever I choose) would make it still "foreign" and not part of the language. My use case is that it is really a DSL that I need. In any case, I already have issues with copy/pasting from the site since ?join is there, and not in my 2.3.19 version. Others are coming that will not allow copy/paste for me.

But agreed. Having a DSL is an endeavor that requires tender love and care. We made a conscious decision to go this way, and certainly it is not for everyone. For us this DSL is core to our offering, so we can support it with a group of developers, writers, PMs, trainers, etc. Having said all that, most of my customization has been possible with new classes and only modifying two of your files: FMParser.jj, ParseException. However, I use package freemarker.core and make use extensively of those protected things you have there.

I understand that this is a matter of time from you. So far the results of your efforts are impressive though. Once I get a list of new goodies in a new version that I can sell to PMs, I will re-implement with that new version of FM.

My only ask would be if more classes were public and non-final. That would be lovely!

Jaime

-----Original Message-----
From: Daniel Dekany [mailto:dd...@fr...] 
Sent: Wednesday, July 23, 2014 3:20 PM
To: Jaime Garza
Cc: Daniel Dekany
Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage

Wednesday, July 23, 2014, 10:56:31 PM, Jaime Garza wrote:

>> -----Original Message-----
>> From: Daniel Dekany [mailto:dd...@fr...]
>> Sent: Wednesday, July 23, 2014 1:06 PM
>> To: Jaime Garza
>> Cc: Daniel Dekany
>> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>> 
>> Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:
>> 
>> > The modifications that I have done have to do with adaptations to 
>> > FM to make it more of a DSL. We use FM for formatting emails and 
>> > SMS messages. I have modified mostly FMParser.jj keeping a series 
>> > of svn patch files for each one of my modifications, hopefully 
>> > helping me move to a more recent FM. We have extensions for sql 
>> > queries with a <#data> directive, data phrasing as a builtin for 
>> > sequences, ?unique for sequences, etc.
>> 
> Yeah I know that. We have a myriad of functions we implement. Our 
> additions look, as you mention, terse. It appears as part of the 
> language, which it now is. It would have been a dream to be able to 
> add additional built-ins, and I have been following the list for a 
> while now, and I know that this is a common ask and that you are 
> thinking on ways to do it. I look forward to the moment we can have 
> "legal" built-ins in the environment. Adding directives in a standard 
> way with <# would have been ideal as well.

In the far future I plan to add namespace prefixes via colon. This is very similar to #import at first glance, but has some important benefits. One of them is that if you have a function (either defined in FTL or in Java) then you can call it both like my:unique(seq) or with postfix suntax, seq?my:unique. (Also then the "built-in" term is gone - they are just core functions.) A side benefit is that <#my:data/> perhaps looks less off than <@my.date/>. And of course, just like #imports can be done on Configuration-level, prefixes could be defined there too, so it doesn't have to be done in each template again and again.

Now, if adding those "my:"-s (or whatever prefix you chose, obviously) and maybe the explicit "/>"-s still seen as too awkward (is it?), this idea could be further extended so that you can define your own template language be directives/functions to the core namespace in the Configuration level (read: pre-parse-time). I'm afraid of that though, as then it's too easy to create custom languages, so people will do it even when it's doesn't really worth the chaos that it causes later heads (manual doesn't mach, copy-pasting from forums doesn't work, IDE plugings complain, etc.). OTOH it would be useful when otherwise someone really had to make a patched FreeMarker version.

>> You can have custom directives and functions without modifying FM, as 
>> you certainly know. Surely they are not as flexible/terse as what can 
>> be achieved with core directives and built-ins... Making them closer is a future goal.
>> 
>> > We maintain our own FM document based on the
>> > 2.3.19 docs. We needed some validation support,
>> 
>> Validation of what?
>
> Users declare their own namespaces in the UI. These are attached to 
> database rows and appear in the root namespace. We need to know if 
> they are really used in the templates at all, for backend 
> optimizations.
> We warn users if they declare them and then they do not use them.

Sounds like that part can be solved by traversing the AST and collect the variables referenced. (There's no publISHED API for that yet
though.)

> For this purpose we follow templates depth-first in a parsing phase as 
> much as possible. If the include uses string expressions, we warn too, 
> as we cannot proceed. We use the parse flag to follow includes.
> Etc. It is kind of elaborate, but it ended being very powerful.

This is maybe also solvable with AST traversal.

> In fact, I have been following the discussion of editor plugins and 
> how we can get the parsing tree, and how people have done this in 
> their own way, and then communicate with you. This is a way in which 
> we discover the namespace for us, in-depth. I instrumented FMParser's 
> PrimaryExpression and Include for this. I did not have to open 
> DollarVariable.java or Include.java for it.
>
>> 
>> > multiple error messages per template (one error was not 
>> > acceptable,)
>> 
>> Is that multiple parsing errors or?
>
> We do multiple parse errors, and lexical errors as well. We 
> implemented a recover mechanism to stabilize the parsing process as 
> much as possible, by skipping tokens, and backtracking. We then return 
> a list of errors as opposed to one error. This was a big change.

This is also in the plans because of the IDE plugins.

So yeah... looks like all well known issues again. As always, the problem is the lack of time.

--
Thanks,
 Daniel Dekany

>> 
>> > and others.
>> 
>> --
>> Thanks,
>>  Daniel Dekany
>> 
>> 
>> > Thanks
>> > Jaime
>> >
>> > -----Original Message-----
>> > From: Daniel Dekany [mailto:dd...@fr...]
>> > Sent: Wednesday, July 23, 2014 11:22 AM
>> > To: Jaime Garza
>> > Cc: fre...@li...
>> > Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>> >
>> > The problem with the approach is that Template-s are bound to a 
>> > Configuration instance (they store a reference to it), and 
>> > sometimes may reach back to it during template execution. However, 
>> > can't you just use a common Configuration instance on all those 
>> > places that share
>> templates?
>> >
>> > (Out of curiosity... what's the modification in 2.3.19 is about?)
>> >
>> > --
>> > Thanks,
>> >  Daniel Dekany
>> >
>> > Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
>> >
>> >> Hi Daniel,
>> >>
>> >> I am trying to share an instance of MruCacheStorage across threads.
>> >> I keep it around across multiple threads and when creating my 
>> >> Configuration, I simply do a Configuration#setCacheStorage(). This 
>> >> internally calls Configuration#createTemplateCache, which calls 
>> >> cache.setConfiguration(this), which calls TemplateCache#clear(), 
>> >> which eventually clears the storage.
>> >>
>> >> My use case is that I know that a set of around 200 threads will 
>> >> be accessing the same templates over and over (executing the same
>> >> template(s) on a different root namespace) concurrently. I want 
>> >> the loading of templates to happen as infrequently as possible. 
>> >> Also, since I am on a slightly customized version of 2.3.19, I 
>> >> cannot come to the
>> greatest new version.
>> >>
>> >> The question (or questions) is: is there a way to implement my 
>> >> strategy? My current workaround is nasty. I have created a 
>> >> subclass of MruCacheStorage, which reads a flag "canClear" from 
>> >> thread local, and then decides whether to let the clear proceed or 
>> >> not. I sandwitched the call to setCacheStorage() with canClear = 
>> >> false and canClear = true
>> >>
>> >>         SharedCacheSettings.disallowClear();
>> >>         config.setCacheStorage(storage);
>> >>         SharedCacheSettings.allowClear();
>> >>
>> >> AND
>> >>
>> >> public class SharedCacheStorage extends MruCacheStorage {
>> >>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>> >>         super(maxStrongSize, maxSoftSize);
>> >>     }
>> >>
>> >>     @Override
>> >>     public void clear() {
>> >>         if (!SharedCacheSettings.canClear())
>> >>             return;
>> >>
>> >>         super.clear();
>> >>     }
>> >> }
>> >>
>> >> What is your opinion about my hack? What kind of trouble am I inviting?
>> >>
>> >> Thanks in advance.
>> >> Jaime Garza