Menu
▾
▴
exist-development
|
From: Andrzej J. T. <an...@ch...> - 2010-02-01 17:29:41
|
Adam: > It breaks then tenement of side-effect free XQuery functions. > Admittedly we have some of these, but they are extension functions and > very specialised. Actually, we have a lot of these: session, request, sql, ldap, email, cache and more! So we've really moved away from the religiously functional world, and I see that as a good thing, since real-world applications typically need to integrated with non-functional real-world systems like servlets, sql database, ldap directories, email and more. That is a far cry from "some of these". Obviously there is a requirement for some side-effect generating, non-functional features in eXist. Quite a few of them in fact. ;-) > The problem was that I had not really discarded the shackles of > procedural programming and not yet really *really* got my head around > functional programming elegance. I now believe that such a facility is > completely unnecessary once you learn the tricks of XQuery programming > - it does take time though. You're getting borderline preachy, Adam! ;-) I don't want to get into a religious war on the benefits of being purely functional, since we're far from that already and I have no interest in an academic argument. I know the tricks and value of functional programming....but sometimes the real world intrudes. > Perhaps you can convince me that such a functionality is really > nessecary, but I would need to understand the use case and why that > cannot be met currently. It's not strictly necessary...more a performance/convenience issue. I'm happy to remove the two get/set functions from trunk, if the others vote that this should be the way we go. My code runs fine by using in-memory fragment creation...it's just slow and a bit convoluted to do that, since I have to create intermediate xml fragments that aggregate multiple xml results so that they can be passed around. No big deal there though, since it does work fine. I may consider putting the functions into a private extension library though, if it buys me enough performance/convenience at the moment, given the state of in-memory fragments. > The request attributes are only designed for setting attributes on the > underlying HttpServletRequest, they were never added for the purpose > of storing and reusing data within a single XQuery. Neither should > they be used for transporting data between XQueries. They are really > for internal interaction of the eXist pipeline, we should come up with > a better mechanism in future as this ties us too tightly to the Java > Http Servlet specification. I don't buy this argument at all. XQuery is a beautiful tool for writing complete, self-contained web applications that run in a servlet container. Heck...we even support REST that way. To do that you need session-based persistence for things like authentication and the like. Regardless of the pragmatic vs theoretical arguments, which are likely to just start a religious war, the cat is out of the bag. These features are heavily used in production systems, and so we need to keep them around. As for what they were intended for, that's irrelevant. If it's available users will use the features....whether they have side-effects or not. > The session attributes are just that, designed for persisting (ideally > small) amounts of data across a users requests during their session. > Again this should not be used for inter-xquery communication. Ideally > sessions should be avoided where possible, this leads to greater > scalability possibilities - there are plenty of articles about > developing stateless web applications so I wont get into a discussion > here. Should, could, would! If it's there, it will be used....and "abused" in some people's minds. That's the reality. > You should not misuse the request or session objects even if they are available. If all you have is a hammer...everything looks like a nail! LOL > I dont see why this would become a requirement then, you should still > be passing data into your XQuery using a different mechanism not the > context. Each executing instance of an XQuery typically has its own > context and in a multi-threaded environment this is no different. Good point. > It sounds like you have a lot of complexity and issues here that need > to be broken down into smaller problems and modularised. Perhaps if > you could give a small specific example of why you need such context > functionality then I could better understand your exacting > requirements? Time demands preclude me from stripping down many thousands of lines of code to create a demonstration case. Sorry....that's my reality right now. Too many real-world deadlines. As for the complexity and such, I've already refactored the code into much smaller, more maintainable, decoupled, independent units as I had described, using a dispatcher paradigm. This has greatly simplified our code. Don't forget, Adam, I've been writing code for nearly 4 decades, and contrary to some opinions, I do know what I'm doing and have a long history of writing extremely modular/maintainable code! Preaching to the converted ain't gonna win you many points with me. ;-) > Actually I think such a feature makes code much dirtyier as you cant > see the flow of data through the system. You just magically stuff > something into the context using some sort of label and then later > extract it (assuming its still there). This is very bad in any > language. Tell that to Google, who use distributed hashmaps all over the place (Map/Reduce is based on the concept). Same with Hadoop. memcached. This is a proven, performant design pattern, that unfortunately does not fit nicely into the purist functional approach. I don't believe it is a bad thing, if used appropriately. In such cases it can clarify the data flows and make code simpler. It's all down to the skill/expertise of the individual, and how "good" their code is. That is independent of functional, distributed hashmap, procedural, declarative or any other approach used. Purist approaches rarely succeed in the real world. Erlang and Lisp come to mind as good examples. LOL > When I got rid of all of this functionality from a large > system that I developed in XQuery, the code actually became much > simpler, and more readable through the extensive refactoring that I > performed. That wouldn't be the case with my code. One day, I'll sit down with you and show you what we've got...when time allows. > Well if you cant show us a concrete example then we definitely cant help you ;-) True enough. Not that I don't want to...just that there isn't enough time at the moment. > Perhaps you could show us something, a small exacting example, and we > could help you refactor it? If we cant help you refactor it, then it > shows that your use case is valid and that we do need a feature such > as this. How about I do some benchmarks and see if the performance gain over building large aggregate in-memory fragments is worth the compromise? ;-) The key point is that I am happy to remove the functionality, if the group as a whole votes that way. Cast yer votes, maties! Alternatives: 1) Leave it in and see if the world collapses 2) Remove it completely 3) Move it to it's own extension module. My vote is for option #3. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Wolfgang M. <wol...@ex...> - 2010-02-01 18:21:05
|
> Alternatives: > > 1) Leave it in and see if the world collapses > 2) Remove it completely > 3) Move it to it's own extension module. > > My vote is for option #3. Since I need more time to think about it, I'd suggest to move it to its own extension module for now. We can further discuss the issue in Prague (unfortunately you won't be there, but I think I understand your point sufficiently well). Maybe we can come up with a clean solution which is acceptable to everyone. Wolfgang |
|
From: Thomas W. <tho...@gm...> - 2010-02-01 19:34:48
|
I hope this will be taken in the best possible way. I could never understand why a functionality that is used by some users could be proposed to be removed by other users either who don't use it or don't like it. If something have got a momentum it should stay. It is there, it does not require any additional resources and brings other routes to more solutions. Richer functionalities allow more developers to find their own specific way in resolving their own specific problems. There could never be a single best way of dealing with any kind of problems. I strongly vote for 1). Kind regards, Thomas ------ Thomas White > > > Alternatives: > > 1) Leave it in and see if the world collapses > 2) Remove it completely > 3) Move it to it's own extension module. > > My vote is for option #3 |
|
From: Wolfgang M. <wol...@ex...> - 2010-02-01 20:14:55
|
> I could never understand why a functionality that is used by some > users could be proposed to be removed by other users either who don't use it > or don't like it. Wait, you miss the point here. Andrzej's was 1) a newly committed feature, 2) a change to a core module. Ideally every commit to the repository should be reviewed by more than one person and this is what happened in this case. Two people objected to the commit and commented on it. That's a very positive sign, even if you disagree with them. A commit can be criticized and sometimes even rolled back. That's what you have SVN for. The more people watch and comment on the commits list, the better. Wolfgang |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-01 21:10:32
|
Wolfgang said: > Wait, you miss the point here. Andrzej's was 1) a newly committed > feature, 2) a change to a core module. Ideally every commit to the > repository should be reviewed by more than one person and this is what > happened in this case. Two people objected to the commit and commented > on it. That's a very positive sign, even if you disagree with them. A > commit can be criticized and sometimes even rolled back. That's what > you have SVN for. > > The more people watch and comment on the commits list, the better. I heartily agree! This makes eXist stronger! Even though it was one of my features that was being commented on, and it has fostered a lively and interesting discussion, I welcome the input, and as a result have made it an optional extension module for the moment (though it still might disappear down the road if it's really not required). -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Thomas W. <tho...@gm...> - 2010-02-02 08:32:12
|
Sorry for the confusion. You are absolutely right. Thomas ----------------- On 1 February 2010 20:14, Wolfgang Meier <wol...@ex...> wrote: > > I could never understand why a functionality that is used by some > > users could be proposed to be removed by other users either who don't use > it > > or don't like it. > > Wait, you miss the point here. Andrzej's was 1) a newly committed > feature, 2) a change to a core module. Ideally every commit to the > repository should be reviewed by more than one person and this is what > happened in this case. Two people objected to the commit and commented > on it. That's a very positive sign, even if you disagree with them. A > commit can be criticized and sometimes even rolled back. That's what > you have SVN for. > > The more people watch and comment on the commits list, the better. > > Wolfgang > |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-01 20:19:03
|
Wolfgang: > Since I need more time to think about it, I'd suggest to move it to > its own extension module for now. We can further discuss the issue in > Prague (unfortunately you won't be there, but I think I understand > your point sufficiently well). Maybe we can come up with a clean > solution which is acceptable to everyone. I have moved the functionality to it's own extension module (context), and have left this module disabled in conf.xml. So a user will have to consciously turn it on to use it for the moment. Sure wish I could make it to Prague. I heard rumours of an eXist gathering in North America this year...? -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Wolfgang M. <wol...@ex...> - 2010-02-01 20:52:05
|
> Sure wish I could make it to Prague. I heard rumours of an eXist gathering in North America this year...? Sure, I would like the idea and I think we should organize one. Probably something to talk about in Prague as well. Wolfgang |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-01 20:44:03
|
Wolfgang:
> Andrzej: maybe you could provide a concrete example for which you
> would need your extensions. This would make it easier for other users
> to follow.
Here's a rough outline of what we've built....
We've created a generic analytics reporting module, where you specify what analytical functions you want run, input
variables with defaults and relationships between hierarchical metrics using a declarative XML specification. That is,
you define your report and analysis in XML. No procedural code per se.
There is an XQuery called request.xql which will read all report definitions stored in the database, and will generate a
user interface in HTML allowing the user to request execution of one of the reports. It reads the xml definition,
creates required input parameters and the like, and will do a post with all this, including the id of the selected report.
The post is handled by render.xql. The issue we ran into is that some of the reports require custom xquery code which
is fairly extensive, and others use common code. In all cases, the user can elect to receive the output in raw xml
format (for submission to upstream systems perhaps) or in a human readable HTML format. The HTML format is created from
the xml output as a precursor step.
So the render.xql has a dispatcher table which lists, for each report definition in the database, what .xqm module to
load dynamically, which function to call to generate the xml and which function to call to generate html from the xml.
This allowed us to split one massive render xquery into multiple modules which can be dynamically imported at runtime,
depending on which report the user requested. In a subsequent step, we'll be splitting each report .xqm module into one
to generate xml and the other to generate html, to further separate things. All of these modules are rather
large...these are extremely complicated reports which use dynamic analytics against healthcare data.
So...if you as a user requested report "A" with human readable HTML output, render.xql would look up the the "A" module
document name (say a-report.xqm or some such), along with the prefix and namespace the module uses, and would load this
module dynamically. Then it would look up the function to call to generate the XML and will use eval() to call that
function. The function will return the xml as a result back to request.xql, which will then look up the function to
call to generate the final HTML, and will call that function using eval() passing it the xml results.
The complicating factor is that some reports can return more than one xml document as a result, and so we needed the xml
generator to do something like this:
return <xml>
<result1>{ $result1 }</result1>
<result2>{ $result2 }</result2>
</xml>
from the xml generation function and then the html renderer needs to then index into the two result values. That is a
very slow and expensive (not to mention memory consuming) operation since it's using in-memory fragments to create the
result document, when we already have the two results available.
For us, it would be a lot more efficient if we did something like:
let $set1 := context:set-context-attribute( "result1", $result1 )
let $set2 := context:set-context-attribute( "result2", $result2 )
in the xml function, and just did the following to get the data back in the html function:
let $get1 := context:set-context-attribute( "result1" )
let $get2 := context:set-context-attribute( "result2" )
rather than go to the time and expense of building the composite xml document.
This is all because we use render.xql as a dynamic dispatcher which dynamically calls and/or chains functions together
from modules that were dynamically loaded, depending on what report the user wanted to run.
Like I said, the composite XML document approach works fine, it's just a bit slow is all, so we don't really "need" the
context attributes. But they will come in handy. ;-)
> You both provided valid arguments. It's a difficult question and I'm
> not yet sure towards which side I tend (in particular since I have a
> terrible cold which currently limits my thinking).
Sorry to hear that.
> Adam is certainly
> right in that functional languages should avoid side-effects wherever
> possible. On the other hand, nearly all functional languages violate
> this principle and provide a way to set variables with side effects.
> In the XQuery world, the XQueryP extension introduces a "set
> $variable" expression
> (http://www.flworfound.org/pubs/Dana-XML-2006.pdf). MarkLogic has a
> dictionary data type, whose key/value pairs can be set from anywhere
> within a query! Apparently, users like this feature, though it
> encourages them to stick to procedural style instead of writing
> functional code.
I think that in the real world, as you said, all functional languages have to violate the principle, to get any "real
work" done and to integrate with external systems.
> On the other hand, I guess there are situations in which I wished I
> could set a global variable, e.g. to pass information between modules.
The newish cache extension module would work nicely for that. I've used it to generate and cache expensive lookup tables
at startup, that take many minutes, and would not be very user friendly to generate in response to a user request.
> Sometimes my modules need to save state between function calls. Sure,
> you could use some XML fragment to save the state and pass it back and
> forth between caller and module. This leads to complex function calls
> though, which are irritating to the user. Fortunately, a module can
> always store state into the database by saving an XML fragment.
> However, this is no different from setting session variables or
> whatever: it again introduces side effects.
That is my point. Trying to support a pure functional approach just doesn't work with real world applications, as all
functional language implementors have discovered.
> I think it is difficult to use XQuery for complex real-world
> application without allowing some functions to have side effects
> (storing a document).
I would venture to say it would be pretty much impossible. And even if it was possible technically, a pure functional
approach would turn off the majority of potential users. You don't see much Haskel used for just this reason.
My leanings are towards a more pragmatic approach.
> But we have to be very cautious. I rejected the
> idea to provide a dictionary data type, because it introduces side
> effects through the backdoor. From this point of view, I tend to
> prefer the more radical approach of a "set" operation. If a function
> has side effects or not should at least be obvious.
I agree with you, though the cache extension module is very much like a dictionary data type, albeit at a global level.
--
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com
|
|
From: Loren C. <lor...@gm...> - 2010-02-01 20:45:58
|
We have talked about this at our office. We are discussing hosting an eXist users conference in Minneapolis. How many people would be interested in an eXist user group meeting in Minneapolis this summer? On Feb 1, 2010, at 02:18 PM, Andrzej Jan Taramina wrote: > Sure wish I could make it to Prague. I heard rumours of an eXist gathering in North America this year...? > > -- > Andrzej Taramina > Chaeron Corporation: Enterprise System Solutions > http://www.chaeron.com > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development |
|
From: Andrzej J. T. <an...@ch...> - 2010-02-01 23:22:21
|
Loren: > We have talked about this at our office. We are discussing hosting an eXist users conference in Minneapolis. How > many people would be interested in an eXist user group meeting in Minneapolis this summer? Count me in! In fact, I'll offer to do a detailed presentation and demo of what we've been up to in the healthcare space. -- Andrzej Taramina Chaeron Corporation: Enterprise System Solutions http://www.chaeron.com |
|
From: Adam R. <ad...@ex...> - 2010-02-02 09:58:52
|
On 1 February 2010 19:34, Thomas White <tho...@gm...> wrote:
> I hope this will be taken in the best possible way.
>
> I could never understand why a functionality that is used by some
> users could be proposed to be removed by other users either who don't use it
> or don't like it.
>
> If something have got a momentum it should stay. It is there, it does not
> require any additional resources and brings other routes to more solutions.
This is not how good software is built, but more importantly its not
how it is maintained. We have had plenty of contributions in the past,
some of excellent quality and some of not. But if we just allowed
anyone to add anything we would end up with a mess, everyone thinks
eXist should be something different, at present it is a lot of
different things to different people - but the core team tries not to
dilute the product too much. Maintainability is also a huge issue for
us. Basically we like features to be discussed by the core team.
> Richer functionalities allow more developers to find their own specific
> way in resolving their own specific problems.
> There could never be a single best way of dealing with any kind of problems.
>
> I strongly vote for 1).
>
> Kind regards,
> Thomas
>
> ------
>
> Thomas White
>
>
>>
>> Alternatives:
>>
>> 1) Leave it in and see if the world collapses
>> 2) Remove it completely
>> 3) Move it to it's own extension module.
>>
>> My vote is for option #3
--
Adam Retter
eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb
|
|
From: Thomas W. <tho...@gm...> - 2010-02-02 12:53:41
|
Adam, All points you make are both valid and important. I agree it is vital to know when to listen and when (not) to talk, especially when one misunderstands completely the context of the conversation. Please accept my apologies for butting in. Thomas On 2 February 2010 09:58, Adam Retter <ad...@ex...> wrote: > On 1 February 2010 19:34, Thomas White <tho...@gm...> wrote: > > I hope this will be taken in the best possible way. > > > > I could never understand why a functionality that is used by some > > users could be proposed to be removed by other users either who don't use > it > > or don't like it. > > > > If something have got a momentum it should stay. It is there, it does not > > require any additional resources and brings other routes to more > solutions. > > This is not how good software is built, but more importantly its not > how it is maintained. We have had plenty of contributions in the past, > some of excellent quality and some of not. But if we just allowed > anyone to add anything we would end up with a mess, everyone thinks > eXist should be something different, at present it is a lot of > different things to different people - but the core team tries not to > dilute the product too much. Maintainability is also a huge issue for > us. Basically we like features to be discussed by the core team. > > > |
|
From: Adam R. <ad...@ex...> - 2010-02-02 13:13:27
|
Not at all - all comments are always welcome :-)
I just wanted you to understand that each new feature is something
else for the core team to maintain should the contributor then
disappear. This can be difficult, especially if the feature is not
understood or been discussed.
Please do comment in future. Cheers Adam.
On 2 February 2010 12:53, Thomas White <tho...@gm...> wrote:
> Adam,
>
> All points you make are both valid and important. I agree it is vital to
> know when to listen and when (not) to talk, especially when one
> misunderstands completely the context of the conversation.
>
> Please accept my apologies for butting in.
>
> Thomas
>
> On 2 February 2010 09:58, Adam Retter <ad...@ex...> wrote:
>>
>> On 1 February 2010 19:34, Thomas White <tho...@gm...> wrote:
>> > I hope this will be taken in the best possible way.
>> >
>> > I could never understand why a functionality that is used by some
>> > users could be proposed to be removed by other users either who don't
>> > use it
>> > or don't like it.
>> >
>> > If something have got a momentum it should stay. It is there, it does
>> > not
>> > require any additional resources and brings other routes to more
>> > solutions.
>>
>> This is not how good software is built, but more importantly its not
>> how it is maintained. We have had plenty of contributions in the past,
>> some of excellent quality and some of not. But if we just allowed
>> anyone to add anything we would end up with a mess, everyone thinks
>> eXist should be something different, at present it is a lot of
>> different things to different people - but the core team tries not to
>> dilute the product too much. Maintainability is also a huge issue for
>> us. Basically we like features to be discussed by the core team.
>>
>>
--
Adam Retter
eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb
|
|
From: Evgeny G. <gaz...@gm...> - 2010-02-02 15:28:34
|
Offtop! Todays trigger's events is not implemented as well. Suggest discus the one important features too - new triggers events. Who don't know what is it - look: /branches/gev/triggers/eXist. -------- Evgeny |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X