[Exist-development] How 'at-relative' extention of the import clause that will facilitate module sharing

[Thomas W. <tho...@gm...>]

Joe,

Using the correct terminology always helps to be understood. Thank you for
your suggestion.

First let explain what I mean when I say * 'at-relative'.*

In the current implementation when a library file is imported the path in
the 'at' part of the import clause should
relative to the main XQuery module.  If the library module A imports another
library file B the path for importing B is not relative to the location of
library A but relative to the main module. So if we organise our modules in
directories we need to hardcode the path of the second level library files
(B) into the first level library (A).
If we decide to change the directory structure or move some folder with
library files, we need to hardcode manually again the new relative path in
the first level library (A). It is so inconvenient that I do not believe
there are people out there brave enough who have tried three levels of
module importing, located in different directories. ( For those of you who
may asks who in earth will needs three or more levels of library module
importing I can only say the size matters. If you have multiple teams, a
number of applications and many client specific features you may need even
more then three levels of importing).

What I am proposing is to extent the import clause with *
'at-relative'*where the path is relative to the file that does the
importing. Then it is
much simpler to maintain any library module import structure and when we
change or relocate a library we do not need to change much. Actually if we
use Evgeny's dynamic module registration we do not need to change anything
:-).

I am going to illustrate my point describing three cases, pretending*
'at-relative'* was available.

*Case 1. Hotel management system*
**
The system may have the following sections developed by different teams:

   - Rooms
   - Reservations
   - Guests
   - Inventory
   - Billing and Accounting
   - Online payments
   - Reports

The common functionality for each section of the system is located in a
number of *library* modules.
Every section has its own set of *main* XQuery modules for the screens
required.
Every main XQuery imports library modules from their own section and some
library modules from the other sections of the system.

To reduce the impact of the changes, library files are not used directly.
Every set of section library files has a single library file that I am going
to call *main library* file (new term: A library module that imports other
library modules; we can have as many main libraries as we need) . It is more
or less a wrapper for the functions in the other library files.
The main library module imports other library files using
*'at-relative'*where all paths are relative to it (the file that does
the importing). It
provides both a stable API for this section and *a single library file to
import* when any of the functionalities from this section is needed.
When a main library module is used we do not need to know anything about the
location and the structure of the other library modules. Implementation of
the library files may change, more files or functions can be added, removed
or merged but this will not impact the rest of the application, because the
main library module will reflect the changes.

*Case 2. The well known XQuery library
**http://www.xqueryfunctions.com/xq/*<http://www.xqueryfunctions.com/xq/>

At the moment the library is in one monolithic file. There are ten groups of
function

   - Strings
   - Numbers
   - Dates, Times and Durations
   - Atomic Values of All Types
   - Sequences
   - XML Elements and Attributes
   - XML Nodes
   - XML Namespaces and Names
   - XML Documents, URIs and IDs
   - Miscellaneous

We can separate functions in ten library modules (to reduce the memory
footprint in a busy site)  and if a function from another module is required
then *'at-relative'* can be used to import the other small library module.
Again when we use any of the ten library modules we no not need to know it
they import another modules, both local or remote.

*Case 3. An XQuery library distributed as a JAR file*

A library may have a complex internal file and directory structure where the
library files are kept.
Providing a number of main library modules that import library modules from
the JAR file using  *'at-relative'* is a convenient way to hide the
complexity and provide a stable API. (At the moment it is not possible to
create this type of libraries).

To me organising my code in libraries like the cases described above using
*'at-relative'* is much more appealing then keeping all main and library
modules in a single directory.

What about you?

I will be more then happy to give more clarification if needed.

Thomas


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007




On 31 May 2010 01:00, Joe Wicentowski <jo...@gm...> wrote:


> Evgeny - Cool.  I am looking forward to giving these new functions a
> try.  I think Dannes asked for unit tests too - something I think is
> necessary to ensure the functions can stay in trunk.  Could you
> contribute unit tests when you have the chance?  Also is there a
> deregister/unmap function, or only a register function?
>
> Thomas - I am definitely in favor of features that enable sharing.
> But I'm sorry, I'm having trouble understanding what you mean by
> "at-relative"; I'm not dubious, I just don't get it yet.  Could you
> illustrate with a simple, concrete scenario?  A few requests: Could
> you define your terms a bit?  (.xql is ambiguous.  Library module vs.
> main module is a distinction we agree on; namely,
> http://www.w3.org/TR/xquery/#dt-main-module vs.
> http://www.w3.org/TR/xquery/#dt-library-module.)  Also, should this go
> in a separate thread?
>
> Thanks,
> Joe
>


On Sun, May 30, 2010 at 5:54 PM, Thomas White <tho...@gm...> wrote:
> Evgeny,
>
> I like the functionality you are working on and I can see its potential
> for using it with single file self-contained modules. I am sure when your
> extension is available we are going to find a lot of ways to make our code
> more flexible, more dynamic  and less dependent of config files.
>
> Something that is missing in our eXist community is a repository of
reusable
> library files where we can share what we have done and help each other not
> to reinvent the hot watter again and again. I bet almost everyone of us
> after a couple of months work with eXist has something useful to share
with
> others. We miss a huge opportunity to have an exponential growth of shared
> knowledge and expertise.
>
> I am a strong believer that 'at-relative' extension of the import clause
> will change the situation dramatically in our favour. When it is easier to
> share and reuse other developer's work then the community will pick up
this
> opportunity quickly and in short period of time we can accumulate a
> significant repository of reusable eXist modules, both XQuery and JAVA
> based.
>
> I hope I have convinced most of you about the importance of having
> 'at-relative' extension of the import clause.
>
> Thomas



>  On 30 May 2010 14:25, Thomas White <tho...@gm...> wrote:
>
>
>> Evgeny,
>>
>> I think you are missing my point. It is not question of dynamic or static
>> import it is about the import context.
>>
>> If we have a library that consists of 50 .xql modules and there are 5 main
>> modules that combine a subset of these modules for a specific  functionality
>> imported into the XQueries what generate the user interface.
>>
>> There are many reasons why we will prefer to use the main 5 import files
>> instead of  the mini module files:
>> 1) The library vendor may decide to change some of the implementations and
>> reshuffle of some some of the function from one module to another. When we
>> import a main module file that imports what is necessary we do not need to
>> know which modules are used and what are their names, locations and name
>> spaces.
>> 2) We may want to reduce the memory footprint in a busy site because for
>> every module loaded an instance of the module is created into the memory.
>>
>> At the moment we can not have libraries of XQuery modules files because
>> the module import context is always the main XQuery file. If the a module
>> 'A'  imports some other modules the paths in 'at' clause would be updated to
>> reflect the relative path to the main modules. This is not practical (if the
>> module file is imported from more then one main XQuery,  and some times it
>> is not possible - if a module is in a JAR file we  can not update the import
>> paths.
>> The only possible way to deal with this serious limitation is to put all
>> XQuery and module files in the same directory.
>>
>> It is a number game after all. If we do small applications then we can
>> have all files in a single directory. If we have a bigger project it is
>> obvious to use directories to separate sections of the project for every
>> team and have all modules into a single directory. But it is so inconvenient
>> when the functionality is shared - then instead of giving a single import
>> file to be used by other teams, we have to give a list of all our modules to
>> be included one by one. The obvious solution - have own copy of the modules
>> in every project is a single directory. :-(
>>
>> Can you imagine if we have to produce a content with XSLT and if we have
>> had the same import context limitation and all imported XSLT should always
>> have an URI relative to the main XSLT.  I do not know about you but I can't.
>>
>>
>> What I am have been advocating for quite some time is to introduce a
>> variation of the 'at' clause in the  import command *'at-relative'* that
>> will resolve all import files path relative to the file that does the
>> importing.
>>
>> Then we can create libraries of XQuery modules and share functionality
>> much easily and we can group our XQuery code in directories anyway we like.
>>
>> I really hope some of the leading developers will be able to spend some
>> time and add *'at-relative'* to the import command.
>>
>> Regards,
>> Thomas
>>
>>  On 27 May 2010 17:25, Thomas White <tho...@gm...> wrote:
>>
>>
>>> Evgeny,
>>>
>>> I am glad you are working on the modules, because once you are there
>>> you may be able to help resolving an important issue - relative module
>>> importing context ie an ability to import modules relatively to the
>>> importing file:
>>>
>>> If I have a file A the imports module B, that imports module C at the
>>> moment all imports are resolved to the main file A.
>>>
>>> If we want to make a libraries made of multiple files then imports
>>> should be able to be relative to the current  importing file - in this
>>> case the path for importing module C should be relative to module B.
>>>
>>> In order to keep existing functionality we can use 'at-relative'
>>> instead of 'at' when we import a module in a relative fashion.
>>> Back in Prague Dmitriy said it should be relatively simple to
>>> implement this functionality.
>>>
>>> I would really appreciate if you can solve this issue, it is a show
>>> stopper for me for my eXe application framework.
>>>
>>> Regards,
>>> Thomas
>>>
>>>
>>>
>>>
>>