Improve external Module Mechanism
Brought to you by:
daldei
Menu
▾
▴
#10 Improve external Module Mechanism
* Currently, installing a Module (even MakrLogic XCC/J) is not entirely intuitive.
* It would be good if a Module provider could give ONE-SINGLE-PACKAGE and just place it in a directory like ext
** With a view to the commands being "automatically available" to run when running XMLSH.
** Does not need to be as complex as a maven dependency "plug in" just yet ...
I suspect that other people may not have the same level of patience as myself when trying to use XMLSH.
Discussion
This is an excellent suggestion. There are some complexities that have kept me from implementing this so far. Dependancies and licensing issues are the biggest issue. For example with the MarkLogic extension I dont supply the dependent library, you have to get it yourself. I am not sure if I am allowed to get it myself. If I cannot pre-package all the components then making a single package module is not possible by me ... the end user would have to do this. In which case, where is the value add of forcing them to not only get the dependent files but also package them ?
I have thought about auto downloading like Maven uses, but from my research there are legal issues involved with distributing third party libraries with Maven because it does not include the required license files. Last thing I want is an army of lawyers at my door. (Plus I would have to figure out maven ... ug).
Suggestions on how to solve these problems very much welcome. I definately would like it easier to install. I have been looking at Java installers but so far have no positive feedback that they would be useful by anyone. I'd also like a "full everything" install that installs xmlsh and all the modules ... but again this has packaging and legal issues.
Maybe you are refering to third party extension developers ? In which case simply supporting a zip file with the needed directory structure might be enough ?
Also what do you mean by "automatically available" ... I still feel somewhat strongly about needing to use an import statement so that extensions are not all in the same namespace and that they are not all loaded unless you need them.
I personally don't see too much of a legal issue going down the Maven root.
MarkLogic provide a Maven URL for the XCC/J drivers, this could be a potential avenue to explore ... However, I certainly am aware of a large financial institution which is a heavy user of XMLSH who have locked down firewalls.
I know that when they deploy an XMLSH script onto a Production Unix/Linux server it has no access to the internet what so ever (which also makes sense from there point of view). MarkLogic DBAs are given a zip package to extract and a set of instructions to run the script, if it doesn't run because a certain URL it's back to often back to development and testing.
Here is a possibility,
When extracting the XMLSH binaries, it already has the folder structure:
/xmlsh_X_X_X/etc/marklogic
/xmlsh_X_X_X/etc/marklogic/marklogic_ext.jar
Where no "module.xml" exists (or more importantly does not need to be configured). It would be XMLSH's responsibility to dynamically load jar's contained either in /etc/marklogic (or perhaps some other location).
So all the user had to do was just "drop" the a marklogic xcc/j binary straight in. When they ran XMLSH the commands were just ready to go.
Perhaps it should be the responsibility of the plugin, e.g. marklogic_ext.jar to do the clever work and the dynamic loading?
Regards,
Charles
Using an import statement, to only load extensions if they are required is fine.
The XMODPATH and configuring module.xml in the relevent ext/package directory is the bit that will throw users in my opinion.
Perhaps the problem is with the documentation not the process.
The module.xml file does NOT need to be configured, it is supplied perfectly 'as is' with the package zip (for all packages on xmlsh.org atleast ...).
All you need to do is unzip the package into the correct directory and your done.
XMODPATH *could* be pre-configured, but to what ? perhaps it can default to $XMLSH/ext ... but then every time you update xmlsh you have to move all your modules around even those that dont need updating.
If you are making your own extension module, there is a lack of documentation (aka none) I had no idea anyone was doing that and if they asked me I would write something up. But making the module.xml file is trivial compared to writing the java code.
So I guess I really dont understand what the problem is.
In my /xmlsh_1_2_0/ext/marklogic/ directory I have 3 files
* marklogic_ext.jar
* marklogic-xcc-4.2.9.jar
* module.xml
IF module.xml does not have the following exact element:
<file url="marklogic-xcc-4.2.9.jar"/>
When trying to run MarkLogic commands, nothing happens, I assume it errors and I simply get the response:
com/marklogic/xcc/types/XdmValue
If the jar file name is not exactly right, this happens. If I am putting the marklogic xcc jar in the wrong location, because it would auto-loads from somewhere else then the documentation should really be a little bit clearer.
Thanks for the ideas. Just for history, the reason I list the jar files in the module is I once had an idea (and this actually is supported) of ONLY needing the module.xml file ... the JAR files can reside anywhere. You dont need an XMODPATH or a directory or even the files locally.
If you only have a module.xml file you can import a module even if it resides entirely on the net.
e.g.
see: http://www.xmlsh.org/CommandImport
locally:
import module mine=/dir/module.xml
remotely:
import module mine=http://www.mymodules.com/module.xml
If the module.xml file uses http addresses for all the jar files they should run (just like applets do).
BUT ... I never used this and its probably not a great feature.
The other reason is that in theory you can share common jar files amoung your modules. You could have a common directory of things like loggers and such and your module.xml file tells you where to look. The module.xml also tells you a minimum version # of xmlsh which is quite useful if you try to load an old module.
Possible solutions:
1) Supply ALL files with the module as a zip. Then there is never "the user has to fetch file and configure things". I am concerned legally about this as most 3rd party modules have restrictions on how they can be deployed by others. But it would be the easiest for users.
2) Provide a common default directory for XMODPATH (like inside xmlsh). That way you dont have to set it. (but you DO have to know where to install your modules).
3) Remove or make optional the dependant jar file section in the library. If missing then just load all .jar files to the classpath.
4) Provide installers. I am looking at a java installer but I am concerned they are more hassle then they are worth.
5) Screw modules as separate installations and just build them all into xmlsh. Same legal issue as #1 but atleast I can try to supply all the legal notices. Side effect, this eliminates or obscures the idea of user developed modules. I still have a goal of a repository of 3rd party contributed modules like CPAN.
I looked at the module configuration some more. There is definately a need for some kind of metadata such as in the module.xml . Bare minimum it needs the package name to use. I cant assume a hard-coded name because multiple modules are loaded and cant all use the same name - atleast how I have the classloaders configured (and I dont want to make more complex class loaders like Tomcat uses , it quickly turns into hell).
So th
Silly SF, cant edit comments.
Ok given that metadata is needed it can either be external where it is viewable and changable (module.xml) or it could be within the jar file itself as is common with say WAR files or the META-INF directory for jar files.
Given that, can this be improved. The simplest thing I can think of is letting the <classpath> attribute contain a directory wildcard. This would solve the case of say updating the marklogic XCC jar with a new version without editing the module.xml . On the other hand there is no guarantee that a module will work if you don't have exactly the right files. So this could be more dangerous then having the module be explicit. I am still not convinced (yet) this would be of actual benefit.
I am open to suggestions on how to make this easier for users and developers (mainly users as developers should be expected to know more).
I checked in an improvement to Modules so that you can specify directories in addition to files for classpath
<directory uri="."/>
causes the module directory to be scanned for .jar files
That has obvious benifits, but also is risky because it will pick up all jar files in the directory.
Do you think this would be sufficient ?
As per discussions, closing.
I have made an improvement so the module.xml can optionally contain a directory instead of a file for dependant jars. This should eliminate any modifying of the module.xml file necessary by users. As for XMODPATH ... I dont see a way around that without making too many hard coded decisions in xmlsh. (similar to PATH, or XMLSH variables I dont think they should be hard coded).
Other suggestions welcome.