exist-development — eXist Developer's List

[Exist-development] thoughts about next steps with RDBMS integration

[James F. <jam...@ex...>]

whilst doing xproc development, I have had great fun integrating some of
eXist existing extension functionality as extension steps ... a thought
occurred to me about the current state of integration with RDBMS.

Currently we have the nice SQL extension module (written by Adam Retter)
which uses JDBC to query a relational database ... this means we can easily
query an RDBMS through xquery using SQL statements. I think this module
enables a lot of scenarios but misses off the primary scenario for the
masses of people out there with data currently existing in RDBMS but want to
use XML database.

Take the following imaginary developer scenario:

'I have an existing RDBMS and I want to use XML database ... I am worried
about going 'all xml' because RDBMS has served me well, not to mention I
have a lot of code hanging off the existing database ... if I migrate to an
XML database I will have to migrate all those other bits ... this is too
hard and too much change to attempt in one risky step. I would like to
install eXist and provide in some registry a connection string which
effectively 'mounts' the RDBMS as a collection ... from which here on out I
can use pure xquery to work on. For now a read only view of the data is all
I need but someday I would like to be able to do updates as well on the
RDBMS data. Lastly I should be able to configure cache characteristics if
the database connection is intermittent'

Thoughts ? Does anyone see this scenario as something that needs to be
addressed ?

James Fuller

Re: [Exist-development] DVCS - Distributed Version Control?

[Andrzej J. T. <an...@ch...>]

Adam:

> Moving away from SourceForge would seem a little extreme to me, and
> certainly would raise larger questions than just what SCM we use.

I'm not sure that who provides the hosting is such a big deal.  So long as they have been around for a bit and have a
good track record. Both github and bitbucket would seem to be pretty "safe" alternatives.

> Perhaps it would be worth contacting sourceforge and finding out what
> their plans for Mercurial are? They are only trialling it at the
> moment, so I guess they will have a roadmap for making it a primary
> service and probably offering a http service.

Good suggestion....I'll see if I can get through to someone there in the next week or so.


-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

Re: [Exist-development] DVCS - Distributed Version Control?

[Andrzej J. T. <an...@ch...>]

Dannes:

> For me, tooling support is the most important factor here. At this
> moment, as far as I see, SVN is the most commonly used way for code
> management. Existing tooling is just excellent (tortoisesvn, svnkit,
> etc) and .... existent.

Yup...but the tooling is coming up fast with DVCS implementations. The fact that Linux uses git says a lot.

> I fear that a switch would 'ruin' this situation. The current SVN does
> not hurt me in any way, so I do not see the benefit of another system,
> at least not at this moment.

It's not that SVN is "broken" in any way.  I'm very comfortable and used to it as well.

I prompted this discussion only because DVCS systems like git or mercurial might make it a bit easier for all of us to
branch and merge experimental code (since we've had a few pain points that this recently).

So I thought it would be worth discussing whether those possible benefits might outweigh the negatives of doing a switch.

One thing that is very interesting is that you can use git, backed by an SVN repository as an interim step. Check this
out if you are interested in that idea:

	http://progit.org/book/ch8-1.html

Interesting concept....

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

Re: [Exist-development] Wikibook and xqDoc

[Dan M. <dan...@gm...>]

> Will it be possible to distribute the demo for other users to study?

Yes, this is no problem for the XQuery code.  I will need to work with Dr.
Mueller to make sure that he feels comfortable with distributing the XML
markup of the Shakespeare code.  Either way, I will send the XQuery code to
anyone that is interested!

I must say that creating this demo with all of Shakespeare documents is a
great demonstration of the capabilities of the eXist system and the Lucene
search.  I only spent a bit of time on it and I am sure my code is not
"optimal" and could be cleaned up quite a bit.

One other note, I have found that there are some missing TEI tags in the
sample files that really need to be added to make the XQueries simpler.  For
example <div type="act"> and <div type="scene"> are missing.  I hope to work
with Dr. Mueller to encourage better TEI tags that make the XQueries much
simpler.  Right now they are more complex then they need to be.

- Dan

On Tue, Mar 30, 2010 at 1:09 PM, Wolfgang Meier <wol...@ex...>wrote:

> Dan,
>
> > Here is an early demo of what we need to document for the TEI community
> that
> > is building navigation and search:
> >
> > http://www.syntactica.com/rest/db/org/northwestern/apps/tei/index.xq
>
> Cool. I think this will become a helpful demo. Martin has repeatedly
> offered his help with writing documentation, but I'm flooded by so
> many things, that I was unable to follow up in time. It is good to see
> other people jump in ;-)
>
> > Dr. Muller is willing to help writeup these examples (once he understand
> > them a bit more) so we might be creating a "Beginners Guide to TEI for
> > eXist" based on this work.
>
> This would surely be a very welcome addition since many of our users
> start with TEI. Will it be possible to distribute the demo for other
> users to study?
>
> Wolfgang
>



-- 
Dan McCreary
Semantic Solutions Architect
syntactica.com
952-460-1674
VOIP: 111@69.199.167.229

Re: [Exist-development] Wikibook and xqDoc

[Wolfgang M. <wol...@ex...>]

Dan,

> Here is an early demo of what we need to document for the TEI community that
> is building navigation and search:
>
> http://www.syntactica.com/rest/db/org/northwestern/apps/tei/index.xq

Cool. I think this will become a helpful demo. Martin has repeatedly
offered his help with writing documentation, but I'm flooded by so
many things, that I was unable to follow up in time. It is good to see
other people jump in ;-)

> Dr. Muller is willing to help writeup these examples (once he understand
> them a bit more) so we might be creating a "Beginners Guide to TEI for
> eXist" based on this work.

This would surely be a very welcome addition since many of our users
start with TEI. Will it be possible to distribute the demo for other
users to study?

Wolfgang

Re: [Exist-development] DVCS - Distributed Version Control?

[Dmitriy S. <sha...@gm...>]

On Tue, 2010-03-30 at 12:15 +0200, Dannes Wessels wrote:
> Hi,
> 
> On Tue, Mar 30, 2010 at 12:02 PM, Adam Retter <ad...@ex...> wrote:
> > Moving away from SourceForge would seem a little extreme to me, and
> > certainly would raise larger questions than just what SCM we use.
> >
> > Perhaps it would be worth contacting sourceforge and finding out what
> > their plans for Mercurial are? They are only trialling it at the
> > moment, so I guess they will have a roadmap for making it a primary
> > service and probably offering a http service.
> 
> For me, tooling support is the most important factor here. At this
> moment, as far as I see, SVN is the most commonly used way for code
> management. Existing tooling is just excellent (tortoisesvn, svnkit,
> etc) and .... existent.

Mercurial don't have problems in tools with GUI
( http://en.wikipedia.org/wiki/Mercurial ). Current problem is http
assess only, proxy can solve it.

I did start one project on Mercurial & I'll share my experience as soon
as get them ;) 

> I fear that a switch would 'ruin' this situation. The current SVN does
> not hurt me in any way, so I do not see the benefit of another system,
> at least not at this moment.
> 
> D.
> 
-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] Communication problem between Exist & Android

[Adam R. <ad...@ex...>]

You need to specify the resource name, i.e. -

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.commons.httpclient.methods.StringRequestEntity;

public class App
{
    public static void main( String[] args ) throws
UnsupportedEncodingException, IOException
    {
        HttpClient client = new HttpClient();
        PutMethod post = new
PutMethod("http://localhost:8080/exist/rest/db/test.xml");

        RequestEntity entity = new StringRequestEntity("<hello/>",
"text/xml", "UTF-8");
        post.setRequestEntity(entity);

        int code = client.executeMethod(post);

        System.out.println("Response Code: " + code);
    }
}

On 30 March 2010 14:55, Jérémy Cazaux <jer...@gm...> wrote:
> Hello,
>
> I have a communication problem between android and eXist. I need to send
> information/file from android to eXist.
>
> My terminal in Android should be able to submit files of type XML to eXist
> database. I'm using android in VPN.
> I tried using the PUT&GET method of HTTP (i think GET&PUT method are aviable
> with eXIst, see=>http://exist.sourceforge.net/devguide_rest.htm ):
>
> private static String url = "http://localhost:8080/exist/rest/db/";
>
> HttpClient httpclient = new DefaultHttpClient ();
> HttpPut httpput = new HttpPut (url);
>
> try (
> httpput.setEntity (new StringEntity ( "<the goes here> xml"));
> httpput.setHeader ( "Content-Type", "text / xml");
> HttpResponse response;
> response = httpclient.execute (httpput);
>
> ) Catch (ClientProtocolException e) (
> e.printStackTrace ();
>
> ) Catch (IOException e) (
> e.printStackTrace ();
> )
>
> I get no result and no error in compilation or execution.
>
> I therefore ask you one example of Java code, or if an explanation of the
> "right way" to use.
>
> Thanks you for your help
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] Communication problem between Exist & Android

[James F. <jam...@gm...>]

On Tue, Mar 30, 2010 at 3:55 PM, Jérémy Cazaux <jer...@gm...> wrote:
> Hello,
>
> I have a communication problem between android and eXist. I need to send
> information/file from android to eXist.
>
> My terminal in Android should be able to submit files of type XML to eXist
> database. I'm using android in VPN.
> I tried using the PUT&GET method of HTTP (i think GET&PUT method are aviable
> with eXIst, see=>http://exist.sourceforge.net/devguide_rest.htm ):

ok first step, u sure you can access eXist e.g. u sending right creds
or have it setup to be writeable by guest ?

>
> private static String url = "http://localhost:8080/exist/rest/db/";

you will need to give a name to the new document e.g.
http://localhost:8080/exist/rest/db/test.xml

> HttpClient httpclient = new DefaultHttpClient ();
> HttpPut httpput = new HttpPut (url);
>
> try (
> httpput.setEntity (new StringEntity ( "<the goes here> xml"));

ummm, unsure of what this line is ... cause its not xml ... perhaps u
are trying to upload binary document ?

> httpput.setHeader ( "Content-Type", "text / xml");
> HttpResponse response;
> response = httpclient.execute (httpput);
>
> ) Catch (ClientProtocolException e) (
> e.printStackTrace ();
>
> ) Catch (IOException e) (
> e.printStackTrace ();
> )
>
> I get no result and no error in compilation or execution.
>
> I therefore ask you one example of Java code, or if an explanation of the
> "right way" to use.
>
> Thanks you for your help

a simple HTTP PUT should work but u need to check access, make sure u
are slinging xml (if thats your intention), give it a name ... u sure
u dont see anything in the eXist logs (usually here
webapp/WEB-INF/logs/exist.log)

hth, James Fuller
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>

[Exist-development] Communication problem between Exist & Android

[Jérémy C. <jer...@gm...>]

Hello,

I have a communication problem between android and eXist. I need to send
information/file from android to eXist.

My terminal in Android should be able to submit files of type XML to eXist
database. I'm using android in VPN.
I tried using the PUT&GET method of HTTP (i think GET&PUT method are aviable
with eXIst, see=>http://exist.sourceforge.net/devguide_rest.htm ):

private static String url = "http://localhost:8080/exist/rest/db/";

HttpClient httpclient = new DefaultHttpClient ();
HttpPut httpput = new HttpPut (url);

try (
httpput.setEntity (new StringEntity ( "<the goes here> xml"));
httpput.setHeader ( "Content-Type", "text / xml");
HttpResponse response;
response = httpclient.execute (httpput);

) Catch (ClientProtocolException e) (
e.printStackTrace ();

) Catch (IOException e) (
e.printStackTrace ();
)

I get no result and no error in compilation or execution.

I therefore ask you one example of Java code, or if an explanation of the
"right way" to use.

Thanks you for your help

Re: [Exist-development] Wikibook and xqDoc

[Dan M. <dan...@gm...>]

Chris,

Thanks for taking the time to do this!  I think that having great examples
of how to do specific tasks is critical for our new users.  I think that
among all the other open source XQuery systems we are starting to get some
better documentation, especially for our beginning users.

Last week Dr. Martin Muller from Northwestern University in Illinois and a
frequent contributor to the TEI lists, challenged us to make eXist usable by
more of the non-programmers in the TEI community.  He sent us 40 Shakespeare
plays in poems in TEI format and wanted us to demonstrate how we would
create a simple system to do navigation and search.

Here is an early demo of what we need to document for the TEI community that
is building navigation and search:

http://www.syntactica.com/rest/db/org/northwestern/apps/tei/index.xq

Dr. Muller is willing to help writeup these examples (once he understand
them a bit more) so we might be creating a "Beginners Guide to TEI for
eXist" based on this work.

- Dan

On Mon, Mar 29, 2010 at 6:09 PM, Chris Wallace
<kit...@go...>wrote:

> I recently put a bit of time into restructuring the XQuery wikibook.
> Amongst other changes I have built up the section on eXist to include
> several of the modules and links to articles which illustrate the use of
> functions in these modules.  There is much left to do but I hope its moving
> in the right direction (until perhaps a move to some other platform).  Any
> example code is always welcome and I will install it on my UWE server.
>
> In the process of implementing examples of xqDoc, I couldnt find a basic
> stylesheet to transform the output from the scan function to HTML - anyone
> know of one?
>
> Chris
>
>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>


-- 
Dan McCreary
Semantic Solutions Architect
syntactica.com
952-460-1674
VOIP: 111@69.199.167.229

Re: [Exist-development] DVCS - Distributed Version Control?

[Dannes W. <da...@ex...>]

Hi,

On Tue, Mar 30, 2010 at 12:02 PM, Adam Retter <ad...@ex...> wrote:
> Moving away from SourceForge would seem a little extreme to me, and
> certainly would raise larger questions than just what SCM we use.
>
> Perhaps it would be worth contacting sourceforge and finding out what
> their plans for Mercurial are? They are only trialling it at the
> moment, so I guess they will have a roadmap for making it a primary
> service and probably offering a http service.

For me, tooling support is the most important factor here. At this
moment, as far as I see, SVN is the most commonly used way for code
management. Existing tooling is just excellent (tortoisesvn, svnkit,
etc) and .... existent.

I fear that a switch would 'ruin' this situation. The current SVN does
not hurt me in any way, so I do not see the benefit of another system,
at least not at this moment.

D.

-- 
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] DVCS - Distributed Version Control?

[Adam R. <ad...@ex...>]

Moving away from SourceForge would seem a little extreme to me, and
certainly would raise larger questions than just what SCM we use.

Perhaps it would be worth contacting sourceforge and finding out what
their plans for Mercurial are? They are only trialling it at the
moment, so I guess they will have a roadmap for making it a primary
service and probably offering a http service.

On 30 March 2010 00:27, Andrzej Jan Taramina <an...@ch...> wrote:
> Adam:
>
>> Before we moved to Subversion, CVS used to be a major pain in the ass
>> for me, because I work from far too many locations (offices, airports,
>> trains, cafes etc) where the only access out is HTTP, Subversion works
>> perfectly over HTTP. If SourceForge were to enable Mercurial writes
>> over HTTP then I might be sold on the idea ;-)
>
> Bitbucket (mercurial hosting) supports both HTTP and SSH access for both pull and push, and has free services for open
> source projects.
>
> Github (git hosting) has free services for open source, but I'm not sure if you have to use SSH or not yet.
>
> So there are a number of alternatives to Sourceforge for DVCS.
>
> Just FYI.
>
> --
> Andrzej Taramina
> Chaeron Corporation: Enterprise System Solutions
> http://www.chaeron.com
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] Wikibook and xqDoc

[Wolfgang M. <wol...@ex...>]

> Amongst other changes I have built up the section on eXist to include
> several of the modules and links to articles which illustrate the use of
> functions in these modules.

Very useful indeed. Once we have the extended module documentation
system in place, we will use your table to link back to the wikibook
articles from the function documentation.

> In the process of implementing examples of xqDoc, I couldnt find a basic
> stylesheet to transform the output from the scan function to HTML - anyone
> know of one?

The xqDoc structure is rather simple, so we used XQuery to transform
output to HTML. For example, see Joe's module:

webapp/functions/functions.xql

Wolfgang

Re: [Exist-development] SSL for Jetty under trunk

[Dmitriy S. <sha...@gm...>]

On Tue, 2010-03-30 at 08:18 +0200, Dannes Wessels wrote:
> Hi,
> 
> On Tue, Mar 30, 2010 at 7:26 AM, Dmitriy Shabanov <sha...@gm...> wrote:
> > Well, add new mode "jettySSL" should solve problem.
> >
> > If no objections, I can make it this way.
> 
> as long it is switched off by default (and no certificate is deployed)

default will be jetty mode as it now. 

BTW, should I add standaloneSSL?

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] SSL for Jetty under trunk

[Dannes W. <da...@ex...>]

Hi,

On Tue, Mar 30, 2010 at 7:26 AM, Dmitriy Shabanov <sha...@gm...> wrote:
> Well, add new mode "jettySSL" should solve problem.
>
> If no objections, I can make it this way.

as long it is switched off by default (and no certificate is deployed)

regards

Dannes

-- 
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] SSL for Jetty under trunk

[Dmitriy S. <sha...@gm...>]

Well, add new mode "jettySSL" should solve problem.

If no objections, I can make it this way.

-- 
Cheers,

Dmitriy Shabanov

On Tue, 2010-03-30 at 00:18 -0500, Loren Cahlander wrote:
> The problem with that, is that the exist startup uses the following in
> org.exist.start.Main:
> 
> 
>                 String config;
>                 if (_mode.equals("jetty"))
>                     config = "jetty.xml";
>                 else
>                     config = "standalone.xml";
>                 args = new String[]{ System.getProperty("jetty.home")
> + File.separatorChar + "etc"
>                         + File.separatorChar + config };
> 
> 
> There currently is no option to specify any additional jetty
> configuration files.
> 
> 
> On Mar 29, 2010, at 11:55 PM, Dmitriy Shabanov wrote:
> 
> > Loren,
> > 
> > Better keep them separate & add  
> > 
> > <!-- ==============================================================
> > --> 
> > <!-- Configure SSL for the Jetty Server
> >                             --> 
> > <!-- this configuration file should be used in combination with
> >     --> 
> > <!-- other configuration files.  e.g.
> >                               --> 
> > <!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml
> >         --> 
> > <!-- ==============================================================
> > --> 
> > 
> > to jetty.xml as information how enable it.
> > 
> > -- 
> > Cheers,
> > 
> > Dmitriy Shabanov
> > 
> > On Mon, 2010-03-29 at 18:55 -0500, Loren Cahlander wrote:
> > > Since we upgraded to Jetty 7, the information for enabling SSL has
> > > disappeared from the jetty configuration file.  I found
> > > jetty-ssl.xml
> > > in a download of the current release of Jetty.  Shall we add an
> > > option
> > > to look at the jetty-ssl.xml file as well as jetty.xml, or just
> > > add
> > > the contents of the jetty-ssl.xml file to the jetty.xml file?
> > > 
> > > 
> > > Below is the contents of the jetty-ssl.xml file:
> > > 
> > > 
> > >        <?xml version="1.0"?>
> > >        <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"
> > >        "http://www.eclipse.org/jetty/configure.dtd">
> > > 
> > > 
> > >        <!--
> > >        =============================================================== -->
> > >        <!-- Configure SSL for the Jetty Server
> > >        -->
> > >        <!-- this configuration file should be used in combination
> > >        with      -->
> > >        <!-- other configuration files.  e.g.
> > >        -->
> > >        <!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml
> > >        -->
> > >        <!--
> > >        =============================================================== -->
> > >        <Configure id="Server"
> > >        class="org.eclipse.jetty.server.Server">
> > > 
> > > 
> > >          <!-- if NIO is not available, use
> > >        org.eclipse.jetty.server.ssl.SslSocketConnector -->
> > > 
> > > 
> > >          <Call name="addConnector">
> > >            <Arg>
> > >              <New
> > >        class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
> > >        <Set name="Port">8443</Set>
> > >        <Set name="maxIdleTime">30000</Set>
> > >                <Set name="Acceptors">2</Set>
> > >                <Set name="AcceptQueueSize">100</Set>
> > >        <Set name="Keystore"><SystemProperty name="jetty.home"
> > >        default="." />/etc/keystore</Set>
> > >        <Set
> > >        name="Password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
> > >        <Set name="KeyPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
> > >                <Set name="truststore"><SystemProperty
> > >        name="jetty.home" default="." />/etc/keystore</Set>
> > >                <Set
> > >        name="trustPassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
> > >              </New>
> > >            </Arg>
> > >          </Call>
> > >        </Configure>
> > 
> > 
> 
> 

Re: [Exist-development] SSL for Jetty under trunk

[Loren C. <lor...@gm...>]

The problem with that, is that the exist startup uses the following in org.exist.start.Main:

                String config;
                if (_mode.equals("jetty"))
                    config = "jetty.xml";
                else
                    config = "standalone.xml";
                args = new String[]{ System.getProperty("jetty.home") + File.separatorChar + "etc"
                        + File.separatorChar + config };

There currently is no option to specify any additional jetty configuration files.

On Mar 29, 2010, at 11:55 PM, Dmitriy Shabanov wrote:

> Loren,
> 
> Better keep them separate & add  
> 
> <!-- ============================================================== --> 
> <!-- Configure SSL for the Jetty Server                             --> 
> <!-- this configuration file should be used in combination with     --> 
> <!-- other configuration files.  e.g.                               --> 
> <!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml         --> 
> <!-- ============================================================== --> 
> 
> to jetty.xml as information how enable it.
> 
> -- 
> Cheers,
> 
> Dmitriy Shabanov
> 
> On Mon, 2010-03-29 at 18:55 -0500, Loren Cahlander wrote:
>> Since we upgraded to Jetty 7, the information for enabling SSL has
>> disappeared from the jetty configuration file.  I found jetty-ssl.xml
>> in a download of the current release of Jetty.  Shall we add an option
>> to look at the jetty-ssl.xml file as well as jetty.xml, or just add
>> the contents of the jetty-ssl.xml file to the jetty.xml file?
>> 
>> 
>> Below is the contents of the jetty-ssl.xml file:
>> 
>> 
>>        <?xml version="1.0"?>
>>        <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"
>>        "http://www.eclipse.org/jetty/configure.dtd">
>> 
>> 
>>        <!--
>>        =============================================================== -->
>>        <!-- Configure SSL for the Jetty Server
>>        -->
>>        <!-- this configuration file should be used in combination
>>        with      -->
>>        <!-- other configuration files.  e.g.
>>        -->
>>        <!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml
>>        -->
>>        <!--
>>        =============================================================== -->
>>        <Configure id="Server"
>>        class="org.eclipse.jetty.server.Server">
>> 
>> 
>>          <!-- if NIO is not available, use
>>        org.eclipse.jetty.server.ssl.SslSocketConnector -->
>> 
>> 
>>          <Call name="addConnector">
>>            <Arg>
>>              <New
>>        class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
>>        <Set name="Port">8443</Set>
>>        <Set name="maxIdleTime">30000</Set>
>>                <Set name="Acceptors">2</Set>
>>                <Set name="AcceptQueueSize">100</Set>
>>        <Set name="Keystore"><SystemProperty name="jetty.home"
>>        default="." />/etc/keystore</Set>
>>        <Set
>>        name="Password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
>>        <Set name="KeyPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
>>                <Set name="truststore"><SystemProperty
>>        name="jetty.home" default="." />/etc/keystore</Set>
>>                <Set
>>        name="trustPassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
>>              </New>
>>            </Arg>
>>          </Call>
>>        </Configure>
> 

Re: [Exist-development] SSL for Jetty under trunk

[Dmitriy S. <sha...@gm...>]

Loren,

Better keep them separate & add  

<!-- ============================================================== --> 
<!-- Configure SSL for the Jetty Server                             --> 
<!-- this configuration file should be used in combination with     --> 
<!-- other configuration files.  e.g.                               --> 
<!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml         --> 
<!-- ============================================================== --> 

to jetty.xml as information how enable it.

-- 
Cheers,

Dmitriy Shabanov

On Mon, 2010-03-29 at 18:55 -0500, Loren Cahlander wrote:
> Since we upgraded to Jetty 7, the information for enabling SSL has
> disappeared from the jetty configuration file.  I found jetty-ssl.xml
> in a download of the current release of Jetty.  Shall we add an option
> to look at the jetty-ssl.xml file as well as jetty.xml, or just add
> the contents of the jetty-ssl.xml file to the jetty.xml file?
> 
> 
> Below is the contents of the jetty-ssl.xml file:
> 
> 
>         <?xml version="1.0"?>
>         <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"
>         "http://www.eclipse.org/jetty/configure.dtd">
>         
>         
>         <!--
>         =============================================================== -->
>         <!-- Configure SSL for the Jetty Server
>         -->
>         <!-- this configuration file should be used in combination
>         with      -->
>         <!-- other configuration files.  e.g.
>         -->
>         <!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml
>         -->
>         <!--
>         =============================================================== -->
>         <Configure id="Server"
>         class="org.eclipse.jetty.server.Server">
>         
>         
>           <!-- if NIO is not available, use
>         org.eclipse.jetty.server.ssl.SslSocketConnector -->
>         
>         
>           <Call name="addConnector">
>             <Arg>
>               <New
>         class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
>         <Set name="Port">8443</Set>
>         <Set name="maxIdleTime">30000</Set>
>                 <Set name="Acceptors">2</Set>
>                 <Set name="AcceptQueueSize">100</Set>
>         <Set name="Keystore"><SystemProperty name="jetty.home"
>         default="." />/etc/keystore</Set>
>         <Set
>         name="Password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
>         <Set name="KeyPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
>                 <Set name="truststore"><SystemProperty
>         name="jetty.home" default="." />/etc/keystore</Set>
>                 <Set
>         name="trustPassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
>               </New>
>             </Arg>
>           </Call>
>         </Configure>

[Exist-development] SSL for Jetty under trunk

[Loren C. <lor...@gm...>]

Since we upgraded to Jetty 7, the information for enabling SSL has disappeared from the jetty configuration file.  I found jetty-ssl.xml in a download of the current release of Jetty.  Shall we add an option to look at the jetty-ssl.xml file as well as jetty.xml, or just add the contents of the jetty-ssl.xml file to the jetty.xml file?

Below is the contents of the jetty-ssl.xml file:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">

<!-- =============================================================== -->
<!-- Configure SSL for the Jetty Server                              -->
<!-- this configuration file should be used in combination with      -->
<!-- other configuration files.  e.g.                                -->
<!--    java -jar start.jar etc/jetty.xml etc/jetty-ssl.xml          -->
<!-- =============================================================== -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">

  <!-- if NIO is not available, use org.eclipse.jetty.server.ssl.SslSocketConnector -->

  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
	<Set name="Port">8443</Set>
	<Set name="maxIdleTime">30000</Set>
        <Set name="Acceptors">2</Set>
        <Set name="AcceptQueueSize">100</Set>
	<Set name="Keystore"><SystemProperty name="jetty.home" default="." />/etc/keystore</Set>
	<Set name="Password">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
	<Set name="KeyPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
        <Set name="truststore"><SystemProperty name="jetty.home" default="." />/etc/keystore</Set>
        <Set name="trustPassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
      </New>
    </Arg>
  </Call>
</Configure>

Loren

Re: [Exist-development] DVCS - Distributed Version Control?

[Andrzej J. T. <an...@ch...>]

Adam:

> Before we moved to Subversion, CVS used to be a major pain in the ass
> for me, because I work from far too many locations (offices, airports,
> trains, cafes etc) where the only access out is HTTP, Subversion works
> perfectly over HTTP. If SourceForge were to enable Mercurial writes
> over HTTP then I might be sold on the idea ;-)

Bitbucket (mercurial hosting) supports both HTTP and SSH access for both pull and push, and has free services for open
source projects.

Github (git hosting) has free services for open source, but I'm not sure if you have to use SSH or not yet.

So there are a number of alternatives to Sourceforge for DVCS.

Just FYI.

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

[Exist-development] Wikibook and xqDoc

[Chris W. <kit...@go...>]

I recently put a bit of time into restructuring the XQuery wikibook.
Amongst other changes I have built up the section on eXist to include
several of the modules and links to articles which illustrate the use of
functions in these modules.  There is much left to do but I hope its moving
in the right direction (until perhaps a move to some other platform).  Any
example code is always welcome and I will install it on my UWE server.

In the process of implementing examples of xqDoc, I couldnt find a basic
stylesheet to transform the output from the scan function to HTML - anyone
know of one?

Chris

[Exist-development] The new triggers staff

[Evgeny G. <gaz...@gm...>]

Hello!

I want present the new triggers staff here.

Current trunk has triggers for six events for document and collections
Some of them (all collection's triggers) is not implemented.
I had implemented all collection's triggers and adding new evens.
So just we can have triggers fired for create, move, rename, remove
collections and documents and update documents.

Every trigger's interfaces (CollectionTrigger and DocumentTrigger)
were divided for two interfaces CollectionTriggerUnary, CollectionTriggerBinary
and DocumentTriggerUnary, DocumentTriggerBinary.

The "unary" triggers are for create, update, events and equal old
trigger's model.
The "binary" triggers are for rename or move events.
They have additional parameter for prepare() and finish() methods.
This parameter is destination path of collection or document.

So collection triggers were not implemented and it's methods not called,
I'd remove one.
DocumentTriggerUnary, DocumentTriggerBinary has DocumentTrigger
as parent interface.
Also DocumentTrigger still have old methods prepare() and finish(),
which I'd deprecated.

All events have a new names. but old can be used as well too.
If user use own triggers compiled with old triggers,
he can use them and wit new triggers interfaces too.
User will be warned in a logs, when "binary" event will be fired for
such trigger.

In XQuery trigger I'd remove one external variable that was passed docImpl.
Think we had loose performance here.
Some external variables are have a new names.
But old can be used too.

See new trigger's documentation in webapp/trigger.xml.

All triggers tests, fluent and versioning extensions were refactored
for new event's names.

All tests are worked as well.

ToDo: test cases for collection's triggers and move/rename events.

Roadmap: triggers for meta information (owner, group, permissions) changes.
For this need a lot investigation into another kernel's modules.
Like the "copy/paste" pattern was used there very often.
Think need to refactor.

---------
Evgeny

Re: [Exist-development] How big can you supersize this puppy?

[Dmitriy S. <sha...@gm...>]

On Mon, 2010-03-29 at 10:59 -0400, Andrzej Jan Taramina wrote:
> Looking to get some guidance on how big you can scale an eXist database.
> 
> Right now, our instances are about 15-25K documents where each document is in the 25K-2M range, probably averaging
> around 150-200K.  This results in a dom.dbx = 3.5G, structure.dbx = 1.8G, collections.dbx = 4.2M and values.dbx = 155M,
> which is not all that large compared to some relational databases.
> 
> What if we scale up 10x to nearly quarter of a million documents?  The file sizes still shouldn't be all that big for
> modern hardware, but will the performance scale linearly or close to it, assuming a powerful enough server (say a
> dual-cpu, 6-Core machine (12 cores, 24 native threads) with gobs of memory)?
> 
> OK.....if that works how about two orders of magnitude (100x current size)?  That would give us 2.5M documents, 250GB
> dom.dbx and a structure.dbx in the 180GB range.  Bit too big or practical to cache the whole structure.dbx in memory,
> regardless of the size of the memory in the server.
> 
> At what point do I start looking at alternative storage mechanisms, (RDBMS, Hadoop, memcached, etc.) or co-operating
> distributed eXist instances?
> 
> Thanks for any insights from those that have pushed big databases in eXist...

eXist can store as much as you have hdd space. 
The main question is what eXist can do with it & what it can't.

It be very good on "single" small result selection from that big amount
of data, but the problem starts as soon as you increase evaluations &
"special" operations like order by.

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] How big can you supersize this puppy?

[Wolfgang M. <wol...@ex...>]

> What if we scale up 10x to nearly quarter of a million documents?  The file sizes still shouldn't be all that big for
> modern hardware, but will the performance scale linearly or close to it, assuming a powerful enough server (say a
> dual-cpu, 6-Core machine (12 cores, 24 native threads) with gobs of memory)?

I have collections with nearly half a million documents. This doesn't say much.

The main question is what queries you intend to run and how well they
can be optimized (from previous conversations, I believe you should
have lots of optimization potential in your queries). A brute force
query which has to scan huge node sets will maybe scale worse than
linearly. Throwing more hardware at eXist may help to speed up a slow
query by a few percent, but you will nevertheless hit a limit at some
point. Improvements by an order of magnitude are only possible by
having a very close look at the queries and how they are evaluated.
Profiling is key.

I'm continuously trying to push the limit in the query engine. I still
see so many things to be improved and optimized. Right now, I'm
working on a facility to speed up complex "order by" expressions by
using pre-computed indexes. I also just found out that some queries
generate a huge amount of intermediate XQuery value instances which
are completely unnecessary. There's a lot more of this to look at.

Those are the areas we should invest into and I would really welcome
more interest in this.

> Bit too big or practical to cache the whole structure.dbx in memory,
> regardless of the size of the memory in the server.

eXist never caches the whole structure.dbx in memory. Doing so has a
very *negative* impact on performance as repeated tests showed. You
mainly want the inner btree pages to be in memory, while the leaf
pages reside on disk. So increasing the cache memory does only make
sense up to a certain limit.

> At what point do I start looking at alternative storage mechanisms, (RDBMS, Hadoop, memcached, etc.) or co-operating
> distributed eXist instances?

Or: use something like Hadoop or Cassandra as backend for eXist? My
recent index redesigns were preparing the ground for things like that
and I was already testing a clustered big btree implementation with
the structural index (it worked, though I was running out of time and
would need to invest some more time to make it fast).

Wolfgang

[Exist-development] How big can you supersize this puppy?

[Andrzej J. T. <an...@ch...>]

Looking to get some guidance on how big you can scale an eXist database.

Right now, our instances are about 15-25K documents where each document is in the 25K-2M range, probably averaging
around 150-200K.  This results in a dom.dbx = 3.5G, structure.dbx = 1.8G, collections.dbx = 4.2M and values.dbx = 155M,
which is not all that large compared to some relational databases.

What if we scale up 10x to nearly quarter of a million documents?  The file sizes still shouldn't be all that big for
modern hardware, but will the performance scale linearly or close to it, assuming a powerful enough server (say a
dual-cpu, 6-Core machine (12 cores, 24 native threads) with gobs of memory)?

OK.....if that works how about two orders of magnitude (100x current size)?  That would give us 2.5M documents, 250GB
dom.dbx and a structure.dbx in the 180GB range.  Bit too big or practical to cache the whole structure.dbx in memory,
regardless of the size of the memory in the server.

At what point do I start looking at alternative storage mechanisms, (RDBMS, Hadoop, memcached, etc.) or co-operating
distributed eXist instances?

Thanks for any insights from those that have pushed big databases in eXist...

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com