Re: [Exist-development] eXist replication & clustering

[Dannes W. <di...@ex...>]

Please note that all is in trunk right now, with updated docs. The branche must be removed ..

--
Dannes Wessels

On 2 okt. 2012, at 08:04, Dan McCreary <dan...@gm...> wrote:

> Hello Alexei,
> 
> Thanks for noting the differences.  I have updated the Wikibook
> article to include the new collection.xconf and the new conf.xml
> files.
> 
> - Dan
> 
> On Tue, Oct 2, 2012 at 6:47 AM, Alexei Tarnakin
> <ale...@gm...> wrote:
>> I made a small research on eXist clustering capabilities. According to
>> Wikibooks article I checked out dizzzz's branch and tuned up ActiveMQ server
>> and a couple of eXist servers.
>> 
>> First of all, I've discovered that article is totally out of date.
>> collection.xconf should look like
>> 
>>> <collection xmlns="http://exist-db.org/collection-config/1.0">
>>> <triggers>
>>> <trigger class="org.exist.replication.jms.publish.ReplicationTrigger">
>>> <parameter name="java.naming.factory.initial"
>>> value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
>>> <parameter name="java.naming.provider.url" value="tcp://localhost:61616"/>
>>> <parameter name="connectionfactory" value="ConnectionFactory"/>
>>> <parameter name="destination" value="dynamicTopics/eXistdb"/>
>>> <parameter name="client-id" value="id1"/>
>>> </trigger>
>>> </triggers>
>>> </collection>
>> 
>> 
>> If  you dig svn log, you will notice that
>> 
>> org.exist.clustering.triggers.send.ClusterTrigger
>> 
>> was renamed to
>> 
>> org.exist.replication.jms,publish.ReplicationTrigger
>> 
>> And also notice that client-id looks like mandatory, but not optional field
>> and separated with a dash, not a dot (i.e. client-id).
>> 
>> Then lets go check slave eXist server. Article is wrong again. Proper
>> conf.xml should look like
>> 
>>> <job type="startup" name="clustering"
>>> class="org.exist.replication.jms.subscribe.MessageReceiverJob">
>>>        <parameter name="java.naming.factory.initial"
>>> value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
>>>        <parameter name="java.naming.provider.url"
>>> value="tcp://localhost:61616"/>
>>>        <parameter name="connectionfactory" value="ConnectionFactory"/>
>>>        <parameter name="destination" value="dynamicTopics/eXistdb"/>
>>>        <parameter name="client-id" value="id2"/>
>>>        <parameter name="subscriber-name" value="sub_name"/>
>>> </job>
>> 
>> 
>> OK, now looks like we configured everything properly, bot does cluster work
>> now?... Nope!
>> Let's try discover why it does not work by means of xdebug. So simply add a
>> couple of breakpoints to ReplicationTrigger and JMSMessageListener to check
>> what we actually send to ActiveMQ and get on other side (yep, I actually
>> skipped several steps like reconfiguring log4j.xml, restarting server &
>> digging logs, cause it's boring).
>> So the problem is that master eXist sends eXistMessage with document type
>> literally "XML" or "BINARY", while slave expects "DOCUMENT" or "COLLECTION".
>> The problem as always was in indian style of code. Lets analyze it a little
>> bit
>> 
>>>    @Override
>>>    public void afterCreateDocument(DBBroker broker, Txn transaction,
>>>            DocumentImpl document) throws TriggerException {
>>> 
>>>        ......
>>>        // Create Message
>>>        eXistMessage msg = new eXistMessage();
>>>        msg.setResourceType(eXistMessage.ResourceType.DOCUMENT);
>>> 
>>> msg.setResourceOperation(eXistMessage.ResourceOperation.CREATE_UPDATE);
>>>        msg.setResourcePath(document.getURI().toString());
>>> 
>>>        // Retrieve Metadata
>>>        Map<String, Object> md = msg.getMetadata();
>>>        MessageHelper.retrieveDocMetadata(md, document.getMetadata());
>>>        MessageHelper.retrieveFromDocument(md, document);
>>>        MessageHelper.retrievePermission(md, document.getPermissions());
>>>        .......
>>>        sendMessage(msg);
>>>    }
>> 
>> 
>> so resourcePath in eXistMessage was setted twice
>>> 
>>> msg.setResourceType(eXistMessage.ResourceType.DOCUMENT);
>> 
>> setted to "DOCUMENT" and then one more time
>>> 
>>> MessageHelper.retrieveFromDocument(md, document);
>> 
>> setted to "XML" due to a bit of hardcode in MessageHelper. To fix it we need
>> to patch it
>> 
>>> Index:
>>> extensions/replication/src/org/exist/replication/shared/MessageHelper.java
>>> ===================================================================
>>> ---
>>> extensions/replication/src/org/exist/replication/shared/MessageHelper.java
>>> (revision 17019)
>>> +++
>>> extensions/replication/src/org/exist/replication/shared/MessageHelper.java
>>> (working copy)
>>> @@ -159,7 +159,9 @@
>>>     public static void retrieveFromDocument(Map<String, Object> props,
>>> DocumentImpl document){
>>> 
>>>             props.put(EXIST_RESOURCE_TYPE,
>>> -                    (document.getResourceType() == DocumentImpl.XML_FILE
>>> ? "XML" : "BINARY"));
>>> +                      (document.getResourceType() ==
>>> DocumentImpl.XML_FILE ?  eXistMessage.ResourceType.DOCUMENT : "BINARY"));
>>>             props.put(EXIST_RESOURCE_DOCUMENTID, document.getDocId());
>>>             props.put(EXIST_RESOURCE_CONTENTLENGTH,
>>> document.getContentLength());
>> 
>> 
>> now we got document updates enabled, but no collection operations yet. Let's
>> try to fix it....
>>                ...well, I'm not going to test your patience, so just
>> attached svn diff below and I'll keep on working. And I do hope that 2.1
>> release will be clustering/replication enabled, cause eXist really needs it
>> due to lack of reliability under pressure.
>> 
>> 
>> P.S. repo moved so proper link for checkout is
>> http://svn.code.sf.net/p/exist/code/branches/dizzzz/clustering/
>> 
>> P.P.S. Sorry for my English, it's not my native
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Don't let slow site performance ruin your business. Deploy New Relic APM
>> Deploy New Relic app performance management and know exactly
>> what is happening inside your Ruby, Python, PHP, Java, and .NET app
>> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
>> http://p.sf.net/sfu/newrelic-dev2dev
>> _______________________________________________
>> Exist-development mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-development
>> 
> 
> 
> 
> -- 
> Dan McCreary
> NoSQL Evangelist
> office: (952) 931-9198
> cell: (612) 986-1552
> 
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development