[Vufind-admins] [VuFind Issues] (VUFIND-834) SolrCloud Configuration

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and (-webkit-min-device-pixel-ratio: 2) {
    #email-body { min-width: 30em !important; }
    #email-page { padding: 8px !important; }
    #email-banner { padding: 8px 8px 0 8px !important; }
    #email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
    #email-fields { padding: 0 8px 8px 8px !important; }
    #email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#f0f0f0;color:#000000;width:100%;">
    <tr valign="top">
        <td id="email-page" style="padding:16px !important;">
            <table align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;">
                <tr valign="top">
                    <td bgcolor="#FFFFFF" style="background-color:#FFFFFF;color:#395fdb;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img src="http://vufind.org/jira/s/en_USitrgpf-418945332/850/10/_/jira-logo-scaled.png" alt="" style="vertical-align:top;" /></td>
                </tr><tr valign="top">
    <td id="email-banner" style="padding:32px 32px 0 32px;">

            <table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="width:100%;">
    <tr valign="top">
        <td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
                                        <img id="email-avatar" src="http://vufind.org/jira/secure/useravatar?ownerId=bemosior&avatarId=11550" alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px 16px 0;" />
                        <div id="email-action" style="padding: 0 0 8px 0;font-size:12px;line-height:18px;">
                                    <a class="user-hover" rel="bemosior" id="email_bemosior" href="http://vufind.org/jira/secure/ViewProfile.jspa?name=bemosior" style="color:#326ca6;">Benjamin Mosior</a>
     created <img src="http://vufind.org/jira/images/icons/issuetypes/improvement.png" height="16" width="16" border="0" align="absmiddle" alt="Improvement"> <a style='color:#326ca6;text-decoration:none;' href='http://vufind.org/jira/browse/VUFIND-834'>VUFIND-834</a>
            </div>
                        <div id="email-summary" style="font-size:16px;line-height:20px;padding:2px 0 16px 0;">
                <a style='color:#326ca6;text-decoration:none;' href='http://vufind.org/jira/browse/VUFIND-834'><strong>SolrCloud Configuration</strong></a>
            </div>
                    </td>
    </tr>
</table>
    </td>
</tr>
<tr valign="top">
    <td id="email-fields" style="padding:0 32px 32px 32px;">
        <table border="0" cellpadding="0" cellspacing="0" style="padding:0;text-align:left;width:100%;" width="100%">
            <tr valign="top">
                <td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
                <td>
                    <table border="0" cellpadding="0" cellspacing="0" width="100%">
                        <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Issue Type:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
                <img src="http://vufind.org/jira/images/icons/issuetypes/improvement.png" height="16" width="16" border="0" align="absmiddle" alt="Improvement">        Improvement
    </td>
</tr>                                                <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Assignee:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
                    Unassigned            </td>
</tr>                        <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Attachments:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
                    solr.xml,                     solrconfig.xml            </td>
</tr>
                                                <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Created:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
        02/Jul/13 5:50 PM
    </td>
</tr>                        <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Description:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
        This ticket is mostly to start discussion of SolrCloud configuration inclusion in the VuFind project. I think it would certainly be nice if VuFind could be HA-ready out of the box.
<br/>

I&#39;m attaching configurations and general information for creating a simple 2-replica setup in a RHEL6 environment. I&#39;m sure that this process would work fine for other typical Linux distros, though I have no idea about Windows.

<br/>

<br/>
I used this post as a general guide: <a href="http://systemsarchitect.net/painless-guide-to-solr-cloud-configuration/">http://systemsarchitect.net/painless-guide-to-solr-cloud-configuration/</a>
<br/>

<br/>

============ ZooKeeper Part One ============
 
First, I set up a zookeeper server per the guide. For the sake of simplicity, I have it running on the first vufind server.

In order to blend the installation into VuFind, I downloaded zookeeper (zookeeper-3.4.5) and extracted it to vufind/solr/zookeeper. Following the instructions, I set dataDir=/var/lib/zookeeper in zoo.cfg.

Other necessary components here include the &quot;cloud scripts&quot;, which are located in the example/cloud-scripts directory of a fresh solr-4.2.1 download. I placed the cloud-scripts directory in vufind/solr/cloud-scripts. In order to get the vufind/solr/cloud-scripts/zkcli.sh working, I had to copy the following libraries from the solr-4.2.1 example/solr-webapp/webapp/WEB-INF/lib/ directory into vufind/solr/lib:

<br/>
commons-cli-1.2.jar
<br/>
commons-io-2.1.jar
<br/>
lucene-analyzers-common-4.2.1.jar
<br/>
lucene-core-4.2.1.jar
<br/>
slf4j-api-1.6.4.jar
<br/>
slf4j-jdk14-1.6.4.jar
<br/>
solr-solrj-4.2.1.jar
<br/>
zookeeper-3.4.5.jar
<br/>

I believe I had to run the solr-4.2.1 example server as they mention in the guide in order to extract the libraries from solr.war.

Run &quot;vufind/solr/zookeeper/bin/zkServer.sh start&quot; on the first vufind server and make sure zookeeper starts without errors.

<br/>

============ Biblio Configuration ============
 
I have attached my resulting vufind/solr/biblio/conf/solrconfig.xml. I imagine this sort of config would have to be used for all cores, though the &lt;dataDir /&gt; would have to be updated to reference each specific core in order to preserve VuFind&#39;s default solr directory configuration.

<br/>

============ Solr Core Configuration ============
 
I have attached my resulting vufind/solr/solr.xml. 
 
Note that &quot;zkHost&quot; will be the same on each Solr replica if you use the IP address (not localhost or 127.0.0.1) of the zookeeper server.
 
Also note, I used the hostname of my VM in the &quot;host&quot; field in order to make it more readable in the &quot;cloud&quot; control panel located at <a href="http://example.solr.hostname:8080/solr/#/~cloud">http://example.solr.hostname:8080/solr/#/~cloud</a>

<br/>

<br/>
============ ZooKeeper Part Two ============
<br/>
Change into the vufind/solr/ directory on the first VuFind server and run these commands:
<br/>
&quot;./cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2181 -d biblio/conf/ -n biblio&quot;
<br/>
&quot;./cloud-scripts/zkcli.sh -cmd linkconfig -zkhost 127.0.0.1:2181 -collection biblio -confname biblio -solrhome biblio&quot;
<br/>

You would likely need to run the above for cores other than biblio.

<br/>
Then:
<br/>
&quot;./cloud-scripts/zkcli.sh -cmd bootstrap -zkhost 127.0.0.1:2181 -solrhome .&quot;
<br/>

At this point, it&#39;s probably a good idea to start up VuFind (/vufind/vufind.sh start) on the first core and see if you get any errors. Definitely check the &quot;cloud&quot; control panel mentioned above, in addition the logging section.

<br/>

============ Second VuFind Server ============
 
Configuration on the second VuFind server should be as easy as setting the vufind/solr/solr.xml (replacing it with the attached solr.xml and updating the host field) and running &quot;vufind/vufind.sh start&quot;. No messy modifications are necessary. After starting the second VuFind, Solr should automatically copy the index over from the first VuFind server.

    </td>
</tr>
                                                                                                <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Project:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
        <a style="color:#326ca6;" href="http://vufind.org/jira/browse/VUFIND">VuFind</a>
    </td>
</tr>                                                <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Priority:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
                        <img src="http://vufind.org/jira/images/icons/priorities/minor.png" height="16" width="16" border="0" align="absmiddle" alt="Minor">                Minor
    </td>
</tr>
                        <tr valign="top">
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
        <strong style="font-weight:normal;color:#505050;">Reporter:</strong>
    </td>
    <td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
                                        <a class="user-hover" rel="bemosior" id="email_bemosior" href="http://vufind.org/jira/secure/ViewProfile.jspa?name=bemosior" style="color:#326ca6;">Benjamin Mosior</a>
                </td>
</tr>                                                                    </table>
                </td>
            </tr>
        </table>
    </td>
</tr>

            </table>
        </td><!-- End #email-page -->
    </tr>
    <tr valign="top">
        <td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: 0 16px 16px 16px;text-align:center;">
            This message is automatically generated by JIRA.<br />
            If you think it was sent incorrectly, please contact your JIRA administrators<br />
            For more information on JIRA, see: <a style='color:#326ca6;' href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
        </td>
    </tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->