oaikepler Files

#LastUpdated: 07/07/2004coc
Kepler Project Installation Package Version 1.2
DLib Research Group, Old Dominion Univ

CONTENTS
========

* Directory Configuration

* Introduction

* System Requirements

* General Setup

* Classpath Setup

* Database Setup

* Harvester Setup

* Search Engine Setup

* Admin Page

* Archivelet Setup

* Putting your own logo into the group web site

* Compile

* Get Newest Code From ODU CVS Server

* Uninstallation of the old kepler

* Validation module readme

* Support

* License



DIRECTORY CONFIGURATION
=======================

This distribution includes the following contents:


  readme.txt			readme file.
  prj/					Configuration files about database schema and 
						make file
  src/					Source code of Kepler service provider, data 
						provider and archivelet

		dataprovider20/		Source code of dataprovider
			token/				Implementation of the resumption tokens
			error/				Oai exceptions
		util/xml/			xmltool class 
		kepler/				Source code of the kepler archivelet
			conf/				Archivelet configuration files and metatdata description file
			data/				Data files (documents) along with the .dc files
			image/				Image files used for the kepler interface
			oai/				kepler record factory and supprting classes
			share/				classes for archivelet configuration
			ui/					Archivelet user interface classes
			webserver/			web server for kepler
			help/				Files related to support help
			metadata/			Support various metadata format
			readmefiles/			
			shellfiles/			
			xmlndom				xml functionality support


		server/				Source code for search and harvester services
			admin/				servlets for database setup and loading configuration values
			arc20/				Dataprovider and oai for the group/community
			conf/				Configuration files for server
			db/					Classes for accessing the database
			html_new/			Web pages for serach interface
			interactive/		
			log/				
			monitor/
			ns/					Classes used by the harvester/search to get	the dp list from the database
			oairequester/		Processing of oai requests
			parser/				Parsing meatadata
			register/			Registering dataproviders
			scheduler/			Harvester code
			search/				search code
			lib/				JDBC drivers
			serverside/		Source code for Kepler Serverside (online) archivelet.

  bin					Binary executables and html scripts
        Archivelet/			Archivelet code.

		oai2.harvest/		Harvester code.

		war/				Service provider code for Tomcat.
 lib                    Related Java Library
  package				jar/tar/zip files for the projects



INTRODUCTION
============

Kepler is distributed as a .tar.gz and .zip archive containing the sources, class files and Windows downloadable.

To untar the package, for Linux/Unix systems, at shell prompt, type:

	gunzip kepler.tar.gz

and this will convert kepler.tar.gz into kepler.tar file. Then move the tar file into the desired directory and type:

	tar -xvf kepler.tar

The package will be installed on the desired directory on your machine.

If installed on Windows system, use Winzip or any other unzip software to unzip the kepler.tar.gz or kepler.zip file into the desired directory on your machine. Once you have un-archived the distribution, We suggest you follow the steps below to setup group service and individual archivelet.


SYSTEM REQUIREMENTS
===================

To install group service, you need the following: 

* The J2SDK or J2RE 1.4  
* Oracle 8i (or above) JDBC-thin driver or MySQL JDBC driver  
* A database account in Oracle or MySQL database    
* Tomcat 4.0 (or above).
* Apache Ant. (If you want to compile the source code)


You can get the JDK/JRE/JAXP from http://www.javasoft.com.

You can get the mysql database from http://www.mysql.org. 
Oracle JDBC driver is available at Oracle website or you may find it in your oracle distribution.
The additional class files should be put in your classpath.

We have tested the system in Tomcat 4.0, it should also work fine in Tomcat3.2, but the configuration steps may be different.  

GENERAL SETUP
=============
 1. Add to the file {$TOMCAT_HOME}/conf/tomcat-users.xml the following

   <user name="admin"   password="admin" roles="administrator" />

   You will need to enter these login name and password when you try to access the administration/configuration
   pages from the web browser.

   You may change the "name" and the "password" to whatever you want but don't touch the roles attribute.

CLASSPATH SETUP
===============

We strongly suggest you download and install required java packages by yourself, but we have related packages packed for your convienence. If you decide to install these packages by yourself, it's not necessary to read this section.

These files are available at {KEPLER_HOME}/lib,including:
ojdbc14.jar: Oracle JDBC-thin Driver
jaxp.jar,xalan.jar,crimson.jar : JAXP XML package
servlet.jar: Servlet/JAXP package from Tomcat
mm.mysql-2.0.8-bin.jar: MySQL JDBC Driver.

*Setup classpath

Unix:
----
We have a csh script {KEPLER_HOME}/prj/kepler.cshrc, you need to change this file to
 reflect the KEPLER_HOME directory, then under csh run
>source kepler.cshrc

Windows:
-------
For Windows 2000 or XP 

Right click on the "My Computer" icon.  Select Properties.  Select Advanced.  Click on Environment variables,
Add the above jar files into CLASSPATH enironment variables.

{KEPLER_HOME}/lib/*


DATABASE SETUP
==============

*Oracle  

1.In oracle 8i, create a user with "ctxapp" privilege.
2.Create a folder of your choice(say.. kepler) under {$TOMCAT_HOME}/webapps/. (Note: We will assume that you specified the folder name as 'kepler' to explain further setup process).
3.Copy $KEPLER_HOME/bin/war/* to your {$TOMCAT_HOME}/webapps/kepler/ directory. (Note: $KEPLER_HOME is directory where kepler distribution was installed).
4.Modify the file under {$TOMCAT_HOME}/webapps/kepler/WEB-INF/classes/conf/ConnectionPool.cfg, to change the database connnection and username/password. 
5.Restart the tomcat server.
6.Open the Url-->'http://yourhost:yourport/kepler/servlet/admin.SetUp' in your Web Browser. Enter Valid information regarding your group service & click SetUp button. This will make sure all the database settings are done for you automatically.
If the SetUp completed succesfully skip the NOTE & steps below & proceed to Harvester Configuration.

NOTE: Reinstalling the database  (CAUTION: This will ERASE ALL THE DATA in the database)

If admin page above shows that you have already SetUp the service(because of a previous setup) &  you need to do a new clean setup again, login to the database server and execute the following SQL command
		
	drop table dpidentify;
			 
then open the setup Url-->'http://yourhost:yourport/kepler/servlet/admin.SetUp in your Web Browser. This would enable you to install a fresh group service. 

*MySQL

For MySQL database setup, follow the same setup as mentioned for Oracle but in step 4 overwrite ConnectionPool.cfg with information specified in ConnectionPool.mysql file and specify the correct userid,password and connection information to connect to MySQL.

**Some Notes about community URL concept
--------------------------------------
Community URL is the URL for the kepler community the group belongs to.Registratin to the comunity allows it to keep track of what are the
available Kepler groups out there. This information is typically made avilable at to the public at the community web site which helps redirect
intersted users to you kepler group. We have a Kepler community site at ODU to which you might register your Kepler group. The URL for this community is
http://kepler.cs.odu.edu/kepler

It is optional. You don't have to register to a community if you do not want to. If you would like not to register your group, just uncheck the
check box on the group setup page (immediatly below the community URL text box) while setting up you group 


HARVESTER SETUP
===============

2.0 version of Harvester

The current version of Kepler supports OAI-PMH2.x.  
The OAI-PMH2.x compliant harvester is available at bin/oai2.harvester. 


*Change Database connection 

Modify the ConnectionPool file under ${KEPLER_HOME}/bin/oaiharvester/conf similar to the configuration specified in Database Setup section, change the database connection string and username/password.
There are two templates available, The ConnectionPool.cfg is fit for the Oracle Configuration, and  ConnectionPool.mysql is for MySQL. You need overwrite ConnectionPool.cfg with your version. 


*Admin Interface

We provide an admin page , wherein you can view/manage Scheduler, dplist configuration  at http://{your tomcat server}:{your port}/kepler/admin_dp.jsp. You need to enter valid Url to harvest from the corresponding sites.


*Command line

(1) Harvest

Change to the bin/oai2.harvester directory and execute the following commands at the prompt.

	> java scheduler/Harvest [-t]  

	-t  Enable internal timer

When the internal timer is started, the harvester will periodically harvest  new published document.
When the internal timer is disable, the harvester will finish after one round  of harvesting. However, the system user has the flexibilty of setup it as a  cron job for example.


(2) Parse

The parser is under in 2.0 harvester distribution.

Change to the bin/oai2.harvester directory and execute the following commands at the prompt.

  >java parser/Parse [-t]<-a || -n>  <MetadataFormat>

  -a		    parse all harvested records

  -n		    parse only new harvested records(Recommended)

  -t		    Enable internal timer

  metadataprefix     the metadata format (e.g. oai_dc)  
  

(3) Index

Change to the bin/oai2.harvester directory and execute the following commands at the prompt.

  > java scheduler.Harvest -i [-t]

  -t	Enable internal timer

  

SEARCH ENGINE SETUP 
===================

The addtional class files should be put in either your classpath or 

{$TOMCAT_HOME}/webapps/kepelr/WEB-INF/lib.

You should be able to see search engine at the URL

	http://{your tomcat server}:{port}/{application name}

And OAI support works in 

	http://{your tomcat server}:{port}/{application name}/servlet/OAI


ADMIN PAGE
==========

The admin page and its help is available at http://yourhost:yourport/kepler/admin_dp.jsp.
The admin page is responsible for scheduler configuration, harvester configuration. 


ARCHIVELET SETUP
================

If a user would like to setup archivelet on his/her desktop running Windows operating system, there are Windows downloadables available under $KEPLER_HOME/bin/war/protected/archivelet, where $KEPLER_HOME is the root installation directory at group level. There is also a link in homepage of the service provider to the archivelet download.

1. Click on the Setup.exe and InstallShield will launch the Windows Installer automatically.

2. Follow the default setup configuration and Installer will install the package on your machine including shortcuts on desktop and program group item.


If a user choose to run archivelet from UNIX or Linux machine, he/she needs to copy appropriate .gz or from $KEPLER_HOME/bin/war/protected/archivelet or download from group server website. 

After unzipping and untaring, the distribution, go to archivelet's bin directory and enter the follwing command..

   java kepler/App


**Some Notes about DC and OLAC metadata editors
---------------------------------------------

The values for ID field in DC and OLAC metadata editors should not contain special characters especially *, ', ", ^. 
The editor does not validate for these characters explicitly, but it accepts the values. But the record with such IDs are not displayed on the main interface.


Putting your own logo into the group website
============================================
To put your own logo into the group website, overwrite the file 'logo.gif' with your own logo gif image. Make sure to keep the file name as is just overwrite the content.


COMPILE
=======

You need have Apache ant installed before you compile source code, the classpath should also be correctly configured.

ALso, make sure that {KEPLER_HOME}\prj is your current directory for all of the  build tasks below.

 1)To build the kepler archivelet use the following command

			ant -buildfile archivelet.xml dist

 2)To build the Search use the following command

			ant -buildfile search.xml dist

 3)To build the Harvester use the following command

			ant -buildfile oai2.harvester.xml dist

Put the lib/ojdbc14.jar into classpath if you are using oracle database or lib/mm.mysql-2.0.8-bin.jar ,if you are using mysql database.

GET NEWEST CODE FROM ODU CVS SERVER
===================================
The kepler project is ongoing and we anticipate future enhancements and bug fixes. We suggest you update code only after consulting with kepler group at ODU, as the code in CVS server is changed on a regular basis and may not be stable.

Link to ODU CVS server.

 > setenv CVSROOT :pserver:anoncvs@globus.cs.odu.edu:/home/xliu/cvs
 > cvs login (password is also anoncvs)
 > cvs checkout kepler3.0


Uninstallation of the old kepler
================================

For those who has already downloaded kepler archivelet or even group service package in tar file before, please follow the following procedure to uninstall the old kepler project on your site.

1. To uninstall the old kepler archivelet on your Windows operating system: 

(NOTE:Uninstallation of the software would automatically delete the published documents, inorder to retain the documents information, save a copy of the Folder 'data' under ({Kepler-Home}/data) go to Settings/Control Panel/Add Remove Programs. At the Current installed programs list, select kepler and click Change/Remove button. This will trigger the Windows installer. 

At the prompt, select Remove the whole program and proceed. Windows installer should  uninstall the archivelet code and related shortcuts. 


2. For group services, shut down Tomcat services first. Untar the tar file for group  services and overwrite the old service provider directory. Copy files under bin/war to  overwrite the corresponding directory under Tomcat (e.g. {Tomcat_home}/webapps/kepler).


Validation module README
================================

PART 1. How to run 

PART 2. Configuration tool in detail

PART 3. Configuration parser tool in detail

---------------------------------------------------------------------------------------------------

PART 1. How to run

(1) to run the "configuration tool":

a) Using the Validation Configuration Interface..Change to
    groupserver_home/WEB-INF/classes/ 
b) Start Exceed or XWin Server on your machine ( or any machine where you are givign demo).
    Set the DISPLAY variable using "export DISPLAY=IP_of_machine:0.0"
    (Ex. export DISPLAY=128.82.7.93:0.0)
c)  Now run the following command
d)  java validationclient/TreeViewer

(2) to run the "configuration parser tool":

go to: group server/bin/war/WEB-INF/classes
run:  java configurationparser/DOMTreeReporter  
---------------------------------------------------------------------------------------------------
PART 2. Configuration tool (TreeViewer) in detail

The purpose of TreeViewer is to provide the group administrator with a GUI to 
change the Mandatory/RegularExpression/DropDownList configuration of a specific element.

TreeViewer works in a client/server way. When the group administrator selects a metadata "driver name", say, DC or OLAC, the latest configuration file is downloaded from the specified $KEPLER_HOME/src/server/validationserver/ValidationDownload.java).
When the group administrator finishes the configuration by clicking "Upload" button, 
the latest configuration file is uploaded to the specified group server
(group server/src/server/validationserver/ValidationUpload.java).

The GUI of TreeViewer consists of two parts: the left part displays elements in 
a JTree, which is controlled by a "GUI configuration file" as described in PART 3; 
the right part displays the configuration of the element being selected in the JTree.

The current implementation of this GUI (as well as the design decision on using 
a supporting GUI configuration file) is the result of many tradeoffs. Initially, the user of 
this GUI is assumed to be someone who knows "XML schema language". So a JTree 
that displays all the elements/types defined in the schema files was implemented.
Such a GUI reads directly from schema files and doesn't require any supporting configuration 
file. However, this GUI caused many confusions when shown to the advisors. The key reason is: 
some keywords in XML Schema language, such as "namespace" "element" "simpleType" 
"complexType" ... are considered as NOT user-friendly, and must NOT appear in this GUI.

To satisfy this requirement, I created such a GUI configuration file to achieve 
the transformations between hidden info and explicit info. The user of the GUI will see 
a very neat (or even over-simplified) GUI, while the writer of the GUI configuration file must 
specify various XML schema-related information carefully. 

A GUI configuration file is an XML instance file. Each element in the file becomes
a node of the JTree, and the parent-child relationship between elements is reflected in the JTree.

Following is an example of an element defined in a GUI configuration file:

   <creator
       namespace="http://purl.org/dc/elements/1.1/" 
       type="element" name="creator"       
       quicklist ="[A-Z][a-zA-Z ]{1,100}!!!.+!!![^\r\t\n\f]"
    >
    </creator>

   The above means: 

   (1) what is shown in the JTree is a tree node called "creator" (tag name of the XML element);
 
   (2) The "type", "name", and "namespace" attributes imply that: 
       this tree node is in fact an "element" with local name "creator" defined in 
       the namespace "http://purl.org/dc/elements/1.1/" in XML schema language; 

   (3) The "quicklist" attribute lists all the candidate regular expressions, separated by "!!!";


Some elements have "externalTypeList" and "internalTypeList" attributes, which are related to
the "Configure Extension" part of the GUI. The meaning of these two attributes can be illustrated 
using the following example:

     <language 
       namespace="http://purl.org/dc/elements/1.1/" 
       type="element" name="language"       
       internalTypeList ="http://www.language-archives.org/OLAC/1.0/###attribute###code.language"
       externalTypeList="language"
     >
     </language>


   The above means:

   "language" (the value of attribute "externalTypeList") is what is displayed in 
   the drop-down list next to the "xsi:type" lable.

   When a user selects the "language" item from the drop-down list, this program (TreeViewer) searches 
   in the configuration file for the corresponding "internalType", (in this case, the internalType is 
   the "code" "attribute" of complexType "language" defined in namespace 
   "http://www.language-archives.org/OLAC/1.0/")  and displays all the possible values in the scroll list.

---------------------------------------------------------------------------------------------------

PART 3: Configuration parser tool (DOMTreeReporter) in detail

The purpose of DOMTreeReporter is to generate "configuration files" 
from "XML Schema files". This program should be run only
when a new metadata format is introduced into the Kepler system,
and should only be run by the programmer who writes the driver
for the new metadata format.

Specifically, when a new metadate format is available, 
the driver programmer needs to do the following steps:

(1) Give the metadata format a "driver name". This driver name
is the String that is passed into Validation constructor as 
parameter. For example, if you give the new metadata format a driver name
of "ABC", then you should use the following way to construct a Validation
object: Validation v = new Validation("ABC");

Currently, I use "DC" as the "driver name" for dc metadata, and 
"OLAC" for olac metadata.

(2) create a directory, say, "/home/yourname/xxx";

(3) copy all the "XML Schema files" into the above directory;

(4) run DOMTreeReporter in the following way:
    java DOMTreeReporter /home/yourname/xxx

Then, a file called "Metadafile.properties" is generated 
in the above directory. This is the "configuration file" 
for this new metadata format.

(5) Then, go to:

$KEPLER_HOME/src/kepler/metadata/validation/schemafiles,
create a new directory with the "driver name", say "ABC".
copy the above Metadatafile.properties into the "ABC" directory.

(6) go to:
$KEPLER_HOME/src/server/validation/schemafiles,
create a new directory with the "driver name", say "ABC".
copy the above Metadatafile.properties into the "ABC" directory.

After the above steps, a driver programmer can call 
the Validation APIs in his own code.

In addition, the metadata driver programmer is also supposed to
provide a "GUI configuration file" for the group administrator 
so that the group administrator can change the configuration file 
(Mandatory, RegularExpression, Drop-down list) using the GUI.

Such a "GUI configuration file" is located at:
$KEPLER_HOME/src/kepler/metadata/validation/schemafiles/your_driver_name/GUI.properties
 

SUPPORT
=======

See http://kepler.cs.odu.edu:8080/kepler, or through e-mail to <kepler@list.odu.edu>


LICENSE
=======

This software is distributed in an UIUC/NCSA Open Source License, which is approved by 
Open Source Initiative in http://www.opensource.org/licenses/UoI-NCSA.php

University of Illinois/NCSA Open Source License

Copyright (c) 2002 Old Dominion University 
All rights reserved.

Developed by: 		Digital Library Group
                   	Old Dominion University
                    http://dlib.cs.odu.edu



Permission is hereby granted, free of charge, to any person obtaining a copy of 
this software and associated documentation files (the "Software"), to deal with 
the Software without restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
Software, and to permit persons to whom the Software is furnished to do so, 
subject to the following conditions: 

Redistributions of source code must retain the above copyright notice, this list 
of conditions and the following disclaimers. 
Redistributions in binary form must reproduce the above copyright notice, this 
list of conditions and the following disclaimers in the documentation and/or 
other materials provided with the distribution. 
Neither the names of Digital Library Group, Old Dominion University, nor the 
names of its contributors may be used to endorse or promote products derived from 
this Software without specific prior written permission. 
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN 
AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE SOFTWARE.