batchserver-cvs Mailing List for Enterprise Batch Server (Page 6)
Brought to you by:
suresh_pragada
Menu
▾
▴
batchserver-cvs — List to get the emails from cvs for check-ins.
You can subscribe to this list here.
| 2006 |
Jan
|
Feb
(10) |
Mar
(159) |
Apr
(5) |
May
(52) |
Jun
(70) |
Jul
|
Aug
(28) |
Sep
(256) |
Oct
(38) |
Nov
|
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(3) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Suresh <sur...@us...> - 2006-09-15 20:17:20
|
Update of /cvsroot/batchserver/batchserver/test/org/jmonks/batch/framework In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv24876/framework Log Message: Directory /cvsroot/batchserver/batchserver/test/org/jmonks/batch/framework added to the repository |
|
From: Suresh <sur...@us...> - 2006-09-15 20:17:04
|
Update of /cvsroot/batchserver/batchserver/test/org/jmonks/batch In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv24499/batch Log Message: Directory /cvsroot/batchserver/batchserver/test/org/jmonks/batch added to the repository |
|
From: Suresh <sur...@us...> - 2006-09-15 20:15:59
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/io In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv24080 Removed Files: FieldSpec.java FileParseException.java FileReader.java FileSpec.java FileSpecException.java FileType.java FileWriter.java package.html ReaderRecord.java Record.java RecordSpec.java RecordType.java WriterRecord.java Log Message: no message --- FileSpecException.java DELETED --- --- RecordType.java DELETED --- --- package.html DELETED --- --- Record.java DELETED --- --- FileParseException.java DELETED --- --- RecordSpec.java DELETED --- --- WriterRecord.java DELETED --- --- FileSpec.java DELETED --- --- ReaderRecord.java DELETED --- --- FieldSpec.java DELETED --- --- FileWriter.java DELETED --- --- FileReader.java DELETED --- --- FileType.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:15:44
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/io/xml In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv24054 Removed Files: package.html PrettyXMLIndentationEngine.java sample-xml-file-spec.xml sample-xml-file.xml XMLFileReader.java XMLFileSpec.java XMLFileWriter.java XMLIndentationEngine.java XMLRecordSpec.java Log Message: no message --- XMLIndentationEngine.java DELETED --- --- XMLFileReader.java DELETED --- --- sample-xml-file.xml DELETED --- --- XMLFileWriter.java DELETED --- --- package.html DELETED --- --- sample-xml-file-spec.xml DELETED --- --- XMLFileSpec.java DELETED --- --- PrettyXMLIndentationEngine.java DELETED --- --- XMLRecordSpec.java DELETED --- |
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/io/flat In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv24026 Removed Files: DelimitedFlatFileFieldSpec.java DelimitedFlatFileFileSpec.java DelimitedFlatFileReader.java DelimitedFlatFileRecordSpec.java DelimitedFlatFileWriter.java FixedWidthFlatFileFieldSpec.java FixedWidthFlatFileFileSpec.java FixedWidthFlatFileReader.java FixedWidthFlatFileRecordSpec.java FixedWidthFlatFileWriter.java FlatFileReader.java FlatFileWriter.java package.html sample-delimited-file-spec.xml sample-delimited-file.dat sample-fixed-width-file-spec.xml sample-fixed-width-file.dat Log Message: no message --- DelimitedFlatFileRecordSpec.java DELETED --- --- FixedWidthFlatFileWriter.java DELETED --- --- DelimitedFlatFileWriter.java DELETED --- --- sample-delimited-file-spec.xml DELETED --- --- DelimitedFlatFileFileSpec.java DELETED --- --- FixedWidthFlatFileFileSpec.java DELETED --- --- FixedWidthFlatFileReader.java DELETED --- --- FixedWidthFlatFileFieldSpec.java DELETED --- --- package.html DELETED --- --- FlatFileWriter.java DELETED --- --- FixedWidthFlatFileRecordSpec.java DELETED --- --- sample-fixed-width-file-spec.xml DELETED --- --- sample-fixed-width-file.dat DELETED --- --- sample-delimited-file.dat DELETED --- --- DelimitedFlatFileFieldSpec.java DELETED --- --- DelimitedFlatFileReader.java DELETED --- --- FlatFileReader.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:14:37
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/management/jmxmp In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23523 Removed Files: JMXMPConnectorHelper.java RepositoryJMXMPConnectorHelper.java Log Message: no message --- RepositoryJMXMPConnectorHelper.java DELETED --- --- JMXMPConnectorHelper.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:14:25
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/management In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23493 Removed Files: JobConnectorHelper.java JobManagementAgent.java JobManager.java JobManagerMBean.java JobMonitor.java JobMonitorMBean.java JobNotification.java ProcessorState.java ProcessorStatus.java Log Message: no message --- JobMonitorMBean.java DELETED --- --- ProcessorStatus.java DELETED --- --- JobConnectorHelper.java DELETED --- --- JobManagerMBean.java DELETED --- --- JobNotification.java DELETED --- --- JobManagementAgent.java DELETED --- --- JobMonitor.java DELETED --- --- JobManager.java DELETED --- --- ProcessorState.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:13:40
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/controller/pool In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23090 Removed Files: AbstractPoolJobLoader.java AbstractPoolJobProcessor.java CollectionJobPool.java JobPool.java PoolJobController.java PoolJobLoader.java PoolJobProcessor.java Log Message: no message --- PoolJobLoader.java DELETED --- --- JobPool.java DELETED --- --- PoolJobProcessor.java DELETED --- --- AbstractPoolJobProcessor.java DELETED --- --- AbstractPoolJobLoader.java DELETED --- --- PoolJobController.java DELETED --- --- CollectionJobPool.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:13:26
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/controller/basic In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv23061 Removed Files: BasicJobController.java BasicJobProcessor.java Log Message: no message --- BasicJobProcessor.java DELETED --- --- BasicJobController.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:13:08
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/controller In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22921 Removed Files: JobController.java Log Message: no message --- JobController.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:12:29
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/config/xml In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22639 Removed Files: XMLBasicJobControllerConfig.java XMLJobConfig.java XMLJobConfigFactory.java XMLJobControllerConfig.java XMLJobLoggingConfig.java XMLPoolJobControllerConfig.java Log Message: no message --- XMLJobConfig.java DELETED --- --- XMLJobLoggingConfig.java DELETED --- --- XMLPoolJobControllerConfig.java DELETED --- --- XMLJobControllerConfig.java DELETED --- --- XMLJobConfigFactory.java DELETED --- --- XMLBasicJobControllerConfig.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:12:03
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/config/db In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22247 Removed Files: DBBasicJobControllerConfig.java DBJobConfig.java DBJobConfigFactory.java DBJobControllerConfig.java DBJobLoggingConfig.java DBPoolJobControllerConfig.java Log Message: no message --- DBJobConfigFactory.java DELETED --- --- DBJobConfig.java DELETED --- --- DBPoolJobControllerConfig.java DELETED --- --- DBJobControllerConfig.java DELETED --- --- DBBasicJobControllerConfig.java DELETED --- --- DBJobLoggingConfig.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:11:46
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/config In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv22230 Removed Files: BasicJobControllerConfig.java ConfigurationException.java FrameworkConfig.java JobConfig.java JobConfigFactory.java JobControllerConfig.java JobLoggingConfig.java PoolJobControllerConfig.java Log Message: no message --- JobConfig.java DELETED --- --- BasicJobControllerConfig.java DELETED --- --- FrameworkConfig.java DELETED --- --- JobConfigFactory.java DELETED --- --- JobControllerConfig.java DELETED --- --- JobLoggingConfig.java DELETED --- --- PoolJobControllerConfig.java DELETED --- --- ConfigurationException.java DELETED --- |
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv21295 Removed Files: ErrorCode.java JobContext.java JobStatistics.java LoggingManager.java Main.java Repository.java Log Message: no message --- JobContext.java DELETED --- --- JobStatistics.java DELETED --- --- LoggingManager.java DELETED --- --- Repository.java DELETED --- --- Main.java DELETED --- --- ErrorCode.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:09:02
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batchserver/framework/util In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20899 Removed Files: FrameworkUtil.java JdbcConnectionHelper.java Log Message: no message --- FrameworkUtil.java DELETED --- --- JdbcConnectionHelper.java DELETED --- |
|
From: Suresh <sur...@us...> - 2006-09-15 20:08:18
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/io/xml In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20799 Added Files: package.html PrettyXMLIndentationEngine.java sample-xml-file-spec.xml sample-xml-file.xml XMLFileReader.java XMLFileSpec.java XMLFileWriter.java XMLIndentationEngine.java XMLRecordSpec.java Log Message: no message --- NEW FILE: XMLIndentationEngine.java --- /* * XMLIndentationEngine.java * * Created on June 12, 2006, 10:01 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; /** * <p> * XMLIndentationEngine returns the indentation information to be written to the stream * based on events received. It receives the events before start and end tags being written * to the stream. * * @author Suresh Pragada * @version 1.0 * @since 1.0 * </p> */ public interface XMLIndentationEngine { /** * Returns the indentation string needs to be written to tbe stream before * the start element being written. */ public String startElement(); /** * Returns the indentation string needs to be written to tbe stream before * the element element being written. */ public String endElement(); } --- NEW FILE: XMLFileReader.java --- /* * XMLFileReader.java * * Created on June 2, 2006, 11:05 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; import java.io.InputStream; import java.io.InputStreamReader; import java.io.Reader; import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Stack; import javax.xml.stream.XMLEventReader; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamException; import javax.xml.stream.events.Characters; import javax.xml.stream.events.EndElement; import javax.xml.stream.events.StartElement; import javax.xml.stream.events.XMLEvent; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileReader; import org.jmonks.batch.io.RecordType; import org.jmonks.batch.io.FileParseException; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.ReaderRecord; /** * <p> * XMLFileReader reads the specified xml file according to the given file spec * and returns the recrods on the needed basis. It finds the record location * using the xpath given in each xml record spec and loads all the child elements * as a fields. Field values should be accessed using the element names. * If element is a complex element, returned value will be a reader record and * it again follow the same convention. If element is repeated more than once, * the value will be loaded as a list. ReaderRecord returned by this reader will * have additional methods used to read the data accurately. To find out how to read * each record from the file and to read the each field from the record, refer to the * package javadoc. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class XMLFileReader extends FileReader { /** * File spec to read the file. */ protected XMLFileSpec fileSpec=null; /** * XML Event reader reads and returns everything as events. */ private XMLEventReader reader=null; /** * Tracks where are we in xml file. */ private String xpath=""; private static Logger logger=Logger.getLogger(XMLFileReader.class); /** * Constructs and initializes the XML File reader. * * @param fileInputStream Input stream to the file. * @param fileSpec File spec to be used to read the file. * * @throws org.jmonks.batch.io.FileParseException If root element doesnt match * with the element specified in file spec and problems to initializes the reader. */ public XMLFileReader(InputStream fileInputStream,FileSpec fileSpec) { this(new InputStreamReader(fileInputStream),fileSpec); } /** * Constructs and initializes the XML File reader. * * @param reader Reader to the file. * @param fileSpec File spec to be used to read the file. * * @throws org.jmonks.batch.io.FileParseException If root element doesnt match * with the element specified in file spec and problems to initializes the reader. */ public XMLFileReader(Reader reader,FileSpec fileSpec) { logger.trace("Entering XMLFileReader constructor"); this.fileSpec=(XMLFileSpec)fileSpec; try { XMLInputFactory inputFactory=XMLInputFactory.newInstance(); this.reader=inputFactory.createXMLEventReader(reader); logger.debug("Created the XML Event reader"); if(this.validateRootElement()) xpath="/"+this.fileSpec.getRootElement(); else { this.reader.close(); throw new FileParseException("Unexpected root element found. Expecting the root element " + this.fileSpec.getRootElement()); } logger.debug("Validate the root tag of the file"); } catch(XMLStreamException exception) { exception.printStackTrace(); this.reader=null; logger.fatal("Exception while initializing the xml stream reader. Message = " + exception.getMessage(),exception); throw new FileParseException("Exception while initializing the xml stream reader. Message = " + exception.getMessage()); } logger.trace("Exiting XMLFileReader constructor"); } /** * Gets the next available record from the file. If file doesnt have any more * records, it returns null. * * @return Returns the next available record. * * @throws org.jmonks.batch.io.FileParseException Problems getting the next record. * */ public ReaderRecord getNextRecord() { logger.trace("Entering getNextRecord"); XMLReaderRecord readerRecord=null; if(this.reader==null) return readerRecord; else { try { /** * Read events until the xpath matches with any of the record specs xpath. */ while(reader.hasNext()) { XMLEvent event=reader.nextEvent(); if(event.isStartElement()) { StartElement startElement=(StartElement)event; String startElementName=startElement.getName().getLocalPart(); xpath=xpath+"/"+startElementName; XMLRecordSpec recordSpec=this.getRecordSpec(); if(recordSpec!=null) { logger.trace("Found configured " + xpath); readerRecord=retrieveRecord(recordSpec,startElementName); int index=xpath.lastIndexOf("/"+startElementName); if(index!=-1) xpath=xpath.substring(0, index); break; } } else if(event.isEndElement()) { EndElement endElement=(EndElement)event; String endElementName=endElement.getName().getLocalPart(); int index=xpath.lastIndexOf("/"+endElementName); if(index!=-1) xpath=xpath.substring(0, index); } } } catch(XMLStreamException exception) { exception.printStackTrace(); logger.fatal("Exception while reading the record. Message = " + exception.getMessage(),exception); throw new FileParseException("Exception while reading the record. Message = " + exception.getMessage()); } } logger.trace("Exiting getNextRecord"); return readerRecord; } /** * Closes the reader. */ public void close() { logger.trace("Entering close"); if(this.reader!=null) { try { this.reader.close(); logger.debug("Closed the reader"); } catch(XMLStreamException exception) { logger.debug("Streamexception while closing the reader. Message = " + exception.getMessage(),exception); } finally { this.reader=null; } } } /** * Validates the root element. Read all the events until it finds first start * element and validate its name against the configured element name. * * @return Returns true if the first start element name matches with configured element name, * false otherwise. */ private boolean validateRootElement() { boolean valid=false; try { while(reader.hasNext()) { XMLEvent event=reader.nextEvent(); if(event.isStartElement()) { StartElement startElement=(StartElement)event; if(this.fileSpec.getRootElement().equalsIgnoreCase(startElement.getName().getLocalPart())) { valid=true; break; } } } } catch(XMLStreamException exception) { exception.printStackTrace(); logger.fatal("Exception while validating the root element. Message = " + exception.getMessage(),exception); throw new FileParseException("Exception while validating the root element. Message = " + exception.getMessage()); } return valid; } /** * Looks for a record spec whose xpath matches with the current xpath. * * @return Returns the record spec matches the current xpath. */ private XMLRecordSpec getRecordSpec() { Collection recordSpecs=this.fileSpec.getRecordSpecs(); for(Iterator iterator=recordSpecs.iterator();iterator.hasNext();) { XMLRecordSpec recordSpec=(XMLRecordSpec)iterator.next(); if(recordSpec.isMatch(this.xpath)) return recordSpec; } return null; } /** * <p> * Loads all the elements available in the stream from now onwards as a fields * into the record until we reach an end element matches the recordElement name. * </p> * * @param recordSpec */ private XMLReaderRecord retrieveRecord(XMLRecordSpec recordSpec,String recordElement) { XMLReaderRecord readerRecord=new XMLReaderRecord(recordSpec.getRecordType()); Stack elementStack=new Stack(); boolean isPreviousElementStart=false; try { while(reader.hasNext()) { XMLEvent nextEvent=reader.peek(); if(nextEvent.isStartElement()) { /** * If the previous element is also a start element, retrieve the * nested element and returns it as a nested record. */ if(isPreviousElementStart) { StartElement previousStartElement=(StartElement)elementStack.pop(); XMLReaderRecord nestedRecord=this.retrieveRecord(recordSpec, previousStartElement.getName().getLocalPart()); readerRecord.writeElement(previousStartElement.getName().getLocalPart(),nestedRecord); isPreviousElementStart=false; } else { StartElement startElement=(StartElement)reader.nextEvent(); isPreviousElementStart=true; elementStack.push(startElement); } } else if(nextEvent.isEndElement()) { EndElement endElement=(EndElement)reader.nextEvent(); isPreviousElementStart=false; if(recordElement.equalsIgnoreCase(endElement.getName().getLocalPart()) && elementStack.empty()) { /** * End element name matches the starting element name of this record and stack is empty. */ break; } else { /** * Check the value on top of the stack. If it is start element, no value has * been provided for this element in the file, otherwise pop the values and start element. */ Object topValue=elementStack.peek(); if(topValue instanceof StartElement) { StartElement startElement=(StartElement)elementStack.pop(); readerRecord.writeElement(startElement.getName().getLocalPart(),""); } else { Object fieldValue=elementStack.pop(); StartElement startElement=(StartElement)elementStack.pop(); readerRecord.writeElement(startElement.getName().getLocalPart(), fieldValue); } } } else if(nextEvent.isCharacters()) { Characters chars=(Characters)reader.nextEvent(); if(!chars.isWhiteSpace()) { isPreviousElementStart=false; /** * Loop through all the next available character events. If there are * any escape characters, stax is returning them as individual character * events. So, loop through all the events and save them as a value. */ StringBuffer elementData=new StringBuffer(chars.getData()); while(reader.peek().isCharacters()) { elementData.append(((Characters)reader.nextEvent()).getData()); } elementStack.push(elementData.toString()); } } else { /** * Ignore the other events for now. */ reader.nextEvent(); } } } catch(XMLStreamException exception) { exception.printStackTrace(); logger.fatal("XMLStream exception while retrieving the record. Message = " + exception.getMessage(), exception); throw new FileParseException("XMLStream exception while retrieving the record. Message = " + exception.getMessage()); } return readerRecord; } /** * XMLReaderRecord implements ReaderRecord by maintaing the * field names and values as a map and provides the methods with proper * access privileges to read into and write from the record. It provides * the additional method to conveniently read the data from the record. * * @author Suresh Pragada */ public class XMLReaderRecord extends ReaderRecord { /** * Map holds the field names and values. */ private Map fieldMap=null; /** * Constructs XML Reader record. */ private XMLReaderRecord(RecordType recordType) { super(recordType); fieldMap=new HashMap(); } /** * Reads the values associated with the given field name and returns. * The return value will vary based on the element type in file. If it is * a simple element it would be string, if it is a nested element it would * reader record, it it is an element repeats multiple times it will be * list. Use the other methods available on XMLReaderRecord for * convenience. * * @param fieldName Name of the element. * * @return Returns the field value as object. */ public Object readField(String fieldName) { return this.fieldMap.get(fieldName); } /** * Expectes the element as a simple element and returns it as a string. * * @param elementName Name of the simple element. * * @return Returns the value associated with this element. */ public String readSimpleElement(String elementName) { return (String)fieldMap.get(elementName); } /** * Expectes the element as a nested/composite element and returns it as a reader record. * * @param elementName Name of the nested/composite element. * * @return Returns the value associated with this element. */ public ReaderRecord readComplexElement(String elementName) { Object complexElement=fieldMap.get(elementName); if(complexElement!=null) { if(complexElement instanceof ReaderRecord) return (ReaderRecord)complexElement; else if(complexElement instanceof String) return new XMLReaderRecord(super.getRecordType()); else return null; } else return null; } /** * Expectes the element as repeated element and returns it as a list. * * @param elementName Name of the repeated element. * * @return Returns the value associated with this element. */ public List readRepeatElement(String elementName) { Object fieldValue=this.fieldMap.get(elementName); if(fieldValue==null) return null; else if(fieldValue instanceof List) return (List)fieldValue; else { List fieldValueList=new ArrayList(); fieldValueList.add(fieldValue); return fieldValueList; } } /** * Writes the field name and values to the map. It will take care * of converting the values into a list, if they are repeated. */ private void writeElement(String fieldName, Object fieldValue) { if(fieldMap.containsKey(fieldName)) { Object existingFieldValue=fieldMap.remove(fieldName); if(existingFieldValue instanceof List) { List existingFieldValueList=(List)existingFieldValue; existingFieldValueList.add(fieldValue); fieldMap.put(fieldName, existingFieldValueList); } else { List fieldValueList=new ArrayList(); fieldValueList.add(existingFieldValue); fieldValueList.add(fieldValue); fieldMap.put(fieldName,fieldValueList); } } else fieldMap.put(fieldName,fieldValue); } /** * @see java.lang.Object#toString() */ public String toString() { return this.fieldMap.toString(); } } } --- NEW FILE: sample-xml-file.xml --- <?xml version='1.0' encoding='ISO-8859-1'?> <sample-root> <sample-header> <file-type>Employee Records</file-type> </sample-header> <sample-detail> <first-name>Suresh</first-name> <last-name>Pragada</last-name> <dept-info> <dept-name>IT</dept-name> <dept-location>LOC1</dept-location> </dept-info> <addresses> <address> <address-type>home</address-type> <city>Menomonee Falls</city> <zip-code>53051</zip-code> </address> <address> <address-type>office</address-type> <city>Menomonee Falls</city> <zip-code>53051</zip-code> </address> <address>Unidentified</address> </addresses> </sample-detail> <sample-trailer> <transaction-count>1</transaction-count> </sample-trailer> </sample-root> --- NEW FILE: XMLFileWriter.java --- /* * XMLFileWriter.java * * Created on June 5, 2006, 10:36 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; import java.io.OutputStream; import java.io.OutputStreamWriter; import java.io.Writer; import java.util.ArrayList; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import javax.xml.stream.XMLOutputFactory; import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLStreamWriter; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileParseException; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.FileWriter; import org.jmonks.batch.io.RecordType; import org.jmonks.batch.io.WriterRecord; /** * <p> * XMLFileWriter writes the specified xml file according to the given file spec * with the the recrods submitted to write into the file. It generates the root tag * from the attribute given in file-spec. It uses the last part in record xpath to * write the record element. If nested elements or repeated elements needs to be written, * writer record provides the methods to write these elements. * </p> * <p> * XMLFileWriter looks for two attributes "indentation-engine" and "encoding" in the * file spec to format the generated xml and use the encoding value in generated * xml processing instruction. The value to the "indentation-engine" should be * the class name implements XMLIndentationEngine interface. If it doesnt find this * attribute or not a valid value in this attribute, it uses the default indentation * engine. The value specified in "encoding" attribute will be used in the processing * instruction. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class XMLFileWriter extends FileWriter { /** * File spec to be adhered with. */ protected XMLFileSpec fileSpec=null; private XMLStreamWriter writer=null; /** * Egnine to be used for the indentation. */ private XMLIndentationEngine indentationEngine=null; private static Logger logger=Logger.getLogger(XMLFileWriter.class); /** * Constructs and initializes the writer with the given values. * * @param outputStream Output stream to write the records. * @param fileSpec File spec to be used to generate the file. */ public XMLFileWriter(OutputStream outputStream,FileSpec fileSpec) { this(new OutputStreamWriter(outputStream), fileSpec); } /** * Constructs and initializes the writer with the given values. * * @param writer Writer to write the records. * @param fileSpec File spec to be used to generate the file. */ public XMLFileWriter(Writer writer,FileSpec fileSpec) { logger.trace("Entering XMLFileWriter constructor"); try { this.fileSpec=(XMLFileSpec)fileSpec; XMLOutputFactory outputFactory=XMLOutputFactory.newInstance(); this.writer=outputFactory.createXMLStreamWriter(writer); logger.debug("Writer has been created."); this.indentationEngine=this.getXMLIndentationEngine(this.fileSpec.getXMLIndentationEngineClassName()); String encoding=this.fileSpec.getEncoding(); if(encoding!=null && !"".equals(encoding)) this.writer.writeStartDocument(encoding, "1.0"); else this.writer.writeStartDocument("UTF-8", "1.0"); this.writer.writeCharacters(this.indentationEngine.startElement()); this.writer.writeStartElement(this.fileSpec.rootElement); } catch(XMLStreamException exception) { exception.printStackTrace(); logger.fatal("XMLStream exception while creating the writer. Message = " + exception.getMessage(), exception); throw new FileParseException("XMLStream exception while creating the writer. Message = " + exception.getMessage()); } logger.trace("Exiting XMLFileWriter constructor"); } /** * Writes the record into the file. This record should be obtained from * this file writer only by using the createWriterRecord method. * * @param writerRecord Record to be written into the file. * * @throws IllegalStateException If writer is closed and trying to write the recrod. * @throws IllegalArgumentException If writer record is null. * @throws org.jmonks.batch.io.FileParseException Problems while trying to write the record. */ public void writeRecord(WriterRecord writerRecord) { logger.trace("Entering writeRecord"); if(writerRecord!=null) { try { XMLWriterRecord record=(XMLWriterRecord)writerRecord; if(record.isNestedElementRecord()) throw new IllegalArgumentException("Writer record should be obtained from the file writer to write into the file."); XMLRecordSpec recordSpec=(XMLRecordSpec)this.fileSpec.getRecordSpec(record.getRecordType()); String recordXpath=recordSpec.getRecordXPath(); int index=recordXpath.lastIndexOf('/'); String recordElement=recordXpath.substring(((index!=-1)?(index+1):0)); writeComplexElement(recordElement, record); writer.flush(); } catch(XMLStreamException exception) { exception.printStackTrace(); logger.fatal("XMLStream exception while writing the record into the writer. Message = " + exception.getMessage(), exception); throw new FileParseException("XMLStream exception while writing the record into the writer. Message = " + exception.getMessage()); } } else throw new IllegalArgumentException("Writer record cannot be null to write record into the file."); logger.trace("Exiting writeRecord"); } /** * Writes the simple element into the file. * * @param fieldName Name of the element. * @param fieldValue Value of the element. */ protected void writeSimpleElement(String fieldName,Object fieldValue) throws XMLStreamException { writer.writeCharacters(indentationEngine.startElement()); writer.writeStartElement(fieldName); writer.writeCharacters((fieldValue!=null)?fieldValue.toString():""); writer.writeCharacters(indentationEngine.endElement()); writer.writeEndElement(); } /** * Writes the nested/composite element into the file. * * @param recordElementName Name of the nested element. * @param fieldValueAsRecord Value of the nested element. */ protected void writeComplexElement(String recordElementName,XMLWriterRecord fieldValueAsRecord) throws XMLStreamException { writer.writeCharacters(indentationEngine.startElement()); writer.writeStartElement(recordElementName); for(Iterator iterator=fieldValueAsRecord.getFieldNameIterator();iterator.hasNext();) { String fieldName=(String)iterator.next(); Object fieldValue=fieldValueAsRecord.readField(fieldName); if(fieldValue instanceof XMLWriterRecord) { writeComplexElement(fieldName, (XMLWriterRecord)fieldValue); } else if(fieldValue instanceof List) { writeRepeatElement(fieldName, (List)fieldValue); } else { writeSimpleElement(fieldName, fieldValue); } } writer.writeCharacters(indentationEngine.endElement()); writer.writeEndElement(); } /** * Writes the repeated element into the file. * * @param multiElementName Name of the repeated element. * @param fieldValueAsList Value of the repeated element. */ protected void writeRepeatElement(String multiElementName,List fieldValueAsList) throws XMLStreamException { List multiElementList=(List)fieldValueAsList; for(Iterator iterator=multiElementList.iterator();iterator.hasNext();) { Object fieldValue=iterator.next(); if(fieldValue instanceof XMLWriterRecord) { writeComplexElement(multiElementName, (XMLWriterRecord)fieldValue); } else if(fieldValue instanceof List) { writeRepeatElement(multiElementName, (List)fieldValue); } else { writeSimpleElement(multiElementName, fieldValue); } } } /** * Creates the writer record assocites with the given record type. * IllegalArgumentException will be thrown, if there is no record * spec is found with this record type. * * @param recordType Type fo the record to be created. * * @return Returns the requested writer record. * * @throws IllegalArgumentException No record spec is found with this record type. */ public WriterRecord createWriterRecord(RecordType recordType) { if(this.fileSpec.isValidRecordType(recordType)) return new XMLWriterRecord(recordType,false); else throw new IllegalArgumentException("Record type " + recordType + " doesnt match with any record specs."); } /** * Closes the writer. */ public void close() { logger.trace("Entering close"); if(this.writer!=null) { try { this.writer.writeCharacters(indentationEngine.endElement()); this.writer.writeEndElement(); this.writer.writeEndDocument(); this.writer.flush(); this.writer.close(); logger.debug("Writer has been closed"); } catch(XMLStreamException exception) { logger.debug("XML Stream Exception while closing the writer. Message = " + exception.getMessage(), exception); } finally { this.writer=null; } } logger.trace("Exiting close"); } /** * Instantiates and returns the instance of the given class name. If it couldnt be instantiated, * it returns default XMLIndentationEngine. * * @param xmlIndentationEngineClassName Class name of engine defined in the file spec. * * @return Returns the instance of xml indentation engine. */ private XMLIndentationEngine getXMLIndentationEngine(String xmlIndentationEngineClassName) { logger.trace("Entering getXMLindentationEngine = " + xmlIndentationEngineClassName); XMLIndentationEngine engine=null; if(xmlIndentationEngineClassName!=null && !"".equals(xmlIndentationEngineClassName)) { try { engine=(XMLIndentationEngine)Class.forName(xmlIndentationEngineClassName).newInstance(); logger.debug("Successfully created engine from the configured class"); } catch(Exception exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); logger.info("Unable to instantiate the configured xml indentation engine class. Creating the default xml indentation engine."); engine=new PrettyXMLIndentationEngine(); } } else { logger.info("No xml indentation engine class has been configured in file spec. Creating the default xml indentation engine."); engine=new PrettyXMLIndentationEngine(); } logger.trace("Exiting getXMLIndentationEngine = " + (engine!=null)); return engine; } /** * XMLWriterRecord implements WriterRecord by maintaing the * field names and values as a map and provides the methods with proper * access privileges to read into and write from the record. It provides * the additional method to conveniently write the data into the record. * * @author Suresh Pragada */ public class XMLWriterRecord extends WriterRecord { /** * Map holds the field names and values. */ private Map fieldMap=null; private boolean isNestedElementRecord=false; /** * Constructs the XML writer record. */ private XMLWriterRecord(RecordType recordType, boolean isNestedElementRecord) { super(recordType); this.isNestedElementRecord=isNestedElementRecord; fieldMap=new LinkedHashMap(); } /** * Tells whether this record has been created to represent the nested element. * * @return Returns true if this has been created for the nested element, false otherwise. */ private boolean isNestedElementRecord() { return this.isNestedElementRecord; } /** * Writes the given field name and value into the record. * * @param fieldName Name of the field. * @param fieldValue Value of the field. */ public void writeField(String fieldName, Object fieldValue) { if(fieldName==null) throw new IllegalArgumentException("Field name cannot be null to write the field."); this.fieldMap.put(fieldName, fieldValue); } /** * Write simple element into the record. */ public void writeSimpleElement(String fieldName, String fieldValue) { if(fieldName==null) throw new IllegalArgumentException("Field name cannot be null to write simple element."); this.fieldMap.put(fieldName, fieldValue); } /** * Create complex element and add it to the record and returns the writer * record instance to which all the elements can be adde. */ public WriterRecord createComplexElement(String fieldName) { if(fieldName==null) throw new IllegalArgumentException("Field name cannot be null to create the complex element."); WriterRecord complexElement=new XMLWriterRecord(super.getRecordType(), true); this.fieldMap.put(fieldName, complexElement); return complexElement; } /** * Creates an orphan record that can be added to the record or repeated * element later. Once you are done working with this record, * you should add this one to either the list or any another element, * but you can not submit to the writer for writing this (orphan) record into the file. * This will not be associated to any record by default. * * @return Returns an XMLWriterRecord. */ public WriterRecord createComplexElement() { return new XMLWriterRecord(super.getRecordType() , true); } /** * Creates the list and add it to the record with the given field name * and returns the list to which values can be added. * * @param fieldName Name of the field repated more than once. * * @return Returns the list. */ public List createRepeatElement(String fieldName) { if(fieldName==null) throw new IllegalArgumentException("Field name cannot be null to create the repeat element."); List repeatElement=new ArrayList(); this.fieldMap.put(fieldName, repeatElement); return repeatElement; } /** * Reads the field from the record. */ private Object readField(String fieldName) { return this.fieldMap.get(fieldName); } /** * Gets the iterator for the field names key set. */ private Iterator getFieldNameIterator() { return this.fieldMap.keySet().iterator(); } } } --- NEW FILE: package.html --- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>XML File Implementation Overview</title> </head> <body> XML file implementation for the IO API. <p> XML files are becoming popular choice by the enterprises to exchange the data. This package provides the implementation to work with the XML files. Implementation assumes that file consists of set of records and each can be identified using the xpath. </p> <p> <h3>XML File Implementation</h3> XML files consists of set of different records where each record can be identified by using xpath expression and all the elements beneath the element identified by xpath are treated as fields of that record. <h5>Defining the file spec for xml files</h5> File spec which descibes the xml file expects the <code>file-type</code> attribute value should be "xml". It requires one additional attribute along with the <code>file-type</code> attribute, which is <code>root-element</code> which will be used highly while generating the xml file from the set of records. The value in the attribute specifies the root element of the xml document. This file spec optionally requires 2 more attributes "indentation-engine" and "encoding". "indentation-engine" attribute value tells the class to be used for the indentation purposes. This class should implement the XMLIndentationEngine interface. If this attribute is not specified and the given class is not valid, it uses default indentation engine. The value specified in "encoding" attribute will be used in generated xml processing instruction. <pre> <file-spec file-type="xml" root-element="sample-root" indentation-engine="org.jmonks.batch.io.xml.PrettyXMLIndentationEngine" encoding="ISO-8859-1"> </file-spec> </pre> There could be multiple record specs exists in a file spec. Along with the record-type attribute, it requires an additional attribute <code>record-xpath</code> which tells the xpath in the document to identify the record. <pre> <file-spec file-type="xml" root-element="sample-root" indentation-engine="org.jmonks.batch.io.xml.PrettyXMLIndentationEngine" encoding="ISO-8859-1"> <record-spec record-type="DETAIL" record-xpath="/sample-root/detail-record"/> </file-spec> </pre> Since all the elements beneath the element represented by record spec will be exposed as a field values, record spec doesnt require any field specs. All the simple elements in the record element will be exposed as field name and values where element name represents the field name and element value represents the field value. All the complex elements(nested elements or elements contains child elements) will be exposed as a records again. All the repeat elements(elements which can repeat more than once) values will be exposed as list. <br> Here onwards will try to show how to process/read and generates/write the following xml file. Lets assume we need to read and write and following xml file. <br> <pre style="color:red"> <b> <?xml version='1.0' encoding='ISO-8859-1'?> <sample-root> <sample-header> <file-type>Employee Records</file-type> </sample-header> <sample-detail> <first-name>Suresh</first-name> <last-name>Pragada</last-name> <dept-info> <dept-name>IT</dept-name> <dept-location>LOC1</dept-location> </dept-info> <addresses> <address> <address-type>home</address-type> <city>Menomonee Falls</city> <zip-code>53051</zip-code> </address> <address> <address-type>office</address-type> <city>Menomonee Falls</city> <zip-code>53051</zip-code> </address> <address>Unidentified</address> </addresses> </sample-detail> <sample-trailer> <transaction-count>1</transaction-count> </sample-trailer> </sample-root> </b> </pre> <br> Following is the file spec to read and write the above given file. <br> <pre style="color:red"> <b> <?xml version="1.0" encoding="UTF-8"?> <file-spec file-type="xml" root-element="sample-root" indentation-engine="org.jmonks.batch.io.xml.PrettyXMLIndentationEngine" encoding="ISO-8859-1"> <record-spec record-type="header" record-xpath="/sample-root/sample-header"/> <record-spec record-type="detail" record-xpath="/sample-root/sample-detail"/> <record-spec record-type="trailer" record-xpath="/sample-root/sample-trailer"/> </file-spec> </b> </pre> <h5>Code to read the records from xml files</h5> <pre style="color:green"> <i> FileReader fileReader=FileReader.getFileReader(new FileInputStream("C:\\sample-xml-file_2.xml"),this.getClass().getResourceAsStream("sample-xml-file-spec.xml")); ReaderRecord readerRecord=null; while((readerRecord=fileReader.getNextRecord())!=null) { if(readerRecord.getRecordType().equals(RecordType.HEADER)) { // Simple elements in the records can be read using either readFiled or readSimpleElement. String fileType=(String)readerRecord.readField("file-type"); System.out.println("File type in header record = " + fileType); } else if(readerRecord.getRecordType().equals(RecordType.TRAILER)) { // Trying to show that simple elements can be read using readSimpleElement. String transactionCount=((XMLReaderRecord)readerRecord).readSimpleElement("transaction-count"); System.out.println(transactionCount); } else if(readerRecord.getRecordType().equals(RecordType.DETAIL)) { XMLReaderRecord xmlDetailRecord=(XMLReaderRecord)readerRecord; // Simple elements can be read using readSimpleElement method. String firstName=xmlDetailRecord.readSimpleElement("first-name"); String lastName=xmlDetailRecord.readSimpleElement("last-name"); System.out.println(firstName + " " + lastName); // Nested elements can be read using readComplexElement method. XMLReaderRecord deptInfoComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("dept-info"); String deptName=deptInfoComplexRecord.readSimpleElement("dept-name"); String deptLocation=deptInfoComplexRecord.readSimpleElement("dept-location"); System.out.println(deptName + " " + deptLocation); XMLReaderRecord addressesComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("addresses"); List addressesRepeatList=addressesComplexRecord.readRepeatElement("address"); for(Iterator iterator=addressesRepeatList.iterator();iterator.hasNext();) { Object addressRecord=iterator.next(); if(addressRecord instanceof XMLReaderRecord) { XMLReaderRecord addressComplexRecord=(XMLReaderRecord)addressRecord; String addressType=addressComplexRecord.readSimpleElement("address-type"); String city=addressComplexRecord.readSimpleElement("city"); String zipCode=addressComplexRecord.readSimpleElement("zip-code"); System.out.println(addressType + " " + city + " " + zipCode); } else if(addressRecord instanceof String) { System.out.println((String)addressRecord); } else System.out.println("Unknown type."); } } else System.out.println("Unknown record type = " + readerRecord.getRecordType().toString()); } fileReader.close(); </i> </pre> <h5>Code to write the records into xml files</h5> <pre style="color:green"> <i> // Get the file writer by providing the output stream to write the xml file and input stream to file spec. FileWriter fileWriter=FileWriter.getFileWriter(new FileOutputStream("C:\\sample-xml-file_2.xml"), this.getClass().getResourceAsStream("sample-xml-file-spec.xml")); // Create and write the header record. XMLWriterRecord headerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.HEADER); headerRecord.writeSimpleElement("file-type", "Employee Records"); fileWriter.writeRecord(headerRecord); // Get the empty record you want to create by passing the record type you mentioned in file spec. XMLWriterRecord detailRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.DETAIL); // Write the simple elements using either writeField or writeSimpleElement methods. detailRecord.writeSimpleElement("first-name", "Suresh"); detailRecord.writeField("last-name", "Pragada"); // Create the nested record by passing the nested element name. This automatically attached to detail record. No need to write it back to detail record. XMLWriterRecord deptComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("dept-info"); deptComplexRecord.writeSimpleElement("dept-name", "IT"); deptComplexRecord.writeSimpleElement("dept-location", "LOC1"); XMLWriterRecord addressesComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("addresses"); // Get the list to add all the elements needs to be written with the given name. List addressRepeatList=addressesComplexRecord.createRepeatElement("address"); // Empty nested element record can be created using any XMLWriterRecord instance. XMLWriterRecord homeAddressComplexRecord=(XMLWriterRecord)addressesComplexRecord.createComplexElement(); homeAddressComplexRecord.writeSimpleElement("address-type", "home"); homeAddressComplexRecord.writeSimpleElement("city", "Menomonee Falls"); homeAddressComplexRecord.writeSimpleElement("zip-code", "53051"); addressRepeatList.add(homeAddressComplexRecord); // Empty nested element record can be created using any XMLWriterRecord instance. XMLWriterRecord officeAddressComplexRecord=(XMLWriterRecord)addressesComplexRecord.createComplexElement(); officeAddressComplexRecord.writeSimpleElement("address-type", "office"); officeAddressComplexRecord.writeSimpleElement("city", "Menomonee Falls"); officeAddressComplexRecord.writeSimpleElement("zip-code", "53051"); addressRepeatList.add(officeAddressComplexRecord); // Feel free to drop simple elements value as well. addressRepeatList.add("Unidentified"); // Write the finished record into the file. fileWriter.writeRecord(detailRecord); // Create and write the trailer record. XMLWriterRecord trailerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.TRAILER); trailerRecord.writeSimpleElement("transaction-count", "1"); fileWriter.writeRecord(trailerRecord); fileWriter.close(); </i> </pre> </p> </body> </html> --- NEW FILE: sample-xml-file-spec.xml --- <?xml version="1.0" encoding="UTF-8"?> <file-spec file-type="xml" root-element="sample-root" indentation-engine="org.jmonks.batch.io.xml.PrettyXMLIndentationEngine" encoding="ISO-8859-1"> <record-spec record-type="header" record-xpath="/sample-root/sample-header"/> <record-spec record-type="detail" record-xpath="/sample-root/sample-detail"/> <record-spec record-type="trailer" record-xpath="/sample-root/sample-trailer"/> </file-spec> --- NEW FILE: XMLFileSpec.java --- /* * XMLFileSpec.java * * Created on June 2, 2006, 10:42 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; import java.util.Iterator; import org.apache.log4j.Logger; import org.w3c.dom.Element; import org.w3c.dom.NodeList; import org.jmonks.batch.io.FileType; import org.jmonks.batch.io.RecordSpec; import org.jmonks.batch.io.FileSpecException; import org.jmonks.batch.io.FileSpec; /** * <p> * XMLFileSpec represents the file spec defines the xml file * where each different kind of record identified by the xpath. * This file-spec requires additional attribute root-element which specifies * root element of the xml file along with the file-type attribute, which is supposed * to be "xml". Here is a sample file spec... * </p> * <p> * <pre> * <file-spec file-type="xml" root-element="consumer-request"> * <!-- record specs will follow here --> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class XMLFileSpec extends FileSpec { /** * Holds the root element. */ protected String rootElement=null; /** * Holds the class name of XMLIndentationEngine. */ protected String xmlIndentationEngineClassName=null; /** * Holds the encoding value to be placed in processing instruction. */ protected String encoding=null; /** * Constant defines the root element attribute name. */ public static final String ROOT_ELEMENT_ATTRIB_NAME = "root-element"; /** * Constant defines the xml indentation engine attribute name. */ public static final String XML_INDENTATION_ENGINE_ATTRIB_NAME = "indentation-engine"; /** * Constant defines the encoding attribute name. */ public static final String ENCODING_ATTRIB_NAME = "encoding"; private static Logger logger=Logger.getLogger(XMLFileSpec.class); /** * Constructs the XMLFileSpec instance. */ protected XMLFileSpec() { super(FileType.XML_FILE); } /** * Gets the root element. */ public String getRootElement() { return this.rootElement; } /** * Returns the XMLIndentationEngine class name to be used * to format the output. This is optional in configuration. If * none is specified default xml indentation engine will be used. * * @return Returns the class name of XMLIndentationEngine, null if not configured. */ public String getXMLIndentationEngineClassName() { return this.xmlIndentationEngineClassName; } /** * Get the encoding value to be placed in processing instruction. This will be * conifgured in file spec. * * @return Returns the file spec. */ public String getEncoding() { return this.encoding; } /** * Factory method to create the xml file spec object from the given * DOM Element representing the file-spec element. * * @param fileSpecElement DOM Element representing the file-spec element. * * @return Returns the instance of XMLFileSpec. * * @throws org.jmonks.batch.io.FileSpecException If two record specs * has the same value for the record-xpath and record-type attributes. */ public static FileSpec createXMLFileSpec(final Element fileSpecElement) { XMLFileSpec fileSpec=new XMLFileSpec(); String rootElement=fileSpecElement.getAttribute(XMLFileSpec.ROOT_ELEMENT_ATTRIB_NAME); if(rootElement!=null && !"".equals(rootElement.trim())) fileSpec.rootElement=rootElement; else throw new FileSpecException("XML FileSpec requires attribute root-element in element file-spec."); fileSpec.xmlIndentationEngineClassName=fileSpecElement.getAttribute(XMLFileSpec.XML_INDENTATION_ENGINE_ATTRIB_NAME); fileSpec.encoding=fileSpecElement.getAttribute(XMLFileSpec.ENCODING_ATTRIB_NAME); NodeList recordSpecNodeList=fileSpecElement.getElementsByTagName(RecordSpec.RECORD_SPEC_TAG_NAME); for(int i=0;i<recordSpecNodeList.getLength();i++) { XMLRecordSpec recordSpec=(XMLRecordSpec)XMLRecordSpec.createXMLRecordSpec((Element)recordSpecNodeList.item(i)); /** * Check for the duplicate record-xpath value on the record specs. */ for(Iterator iterator=fileSpec.getRecordSpecs().iterator();iterator.hasNext();) { XMLRecordSpec existingRecordSpec=(XMLRecordSpec)iterator.next(); if(existingRecordSpec.getRecordXPath().equalsIgnoreCase(recordSpec.getRecordXPath())) { throw new FileSpecException("Two record specs in the same file spec cannot have same values for record-xpath attribute."); } } logger.debug("Adding the record spec = " + recordSpec.toString()); fileSpec.addRecordSpec(recordSpec); } return fileSpec; } public String toString() { StringBuffer stringValue=new StringBuffer("{XMLFileSpec "); stringValue.append("[fileType = " + super.fileType.toString() + "]"); stringValue.append("[rootElement = " + this.rootElement + "]"); stringValue.append("[xmlIndentationEngine = " + this.xmlIndentationEngineClassName + "]"); stringValue.append("[encoding = " + this.encoding + "]"); stringValue.append("[recordSpecList = "); for(Iterator iterator=recordSpecMap.values().iterator();iterator.hasNext();) stringValue.append(((XMLRecordSpec)iterator.next()).toString()); stringValue.append("]}"); return stringValue.toString(); } } --- NEW FILE: PrettyXMLIndentationEngine.java --- /* * PrettyXMLIndentationEngine.java * * Created on June 12, 2006, 10:06 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; /** * <p> * PrettyXMLIndentationEngine writes the the elements in XML in the folloing manner. * <br> * <pre> * <root> * <element1>element-data1</element1> * <element2> * <element3>element-data3</element> * <element4>element-data4</element> * </element2> * </root> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class PrettyXMLIndentationEngine implements XMLIndentationEngine { /** * Remembers previous element is start element. */ private boolean isPrevStartElement=false; /** * Remembers previous element is end element. */ private boolean isPrevEndElement=false; private String indentationString="\n"; public PrettyXMLIndentationEngine() { } /** * @see org.jmonks.batch.io.xml.XMLIndentationEngine#startElement */ public String startElement() { isPrevEndElement=false; if(isPrevStartElement) { this.indentationString+=" "; return this.indentationString; } else { isPrevStartElement=true; return this.indentationString; } } /** * @see org.jmonks.batch.io.xml.XMLIndentationEngine#endElement */ public String endElement() { isPrevStartElement=false; if(isPrevEndElement) { this.indentationString=this.indentationString.substring(0, this.indentationString.length()-4); return this.indentationString; } else { isPrevEndElement=true; return ""; } } } --- NEW FILE: XMLRecordSpec.java --- /* * XMLRecordSpec.java * * Created on June 2, 2006, 10:54 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.xml; import org.apache.log4j.Logger; import org.w3c.dom.Element; import org.jmonks.batch.io.RecordType; import org.jmonks.batch.io.RecordSpec; import org.jmonks.batch.io.FileSpecException; /** * <p> * XMLRecordSpec represents record-spec element in the file spec * belongs to the xml file type. Along with the record-type attri... [truncated message content] |
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/io/flat In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20420 Added Files: DelimitedFlatFileFieldSpec.java DelimitedFlatFileFileSpec.java DelimitedFlatFileReader.java DelimitedFlatFileRecordSpec.java DelimitedFlatFileWriter.java FixedWidthFlatFileFieldSpec.java FixedWidthFlatFileFileSpec.java FixedWidthFlatFileReader.java FixedWidthFlatFileRecordSpec.java FixedWidthFlatFileWriter.java FlatFileReader.java FlatFileWriter.java package.html sample-delimited-file-spec.xml sample-delimited-file.dat sample-fixed-width-file-spec.xml sample-fixed-width-file.dat Log Message: no message --- NEW FILE: DelimitedFlatFileRecordSpec.java --- /* * DelimitedFlatFileRecordSpec.java * * Created on June 10, 2006, 10:55 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.util.Iterator; import org.apache.log4j.Logger; import org.jmonks.batch.io.FieldSpec; import org.jmonks.batch.io.FileSpecException; import org.jmonks.batch.io.RecordSpec; import org.jmonks.batch.io.RecordType; import org.w3c.dom.Element; import org.w3c.dom.NodeList; /** * <p> * DelimitedFlatFileRecordSpec represents record-spec element in the file spec * belongs to the delimited flat file type. Along with the <code>record-type</code> attribute * it looks for the two additional attributes <code>delimiter</code> and <code>field-count</code>. * Attribute <code>delimiter</code> tells the value that divides or delimits the fields * and attribute <code>field-count</code> tells the number of fields exists in the record. * There should be only one record spec is allowed in this file spec. * Here is a sample spec snippet... * </p> * <p> * <pre> * <file-spec file-type="delimited-flat"> * <record-spec record-type="detail" delimiter="|" field-count="4"> * <!-- field specs will follow here --> * </record-spec> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class DelimitedFlatFileRecordSpec extends RecordSpec { /** * Represents the delimiter for the record. */ private String delimiter=null; /** * Represents the number of fields available in the record. */ private int fieldCount; /** * Constant defines the delimiter attribute name which the value is "delimiter" */ public static final String DELIMITER_ATTRIB_NAME = "delimiter"; /** * Constant defines the field count attribute name which the value is "field-count" */ public static final String FIELD_COUNT_ATTRIB_NAME = "field-count"; private static Logger logger=Logger.getLogger(DelimitedFlatFileRecordSpec.class); /** * Constructs the record spec by accepting the record type. */ protected DelimitedFlatFileRecordSpec(RecordType recordType) { super(recordType); } /** * Returns the delimiter which divides the fields in the record. */ public String getDelimiter() { return this.delimiter; } /** * Returns the number of fields exists in this record. */ public int getFieldCount() { return this.fieldCount; } /** * Factory method to create the delimited flat record spec from the given * DOM Element representing the record-spec element in the file spec. * * @param recordSpecElement DOM Element representing the record spec. * * @return Returns the delimited flat file record spec. * * @throws org.jmonks.batch.io.FileSpecException If record-spec * doesnt have the delimiter or record-type attribute or index value is repeated on more than one field spec. */ public static DelimitedFlatFileRecordSpec createDelimitedFlatFileRecordSpec(final Element recordSpecElement) { logger.trace("Entering createDelimitedFlatFileRecordSpec"); DelimitedFlatFileRecordSpec recordSpec=null; String configuredRecordType=recordSpecElement.getAttribute(RecordSpec.RECORD_TYPE_ATTRIB_NAME); logger.debug("record specs record-type value = " + configuredRecordType); if(configuredRecordType!=null && !(configuredRecordType.trim().equals(""))) { RecordType recordType=RecordType.toRecordType(configuredRecordType); recordSpec=new DelimitedFlatFileRecordSpec(recordType); } else throw new FileSpecException("Record Spec in Delimited File Spec should have record-type attribute."); String delimiter=recordSpecElement.getAttribute(DelimitedFlatFileRecordSpec.DELIMITER_ATTRIB_NAME); logger.debug("record specs delimiter value = " + delimiter); if(delimiter!=null && !"".equals(delimiter)) recordSpec.delimiter=delimiter; else throw new FileSpecException("Record Spec in Delimited File Spec should have delimiter attribute."); String fieldCount=recordSpecElement.getAttribute(DelimitedFlatFileRecordSpec.FIELD_COUNT_ATTRIB_NAME); logger.debug("record specs field count value = " + delimiter); if(fieldCount!=null && !"".equals(fieldCount.trim())) recordSpec.fieldCount=Integer.parseInt(fieldCount); else throw new FileSpecException("Record Spec in Delimited File Spec should have field-count attribute."); NodeList fieldSpecNodeList=recordSpecElement.getElementsByTagName(FieldSpec.FIELD_SPEC_TAG_NAME); for(int i=0;i<fieldSpecNodeList.getLength();i++) { DelimitedFlatFileFieldSpec fieldSpec=DelimitedFlatFileFieldSpec.createDelimitedFlatFileFieldSpec((Element)fieldSpecNodeList.item(i)); for(Iterator iterator=recordSpec.getFieldSpecs().iterator();iterator.hasNext();) { DelimitedFlatFileFieldSpec existingFieldSpec=(DelimitedFlatFileFieldSpec)iterator.next(); if(existingFieldSpec.getIndex()==fieldSpec.getIndex()) { throw new FileSpecException("Found the index value " + fieldSpec.getIndex() + " on multiple fields."); } } recordSpec.addFieldSpec(fieldSpec); } return recordSpec; } /** * @see java.lang.Object#toString() */ public String toString() { StringBuffer stringValue=new StringBuffer("{DelimitedFlatFileRecordSpec "); stringValue.append("[recordType = " + super.recordType.toString() + "]"); stringValue.append("[delimiter = " + this.delimiter + "]"); stringValue.append("[fieldCount = " + this.fieldCount + "]"); stringValue.append("[fieldSpecList = "); for(Iterator iterator=fieldSpecList.iterator();iterator.hasNext();) stringValue.append(((DelimitedFlatFileFieldSpec)iterator.next()).toString()); stringValue.append("]}"); return stringValue.toString(); } } --- NEW FILE: FixedWidthFlatFileWriter.java --- /* * FixedWidthFlatFileWriter.java * * Created on June 6, 2006, 8:58 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.io.IOException; import java.io.OutputStream; import java.io.Writer; import java.util.Arrays; import java.util.Iterator; import java.util.List; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.WriterRecord; import org.jmonks.batch.io.flat.FlatFileWriter.FlatFileWriterRecord; /** * <p> * FixedWidthFlatFileWriter writes the file according to the given file spec * with the data submitted in the form of WriterRecord's. This provides the methods * to create the required writer records and write the writer record into the file writer. * Once finished writing of all the records file writer should be closed by calling * the "close" method. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FixedWidthFlatFileWriter extends FlatFileWriter { private static Logger logger=Logger.getLogger(FixedWidthFlatFileWriter.class); /** * Constructs and initializes the writer with the given values. * * @param outputStream Output stream to write the records. * @param fileSpec File spec to be used to generate the file. */ public FixedWidthFlatFileWriter(OutputStream outputStream,FileSpec fileSpec) { super(outputStream,fileSpec); } /** * Constructs and initializes the writer with the given values. * * @param writer Writer to write the records. * @param fileSpec File spec to be used to generate the file. */ public FixedWidthFlatFileWriter(Writer writer,FileSpec fileSpec) { super(writer,fileSpec); } /** * Writes the given record into the file/writer. * * @param writerRecord Record consist of field names and values. * * @throws IllegalStateException If writer is closed and trying to write the recrod. * @throws IllegalArgumentException If writer record is null. * @throws org.jmonks.batch.io.FileParseException Problems while trying to write the record. */ protected String generateRecord(WriterRecord writerRecord) throws IOException { logger.trace("Entering generateRecord"); FlatFileWriterRecord record=(FlatFileWriterRecord)writerRecord; FixedWidthFlatFileRecordSpec recordSpec=(FixedWidthFlatFileRecordSpec)this.fileSpec.getRecordSpec(record.getRecordType()); char[] recordBuffer=new char[recordSpec.getRecordLength()]; Arrays.fill(recordBuffer,' '); System.arraycopy(recordSpec.startsWith.toCharArray(), 0, recordBuffer, 0, recordSpec.startsWith.length()); List fieldSpecList=recordSpec.getFieldSpecs(); for(Iterator iterator=fieldSpecList.iterator();iterator.hasNext();) { FixedWidthFlatFileFieldSpec fieldSpec=(FixedWidthFlatFileFieldSpec)iterator.next(); String fieldValue=(String)record.readField(fieldSpec.getFieldName()); if(fieldValue!=null) { char[] fieldValueChars=fieldValue.toCharArray(); System.arraycopy(fieldValueChars, 0, recordBuffer, fieldSpec.getStartPosition()-1, ((fieldValueChars.length>fieldSpec.getFieldWidth())?fieldSpec.getFieldWidth():fieldValueChars.length)); } } logger.trace("Exiting generateRecord"); return new String(recordBuffer); } } --- NEW FILE: DelimitedFlatFileWriter.java --- /* * DelimitedFlatFileWriter.java * * Created on June 10, 2006, 12:38 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.io.OutputStream; import java.io.Writer; import java.util.Iterator; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.WriterRecord; import org.jmonks.batch.io.flat.FlatFileWriter.FlatFileWriterRecord; /** * <p> * DelimitedFlatFileWriter writes the file according to the given file spec * with the data submitted in the form of WriterRecord's. This provides the methods * to create the required writer records and write the writer record into the file writer. * Once finished writing of all the records file writer should be closed by calling * the "close" method. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class DelimitedFlatFileWriter extends FlatFileWriter { private static Logger logger=Logger.getLogger(DelimitedFlatFileWriter.class); /** * Holds the single record spec exists in Delimited file spec. */ private DelimitedFlatFileRecordSpec recordSpec=null; /** * Constructs the Delimited flat file writer from the given output stream * and file spec. * * @param outputStream Output stream representing the file to generate. * @param fileSpec File spec of the delimited flat to be generated. */ public DelimitedFlatFileWriter(OutputStream outputStream,FileSpec fileSpec) { super(outputStream,fileSpec); recordSpec=(DelimitedFlatFileRecordSpec)fileSpec.getRecordSpecs().iterator().next(); } /** * Constructs the Delimited flat file writer from the given output stream * and file spec. * * @param writer Writer representing the file to generate. * @param fileSpec File spec of the delimited flat to be generated. */ public DelimitedFlatFileWriter(Writer writer,FileSpec fileSpec) { super(writer,fileSpec); recordSpec=(DelimitedFlatFileRecordSpec)fileSpec.getRecordSpecs().iterator().next(); } /** * Generates the string represenatation of the record from the writer record given by client. * * @param writerRecord Writer record consists of field names and values. * * @return Returns the string representation of the record. */ protected String generateRecord(WriterRecord writerRecord) throws java.io.IOException { FlatFileWriterRecord flatFileWriterRecord=(FlatFileWriterRecord)writerRecord; String[] fieldValuesArray=new String[recordSpec.getFieldCount()]; for(Iterator iterator=recordSpec.getFieldSpecs().iterator();iterator.hasNext();) { DelimitedFlatFileFieldSpec fieldSpec=(DelimitedFlatFileFieldSpec)iterator.next(); String fieldValue=(String)flatFileWriterRecord.readField(fieldSpec.getFieldName()); fieldValuesArray[fieldSpec.getIndex()-1]= fieldValue!=null?fieldValue:""; } StringBuffer recordBuffer=new StringBuffer(); for(int i=0;i<fieldValuesArray.length;i++) { if(i==fieldValuesArray.length-1) recordBuffer.append(fieldValuesArray[i]); else { recordBuffer.append(fieldValuesArray[i]); recordBuffer.append(recordSpec.getDelimiter()); } } return recordBuffer.toString(); } } --- NEW FILE: sample-delimited-file-spec.xml --- <?xml version="1.0" encoding="UTF-8"?> <file-spec file-type="delimited-flat"> <record-spec record-type="detail" delimiter="|" field-count="4"> <field-spec field-name="field1" index="1"/> <field-spec field-name="field2" index="2"/> <field-spec field-name="field3" index="3"/> <field-spec field-name="field4" index="4"/> </record-spec> </file-spec> --- NEW FILE: DelimitedFlatFileFileSpec.java --- /* * DelimitedFlatFileFileSpec.java * * Created on June 10, 2006, 10:47 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.util.Iterator; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.FileSpecException; import org.jmonks.batch.io.FileType; import org.jmonks.batch.io.RecordSpec; import org.w3c.dom.Element; import org.w3c.dom.NodeList; /** * <p> * DelimitedFlatFileFileSpec represents the file spec defines the flat file * where all the fields in the records delimited with a particular character or value. * This file spec doesn't require any special attributes other than <code>file-type</code> attribute, * which should be "delimited-flat". * Here is a sample file spec... * </p> * <p> * <pre> * <file-spec file-type="delimited-flat"> * <!-- record specs will follow here --> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class DelimitedFlatFileFileSpec extends FileSpec { private static Logger logger=Logger.getLogger(DelimitedFlatFileFileSpec.class); /** * Constructs the DelimitedFlatFileFileSpec. */ protected DelimitedFlatFileFileSpec() { super(FileType.DELIMITED_FLAT_FILE); } /** * Factory method create the delimited flat file spec object from the given * DOM Element representing the file-spec element. * * @param fileSpecElement DOM Element representing the file-spec element. * * @return Returns the instance of DelimitedFlatFileFileSpec. * * @throws org.jmonks.batch.io.FileSpecException If more than one record spec is configured. */ public static FileSpec createDelimitedFlatFileFileSpec(final Element fileSpecElement) { logger.trace("Entering createDelimitedFlatFileFileSpec"); DelimitedFlatFileFileSpec fileSpec=new DelimitedFlatFileFileSpec(); NodeList recordSpecNodeList=fileSpecElement.getElementsByTagName(RecordSpec.RECORD_SPEC_TAG_NAME); if(recordSpecNodeList.getLength()>1) { logger.fatal("Delimited file spec should not have more than 1 record spec. Number of record specs found = " + recordSpecNodeList.getLength()); throw new FileSpecException("Delimited file spec should have only one record spec."); } else if(recordSpecNodeList.getLength()<1) { logger.fatal("Delimited file spec should have atleast one record spec."); throw new FileSpecException("Delimited file spec should have atleast one record spec."); } else { DelimitedFlatFileRecordSpec recordSpec=DelimitedFlatFileRecordSpec.createDelimitedFlatFileRecordSpec((Element)recordSpecNodeList.item(0)); logger.debug("Adding the record spec = " + recordSpec.toString()); fileSpec.addRecordSpec(recordSpec); return (FileSpec)fileSpec; } } /** * @see java.lang.Object#toString() */ public String toString() { StringBuffer stringValue=new StringBuffer("{DelimitedFlatFileFileSpec "); stringValue.append("[fileType = " + super.fileType.toString() + "]"); stringValue.append("[recordSpecList = "); for(Iterator iterator=recordSpecMap.values().iterator();iterator.hasNext();) stringValue.append(((DelimitedFlatFileRecordSpec)iterator.next()).toString()); stringValue.append("]}"); return stringValue.toString(); } } --- NEW FILE: FixedWidthFlatFileFileSpec.java --- /* * FixedWidthFlatFileFileSpec.java * * Created on June 1, 2006, 2:06 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.util.Iterator; import org.apache.log4j.Logger; import org.w3c.dom.Element; import org.w3c.dom.NodeList; import org.jmonks.batch.io.FileType; import org.jmonks.batch.io.RecordSpec; import org.jmonks.batch.io.FileSpecException; import org.jmonks.batch.io.FileSpec; /** * <p> * FixedWidthFlatFileFileSpec represents the file spec defines the flat file * where each different kind of record starts with a particular value and the * fields in each record will have start and end position. This file-spec doesn't * require any special attributes other than file-type, which should be "fixed-width-flat". * Here is a sample file spec... * </p> * <p> * <pre> * <file-spec file-type="fixed-width-flat"> * <!-- record specs will follow here --> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FixedWidthFlatFileFileSpec extends FileSpec { private static Logger logger=Logger.getLogger(FixedWidthFlatFileFileSpec.class); /** * Constructs the FixedWidthFlatFileFileSpec. */ protected FixedWidthFlatFileFileSpec() { super(FileType.FIXED_WIDTH_FLAT_FILE); } /** * Factory method create the fixed width flat file spec object from the given * DOM Element representing the file-spec element. * * @param fileSpecElement DOM Element representing the file-spec element. * * @return Returns the instance of FixedWidthFlatFileFileSpec. * * @throws org.jmonks.batch.io.FileSpecException If two record specs * has the same value for the starts-with and record-type attributes. */ public static FileSpec createFixedWidthFlatFileFileSpec(final Element fileSpecElement) { logger.trace("Entering createFixedWidthFlatFileFileSpec "); FixedWidthFlatFileFileSpec fileSpec=new FixedWidthFlatFileFileSpec(); NodeList recordSpecNodeList=fileSpecElement.getElementsByTagName(RecordSpec.RECORD_SPEC_TAG_NAME); logger.debug("Number of record specs found = " + recordSpecNodeList.getLength()); for(int i=0;i<recordSpecNodeList.getLength();i++) { FixedWidthFlatFileRecordSpec recordSpec=FixedWidthFlatFileRecordSpec.createFixedWidthFlatFileRecordSpec((Element)recordSpecNodeList.item(i)); /** * Check for the duplicat start-with value on the record specs. */ for(Iterator iterator=fileSpec.getRecordSpecs().iterator();iterator.hasNext();) { FixedWidthFlatFileRecordSpec existingRecordSpec=(FixedWidthFlatFileRecordSpec)iterator.next(); if(existingRecordSpec.getStartsWith().equalsIgnoreCase(recordSpec.getStartsWith())) { throw new FileSpecException("Two record specs in the same file spec cannot have same values for starts-with attribute."); } } logger.debug("Adding the record spec = " + recordSpec.toString()); fileSpec.addRecordSpec(recordSpec); } return (FileSpec)fileSpec; } /** * @see java.lang.Object#toString() */ public String toString() { StringBuffer stringValue=new StringBuffer("{FixedWidthFlatFileFileSpec "); stringValue.append("[fileType = " + super.fileType.toString() + "]"); stringValue.append("[recordSpecList = "); for(Iterator iterator=recordSpecMap.values().iterator();iterator.hasNext();) stringValue.append(((FixedWidthFlatFileRecordSpec)iterator.next()).toString()); stringValue.append("]}"); return stringValue.toString(); } } --- NEW FILE: FixedWidthFlatFileReader.java --- /* * FixedWidthFlatFileReader.java * * Created on May 26, 2006, 2:15 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.io.InputStream; import java.io.Reader; import java.util.Collection; import java.util.Iterator; import java.util.List; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileParseException; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.ReaderRecord; import org.jmonks.batch.io.flat.FlatFileReader.FlatFileReaderRecord; /** * <p> * FixedWidthFlatFileReader reads the specified fixed width flat file according to the given file spec * and returns the recrods on the needed basis. Each field value from the record * should be read using readField method by passing the fieldName mentioned in * the file spec. To find out how to read each record from the file and to read * the each field from the record, refer to the FileReader javadoc. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FixedWidthFlatFileReader extends FlatFileReader { private static Logger logger=Logger.getLogger(FixedWidthFlatFileReader.class); /** * Constructs and initializes the reader with the given file path and file spec. * * @param fileInputStream Input stream to the file. * @param fileSpec File spec to be used to read the file. */ public FixedWidthFlatFileReader(InputStream fileInputStream,FileSpec fileSpec) { super(fileInputStream,fileSpec); } /** * Constructs and initializes the reader with the given file path and file spec. * * @param reader Reader to the file. * @param fileSpec File spec to be used to read the file. */ public FixedWidthFlatFileReader(Reader reader,FileSpec fileSpec) { super(reader,fileSpec); } /** * Parses the given record string, translates it into the proper ReaderRecord * and returns the reader record. * * @param recordString Record read from the file. * * @throws org.jmonks.batch.io.FileParseException Problems while parsing the * next record. This includes unable to identify the recod with the availble record specs. */ protected ReaderRecord parseRecord(String recordString) { logger.trace("Entering parseRecord"); FlatFileReaderRecord record=null; Collection recordSpecCollection=this.fileSpec.getRecordSpecs(); for(Iterator recordSpecIterator=recordSpecCollection.iterator();recordSpecIterator.hasNext();) { FixedWidthFlatFileRecordSpec recordSpec=(FixedWidthFlatFileRecordSpec)recordSpecIterator.next(); if(recordSpec.isMatch(recordString)) { List fieldSpecList=recordSpec.getFieldSpecs(); record=new FlatFileReaderRecord(recordSpec.getRecordType(),fieldSpecList.size()); for(Iterator fieldSpecIterator=fieldSpecList.iterator();fieldSpecIterator.hasNext();) { FixedWidthFlatFileFieldSpec fieldSpec=(FixedWidthFlatFileFieldSpec)fieldSpecIterator.next(); String fieldValue=null; try { fieldValue=recordString.substring(fieldSpec.getStartPosition()-1,fieldSpec.getEndPosition()); } catch(IndexOutOfBoundsException exception) { exception.printStackTrace(); logger.fatal("Could not retrieve the field " + fieldSpec.getFieldName() + " from the record string " + recordString + " from the positions start = " + fieldSpec.getStartPosition() + " end = " + fieldSpec.getEndPosition()); } record.writeField(fieldSpec.getFieldName(),fieldValue); } break; } if(!recordSpecIterator.hasNext()) throw new FileParseException("Record " + recordString + " cannot be matched with any configured record specs."); } logger.trace("Exiting parseRecord"); return record; } } --- NEW FILE: FixedWidthFlatFileFieldSpec.java --- /* * FixedWidthFlatFileFieldSpec.java * * Created on May 26, 2006, 2:43 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import org.apache.log4j.Logger; import org.w3c.dom.Element; import org.jmonks.batch.io.FieldSpec; import org.jmonks.batch.io.FileSpecException; /** * <p> * FixedWidthFlatFileFieldSpec represents <code>field-spec</code> element in the record spec * belongs to the fixed width flat file type. Along with the <code>field-name</code> attribute * it looks for the <code>start-pos</code> and <code>end-pos</code> attributes in the field-spec element to retrieve * the field from the record. Field name on <code>field-spec</code> should be unique across the record-spec. * Position value starts with "1" in the record and the start-pos and end-pos values are inclusive. * Here is a sample spec snippet... * </p> * <p> * <pre> * <file-spec file-type="fixed-width-flat"> * <record-spec record-type="detail" starts-with="5"> * <field-spec field-name="consumer-id" start-pos="2" end-pos="11"/> * <field-spec field-name="consumer-name" start-pos="12" end-pos="31"/> * </record-spec> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FixedWidthFlatFileFieldSpec extends FieldSpec { /** * Holds the start position of the field. */ private int startPosition=0; /** * Holds the end position of the field. */ private int endPosition=0; /** * Constant defines the start position attribute name. */ public static final String START_POSITION_ATTRIB_NAME = "start-pos"; /** * Constant defines the end position attribute name. */ public static final String END_POSITION_ATTRIB_NAME = "end-pos"; private static Logger logger=Logger.getLogger(FixedWidthFlatFileFieldSpec.class); /** * Creates a new instance of FixedWidthFlatFileFieldSpec by using field name. */ protected FixedWidthFlatFileFieldSpec(String fieldName) { super(fieldName); } /** * Gets the starting position of the field. */ public int getStartPosition() { return this.startPosition; } /** * Gets the ending position of the field. */ public int getEndPosition() { return this.endPosition; } /** * Gets the field width. */ public int getFieldWidth() { return (endPosition-startPosition)+1; } /** * Factory method to create the FixedWidthFlatFileFieldSpec instance from * the given DOM Element representing the filed-spec element in record spec. * * @param fieldSpecElement DOM Element representing the field-spec element. * * @return Returns the FixedWidthFlatFileFieldSpec instance. * * @throws org.jmonks.batch.io.FileSpecException If field-spec * field-name values are not unique across record-spec. */ public static FixedWidthFlatFileFieldSpec createFixedWidthFlatFileFieldSpec(final Element fieldSpecElement) { logger.trace("Entering createFixedWidthFlatFileFieldSpec"); String fieldName=fieldSpecElement.getAttribute(FieldSpec.FIELD_NAME_ATTRIB_NAME); FixedWidthFlatFileFieldSpec fieldSpec=new FixedWidthFlatFileFieldSpec(fieldName); String startPosition=fieldSpecElement.getAttribute(FixedWidthFlatFileFieldSpec.START_POSITION_ATTRIB_NAME); if(startPosition!=null && !"".equals(startPosition.trim())) fieldSpec.startPosition=Integer.parseInt(startPosition); else throw new FileSpecException("start-pos attribute is required in fixed width flat file spec element."); String endPosition=fieldSpecElement.getAttribute(FixedWidthFlatFileFieldSpec.END_POSITION_ATTRIB_NAME); if(endPosition!=null && !"".equals(endPosition.trim())) fieldSpec.endPosition=Integer.parseInt(endPosition); else throw new FileSpecException("end-pos attribute is required in fixed width flat file spec element."); return fieldSpec; } /** * @see java.lang.Object#toString() */ public String toString() { StringBuffer stringValue=new StringBuffer("{FixedWidthFlatFileFieldSpec "); stringValue.append("[fieldName = " + super.fieldName + "]"); stringValue.append("[startPosition = " + this.startPosition + "]"); stringValue.append("[endPosition = " + this.endPosition+ "]"); stringValue.append("}"); return stringValue.toString(); } } --- NEW FILE: package.html --- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>Flat File Implementation Overview</title> </head> <body> Flat file implementation for the IO API. <p> Flat files are the common type of files being used by the enterprises to exchange the data. This package provides the implementation to read and write two flat file formats fixed width and delimited flat files. Implementation assumes that each line in a flat file corresponds to a record and there is a way to identify each record and each field in that record. </p> <p> <h3>Fixed Width Flat File Implementation</h3> Fixed Width flat files consists of set of different records where each record is starts with some value to identify the record and each record consists of set of of fields where each field starts at one position and ends at one position. <h5>Defining the file spec for fixed width flat files</h5> File spec which descibes the fixed width flat file expects the <code>file-type</code> attribute value should be "fixed-width-flat". It doesnt require any additional attributes along with the <code>file-type</code> attribute. <pre> <file-spec file-type="fixed-width-flat"> </file-spec> </pre> There could be multiple record specs exists in a file spec. Along with the <code>record-type</code> attribute, it requires two additional attributes <code>starts-with</code> which tells the value that the record should starts with and <code>record-length</code> which tells the lenght of the record. <pre> <file-spec file-type="fixed-width-flat"> <record-spec record-type="DETAIL" starts-with="5" record-length="42"> </record-spec> </file-spec> </pre> There could be multiple field specs exists in a single record spec. Field spec requires few additional attributes along with the field-name attribute to identify or extract the field from the record. These attributes are <code>start-pos</code> which tells the starting position of the field in the record and <code>end-pos</code> which tells the ending position of the field in the record. Starting index in a record starts with "1". <pre> <file-spec file-type="fixed-width-flat"> <record-spec record-type="DETAIL" starts-with="5" record-length="42"> <field-spec field-name="field-name1" start-pos="12" end-pos="23"/> </record-spec> </file-spec> </pre> <h5>Reading the records from fixed width flat files</h5> Reading the records from the fixed width flat files is fairly simple. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. InputStream fileInputStream= // Get the input stream of the file to be read. FileReader reader=FileReader.getFileReader(fileInputStream,fileSpecInputStream); ReaderRecord record=null; while((record=reader.getNextRecord())!=null) { if(record.getRecordType().equals(RecordType.DETAIL)) { String fieldValue1=record.readField("field-name1"); // Read the rest of the field and does the processing. } } reader.close(); </i> </pre> <h5>Writing the records into fixed width flat files</h5> Writing the records into fixed width flat files is fairly simple. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. OutputStream fileOutputStream= // Get the output stream of the file to be written/generated. FileWriter writer=FileWriter.getFileWriter(fileOutputStream,fileSpecInputStream); WriterRecord record=writer.createWriterRecord(RecordType.DETAIL); record.writeField("field-name1","field-value1"); // Write all the other field values into the writer record. writer.writeRecord(record); writer.close(); </i> </pre> </p> <p> <h3>Delimited Flat File Implementation</h3> Delimited flat files consists of set of records of same type where all the records consists of same number of fields and each field is are delimited by a special value in the record. <h5>Defining the file spec for delimited flat files</h5> File spec which descibes the delimited flat file expects the <code>file-type</code> attribute value should be "delimited-flat". It doesnt require any additional attributes along with the <code>file-type</code> attribute. <pre> <file-spec file-type="delimited-flat"> </file-spec> </pre> Since all the records in delimited flat file of the same type, it allows only one record spec should exists in the file spec. Along with the record-type attribute, it requires two additional attributes <code>delimiter</code> and <code>field-count</code>. Attributes <code>delimiter</code> tells the value that delimites fields among the record and attribute <code>field-count</code> tells the number of fields can exists in the record. <pre> <file-spec file-type="delimited-flat"> <record-spec record-type="DETAIL" delimiter="|" field-count="4"> </record-spec> </file-spec> </pre> There could be multiple field specs exists in a record spec. Field spec requires one additional attributes along with the field-name attribute to identify or extract the field from the record. This attribute is <code>index</code> which tells the position of the field in the record. Starting index in a record starts with "1". <pre> <file-spec file-type="delimited-flat"> <record-spec record-type="DETAIL" delimiter="|" field-count="4"> <field-spec field-name="field-name1" index="3"/> </record-spec> </file-spec> </pre> <h5>Reading the records from delimited flat files</h5> Reading the records from the delimited flat files is fairly simple. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. InputStream fileInputStream= // Get the input stream of the file to be read. FileReader reader=FileReader.getFileReader(fileInputStream,fileSpecInputStream); ReaderRecord record=null; while((record=reader.getNextRecord())!=null) { if(record.getRecordType().equals(RecordType.DETAIL)) { String fieldValue1=record.readField("field-name1"); // Read the rest of the field and does the processing. } } reader.close(); </i> </pre> <h5>Writing the records into delimited flat files</h5> Writing the records into delimited flat files is fairly simple. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. OutputStream fileOutputStream= // Get the output stream of the file to be written/generated. FileWriter writer=FileWriter.getFileWriter(fileOutputStream,fileSpecInputStream); WriterRecord record=writer.createWriterRecord(RecordType.DETAIL); record.writeField("field-name1","field-value1"); // Write all the other field values into the writer record. writer.writeRecord(record); writer.close(); </i> </pre> </p> </body> </html> --- NEW FILE: FlatFileWriter.java --- /* * FixedWidthFlatFileWriter.java * * Created on June 6, 2006, 8:58 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.io.BufferedWriter; import java.io.IOException; import java.io.OutputStream; import java.io.OutputStreamWriter; import java.io.Writer; import java.util.HashMap; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.io.FileParseException; import org.jmonks.batch.io.FileSpec; import org.jmonks.batch.io.FileWriter; import org.jmonks.batch.io.RecordType; import org.jmonks.batch.io.WriterRecord; /** * <p> * FlatFileWriter writes/generates the file according to the given file spec * with the data submitted in the form of WriterRecord's. This provides the methods * to create the required writer records and write the writer record into the file writer. * Once finished writing of all the records file writer should be closed by calling * the "close" method. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class FlatFileWriter extends FileWriter { /** * File spec to be used in writing the file. */ protected FileSpec fileSpec=null; /** * Buffered writer to conveniently write each record as a new line into the file. */ private BufferedWriter writer=null; /** * Flag holds whethe first record has been written or not. This is * to identify whether we need to write EOL character before every record. */ private boolean writtenFirstLine=false; private static Logger logger=Logger.getLogger(FlatFileWriter.class); /** * Constructs and initializes the writer with the given values. * * @param outputStream Output stream to write the records. * @param fileSpec File spec to be used to generate the file. */ public FlatFileWriter(OutputStream outputStream,FileSpec fileSpec) { this(new OutputStreamWriter(outputStream), fileSpec); } /** * Constructs and initializes the writer with the given values. * * @param writer Writer to write the records. * @param fileSpec File spec to be used to generate the file. */ public FlatFileWriter(Writer writer,FileSpec fileSpec) { logger.trace("Entering FlatFileWriter constructor"); this.fileSpec=fileSpec; this.writer=new BufferedWriter(writer); logger.info("Done creating the flat file writer."); } /** * Creates the writer record assocites with the given record type. * IllegalArgumentException will be thrown, if there is no record * spec is found with this record type. * * @param recordType Type fo the record to be created. * * @return Returns the requested writer record. * * @throws IllegalArgumentException No record spec is found with this record type. */ public WriterRecord createWriterRecord(RecordType recordType) { if(this.fileSpec.isValidRecordType(recordType)) return new FlatFileWriterRecord(recordType); else throw new IllegalArgumentException("No record spec has been found with the record type = " + recordType); } /** * Generates the string representation of the record from given * writer record. * * @param writerRecord Record where field values can be read. * * @return Returns the string representation of the record. */ protected abstract String generateRecord(WriterRecord writerRecord) throws IOException; /** * Writes the given record into the file/writer. * * @param writerRecord Record consist of field names and values. * * @throws IllegalStateException If writer is closed and trying to write the recrod. * @throws IllegalArgumentException If writer record is null. * @throws org.jmonks.batch.io.FileParseException Problems while trying to write the record. */ public void writeRecord(WriterRecord writerRecord) { logger.trace("Entering writeRecord"); if(writerRecord!=null) { if(this.writer!=null) { try { String record=generateRecord(writerRecord); if(writtenFirstLine) this.writer.newLine(); else writtenFirstLine=true; this.writer.write(record); } catch(IOException exception) { exception.printStackTrace(); logger.fatal("IOException while writing the record into the reader. Message = " + exception.getMessage(),exception); throw new FileParseException("IOException while creating the reader. Message = " + exception.getMessage()); } } else throw new IllegalStateException("FileWriter is not available to write this record. Writer is either closed or not initialized properly."); } else throw new IllegalArgumentException("Writer record cannot be null to write record into the file."); logger.trace("Exiting writeRecord"); } /** * Closes the writer and all the resource being used by this writer. * This includes the output stream passed to construct the writer. */ public void close() { logger.trace("Entering close"); if(this.writer!=null) { try { this.writer.close(); logger.debug("Writer has been closed"); } catch(IOException exception) { logger.debug("IOException while closing the reader. Message = " + exception.getMessage(),exception); } finally { this.writer=null; } } } /** * FixedWidthFlatFileWriterRecord implements WriterRecord by maintaing the * field names and values as a map and provides the methods with proper * access privileges to read into and write from the record. * * @author Suresh Pragada */ public class FlatFileWriterRecord extends WriterRecord { /** * Map to hold the field names and values. */ private Map fieldMap=null; /** * Constructs and initializes the writer record. */ protected FlatFileWriterRecord(RecordType recordType) { super(recordType); fieldMap=new HashMap(); } /** * Writes the field data into the record. * * @param fieldName Name of the field defined in field spec. * @param fieldValue Value for the field name. */ public void writeField(String fieldName, Object fieldValue) { this.fieldMap.put(fieldName,fieldValue); } /** * Reads and returns the value associated with requested field name. * * @param fieldName Name of the field. * * @return Returns the value associated with the field name. */ protected Object readField(String fieldName) { if(this.fieldMap.containsKey(fieldName)) return this.fieldMap.get(fieldName); else return ""; } } } --- NEW FILE: FixedWidthFlatFileRecordSpec.java --- /* * FixedWidthFlatFileRecordSpec.java * * Created on May 26, 2006, 11:33 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import java.util.Iterator; import org.apache.log4j.Logger; import org.w3c.dom.Element; import org.w3c.dom.NodeList; import org.jmonks.batch.io.RecordType; import org.jmonks.batch.io.FieldSpec; import org.jmonks.batch.io.RecordSpec; import org.jmonks.batch.io.FileSpecException; /** * <p> * FixedWidthFlatFileRecordSpec represents record-spec element in the file spec * belongs to the fixed width flat file type. Along with the <code>record-type</code> attribute * it looks for the <code>starts-with</code> and <code>record-length</code> attributes * in the record-spec element to identify the record. <code>starts-with</code> tells * the value that the record starts with and <code>record-length</code> tells the * length of the record. Record type and starts-with attribute values * should be unique across the file-spec. Here is a sample spec snippet... * </p> * <p> * <pre> * <file-spec file-type="fixed-width-flat"> * <record-spec record-type="detail" starts-with="5" record-length="42"> * <!-- field specs will follow here --> * </record-spec> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FixedWidthFlatFileRecordSpec extends RecordSpec { /** * Holds the record specs starts with attribute value. */ protected String startsWith=null; /** * Holds the length of the record. */ protected int recordLength=0; /** * Constant defines the starts with attribute name. */ public static final String STARTS_WITH_ATTRIB_NAME = "starts-with"; /** * Constant defines the record length attribute name. */ public static final String RECORD_LENGTH_ATTRIB_NAME = "record-length"; private static Logger logger=Logger.getLogger(FixedWidthFlatFileRecordSpec.class); /** * Constructs the record spec by accepting the record type. */ protected FixedWidthFlatFileRecordSpec(RecordType recordType) { super(recordType); } /** * Gets the starts-with value of this record. */ public String getStartsWith() { return this.startsWith; } /** * Gets the length of the record. This will be obtained from the record-legnth * attribute in the record-spec. This will be used while writing/generating the record using * file writer and while reading the record to identify the record type. */ public int getRecordLength() { return recordLength; } /** * Tells whether the given record matches with the record spec. It checks whether * this record string starts with the starts-with value and the length is equals * to the record length. * * @param recordString Record read from the file. * * @return Returns true if the recordString matches this recrd spec, false otherwise. */ public boolean isMatch(String recordString) { if(recordString==null) throw new IllegalArgumentException("Record string cannot be null to match record spec."); return recordString.startsWith(this.startsWith) && recordString.length()==recordLength; } /** * Factory method to create the fixed width flat record spec from the given * DOM Element representing the record-spec element in the file spec. * * @param recordSpecElement DOM Element representing the record spec. * * @return Returns the fixed width flat file record spec. * * @throws org.jmonks.batch.io.FileSpecException If record-spec * doesnt have the starts-with or record-type attribute. */ public static FixedWidthFlatFileRecordSpec createFixedWidthFlatFileRecordSpec(final Element recordSpecElement) { logger.trace("Entering createFixedWidthFlatFileRecordSpec"); FixedWidthFlatFileRecordSpec recordSpec=null; String configuredRecordType=recordSpecElement.getAttribute(RecordSpec.RECORD_TYPE_ATTRIB_NAME); logger.debug("record specs record-type value = " + configuredRecordType); if(configuredRecordType!=null && !(configuredRecordType.trim().equals(""))) { RecordType recordType=RecordType.toRecordType(configuredRecordType); recordSpec=new FixedWidthFlatFileRecordSpec(recordType); } else throw new FileSpecException("Record Spec in Fixed Width File Spec should have record-type attribute."); String startsWith=recordSpecElement.getAttribute(FixedWidthFlatFileRecordSpec.STARTS_WITH_ATTRIB_NAME); logger.debug("record specs starts-with value = " + startsWith); if(startsWith!=null) // && !"".equals(startsWith.trim())) recordSpec.startsWith=startsWith.trim(); else throw new FileSpecException("Record Spec in Fixed Width File Spec should have starts-with attribute."); String recordLength=recordSpecElement.getAttribute(FixedWidthFlatFileRecordSpec.RECORD_LENGTH_ATTRIB_NAME); logger.debug("record specs length value = " + startsWith); if(recordLength!=null && !"".equals(startsWith.trim())) recordSpec.recordLength=Integer.parseInt(recordLength); else throw new FileSpecException("Record Spec in Fixed Width File Spec should have record-length attribute."); NodeList fieldSpecNodeList=recordSpecElement.getElementsByTagName(FieldSpec.FIELD_SPEC_TAG_NAME); for(int i=0;i<fieldSpecNodeList.getLength();i++) { FixedWidthFlatFileFieldSpec fieldSpec=FixedWidthFlatFileFieldSpec.createFixedWidthFlatFileFieldSpec((Element)fieldSpecNodeList.item(i)); recordSpec.addFieldSpec(fieldSpec); } return recordSpec; } /** * @see java.lang.Object#toString() */ public String toString() { StringBuffer stringValue=new StringBuffer("{FixedWidthFlatFileRecordSpec "); stringValue.append("[recordType = " + super.recordType.toString() + "]"); stringValue.append("[startsWith = " + this.startsWith + "]"); stringValue.append("[recordLength = " + this.recordLength + "]"); stringValue.append("[fieldSpecList = "); for(Iterator iterator=fieldSpecList.iterator();iterator.hasNext();) stringValue.append(((FixedWidthFlatFileFieldSpec)iterator.next()).toString()); stringValue.append("]}"); return stringValue.toString(); } } --- NEW FILE: sample-fixed-width-file-spec.xml --- <?xml version="1.0" encoding="UTF-8"?> <file-spec file-type="fixed-width-flat"> <record-spec record-type="header" starts-with="1" record-length="9"> <field-spec field-name="timestamp" start-pos="2" end-pos="9"/> </record-spec> <record-spec record-type="detail" starts-with="5" record-length="41"> <field-spec field-name="field1" start-pos="2" end-pos="11"/> <field-spec field-name="field2" start-pos="12" end-pos="21"/> <field-spec field-name="field3" start-pos="22" end-pos="31"/> <field-spec field-name="field4" start-pos="32" end-pos="41"/> </record-spec> <record-spec record-type="trailer" starts-with="6" record-length="9"> <field-spec field-name="recordCount" start-pos="2" end-pos="9"/> </record-spec> </file-spec> --- NEW FILE: sample-fixed-width-file.dat --- 120060526 512345678910000012.0034343434347878787878 512345678920000013.0034343434347878787878 512345678930000014.0034343434347878787878 512345678940000015.0034343434347878787878 512345678950000016.0034343434347878787878 60000000.00 --- NEW FILE: sample-delimited-file.dat --- 1234567891|0000012.00|3434343434|7878787878 1234567892|0000013.00|3434343434|7878787878 1234567893|0000014.00|3434343434|7878787878 1234567894|0000015.00|3434343434|7878787878 1234567895|0000016.00|3434343434|7878787878 --- NEW FILE: DelimitedFlatFileFieldSpec.java --- /* * DelimitedFlatFileFieldSpec.java * * Created on June 10, 2006, 11:09 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io.flat; import org.apache.log4j.Logger; import org.jmonks.batch.io.FieldSpec; import org.jmonks.batch.io.FileSpecException; import org.w3c.dom.Element; /** * <p> * DelimitedFlatFileFieldSpec represents <code>field-spec</code> element in the <code>record spec</code> * element belongs to the delimited flat file type. Along with the <code>field-name</code> attribute * it looks for the <code>index</code> attribute in the field-spec element to retrieve * the field from the record. It specifies the field position in the record. * Field index value starts with 1. * Filed name on field-spec should be unique across the record-spec. * Here is a sample spec snippet... * </p> * <p> * <pre> * <file-spec file-type="delimited-flat"> * <record-spec record-type="detail" delimiter="|"> * <field-spec field-name="consumer-id" index="1"/> * <field-spec field-name="consumer-name" index="2"/> * </record-spec> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class DelimitedFlatFileFieldSpec extends FieldSpec { /** * Represents the field position in the record. */ private int index; /** * Constant defines the index attribute name which is "index" */ public static final String INDEX_ATTRIB_NAME = "index"; private static Logger logger=Logger.getLogger(DelimitedFlatFileFieldSpec.class); /** * Creates a new instance of DelimitedFlatFileFieldSpec by using field name. */ protected DelimitedFlatFileFieldSpec(String fieldName) { super(fieldName); } /** * Returns the index of field in the record. Index value starts with 1. */ public int getIndex() { return this.index; } /** * Factory method to create the DelimitedFlatFileFieldSpec instance from * the given DOM Element representing the filed-spec element in record spec. * * @param fieldSpecElement DOM Element representing the field-spec element. * * @return Returns the DelimitedFlatFileFieldSpec instance. * * @throws org.jmonks.batch.io.FileSpecException If field-spec * field-name values are not unique across record-spec and it is missing index attribute. */ public static DelimitedFlatFileFieldSpec createDelimitedFlatFileFieldSpec(final Element fieldSpecElement) { logger.trace("Entering createDelimitedFlatFileFieldSpec"); String fieldName=fieldSpecElement.getAttribute(FieldSpec.FIELD_NAME_ATTRIB_NAME); DelimitedFlatFileFieldSpec fieldSpec=new DelimitedFlatFileFieldSpec(fieldName); String index=fieldSpecElement.getAttribute(DelimitedFlatFileFieldSpec.INDEX_ATTRIB_NAME); if(index!=null && !"".equals(index.trim())) fieldSpec.index=Integer.parseI... [truncated message content] |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:54
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/io In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20408 Added Files: FieldSpec.java FileParseException.java FileReader.java FileSpec.java FileSpecException.java FileType.java FileWriter.java package.html ReaderRecord.java Record.java RecordSpec.java RecordType.java WriterRecord.java Log Message: no message --- NEW FILE: FileSpecException.java --- /* * FileSpecException.java * * Created on May 26, 2006, 11:36 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * FileSpecException is a runtime exception thrown when there is a problem creating * the FileSpec object from the given file contains the file specification. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FileSpecException extends RuntimeException { /** * Constructs the FileSpecException by accepting the message describes the * problem. * * @param message Message identifies the problem. */ public FileSpecException(String message) { super(message); } } --- NEW FILE: RecordType.java --- /* * RecordType.java * * Created on May 26, 2006, 11:14 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * RecordType defines the different kind of records can be available in a file. * This defines the constants to identify all the records as enums. This record * type information should be provided when defining the record spec in file spec. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public final class RecordType { /** * Holds the string representing the record type. */ private String type=null; /** * Private constructor to make sure that RecordType cannot be instantiated from * outside of this class. * * @param type String representing the type of record. * * @throws IllegalArgumentException Throws if the type is null or the value is in use. */ private RecordType(String type) { if(type!=null) { this.type=type; } else throw new IllegalArgumentException("Record type cannot be null."); } /** * Utility method returns the RecordType object matches to the given value. If * it doesnt match with the any pre defined record types, it creates and returns * a new record type. * * @param type String representing the type of record. * * @return Returns the RecordType. * * @throws IllegalArgumentException Throws if the recordType is null or the value is in use. */ public static RecordType toRecordType(String recordType) { if(RecordType.HEADER.toString().equalsIgnoreCase(recordType)) return RecordType.HEADER; else if(RecordType.DETAIL.toString().equalsIgnoreCase(recordType)) return RecordType.DETAIL; else if(RecordType.TRAILER.toString().equalsIgnoreCase(recordType)) return RecordType.TRAILER; else if(RecordType.BLOCK_START.toString().equalsIgnoreCase(recordType)) return RecordType.BLOCK_START; else if(RecordType.BLOCK_END.toString().equalsIgnoreCase(recordType)) return RecordType.BLOCK_END; else return new RecordType(recordType); } /** * Equality will be based on the type. * * @see java.lang.Object#equals(java.lang.Object) */ public boolean equals(Object recordType) { return (recordType!=null) && (this.getClass()==recordType.getClass()) && (this.type.equalsIgnoreCase(recordType.toString())); } /** * Hashcode of the type will be returned as a hash code. * * @see java.lang.Object#hashCode() */ public int hashCode() { return this.type.hashCode(); } /** * Returns the string representation of the RecordType which can be used to * define the record spec in file spec. */ public final String toString() { return type; } /** * Represents the header record and the value to be used in the record * spec is "HEADER". */ public static final RecordType HEADER = new RecordType("HEADER"); /** * Represents the header record and the value to be used in the record * spec is "TRAILER". */ public static final RecordType TRAILER = new RecordType("TRAILER"); /** * Represents the detailed record and the value to be used in the record * spec is "DETAIL". */ public static final RecordType DETAIL = new RecordType("DETAIL"); /** * Represents the starting record of the block and the value to be used in the record * spec is "BLOCK-START". */ public static final RecordType BLOCK_START = new RecordType("BLOCK-START"); /** * Represents the ending record of the block and the value to be used in the record * spec is "BLOCK-END". */ public static final RecordType BLOCK_END = new RecordType("BLOCK-END"); } --- NEW FILE: package.html --- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>IO API Overview</title> </head> <body> Defines API for the clients to read records from the file and write records into the file. <p> This package provides the API for the client programs to work with the files easily irrespective of the file format. This API is based on the analogy that all the files to be read and write have some data in the form of records and each record consists of a set of fields. Structure of these files, records and fields can be defined or configured using xml format. API allows the client programs to reads the records in an iterative manner from the file by specifying the xml file defined the file structure (It will be called as file spec here onwards) and file to read. API allows the client programs to create the record and write that record into the file. </p> <p> <h3>Defining file structure</h3> API expects the file consists of the data in the form of mulitple records and each record consists of multiple fields. Structure of the file will be defined using the root tag <file-spec> accepts the attribute <code>file-type</code> which signifies the kind of format of the file it is defining. Allowed values for the <code>file-type</code> attribute should be from FileType class. This class defines all the available file types can be used. Using the value doesnt exist in the FileType class results FileSpecException being thrown. Implementations may expects other attributes along with the <code>file-type</code> attribute for better file reading and file writing. These additional attributes will be mentioned by the FileSpec impelementors. <pre> <file-spec file-type="file-format"> <!-- Record specs will go on here --> </file-spec> </pre> </p> <p> As every file will have its data in the form of records, each record will be defined using the tag <record-spec> in the <file-spec> tag. Every <code>record-spec</code> will have a <code>record-type</code> attribute which signifies the kind of record it is configuring like header record or detail record or trailer record. This <code>record-type</code> should be unique across the file spec and the value can be taken from RecordType class. Allowed to use other record types not defined in the RecordType class. Along with the <code>record-type</code> attribute each implementation will require additional attributes to idnetify the records in the file and to write the records into the file. These additional attributes will be mentioned by the RecordSpec implementations. <pre> <file-spec file-type="file-format"> <record-spec record-type="detail"> <!-- Record specs will go on here --> </record-spec> </file-spec> </pre> </p> <p> As every record spec consists of set of fields, each field is defined using the tag <field-spec> in <record-spec> tag. Each <code>field-spec</code> should have an attribute </code>field-name</code> identifies the name of the field, which will be used to populate the field values into the record. Based on the implementation, additional attributes will be required along with the <code>field-name</code> to identify the field. Some implementations might not require <code>field-spec</code> elements at all. Whether clients neeeds to define the <code>field-spec</code> elements or not will be decided by RecordSpec implementations. <pre> <file-spec file-type="file-format"> <record-spec record-type="detail"> <field-spec field-name="field-name1"/> </record-spec> </file-spec> </pre> </p> <p> <h3>Reading records from the file</h3> FileReader class helps us in obtaining the FileReader instance through various factory methods by accepting the file to be read and file spec in different forms. FileReader reads data from the file according to the given file spec and when it finds a match to any of the provided record spec, it reads that record and returns field values as ReaderRecord instance. ReaderRecord instance has a method to read the field values by passing the field names. FileReader implementations provides ReaderRecord implementations, which allow additional methods to read the field values. Look at the FileReader implementation javadoc for the additional methods. Reader records can be read from FileReader in an iterative manner. When file reader doesnt have any more records, it returns null indicating that reader doenst have any more records. At this point client can stop requesting for further records. Once done reading all the records, client should close the reader. <br/><br/> Folloing example indicates how to obtain a file reader instance and how to obtain reader record and how to read field values from the reader records. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. InputStream fileInputStream= // Get the input stream of the file to be read. FileReader reader=FileReader.getFileReader(fileInputStream,fileSpecInputStream); ReaderRecord record=null; while((record=reader.getNextRecord())!=null) { if(record.getRecordType().equals(RecordType.DETAIL)) { String fieldValue1=record.readField("field-name1"); // Read the rest of the field and does the processing. } } reader.close(); </i> </pre> </p> <p> <h3>Writing records into the file</h3> FileWriter class helps us in obtaining the FileWriter instance through various factory methods by accepting the file to be written and file spec in different forms. FileWriter accepts WriterRecord instances which actually consists of the field values to write the data into the file. WriterRecord instances can be obtained from FileWriter instances. WriterRecord instances provides the method to write the field values associated to field name. FileWriter implementation classes will provide additional methods to write the field values in an effective way into the writer record. Once all the required field values have been pushed into the writer record, this record needs to be submitted to the FileWriter instance. This will writes that record according to the file spec provided while obtaining the file writer instance. Once done writing all the records, file writer needs to be closed. <br/><br/> Folloing example indicates how to obtain a file writer instance and how to obtain writer record and how to write field values into the writer records and how to submit that record to the file writer. <pre style="color:green"> <i> InputStream fileSpecInputStream= // Get the input stream for the XML file contains the file structure. OutputStream fileOutputStream= // Get the output stream of the file to be written/generated. FileWriter writer=FileWriter.getFileWriter(fileOutputStream,fileSpecInputStream); WriterRecord record=writer.createWriterRecord(RecordType.DETAIL); record.writeField("field-name1","field-value1"); // Write all the other field values into the writer record. writer.writeRecord(record); writer.close(); </i> </pre> </p> </body> </html> --- NEW FILE: Record.java --- /* * Record.java * * Created on May 26, 2006, 11:04 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * Record represents the map of field names and values based on the record spec * defined in the file spec. ReaderRecord and WriterRecord classes provide * the methods to read and write the values to this record object for the * FileReader and FileWriter consequently. * <br> * TODO :: Based on the memory statistics on handling huge files introduces a method to * remove the data from the records, once they have been used. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class Record { /** * Holds the type of the record. */ private RecordType recordType=null; /** * Creates the record by accepting the record type. * * @param recordType Type of the record. */ protected Record(RecordType recordType) { this.recordType=recordType; } /** * Gets the record type. */ public RecordType getRecordType() { return this.recordType; } } --- NEW FILE: FileParseException.java --- /* * FileParseException.java * * Created on May 26, 2006, 11:42 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * FileParseException is a runtime exception thrown when there is a problem reading * the records from the file or writing the records into the file. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class FileParseException extends RuntimeException { /** * Constructs the FileParseException by accepting the message describes * the problem. * * @param message Messages describes the problem. */ public FileParseException(String message) { super(message); } } --- NEW FILE: RecordSpec.java --- /* * RecordSpec.java * * Created on May 26, 2006, 2:21 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; import java.util.ArrayList; import java.util.Iterator; import java.util.List; import org.apache.log4j.Logger; /** * <p> * RecordSpec represents the <record-spec> element in file spec. * Record spec specifies the format of the record in the form of multiple * field specs and contains the attribute helps in identifying the type of * record and so. Each record spec contains zero or more field specs based * on the file type. * </p> * <p> * Every record-spec element contains record-type attribute * specifies the kind of record it is. Most of these values will be defined * in RecordType class and additional values can be added by extending the * RecordType. The record-type values should be unique across one file spec. * Here is a sample record-spec configuration. * <br> * <pre> * <record-spec record-type="DETAIL"> * <field-spec field-name="consumer-id"> * </field-spec> * </record-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class RecordSpec { /** * Holds the type of the record this spec is representing. */ protected RecordType recordType=null; /** * Holds all the field spec objects. */ protected List fieldSpecList=null; /** * Constant defines the record spec tag name. */ public static final String RECORD_SPEC_TAG_NAME = "record-spec"; /** * Constant defines the record type attribute name. */ public static final String RECORD_TYPE_ATTRIB_NAME = "record-type"; private static Logger logger=Logger.getLogger(RecordSpec.class); /** * Constructs the RecordSpec object by accepting the RecordType. * * @param recordType Type of the record. */ protected RecordSpec(RecordType recordType) { this.recordType=recordType; fieldSpecList=new ArrayList(); } /** * Adds the given field spec object to the record spec. * * @param fieldSpec Field spec object to be added. * * @return Returns true if field spec has been added successfully, flase otherwise. * * @throws IllegalArgumentException If given field spec is null. * @throws org.jmonks.batch.io.FileSpecException If multiple field specs configured with same field name. */ protected boolean addFieldSpec(FieldSpec fieldSpec) { if(fieldSpec==null) throw new IllegalArgumentException("FiledSpec object to add to the record spec cannot be null."); else { for(Iterator iterator=this.fieldSpecList.iterator();iterator.hasNext();) { FieldSpec existingFieldSpec=(FieldSpec)iterator.next(); if(existingFieldSpec.getFieldName().equalsIgnoreCase(fieldSpec.getFieldName())) throw new FileSpecException("Multiple field specs have been defined with same field name = " + fieldSpec.getFieldName()); } return this.fieldSpecList.add(fieldSpec); } } /** * Gets the field spec objects as a list. * * @returns Returns the list contains of all the field specs. */ public List getFieldSpecs() { return fieldSpecList; } /** * Returns the record type of this record spec. */ public RecordType getRecordType() { return this.recordType; } } --- NEW FILE: WriterRecord.java --- /* * WriterRecord.java * * Created on June 2, 2006, 11:26 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * WriterRecord provides the methods to write the values of the fields into the record. * This record will be created from and submitted to write into the file for the FileWriter's. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class WriterRecord extends Record { /** * Creates a new instance of WriterRecord by accepting the type of record. * * @param recordType Type of the record. */ protected WriterRecord(RecordType recordType) { super(recordType); } /** * <p> * Writes the given field name and value into the record. The value will be * depends on the type of file. Concrete implementors provide the details * on the kind of the object should be written for different fields and what the field * name corresponds to. * </p> * * @param fieldName Name of the field. * @param fieldValue Value corresponding the field name. */ public abstract void writeField(String fieldName,Object fieldValue); } --- NEW FILE: FileSpec.java --- /* * FileSpec.java * * Created on May 26, 2006, 2:18 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; import java.io.IOException; import java.io.InputStream; import java.util.Collection; import java.util.HashMap; import java.util.Map; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import javax.xml.parsers.ParserConfigurationException; import org.apache.log4j.Logger; import org.jmonks.batch.io.flat.DelimitedFlatFileFileSpec; import org.w3c.dom.Document; import org.w3c.dom.Element; import org.xml.sax.SAXException; import org.jmonks.batch.io.xml.XMLFileSpec; import org.jmonks.batch.io.flat.FixedWidthFlatFileFileSpec; /** * <p> * FileSpec represents the file contains specification (spec for short) of any * file to read or write using this package. This provides the factory * method to create the FileSpec object from a file contains the spec * and necessary methods to access the record spec objects. * </p> * <p> * File spec defines the file type and possible records could exists in the file. * Every file spec will contains one or * more record specs. Element file-spec should contain one attribute file-type * defines the type of the file this spec is going to represent. Based on the file * type, there could be other attributes in file-spec element along with file-type * attribute. These attributes can be found from the javadoc of the particular file type * file specs. Here is a sample file-spec configuration. * </p> * <p> * <pre> * <file-spec file-type="fixed-width-flat"> * <record-spec record-type="DETAIL"> * </record-spec> * </file-spec> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class FileSpec { /** * Holds the type of the file this spec is representing. */ protected FileType fileType=null; /** * Map holds the record specs with their record types as keys. */ protected Map recordSpecMap=null; /** * Constant defines the file spec tage name. */ public static final String FILE_SPEC_TAG_NAME = "file-spec"; /** * Constant defines the attribute name file type should contain. */ public static final String FILE_TYPE_ATTRIB_NAME = "file-type"; private static Logger logger=Logger.getLogger(FileSpec.class); /** * Constructs the FileSpec object by accepting file type. * * @param fileType Type of the file this spec is representing. */ protected FileSpec(FileType fileType) { this.fileType=fileType; recordSpecMap=new HashMap(); } /** * Adds the record spec to the file spec. * * @param recordSpec Record spec to be added to the file spec. * * @return Returns true, if record spec is added, false otherwise. * * @throws IllegalArgumentException If recordSpec is null. * @throws org.jmonks.batch.io.FileSpecException If multiple record specs configured with same record type. */ protected boolean addRecordSpec(RecordSpec recordSpec) { if(recordSpec==null) throw new IllegalArgumentException("RecordSpec cannot be null to add to the file spec."); else { if(this.recordSpecMap.containsKey(recordSpec.getRecordType())) { throw new FileSpecException("Duplicate record spec with the same record type " + recordSpec.getRecordType().toString()); } else { this.recordSpecMap.put(recordSpec.getRecordType(),recordSpec); return true; } } } /** * <p> * Gets all the record spec's in this file spec as a list. * * TODO :: There is chance that client might misuse this list. Based on the * performance create a new list and return it. * </p> * * @return Returns record spec's as a collection. */ public Collection getRecordSpecs() { return this.recordSpecMap.values(); } /** * Gets the record spec associated with the given record type. * * @param recordType Record types associated with the record spec. * * @return Retruns the record spec associated with the record type, * null if record spec could not be found. * * @throws IllegalArgumentException If recordType is null. */ public RecordSpec getRecordSpec(RecordType recordType) { if(recordType==null) throw new IllegalArgumentException("RecordType cannot be null to get the record spec from file spec."); else return (RecordSpec)this.recordSpecMap.get(recordType); } /** * Tells whether a record spec is exists with the given record type in the file spec. * * @param recordType Type of the record to check in the file spec. * * @return Return true if there is a record spec with this type, false otherwise. */ public boolean isValidRecordType(RecordType recordType) { return this.recordSpecMap.containsKey(recordType); } /** * Gets the file type. * * @return Returns the file type. */ public FileType getFileType() { return this.fileType; } /** * <p> * Creates the file spec object from the file contains the spec. This looks for * the file-type attribute in file-spec element and creates the appropriate * file spec object. * </p> * @param fileSpecInputStream Input stream represents the file spec. * * @return Returns the file spec object. * * @throws org.jmonks.batch.io.FileSpecException If file-type attribute * is missing or doesnt consists of the type available FileType class * or given file cannot be found or cannot parse the file. */ public static FileSpec createFileSpec(InputStream fileSpecInputStream) { logger.trace("Entering createFileSpec"); try { if(fileSpecInputStream==null) throw new IllegalArgumentException("File spec input stream to read the file spec cannot be null."); DocumentBuilderFactory documentBuilderFactory=DocumentBuilderFactory.newInstance(); DocumentBuilder documentBuilder=documentBuilderFactory.newDocumentBuilder(); Document document=documentBuilder.parse(fileSpecInputStream); Element fileSpecElement=document.getDocumentElement(); if(FileSpec.FILE_SPEC_TAG_NAME.equals(fileSpecElement.getTagName())) { String configuredFileType=fileSpecElement.getAttribute(FileSpec.FILE_TYPE_ATTRIB_NAME); logger.info("Configured file-type value = " + configuredFileType); FileType fileType=FileType.toFileType(configuredFileType); if(fileType==FileType.FIXED_WIDTH_FLAT_FILE) return FixedWidthFlatFileFileSpec.createFixedWidthFlatFileFileSpec(fileSpecElement); else if(fileType==FileType.DELIMITED_FLAT_FILE) return DelimitedFlatFileFileSpec.createDelimitedFlatFileFileSpec(fileSpecElement); else if(fileType==FileType.XML_FILE) return XMLFileSpec.createXMLFileSpec(fileSpecElement); else throw new FileSpecException("Configured file type " + fileType + " in the file spec is not recognizable."); } else throw new FileSpecException("File spec root element doesnt match the expected " + FileSpec.FILE_SPEC_TAG_NAME + " value in the file spec obtained from the given stream."); } catch(ParserConfigurationException exception) { exception.printStackTrace(); logger.fatal("Couldnt find DOM parser in the classpath to parse the file spec",exception); throw new FileSpecException("Couldnt find DOM parser in the classpath to parse the file spec. Message = " + exception.getMessage()); } catch(SAXException exception) { exception.printStackTrace(); logger.fatal("Parsing exception while parsing the given inputstream.",exception); throw new FileSpecException("Parsing exception while parsing the file spec in given input stream. Message = " + exception.getMessage()); } catch(IOException exception) { exception.printStackTrace(); logger.fatal("IO exception while parsing the given input stream to create the file spec.",exception); throw new FileSpecException("IO exception while parsing the given input stream to create the file spec. Message = " + exception.getMessage()); } } } --- NEW FILE: ReaderRecord.java --- /* * ReaderRecord.java * * Created on June 2, 2006, 11:24 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * ReaderRecord provides the methods to read the values of the fields from a record. * This record will be generated from the FileReader's. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class ReaderRecord extends Record { /** * Constructs the ReaderRecord by accepting the record type. * * @param recordType Type of the record. */ protected ReaderRecord(RecordType recordType) { super(recordType); } /** * <p> * Reads the value of the given field from the record. The value will be * depends on the type of file. Concrete implementors provide the details * on the kind of the object they return for different fields and what the field * name corresponds to. * </p> * * @param fieldName Name of the field. * * @return Returns the value of the given field from the record, null if field doesnt exists. */ public abstract Object readField(String fieldName); } --- NEW FILE: FieldSpec.java --- /* * FieldSpec.java * * Created on May 26, 2006, 2:25 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; import org.apache.log4j.Logger; /** * <p> * FieldSpec represents the <field-spec> element in file spec. * Field spec specifies the name of the field and additional attributes to * identifies the field among the record and other properties based on the file type. * Every field-spec element contains <code>field-name</code> attribute identifies among the * record spec. The <code>field-name</code> values should be unique across one record spec. * </p> * <p> * <pre> * <field-spec field-name="consumer-id"/> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class FieldSpec { /** * Holds the name of the field. */ protected String fieldName=null; /** * Constant defines the field spec tag name. */ public static final String FIELD_SPEC_TAG_NAME = "field-spec"; /** * Constant defines the field name atttribute name. */ public static final String FIELD_NAME_ATTRIB_NAME = "field-name"; private static Logger logger=Logger.getLogger(FieldSpec.class); /** * Constructs the FieldSpec object by accepting the field name. * * @param fieldName Name of the field. */ public FieldSpec(String fieldName) { this.fieldName=fieldName; } /** * Returns the field name. */ public String getFieldName() { return this.fieldName; } } --- NEW FILE: FileWriter.java --- /* * FileWriter.java * * Created on June 6, 2006, 8:59 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; import java.io.InputStream; import java.io.OutputStream; import java.io.Writer; import org.apache.log4j.Logger; import org.jmonks.batch.io.flat.DelimitedFlatFileWriter; import org.jmonks.batch.io.flat.FixedWidthFlatFileWriter; import org.jmonks.batch.io.xml.XMLFileWriter; /** * <p> * FileWriter writes the specified file based on the given file spec with * the data written in the form of WriterRecord's. This provides the methods * to create the required records and write them into the file writer. Once finished * writing of all the records file writer should be closed by calling * the appropriate methods. This provides the factory method to create the file writer * based on the file spec. Here is a sample code snippet to use the file writer. * </p> * <p> * <pre> * InputStream fileSpecInputStream=...; * OutputStream fileOutputStream=...; * FileWriter fileWriter=FileWriter.getFileWriter(fileOutputStream,fileSpecInputStream); * WriterRecord record=fileWriter.createWriterRecord(RecordType.DETAIL); * record.writeField("consumer-id","123456"); * //Write the other field values. * fileWriter.writeRecord(record); * fileWriter.close(); * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class FileWriter { private static Logger logger=Logger.getLogger(FileWriter.class); /** * Writes the given record into the file. The record should be obtained * from this file writer using the <code>createWriterRecord</code> method. * * @param record WriterRecord that you want to write into the file. * * @throws org.jmonks.batch.io.FileParseException Problems while writing the record into file. */ public abstract void writeRecord(WriterRecord record); /** * Closes the writer and releases all the resources associated with the writer. */ public abstract void close(); /** * Creates the writer record to be used to fill all the field values * and submit to the file writer to write the record into the file. * * @param recordType Type of record. * * @return Retruns the writer record. * * @throws IllegalArgumentException If given record type is not associated with any record spec. */ public abstract WriterRecord createWriterRecord(RecordType recordType); /** * <p> * Factory method to get the file writer. This method returns the appropriate file writer * based on the file type specified in the file spec. * </p> * * @param outputStream Output stream to write the records. * @param fileSpecInputStream Input stream to read the file spec. * * @return Returns the appropriate writer. * * @throws IllegalArgumentException If the given file output stream or file spec input stream is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileWriter getFileWriter(OutputStream outputStream,InputStream fileSpecInputStream) { logger.trace("Entering getFileReader"); if(outputStream==null) throw new IllegalArgumentException("Output stream to write the records cannot be null."); if(fileSpecInputStream==null) throw new IllegalArgumentException("Input stream to read the file spec cannot be null."); FileSpec fileSpec=FileSpec.createFileSpec(fileSpecInputStream); return FileWriter.getFileWriter(outputStream, fileSpec); } /** * <p> * Factory method to get the file writer. This method returns the appropriate file writer * based on the file type specified in the file spec. * </p> * * @param outputStream Output stream to write the records. * @param fileSpec File spec to be used to create the writer. * * @return Returns the appropriate writer. * * @throws IllegalArgumentException If the given file output stream or file spec is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileWriter getFileWriter(OutputStream outputStream, FileSpec fileSpec) { logger.trace("Entering getFileReader"); if(outputStream==null) throw new IllegalArgumentException("Output stream to write the records cannot be null."); if(fileSpec==null) throw new IllegalArgumentException("File spec to create the writer cannot be null."); logger.debug("Given file spec = " + fileSpec.toString()); if(fileSpec.getFileType()==FileType.FIXED_WIDTH_FLAT_FILE) return new FixedWidthFlatFileWriter(outputStream,fileSpec); else if(fileSpec.getFileType()==FileType.DELIMITED_FLAT_FILE) return new DelimitedFlatFileWriter(outputStream, fileSpec); else if(fileSpec.getFileType()==FileType.XML_FILE) return new XMLFileWriter(outputStream,fileSpec); else throw new FileSpecException("Unsupported file type in the file spec = " + fileSpec.getFileType().toString()); } /** * <p> * Factory method to get the file writer. This method returns the appropriate file writer * based on the file type specified in the file spec. * </p> * * @param writer Writer to write the records. * @param fileSpecInputStream Input stream to read the file spec. * * @return Returns the appropriate writer. * * @throws IllegalArgumentException If the given file output stream or file spec input stream is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileWriter getFileWriter(Writer writer,InputStream fileSpecInputStream) { logger.trace("Entering getFileReader"); if(writer==null) throw new IllegalArgumentException("Writer to write the records cannot be null."); if(fileSpecInputStream==null) throw new IllegalArgumentException("Input stream to read the file spec cannot be null."); FileSpec fileSpec=FileSpec.createFileSpec(fileSpecInputStream); return FileWriter.getFileWriter(writer, fileSpec); } /** * <p> * Factory method to get the file writer. This method returns the appropriate file writer * based on the file type specified in the file spec. * </p> * * @param writer Writer to write the records. * @param fileSpec File spec to be used to create the writer. * * @return Returns the appropriate writer. * * @throws IllegalArgumentException If the given file output stream or file spec is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileWriter getFileWriter(Writer writer, FileSpec fileSpec) { logger.trace("Entering getFileReader"); if(writer==null) throw new IllegalArgumentException("Writer to write the records cannot be null."); if(fileSpec==null) throw new IllegalArgumentException("File spec to create the writer cannot be null."); logger.debug("Given file spec = " + fileSpec.toString()); if(fileSpec.getFileType()==FileType.FIXED_WIDTH_FLAT_FILE) return new FixedWidthFlatFileWriter(writer,fileSpec); else if(fileSpec.getFileType()==FileType.DELIMITED_FLAT_FILE) return new DelimitedFlatFileWriter(writer, fileSpec); else if(fileSpec.getFileType()==FileType.XML_FILE) return new XMLFileWriter(writer,fileSpec); else throw new FileSpecException("Unsupported file type in the file spec = " + fileSpec.getFileType().toString()); } } --- NEW FILE: FileReader.java --- /* * FileReader.java * * Created on May 26, 2006, 9:37 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; import java.io.InputStream; import java.io.Reader; import org.apache.log4j.Logger; import org.jmonks.batch.io.flat.DelimitedFlatFileReader; import org.jmonks.batch.io.xml.XMLFileReader; import org.jmonks.batch.io.flat.FixedWidthFlatFileReader; /** * <p> * FileReader reads the specified file based on the given file spec and * returns the data in the form of ReaderRecord's to read the required values referring the field names. * This provides the methods to reads the records in an iterative manner. Once finished * reading all or required records from the file reader, it should be closed by calling * the appropriate methods. This provides the factory method to create the file reader * based on the file spec. Here is a sample code snippet to use the file reader. * </p> * <p> * <pre> * InputStream fileSpecInputStream=...; * InputStream fileInputStream=....; * FileReader fileReader=FileReader.getFileReader(fileInputStream,fileSpecInputStream); * ReaderRecord record=null; * while((record=fileReader.getNextRecord())!=null) * { * if(record.getRecordType()==RecordType.DETAIL) * { * String consumerID=record.readField("consumer-id"); * // Read the rest of the fields and does the processing. * } * } * fileReader.close(); * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class FileReader { private static Logger logger=Logger.getLogger(FileReader.class); /** * <p> * Gets the next available record from the file. If file doesnt have any more records * it returns null. * </p> * * @return Returns the next available record from the file, null if it doesnt have any more records. * * @throws org.jmonks.batch.io.FileParseException Problems while parsing the next record. */ public abstract ReaderRecord getNextRecord(); /** * Closes the reader and releases all the resources associated with the reader. */ public abstract void close(); /** * <p> * Factory method to get the file reader. This method returns the appropriate file reader * based on the file type specified in the file spec. * </p> * * @param fileInputStream Input stream to read the data file. * @param fileSpecInputStream Input stream to read the file spec. * * @return Returns the appropriate reader. * * @throws IllegalArgumentException If the given fileInputStream or fileSpecInputStream is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileReader getFileReader(InputStream fileInputStream,InputStream fileSpecInputStream) { logger.trace("Entering getFileReader"); if(fileInputStream==null) throw new IllegalArgumentException("Input stream to read the data file cannot be null."); if(fileSpecInputStream==null) throw new IllegalArgumentException("Input stream to read the file spec cannot be null."); FileSpec fileSpec=FileSpec.createFileSpec(fileSpecInputStream); return FileReader.getFileReader(fileInputStream, fileSpec); } /** * <p> * Factory method to get the file reader. This method returns the appropriate file reader * based on the file type specified in the file spec. * </p> * * @param fileInputStream Input stream to read the data file. * @param fileSpec File spec to create the file reader. * * @return Returns the appropriate file reader. * * @throws IllegalArgumentException If the given fileInputStream or fileSpec is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileReader getFileReader(InputStream fileInputStream, FileSpec fileSpec) { logger.trace("Entering getFileReader"); if(fileInputStream==null) throw new IllegalArgumentException("Input stream to read the data file cannot be null."); if(fileSpec==null) throw new IllegalArgumentException("File spec cannot be null to create the file reader."); logger.debug("Given file spec = " + fileSpec.toString()); if(fileSpec.getFileType()==FileType.FIXED_WIDTH_FLAT_FILE) return new FixedWidthFlatFileReader(fileInputStream,fileSpec); else if(fileSpec.getFileType()==FileType.DELIMITED_FLAT_FILE) return new DelimitedFlatFileReader(fileInputStream,fileSpec); else if(fileSpec.getFileType()==FileType.XML_FILE) return new XMLFileReader(fileInputStream,fileSpec); else throw new FileSpecException("Unsupported file type in the file spec = " + fileSpec.getFileType().toString()); } /** * <p> * Factory method to get the file reader. This method returns the appropriate file reader * based on the file type specified in the file spec. * </p> * * @param reader Reader to read the data file. * @param fileSpecInputStream Input stream to read the file spec. * * @return Returns the appropriate reader. * * @throws IllegalArgumentException If the given reader or fileSpecInputStream is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileReader getFileReader(Reader reader,InputStream fileSpecInputStream) { logger.trace("Entering getFileReader"); if(reader==null) throw new IllegalArgumentException("Reader to read the data file cannot be null."); if(fileSpecInputStream==null) throw new IllegalArgumentException("Input stream to read the file spec cannot be null."); FileSpec fileSpec=FileSpec.createFileSpec(fileSpecInputStream); return FileReader.getFileReader(reader, fileSpec); } /** * <p> * Factory method to get the file reader. This method returns the appropriate file reader * based on the file type specified in the file spec. * </p> * * @param reader Reader to read the data file. * @param fileSpec File spec to create the file reader. * * @return Returns the appropriate reader. * * @throws IllegalArgumentException If the given reader or fileSpec is null. * @throws org.jmonks.batch.io.FileSpecException If the configured file type in file spec doesnt * exist in FileType class. */ public static FileReader getFileReader(Reader reader, FileSpec fileSpec) { logger.trace("Entering getFileReader"); if(reader==null) throw new IllegalArgumentException("Reader to read the data file cannot be null."); if(fileSpec==null) throw new IllegalArgumentException("File spec cannot be null to create the file reader."); logger.debug("Given file spec = " + fileSpec.toString()); if(fileSpec.getFileType()==FileType.FIXED_WIDTH_FLAT_FILE) return new FixedWidthFlatFileReader(reader,fileSpec); else if(fileSpec.getFileType()==FileType.DELIMITED_FLAT_FILE) return new DelimitedFlatFileReader(reader,fileSpec); else if(fileSpec.getFileType()==FileType.XML_FILE) return new XMLFileReader(reader,fileSpec); else throw new FileSpecException("Unsupported file type in the file spec = " + fileSpec.getFileType().toString()); } } --- NEW FILE: FileType.java --- /* * FileType.java * * Created on May 26, 2006, 3:01 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.io; /** * <p> * FileType defines different kinds of file formats this package supports for * reading and writing. This defines the constants to identify those file formats * that it supports as enums. The file type information should be provided when * defining the file spec to identify the kind of file this spec is going to represent. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public final class FileType { /** * Holds the string representing the file type. */ private String type=null; /** * Private constructor to make sure that FileType cannot be instantiated from * outside of this class. * * @param type String representing the type of file. */ private FileType(String type) { this.type=type; } /** * Utility method returns the FileType object matches to the given value. * * @param type String representing the type of file. * * @return Returns the FileType references matches to the given values, null * if it cannot match to the available file types. */ public static FileType toFileType(String type) { if(FileType.FIXED_WIDTH_FLAT_FILE.toString().equalsIgnoreCase(type)) return FileType.FIXED_WIDTH_FLAT_FILE; else if(FileType.DELIMITED_FLAT_FILE.toString().equalsIgnoreCase(type)) return FileType.DELIMITED_FLAT_FILE; else if(FileType.XML_FILE.toString().equalsIgnoreCase(type)) return FileType.XML_FILE; else return null; } /** * Returns the string representation of the FileType which can be used to * defines the file spec. */ public String toString() { return type; } /** * Represents the fixed width flat files. The value to be used in the file * spec for this kind of file types is "fixed-width-flat". */ public static final FileType FIXED_WIDTH_FLAT_FILE = new FileType("fixed-width-flat"); /** * Represents the delimited flat files. The value to be used in the file * spec for this kind of file types is "delimited-flat". */ public static final FileType DELIMITED_FLAT_FILE = new FileType("delimited-flat"); /** * Represents the xml files. The value to be used in the file * spec for this kind of file types is "xml". */ public static final FileType XML_FILE = new FileType("xml"); } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:43
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/util In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20378 Added Files: FrameworkUtil.java JdbcConnectionHelper.java Log Message: no message --- NEW FILE: FrameworkUtil.java --- package org.jmonks.batch.framework.util; import java.util.Map; import java.util.StringTokenizer; import java.util.logging.Logger; import org.w3c.dom.Element; import org.w3c.dom.Node; import org.w3c.dom.NodeList; /** * <p> * FramworkUtil contains utility methods required by the framework. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public final class FrameworkUtil { private static Logger logger=Logger.getLogger(FrameworkUtil.class.getName()); /** * Private constructor to make sure it will not be instantiated. */ private FrameworkUtil() { /** * Just to make sure this class will not be instantiated. */ } /** * <p> * Loads the property elements exists in the given element to the given map. * This looks for the <property> elements in the given element and looks * for the value of "key" attribute and the values of <property> element * and load them as key values pairs. * </p> * <p> * Property elements should be in the following format.<br><br> * <property key="some-config-key1">some-config-value1</property> * </p> * * @param configElement DOM Element consists of <property> elements. * @param propertyMap Map to be loaded with property key values. * * @throws IllegalArgumentException If either configElement or propertyMap is null. */ public static void loadPropertiesFromElementToMap(Element configElement,Map propertyMap) { logger.entering(FrameworkUtil.class.getName(),"loadPropertiesFromElementToMap"); if(configElement==null) throw new IllegalArgumentException("Input configuration element configElement cannot be null"); if(propertyMap==null) throw new IllegalArgumentException("Input properties map propertyMap cannot be null"); NodeList propertyElements=configElement.getElementsByTagName("property"); for(int i=0;i<propertyElements.getLength();i++) { Element propertyElement=(Element)propertyElements.item(i); String key=propertyElement.getAttribute("key"); String value=""; NodeList propertyValueNodes=propertyElement.getChildNodes(); for(int j=0;j<propertyValueNodes.getLength();j++) { Node propertyValueNode=propertyValueNodes.item(j); if(propertyValueNode.getNodeType()==Node.TEXT_NODE || propertyValueNode.getNodeType()==Node.CDATA_SECTION_NODE) { value=propertyValueNode.getNodeValue(); break; } } propertyMap.put(key,value); } logger.exiting(FrameworkUtil.class.getName(),"loadPropertiesFromElementToMap"); } /** * <p> * Loads the property key value pairs exists in the given string to the given map. * This looks for the properties in concantenated by ":" and each property in turn * concatenated by "=". * </p> * <p> * Property elements should be in the following format.<br><br> * key1=value1:key2=value2 * </p> * * @param propertiesString String consists of properties. * @param propertyMap Map to be loaded with property key values. * * @throws IllegalArgumentException If either propertiesString or propertyMap is null. */ public static void loadPropertiesFromStringToMap(String propertiesString,Map propertyMap) { logger.entering(FrameworkUtil.class.getName(),"loadPropertiesFromStringToMap"); if(propertiesString==null) throw new IllegalArgumentException("Input properties string cannot be null"); if(propertyMap==null) throw new IllegalArgumentException("Input properties map propertyMap cannot be null"); StringTokenizer propertiesTokenizer=new StringTokenizer(propertiesString,":"); while(propertiesTokenizer.hasMoreTokens()) { String property=propertiesTokenizer.nextToken(); if(property!=null && !"".equals(property.trim())) { StringTokenizer propertyTokenizer=new StringTokenizer(property,"="); String key=propertyTokenizer.nextToken(); String value=propertyTokenizer.nextToken(); propertyMap.put(key,value); } } logger.exiting(FrameworkUtil.class.getName(),"loadPropertiesFromStringToMap"); } } --- NEW FILE: JdbcConnectionHelper.java --- /* * JdbcConnectionHelper.java * * Created on September 9, 2006, 10:57 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.util; import java.sql.Connection; import java.sql.DriverManager; import java.util.Map; import org.apache.log4j.Logger; /** * JdbcConnectionHelper helps the jdbc repository and db configuration factory * to create the jdbc connections with the given configuration map and closes the * connections. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public final class JdbcConnectionHelper { /** * Constant defines the property name for jdbc driver class name which is * <code>jdbc-driver-class-name</code>. */ public static final String PROPERTY_JDBC_DRIVER_CLASS_NAME = "jdbc-driver-class-name"; /** * Constant defines the property name for database URL which is * <code>jdbc-url</code>. */ public static final String PROPERTY_JDBC_URL = "jdbc-url"; /** * Constant defines the property name for database user name which is * <code>username</code>. */ public static final String PROPERTY_DATABASE_USER_NAME = "username"; /** * Constant defines the property name for database password which is * <code>password</code>. */ public static final String PROPERTY_DATABASE_PASSWORD = "password"; private static Logger logger=Logger.getLogger(JdbcConnectionHelper.class); /** * To restrict the instantiation of JdbcConnectionHelper */ private JdbcConnectionHelper() { } /** * Get the required properties from the given configMap and * establishes the connection. Returns null, if it cannot establish the * connection. * * @param configMap Map contains the properties required to create the connection. * * @return Returns the database connction, null, if it cannot establish the connection. */ public static synchronized Connection getConnection(Map configMap) { logger.trace("Entering getConnection"); logger.info("Connection configuration : " + configMap); Connection connection=null; String driverClassName=(String)configMap.get(JdbcConnectionHelper.PROPERTY_JDBC_DRIVER_CLASS_NAME); String databaseURL=(String)configMap.get(JdbcConnectionHelper.PROPERTY_JDBC_URL); String userName=(String)configMap.get(JdbcConnectionHelper.PROPERTY_DATABASE_USER_NAME); String password=(String)configMap.get(JdbcConnectionHelper.PROPERTY_DATABASE_PASSWORD); if(driverClassName==null || "".equals(driverClassName) || databaseURL==null || "".equals(databaseURL) || userName==null || "".equals(userName) || password==null || "".equals(password)) connection=null; else { try { Class.forName(driverClassName); connection=DriverManager.getConnection(databaseURL,userName, password); connection.setAutoCommit(false); logger.debug("Connection has been created successfully"); } catch(Exception exception) { /** * No need to worry about the specific exception. In any case need to return null. * So catching the generic one. */ exception.printStackTrace(); logger.error(exception.getMessage(), exception); connection=null; } } logger.trace("Exiting getConnection"); return connection; } /** * Closes the given connection. Doesn't throw any exceptions to the caller. */ public static void closeConnection(Connection connection) { try { if(connection!=null && !connection.isClosed()) connection.close(); logger.debug("Connection has been closed successfully"); } catch(Exception exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:34
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/repository/jdbc In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20339 Added Files: JdbcRepository.java Log Message: no message --- NEW FILE: JdbcRepository.java --- /* * JdbcRepository.java * * Created on September 9, 2006, 10:34 AM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.jdbc; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Timestamp; import java.sql.Types; import java.util.HashMap; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobStatistics; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.Repository; import org.jmonks.batch.framework.util.JdbcConnectionHelper; /** * <p> * JdbcRepository implementation of repository uses any database that can be accessed by using * the JDBC technology. This looks for the properties "jdbc-driver-class-name","jdbc-url", * "username" and "password" in repository config in framework configuration contains the * values to establish the connection to implement the Repository. * <br> * Following is the example of repository configuration. * <br><br> * <pre> * <repository-config repository-class-name="org.jmonks.batch.framework.repository.jdbc.JdbcRepository"> * <property key="jdbc-driver-class-name">oracle.jdbc.driver.OracleDriver</property> * <property key="jdbc-url">jdbc:oracle:thin:@hostname:1521:instancename</property> * <property key="username">scott</property> * <property key="password">tiger</property> * </repository-config> * </pre> * <br><br> * The database user specified in the configuration should have update, insert & delete * privileges on the following objects in the database. * <table border="1"> * <tr><td>JOB_DATA_TRANSFER</td></tr> * <tr><td>JOB_STATISTICS</td></tr> * <tr><td>JOB_MGMT_MNTR_INFO</td></tr> * </table> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class JdbcRepository extends Repository { private static Logger logger=Logger.getLogger(JdbcRepository.class); /** * Map holds the jdbc configuration. */ protected Map jdbcConfigMap=null; /** * <p> * Receives the jdbc configuration in a map from the factory method and * validates the given configuration is valid by making a connection to the database. * </p> * * @param configProps Map contains the jdbc configuration. * * @throws ConfigurationException If not able to get the connection by using the properties * given in the map. */ protected void init(Map configProps) { logger.trace("Entering init"); this.jdbcConfigMap=new HashMap(configProps); /** * Do the basic validation to make sure we can get a connection using the given configuration. */ Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Unable to get the jdbc " + "connection with the properties " + this.jdbcConfigMap +"."); else JdbcConnectionHelper.closeConnection(connection); logger.trace("Exiting init"); } /** * @see org.jmonks.batch.framework.repository.Repository#clearDataTransferredToNextJob() */ public boolean clearDataTransferredFromThisJob() { logger.trace("Entering clearDataTransferredToNextJob"); boolean dataCleared=false; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) dataCleared=false; else { try { PreparedStatement statement=connection.prepareStatement("delete from job_data_transfer where source_job_name=?"); statement.setString(1, this.jobName); statement.executeUpdate(); statement.close(); connection.commit(); dataCleared=true; } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while trying to clear the data for requested job = " + this.jobName + ". Message = " + exception.getMessage(), exception); dataCleared=false; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting clearDataTransferredToNextJob"); return dataCleared; } /** * @see org.jmonks.batch.framework.repository Repository#sendDataToNextJob(String,String,Object) */ public boolean sendDataToNextJob(String dataIdentifier, String nextJobName, final Object data) { logger.trace("Entering sendDataToNextJob);"); logger.debug("data identifier = " + dataIdentifier + " next job name = " + nextJobName + " data = " + data); if(dataIdentifier==null || nextJobName==null || data==null) throw new IllegalArgumentException("No arguments cannot be null to the sendDataToNextJob method."); boolean dataSaved=false; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) dataSaved=false; else { try { PreparedStatement deleteStatement=connection.prepareStatement("delete from job_data_transfer where source_job_name=? and destination_job_name=? and job_data_identifier=?"); deleteStatement.setString(1, this.jobName); deleteStatement.setString(2, nextJobName); deleteStatement.setString(3, dataIdentifier); deleteStatement.executeUpdate(); deleteStatement.close(); logger.debug("Data has been cleared before loading = " + this.jobName + " " + nextJobName + " " + dataIdentifier); PreparedStatement insertStatement=connection.prepareStatement("insert into job_data_transfer (source_job_name,destination_job_name,job_data_identifier,job_data) values (?,?,?,?)"); insertStatement.setString(1, this.jobName); insertStatement.setString(2, nextJobName); insertStatement.setString(3, dataIdentifier); byte[] objectByteArray=this.serializeObjectIntoByteArray(data); insertStatement.setBinaryStream(4, new ByteArrayInputStream(objectByteArray), objectByteArray.length); insertStatement.executeUpdate(); insertStatement.close(); logger.debug("Data has been loaded = " + this.jobName + " " + nextJobName + " " + dataIdentifier); connection.commit(); dataSaved=true; } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while trying to send the data from " + this.jobName + " to job = " + nextJobName + " with the identifier " + dataIdentifier + ". Message = " + exception.getMessage(), exception); dataSaved=false; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting sendDataToNextJob"); return dataSaved; } /** * @see org.jmonks.batch.framework.repository.Repository#getDataFromPreviousJob(String,String) */ public Object getDataFromPreviousJob(String dataIdentifier, String previousJobName) { logger.trace("Entering getDataFromPreviousJob"); logger.debug("data identifier = " + dataIdentifier + " previous job name = " + previousJobName); if(dataIdentifier==null || previousJobName==null) throw new IllegalArgumentException("Data identifer and previous job name cannot be null " + "to get the data from previous job."); Object data=null; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) data=null; else { try { PreparedStatement statement=connection.prepareStatement("select job_data from job_data_transfer where source_job_name=? and destination_job_name=? and job_data_identifier=?"); statement.setString(1, previousJobName); statement.setString(2, this.jobName); statement.setString(3, dataIdentifier); ResultSet rs=statement.executeQuery(); if(rs.next()) { InputStream inputStream=rs.getBinaryStream(1); data=deserializeObjectFromInputStream(inputStream); } else data=null; rs.close(); statement.close(); } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while trying to send the data from " + previousJobName + " to job = " + this.jobName + " with the identifier " + dataIdentifier + ". Message = " + exception.getMessage(), exception); data=null; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting getDataFromPreviousJob"); return data; } /** * @see org.jmonks.batch.framework.repository.Repository#logStatistics(org.jmonks.batch.framework.JobStatistics) */ public boolean logStatistics(final JobStatistics statistics) { logger.trace("Entering logStatistics"); if(statistics==null) throw new IllegalArgumentException("Null statistic objects will not be saved in repository"); if(!statistics.getJobname().equals(this.jobName)) throw new IllegalArgumentException("Statistics object is not related to the job configured for the repository."); logger.debug("Logging = " + statistics.toString()); boolean logged=false; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) logged=false; else { try { PreparedStatement statement=connection.prepareStatement("insert into job_statistics (job_name,job_start_time,job_end_time," + "job_exit_code,job_exit_reason,records_processed,memory_usage) values(?,?,?,?,?,?,?)"); statement.setString(1, statistics.getJobname()); statement.setTimestamp(2, new Timestamp(statistics.getStartTime().getTime())); statement.setTimestamp(3, new Timestamp(statistics.getEndTime().getTime())); statement.setInt(4, statistics.getExitCode().getCode()); if(statistics.getExitCode().getMessage()!=null) statement.setString(5, statistics.getExitCode().getMessage()); else statement.setNull(5,Types.VARCHAR); statement.setLong(6, statistics.getRecordsProcessed()); statement.setLong(7, statistics.getMaxMemoryUsage()); statement.executeUpdate(); statement.close(); connection.commit(); logged=true; } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while logging the job statistics = " + statistics.toString() + ". Message = " + exception.getMessage(), exception); logged=false; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting logStatistics"); return logged; } /** * @see org.jmonks.batch.framework.repository.Repository#unregisterJobMgmtMntrInfo() */ public boolean unregisterJobMgmtMntrInfo() { logger.trace("Entering unregisterJobMgmtMntrInfo"); boolean unregistered=false; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) unregistered=false; else { try { PreparedStatement statement=connection.prepareStatement("delete from job_mgmt_mntr_info where job_name=?"); statement.setString(1, this.jobName); statement.executeUpdate(); statement.close(); connection.commit(); unregistered=true; } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while trying to lookup mgmt and mntr info = " + this.jobName); unregistered=false; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting unregisterJobMgmtMntrInfo"); return unregistered; } /** * @see org.jmonks.batch.framework.repository.Repository#registerJobMgmtMntrInfo(Object) */ public boolean registerJobMgmtMntrInfo(final Object registrationInfo) { logger.trace("Entering registerJobMgmtMntrInfo"); logger.debug(" registratinfo = " + registrationInfo); if(registrationInfo==null) throw new IllegalArgumentException("Job registration information cannot be null to register the job in repository."); boolean registered=true; Connection connection=JdbcConnectionHelper.getConnection(this.jdbcConfigMap); if(connection==null) registered=false; else { try { PreparedStatement deleteStatement=connection.prepareStatement("delete from job_mgmt_mntr_info where job_name=?"); deleteStatement.setString(1, this.jobName); deleteStatement.executeUpdate(); deleteStatement.close(); logger.debug("Cleared the existing registration info = " + this.jobName); PreparedStatement insertStatement=connection.prepareStatement("insert into job_mgmt_mntr_info (job_name,job_info) values (?,?)"); insertStatement.setString(1, this.jobName); byte[] objectByteArray=this.serializeObjectIntoByteArray(registrationInfo); insertStatement.setBinaryStream(2, new ByteArrayInputStream(objectByteArray), objectByteArray.length); insertStatement.executeUpdate(); insertStatement.close(); logger.debug("Registered mgmt mntr info = " + this.jobName); connection.commit(); registered=true; } catch(SQLException exception) { exception.printStackTrace(); logger.error("Exception while trying to register the mgmt mntr info = " + this.jobName+ ". Message = " + exception.getMessage(), exception); registered=false; } finally { JdbcConnectionHelper.closeConnection(connection); } } logger.trace("Exiting registerJobMgmtMntrInfo"); return registered; } /** * Serializes the given object and return that serialized data as byte array. * * @param object Object to be serialized. * * @return Returns the serialized data as byte array. */ protected byte[] serializeObjectIntoByteArray(Object object) { try { ByteArrayOutputStream bos=new ByteArrayOutputStream(); ObjectOutputStream oos=new ObjectOutputStream(bos); oos.writeObject(object); oos.close(); bos.flush(); bos.close(); return bos.toByteArray(); } catch(IOException exception) { exception.printStackTrace(); logger.error("Error while serializing the object. Message = " + exception.getMessage(), exception); return null; } } /** * Deserializes the given inputstream and return the object. * * @param inputStream InputStream to be deserialized. * * @return Returns the deserialized object. */ protected Object deserializeObjectFromInputStream(InputStream inputStream) { try { ObjectInputStream ois=new ObjectInputStream(inputStream); Object data=ois.readObject(); ois.close(); return data; } catch(ClassNotFoundException exception) { exception.printStackTrace(); logger.error("Error while deserializing the object. Message = " + exception.getMessage(), exception); return null; } catch(IOException exception) { exception.printStackTrace(); logger.error("Error while deserializing the object. Message = " + exception.getMessage(), exception); return null; } } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:28
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/repository/db4o In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20326 Added Files: ClientServerDb4oRepository.java Db4oJobControllerHolder.java Db4oJobControllerHolderPredicate.java Db4oJobDataTransferHolder.java Db4oJobMgmtMntrInfoHolder.java Db4oJobMgmtMntrInfoHolderPredicate.java Db4oJobStatisticsPredicate.java Db4oRepository.java Log Message: no message --- NEW FILE: Db4oJobControllerHolder.java --- /* * Db4oJobControllerHolder.java * * Created on March 15, 2006, 10:52 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; import org.jmonks.batch.framework.controller.JobController; /** * Db4oJobControllerHolder holds the job controller object and job name. * This class will be accessed by only Db4oRepository. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobControllerHolder { /** * Holds the job name */ private String jobName=null; /** * Holds the job controller object */ private JobController jobController=null; /** * Constructor takes job name and job controller to constructs the object. */ Db4oJobControllerHolder(String jobName,JobController jobController) { this.jobName=jobName; this.jobController=jobController; } /** * Returns the jobName. */ public String getJobName() { return this.jobName; } /** * Returns the mgmt & mntr object. */ public JobController getJobController() { return this.jobController; } /** * <p> * Returns the string representation of Db4oJobControllerHolder class in the format * <br> {Db4oJobControllerHolder [jobName = value] [mgmtMntrInfo = value]} * </p> * * @return Returns the string representation of Db4oJobControllerHolder. */ public String toString() { StringBuffer stringValue=new StringBuffer("{Db4oJobControllerHolder "); stringValue.append("[jobName = " + this.jobName + "]"); stringValue.append("[jobController = " + this.jobController + "]"); stringValue.append("}"); return stringValue.toString(); } } --- NEW FILE: Db4oJobControllerHolderPredicate.java --- /* * Db4oJobControllerHolderPredicate.java * * Created on March 16, 2006, 7:32 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; import com.db4o.query.Predicate; /** * Db4oJobControllerHolderPredicate used to find the Db4oJobControllerHolder * objects in Db4o database using native query. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobControllerHolderPredicate extends Predicate { /** * job name to find the controller holder objects for. */ private String jobName=null; /** * Creates a new instance of Db4oJobControllerHolderPredicate with the job name. */ Db4oJobControllerHolderPredicate(String jobName) { if(jobName==null) throw new IllegalArgumentException("job name cannot be null for Db4oJobControllerHolder Predicate."); this.jobName=jobName; } /** * Method to run the the native query. This receives the Db4oJobControllerHolder * objects and look for the job name this predicate initialized with. */ public boolean match(Db4oJobControllerHolder holder) { if(jobName.equals(holder.getJobName())) return true; else return false; } } --- NEW FILE: Db4oRepository.java --- package org.jmonks.batch.framework.repository.db4o; import com.db4o.Db4o; import com.db4o.ObjectContainer; import com.db4o.ObjectSet; import com.db4o.query.Predicate; import java.io.File; import java.io.IOException; import java.util.HashMap; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.Repository; import org.jmonks.batch.framework.JobStatistics; /** * <p> * This implementation of repository uses the DB4O database as the repository. * It expects the property repository-location present in the repository configuration * in framework-cofig.xml file. The value of this property should be a valid directory name. * If directory doesnt exist, it will try to create directory with that path. * <br><br> * <pre> * <repository-config repository-class-name="org.jmonks.batch.framework.repository.db4o.Db4oRepository"> * <property key="db4o-filename">/batchserver/repository/batchserver_repository.db</property> * </repository-config> * </pre> * <br> * Here repository-location defines where the DB4O database needs to create * the repository file. By default framework will be configured with the values * shown in the above XML configuration snippet. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oRepository extends Repository { private static Logger logger=Logger.getLogger(Db4oRepository.class); /** * Property identifies the db4o file name which <code>db4o-filename</code>. */ public static final String PROPERTY_DB4O_FILENAME = "db4o-filename"; /** * Map holds the configuration properties of repository. */ protected Map repositoryConfigProperties=null; /** *<p> * <i> * Do not use this constructor to instantiate Db4oRepository directly. * Access the repository only through the JobContext reference. * This constructor has been provided to make sure this should be * instantiable and accessible to the Repository class. * </i> * </p> */ public Db4oRepository() { } /** * <p> * Initializes the Db4oRepository by accepting the map consist of properties * needed to setup the repository. This make sure repository-location property * has been defined and the value defined for this property is valid. * </p> * @param configProps Map contains the configuration properties. * * @throws ConfigurationException If repository-location property is not defined and * the value specified is invalid. * @throws IllegalArgumentException If given configProps is null. */ protected void init(Map configProps) { logger.trace("Entering init"); if(configProps==null) throw new IllegalArgumentException("Map to call the init cannot be null."); this.repositoryConfigProperties=new HashMap(configProps); String db4oFileName=(String)this.repositoryConfigProperties.get(Db4oRepository.PROPERTY_DB4O_FILENAME); if(db4oFileName==null || "".equals(db4oFileName)) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "db4o-filename is required for Db4oReposiotry implementation."); else { File repositoryFile=new File(db4oFileName); if(!repositoryFile.exists()) { try { boolean created=repositoryFile.createNewFile(); if(!created) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Attempt to create Db4o filename " + db4oFileName + " is failed."); } catch(IOException exception) { exception.printStackTrace(); logger.fatal("Exception while trying to create not nonexistent file = " + db4oFileName, exception); throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Exception while trying to create non existing Db4o filename " + db4oFileName + ". Message = " + exception.getMessage()); } } else if(repositoryFile.isDirectory()) { throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Db4o file name " + db4oFileName + " defined is a directory. File name is expected."); } /** * Just to make sure, would be able to setup the repository. */ ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Unable to create the file " + repositoryFile.getAbsoluteFile() + " to setup repository."); container.close(); } logger.trace("Exiting init"); } /** * Creates the container by looking at the configuration from given Map. * * @configProps Configuration properties defined in repository configuration. * * @return Returns the Db4o object container reference. */ protected ObjectContainer createContainer(Map configProps) { String db4oFileName=(String)configProps.get(Db4oRepository.PROPERTY_DB4O_FILENAME); ObjectContainer container=Db4o.openFile(db4oFileName); return container; } /** * @see org.jmonks.batch.framework.repository Repository#sendDataToNextJob(String,String,Object) */ public boolean sendDataToNextJob(String dataIdentifier, String nextJobName, final Object data) { logger.trace("Entering sendDataToNextJob);"); logger.debug("data identifier = " + dataIdentifier + " next job name = " + nextJobName + " data = " + data); if(dataIdentifier==null || nextJobName==null || data==null) throw new IllegalArgumentException("No arguments cannot be null to the sendDataToNextJob method."); final String finalDataIdentifier=dataIdentifier; final String finalSourceJobName=this.jobName; final String finalNextJobName=nextJobName; boolean dataSaved=false; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) dataSaved=false; else { /** * Delete the entries exist with the identifier, source and target job names. */ ObjectSet dataTransferredSet=container.query(new Predicate() { public boolean match(Db4oJobDataTransferHolder jobDataTransfer) { if(finalDataIdentifier.equals(jobDataTransfer.getDataIdentifier()) && finalSourceJobName.equals(jobDataTransfer.getSourceJobName()) && finalNextJobName.equals(jobDataTransfer.getTargetJobName())) return true; else return false; } }); while(dataTransferredSet.hasNext()) { container.delete(dataTransferredSet.next()); } container.commit(); logger.trace("deleted the data identified by " + dataIdentifier + " from the job name " + this.jobName + " to " + nextJobName); /** * Add the new entry. */ Db4oJobDataTransferHolder dataTransfer=new Db4oJobDataTransferHolder(dataIdentifier, this.jobName, nextJobName, data); container.set(dataTransfer); container.commit(); container.close(); logger.trace("data has been added with the identifier " + dataIdentifier + " from job name " + this.jobName); dataSaved=true; } logger.trace("Exiting sendDataToNextJob"); return dataSaved; } /** * @see org.jmonks.batch.framework.repository.Repository#getDataFromPreviousJob(String,String) */ public Object getDataFromPreviousJob(String dataIdentifier, String previousJobName) { logger.trace("Entering getDataFromPreviousJob"); logger.debug("data identifier = " + dataIdentifier + " target job name = " + previousJobName); if(dataIdentifier==null || previousJobName==null) throw new IllegalArgumentException("Data identifer and previous job name cannot be null " + "to get the data from previous job."); final String finalDataIdentifier=dataIdentifier; final String finalPreviousJobName=previousJobName; final String finalTargetJobName=this.jobName; Object data=null; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) data=null; else { ObjectSet dataTransferResultSet=container.query(new Predicate() { public boolean match(Db4oJobDataTransferHolder jobDataTransfer) { if(finalDataIdentifier.equals(jobDataTransfer.getDataIdentifier()) && finalPreviousJobName.equals(jobDataTransfer.getSourceJobName()) && finalTargetJobName.equals(jobDataTransfer.getTargetJobName())) return true; else return false; } }); if(dataTransferResultSet.hasNext()) { data=((Db4oJobDataTransferHolder)dataTransferResultSet.next()).getData(); logger.debug("data identified by " + dataIdentifier + " has been found from the source job name " + previousJobName); } else data=null; container.close(); } logger.trace("Exiting getDataFromPreviousJob"); return data; } /** * @see org.jmonks.batch.framework.repository.Repository#clearDataTransferredToNextJob() */ public boolean clearDataTransferredFromThisJob() { logger.trace("Entering clearDataTransferredToNextJob"); final String finalJobName=this.jobName; boolean dataCleared=false; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) dataCleared=false; else { ObjectSet dataTransferResultSet=container.query(new Predicate() { public boolean match(Db4oJobDataTransferHolder jobDataTransfer) { if(finalJobName.equals(jobDataTransfer.getSourceJobName())) { return true; } else return false; } }); while(dataTransferResultSet.hasNext()) container.delete(dataTransferResultSet.next()); logger.debug(jobName + " data has been cleared from the repository."); container.commit(); container.close(); dataCleared=true; } logger.trace("Exiting clearDataTransferredToNextJob"); return dataCleared; } /** * @see org.jmonks.batch.framework.repository.Repository#registerJobMgmtMntrInfo(Object) */ public boolean registerJobMgmtMntrInfo(final Object registrationInfo) { logger.trace("Entering registerJobMgmtMntrInfo"); logger.debug("registratinfo = " + registrationInfo); if(registrationInfo==null) throw new IllegalArgumentException("Job registration information cannot be null to register the job in repository."); boolean registered=true; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) registered=false; else { ObjectSet mgmtMntrInfoHolderResultSet=container.query(new Db4oJobMgmtMntrInfoHolderPredicate(this.jobName)); while(mgmtMntrInfoHolderResultSet.hasNext()) container.delete(mgmtMntrInfoHolderResultSet.next()); container.commit(); logger.trace(this.jobName + " previous mgmt and mntr information has been deleted"); container.set(new Db4oJobMgmtMntrInfoHolder(this.jobName, registrationInfo)); container.commit(); container.close(); logger.trace(this.jobName + " mgmt and mntr information has been added to the repository"); registered=true; } logger.trace("Exiting registerJobMgmtMntrInfo"); return registered; } /** * @see org.jmonks.batch.framework.repository.Repository#unregisterJobMgmtMntrInfo() */ public boolean unregisterJobMgmtMntrInfo() { logger.trace("Entering unregisterJobMgmtMntrInfo"); boolean unregistered=false; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) unregistered=false; else { ObjectSet mgmtMntrInfoHolderResultSet=container.query(new Db4oJobMgmtMntrInfoHolderPredicate(this.jobName)); if(mgmtMntrInfoHolderResultSet.hasNext()) { container.delete(mgmtMntrInfoHolderResultSet.next()); container.commit(); logger.debug(this.jobName + " mgmt and mntr information has been deleted."); unregistered=true; } else unregistered=false; container.close(); } logger.trace("Exiting unregisterJobMgmtMntrInfo"); return unregistered; } /** * * @see org.jmonks.batch.framework.repository.Repository#logStatistics(org.jmonks.batch.framework.JobStatistics) */ public boolean logStatistics(final JobStatistics statistics) { logger.trace("Entering logStatistics"); if(statistics==null) throw new IllegalArgumentException("Null statistic objects will not be saved in repository"); if(!statistics.getJobname().equals(this.jobName)) throw new IllegalArgumentException("Statistics object is not related to the job configured for this repository."); boolean logged=false; ObjectContainer container=this.createContainer(this.repositoryConfigProperties); if(container==null) logged=false; else { container.set(statistics); container.commit(); logger.debug(statistics.getJobname() + " has been logged."); logged=true; container.close(); } logger.trace("Exiting logStatistics"); return logged; } } --- NEW FILE: Db4oJobMgmtMntrInfoHolder.java --- /* * Db4oJobMgmtMntrInfoHolder.java * * Created on March 15, 2006, 10:54 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; /** * Db4oJobMgmtMntrInfoHolder holds the job mgmt&mntr information and job name. * This will be used only by Db4oRepository. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobMgmtMntrInfoHolder { /** * Holds the job name */ private String jobName=null; /** * Holds the mgmtMntrInfo object */ private Object mgmtMntrInfo=null; /** * Constructor takes job name and mgmt & mntr info and constructs the object. */ Db4oJobMgmtMntrInfoHolder(String jobName,Object mgmtMntrInfo) { this.jobName=jobName; this.mgmtMntrInfo=mgmtMntrInfo; } /** * Returns the jobName. */ public String getJobName() { return this.jobName; } /** * Returns the mgmt & mntr object. */ public Object getMgmtMntrInfo() { return this.mgmtMntrInfo; } /** * <p> * Returns the string representation of Db4oJobMgmtMntrInfoHolder class in the format * <br> {Db4oJobMgmtMntrInfoHolder [jobName = value] [mgmtMntrInfo = value]} * </p> * * @return Returns the string representation of Db4oJobMgmtMntrInfoHolder. */ public String toString() { StringBuffer stringValue=new StringBuffer("{Db4oJobMgmtMntrInfoHolder "); stringValue.append("[jobName = " + this.jobName + "]"); stringValue.append("[mgmtMntrInfo = " + this.mgmtMntrInfo + "]"); stringValue.append("}"); return stringValue.toString(); } } --- NEW FILE: Db4oJobMgmtMntrInfoHolderPredicate.java --- /* * Db4oJobMgmtMntrInfoHolderPredicate.java * * Created on March 16, 2006, 7:23 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; import com.db4o.query.Predicate; /** * Db4oJobMgmtMntrInfoHolderPredicate used to find the Db4oJobMgmtMntrInfoHolder * objects in Db4o database using native query. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobMgmtMntrInfoHolderPredicate extends Predicate { /** * job name to find the job mgmt and mntr info objects for. */ private String jobName=null; /** * Creates a new instance of Db4oJobMgmtMntrInfoHolderPredicate with the job name. */ Db4oJobMgmtMntrInfoHolderPredicate(String jobName) { if(jobName==null) throw new IllegalArgumentException("job name cannot be null"); this.jobName=jobName; } /** * Method to run the the native query. This receives the Db4oJobMgmtMntrInfoHolder * objects and look for the job name this predicate initialized with. */ public boolean match(Db4oJobMgmtMntrInfoHolder holder) { if(jobName.equals(holder.getJobName())) return true; else return false; } } --- NEW FILE: Db4oJobStatisticsPredicate.java --- /* * Db4oJobStatisticsPredicate.java * * Created on March 16, 2006, 7:41 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; import com.db4o.query.Predicate; import org.jmonks.batch.framework.JobStatistics; /** * Db4oJobStatisticsPredicate used to find the JobStatistics * objects in Db4o database using native query. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobStatisticsPredicate extends Predicate { /** * job name to find the job statistics objects for. */ private String jobName=null; /** * Creates a new instance of Db4oJobStatisticsPredicate with the given job name. */ Db4oJobStatisticsPredicate(String jobName) { if(jobName==null) throw new IllegalArgumentException("job name cannot be null"); this.jobName=jobName; } /** * Method to run the the native query. This receives the JobStatistics * objects and look for the job name this predicate initialized with. */ public boolean match(JobStatistics statistics) { if(jobName.equals(statistics.getJobname())) return true; else return false; } } --- NEW FILE: ClientServerDb4oRepository.java --- /* * ClientServerDb4oRepository.java * * Created on September 14, 2006, 11:13 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; import com.db4o.Db4o; import com.db4o.ObjectContainer; import java.io.IOException; import java.util.HashMap; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.config.ConfigurationException; /** * <p> * ClientServerDb4oRepository access the Db4o Server and use it as a repository. * This requires the db4o server configuration in the repository configuration. * Framework provides a utility shell script to start the server. Following is an * example repository configuration to use Db4o in client and server mode. * <br><br> * <pre> * <repository-config repository-class-name="org.jmonks.batch.framework.repository.db4o.ClientServerDb4oRepository"> * <property key="db4o-server-name">server name or IP Address</property> * <property key="db4o-server-port">4545</property> * <property key="db4o-server-username">scott</property> * <property key="db4o-server-password">tiger</property> * </repository-config> * </pre> * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class ClientServerDb4oRepository extends Db4oRepository { private static Logger logger=Logger.getLogger(ClientServerDb4oRepository.class); /** * Property identifies the Db4o Server name which is <code>db4o-server-name</code>. */ public static final String PROPERTY_DB4O_SERVER_NAME = "db4o-server-name"; /** * Property identifies the Db4o Server port which is <code>db4o-server-port</code>. */ public static final String PROPERTY_DB4O_SERVER_PORT = "db4o-server-port"; /** * Property identifies the Db4o Server user name which is <code>db4o-server-username</code>. */ public static final String PROPERTY_DB4O_SERVER_USERNAME = "db4o-server-username"; /** * Property identifies the Db4o Server password which is <code>db4o-server-password</code>. */ public static final String PROPERTY_DB4O_SERVER_PASSWORD = "db4o-server-password"; /** * Enables the creation of the ClientServerDb4oRepository instance from the factory method. */ public ClientServerDb4oRepository() { } /** * <p> * Initializes the ClientServerDb4oRepository by accepting the map consist of properties * needed to access Db4o server. This make sure required properties like * <code>db4o-server-name</code>, <code>db4o-server-port</code>, * <code>db4o-server-username</code> and <code>db4o-server-password</code> have been defined * and the value defined for this property is valid. * </p> * * @param configProps Map contains the configuration properties. * * @throws ConfigurationException If any one of the required properties are not defined and * the value specified is invalid. * @throws IllegalArgumentException If given configProps is null. */ protected void init(Map configProps) { logger.trace("Entering init"); if(configProps==null) throw new IllegalArgumentException("Map to call the init cannot be null."); super.repositoryConfigProperties=new HashMap(configProps); String db4oServerName=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_NAME); String db4oServerPort=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_PORT); String db4oServerUserName=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_USERNAME); String db4oServerPassword=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_PASSWORD); if(db4oServerName==null || "".equals(db4oServerName) || db4oServerPort==null || "".equals(db4oServerPort) || db4oServerUserName==null || "".equals(db4oServerUserName) || db4oServerPassword==null || "".equals(db4oServerPassword)) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Required properties to access Db4o server are missing."); else { /** * Just to make sure, would be able to setup the repository. */ ObjectContainer container=createContainer(super.repositoryConfigProperties); if(container==null) throw new ConfigurationException(ConfigurationException.REPOSITORY_CONFIG, "Unable to access the db4o server using given properites = " + super.repositoryConfigProperties); container.close(); } logger.trace("Exiting init"); } /** * @see org.jmonks.batch.framework.repository.db4o.Db4oRepository#createContainer(Map) */ protected ObjectContainer createContainer(Map configProps) { ObjectContainer container=null; try { String db4oServerName=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_NAME); int db4oServerPort=Integer.parseInt((String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_PORT)); String db4oServerUserName=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_USERNAME); String db4oServerPassword=(String)this.repositoryConfigProperties.get(ClientServerDb4oRepository.PROPERTY_DB4O_SERVER_PASSWORD); container=Db4o.openClient(db4oServerName, db4oServerPort, db4oServerUserName, db4oServerPassword); } catch(IOException exception) { exception.printStackTrace(); logger.fatal("IOException while opening the client with the configuration = " + configProps.toString(), exception); } return container; } } --- NEW FILE: Db4oJobDataTransferHolder.java --- /* * Db4oJobDataTransferHolder.java * * Created on March 15, 2006, 10:39 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.repository.db4o; /** * <p> * Db4oJobDataTransferHolder holds the data being transferred between the jobs. * This will not have any functionality except holding the data and * this will be used to store in DB4O database. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class Db4oJobDataTransferHolder { /** * Holds the identifier by which jobs will communicated. */ private String dataIdentifier; /** * Source job name. */ private String sourceJobName; /** * Target job name, where data to be sent. */ private String targetJobName; /** * Data object. */ private Object data; /** * Constructor with all the argument to make this class immutable, * once it got created. */ Db4oJobDataTransferHolder(String dataIdentifer,String sourceJobName,String targetJobName,Object data) { this.dataIdentifier=dataIdentifer; this.sourceJobName=sourceJobName; this.targetJobName=targetJobName; this.data=data; } /** * Returns the data identifier. */ public String getDataIdentifier() { return this.dataIdentifier; } /** * Returns the source job name. */ public String getSourceJobName() { return this.sourceJobName; } /** * Returns the target job name. */ public String getTargetJobName() { return this.targetJobName; } /** * Returns the job data. */ public Object getData() { return this.data; } /** * <p> * Returns the string representation of Db4oJobDataTransferHolder class in the format * <br> {Db4oJobDataTransferHolder [dataIdentifier = value] [sourceJN = value] * [targetJN = value] [data = value]} * </p> * * @return Returns the string representation of Db4oJobDataTransferHolder. */ public String toString() { StringBuffer stringValue=new StringBuffer("{Db4oJobDataTransferHolder "); stringValue.append("[dataIdentifier = " + this.dataIdentifier + "]"); stringValue.append("[sourceJobName = " + this.sourceJobName + "]"); stringValue.append("[targetJobName = " + this.targetJobName + "]"); stringValue.append("[Data = " + this.data + "]"); stringValue.append("}"); return stringValue.toString(); } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:12
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/management/jmxmp In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv20312 Added Files: JMXMPConnectorHelper.java RepositoryJMXMPConnectorHelper.java Log Message: no message --- NEW FILE: RepositoryJMXMPConnectorHelper.java --- package org.jmonks.batch.framework.management.jmxmp; import javax.management.remote.JMXConnectorServer; import javax.management.remote.JMXServiceURL; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.Repository; /** * <p> * This connector is based on JMX Messaging protocol and uses the framework * reposiotry as the lookup location to register and unregister the JMX Service URL. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class RepositoryJMXMPConnectorHelper extends JMXMPConnectorHelper { private static Logger logger=Logger.getLogger(RepositoryJMXMPConnectorHelper.class); /** * @see org.jmonks.batch.framework.management.JobConnectorHelper@init(java.util.Map) */ public void init(java.util.Map configProps) { logger.trace("Entering init") ; /** * Because this uses the repository, defined for this framework as the * lookup location, this implementation does not require any properties * in configuration file to initialize the lookup location. */ logger.trace("Exiting init") ; } /** * Registers the jmx connector server with the job name taken from job context in the repository * defined for this framework. Repository reference will also be taken from job context. * * @param jobContext Context of the job is being run. * @param jmxConnectorServer JMX connector server identifies where all the manager and montiro mbeans are configured. * * @return Returns true, if it successfully registred in repository, false, otherwise. * * @throws IllegalArgumentException If job name paramter or jmxConnectorServer parameter is null. */ public boolean registerConnectorServer(JobContext jobContext,JMXConnectorServer jmxConnectorServer) { logger.trace("Entering registerConnectorServer") ; if(jobContext==null) throw new IllegalArgumentException("job context cannot be null to registerd jmx service url."); if(jmxConnectorServer==null) throw new IllegalArgumentException("jmx connector server cannot be null to registerd jmx connector server."); /** * By default repository will be associated with the job. So we can ignore the jobName here. */ JMXServiceURL jmxServiceURL=jmxConnectorServer.getAddress(); boolean registered=jobContext.getRepository().registerJobMgmtMntrInfo(jmxServiceURL.toString()); logger.info(jobContext.getJobName() + " with the service url " + jmxServiceURL.toString() + " registered in repository = " + registered); logger.trace("Exiting registerConnectorServer") ; return registered; } /** * Unregisters the jmx connector server registered in repository with the * job name available in job context. * * @param jobContext Context of the job is begin run. * * @return Returns true, if it successfully unregistered, false otherwise. * * @throws IllegalArgumentException If job name paramter is null. */ public boolean unregisterConnectorServer(JobContext jobContext) { logger.trace("Entering unregisterConnectorServer"); if(jobContext==null) throw new IllegalArgumentException("job context cannot be null to unregister the service url."); /** * By default repository will be associated with the job. So we can ignore the jobName here. */ boolean unregistered=jobContext.getRepository().unregisterJobMgmtMntrInfo(); logger.debug(jobContext.getJobName() + " mgmt and mntr information unregistered = " + unregistered); logger.trace("Exiting unregisterConnectorServer") ; return unregistered; } } --- NEW FILE: JMXMPConnectorHelper.java --- /* * JMXMPConnectorHelper.java * * Created on March 22, 2006, 9:27 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.management.jmxmp; import java.io.IOException; import java.net.MalformedURLException; import java.util.HashMap; import javax.management.remote.JMXConnectorServer; import javax.management.remote.JMXConnectorServerFactory; import javax.management.remote.JMXServiceURL; import org.apache.log4j.Logger; import org.jmonks.batch.framework.management.JobConnectorHelper; /** * <p> * JMXMPConnectorHelper creates the JMX connector server based on the JMX Messaging * protocol and leaves the lookup location details to the class going to implement this * abstract class. The machine the job being run will be used as the host name and * the port will be selected automatically by the JMX connector server factory. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class JMXMPConnectorHelper extends JobConnectorHelper { private static Logger logger=Logger.getLogger(JMXMPConnectorHelper.class); /** * JMX Messaging Protocol URL to be used to create the connector server. */ private static final String JMXMP_SERVICE_URL = "service:jmx:jmxmp://"; /** * Creates the JMX connector server based on JMX Messaging protocol. This does * not defined the host name and port name in the URL. So, the machine job * being run will be used as the host and port will be automatically choosen. * * @return Returns the JMX connector server, null, if it cannot create the connector server. */ public JMXConnectorServer createConnectorServer() { JMXServiceURL jmxServiceURL=null; JMXConnectorServer jmxConnectorServer=null; try { jmxServiceURL=new JMXServiceURL(JMXMPConnectorHelper.JMXMP_SERVICE_URL); jmxConnectorServer=JMXConnectorServerFactory.newJMXConnectorServer(jmxServiceURL, new HashMap(), null); } catch(MalformedURLException exception) { exception.printStackTrace(); logger.error(exception.getMessage(), exception); } catch(IOException exception) { exception.printStackTrace(); logger.error(exception.getMessage(), exception); } return jmxConnectorServer; } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:07:04
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/management In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv19938 Added Files: JobConnectorHelper.java JobManagementAgent.java JobManager.java JobManagerMBean.java JobMonitor.java JobMonitorMBean.java JobNotification.java ProcessorState.java ProcessorStatus.java Log Message: no message --- NEW FILE: JobMonitorMBean.java --- package org.jmonks.batch.framework.management; /** * <p> * JobMonitorMBean interface defines the methods to instrument the job being run * to find out the statistics of the job and to find out all the processor states * running in the controller. This follows the JMX standard MBean paradigm to define * name of the MBean and methods instrument the job. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public interface JobMonitorMBean { /** * Gets the total number of records this job might try to process. * * @return Returns the total number of records this job might try to process. */ public long getExpectedRecordsCount(); /** * Gets the records processed by this job so far. * * @return Returns the number of records processed by this job so far. */ public long getProcessedRecordsCount(); /** * Returns the array of IDs assigned to be processors running under this controller. * * @return Returns the array of IDs being assigned by each processor. */ public String[] getProcessorIDList(); /** * Gets the state of the processor identified by the given processor ID. * This returns the state in a ProcessorState object. * * @return Returns the state of the requested processor. */ public ProcessorState getProcessorState(String processorID); } --- NEW FILE: ProcessorStatus.java --- package org.jmonks.batch.framework.management; /** * <p> * ProcessorStatus defines the statuses that processors can be in the system. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class ProcessorStatus { /** * Status indicates that processor is instantiated. */ public static ProcessorStatus INSTANTIATED = new ProcessorStatus("INSTANTIATED"); /** * Status indicates that processor is being initialized. */ public static ProcessorStatus INITIALIZING = new ProcessorStatus("INITIALIZING"); /** * Status indicates that processor is initailized. */ public static ProcessorStatus INITIALIZED = new ProcessorStatus("INITIALIZED"); /** * Status indicates that processor is running. */ public static ProcessorStatus RUNNING = new ProcessorStatus("RUNNING"); /** * Status indictes that processor is suspended. */ public static ProcessorStatus SUSPENDED = new ProcessorStatus("SUSPENDED"); /** * Status indicates the processor is stopped. */ public static ProcessorStatus STOPPED = new ProcessorStatus("STOPPED"); /** * Status indicates the processor is resumed. */ public static ProcessorStatus RESUMED = new ProcessorStatus("RESUMED"); /** * Status indicates the processor is being cleaned up. */ public static ProcessorStatus CLEANUP = new ProcessorStatus("CLEANUP"); /** * Status indicates the processor is finished. */ public static ProcessorStatus FINISHED = new ProcessorStatus("FINISHED"); /** * Varaible holds the status ID. */ private String statusID; /** * Constructor make sure all the constants have been defined inside this * class and takes the status ID. */ private ProcessorStatus(String statusID) { this.statusID=statusID; } /** * Returns the job status in string format. */ public String toString() { return this.statusID; } } --- NEW FILE: JobConnectorHelper.java --- /* * JobConnectorHelper.java * * Created on March 24, 2006, 8:04 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.management; import java.util.Map; import javax.management.remote.JMXConnectorServer; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.config.FrameworkConfig; /** * <p> * JobConnectorHelper defines the helper methods will be used in creating the * JMX connector server, registering and unregistering the JMX connector server * in lookup location and querying the registerd jobs in look up location and * creating the JMX connector clients. * Because of many possible implementations available in creating * the JMX connector server and lookup locations, this class defines the methods * to choose their own implementations. Mainly, Agent and client looks for the following * information and activities from the concrete implementations of the connector * helper class. * <br> * <ul> * <li>Create the JMX connector server. * <li>Register the given the JMX connector server in the lookup location with the given jobname. * <li>Unregister the JMX connector server available in lookup location with the given jobname. * <li>Get all the registered jobs in look up location as list. * <li>Get the JMX connector from the look up location for the requested job name. * </ul> * <br> * This class provides the factory method to return the configured JobConnectorHelper * instance by looking at the framework configuration and make sure only one * instance will be created for the job(jvm). * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class JobConnectorHelper { private static Logger logger=Logger.getLogger(JobConnectorHelper.class); /** * Singleton job connector helper instance. */ private static JobConnectorHelper jobConnectorHelper=null; /** * This factory method creates the correct implementation of JobConnectorHelper * based on the configuration defined in framework-config.xml file. * This looks for the following configuration and uses the class name provided * at the job-connector-helper-class-name attribute to create the helper * instance. * <br> * <pre> * <job-connector-config job-connector-helper-class-name="org.jmonks.batch.framework.management.jmxmp.repository.RepositoryJMXMPConnectorHelper"> * <property key="mgmt-mntr-config-key">mgmt-mntr-config-value</property> * </job-connector-config> * </pre> * * @return Returns the JobConnectorHelper implemenation. * * @throws ConfigurationException If configuration could not be found or * implementation class cannot be accessed or instantiated. */ public static synchronized JobConnectorHelper getJobConnectorHelper() { logger.trace("Entering getJobConnectorHelper"); if(jobConnectorHelper==null) { FrameworkConfig.JobConnectorConfig jobConnectorConfig=FrameworkConfig.getInstance().getJobConnectorConfig(); if(jobConnectorConfig==null) throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, "job connector configuration is not configured in framework configuration"); String jobConnectorHelperClassName=jobConnectorConfig.getJobConnectorHelperClassName(); logger.trace("Job connector helper class name = " + jobConnectorHelperClassName); try { jobConnectorHelper=(JobConnectorHelper)Class.forName(jobConnectorHelperClassName).newInstance(); jobConnectorHelper.init(jobConnectorConfig.getJobConnectorConfigProperties()); } catch(ClassNotFoundException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, exception.getMessage()); } catch(InstantiationException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, exception.getMessage()); } catch(IllegalAccessException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, exception.getMessage()); } } logger.trace("Exiting getJobConnectorHelper"); return jobConnectorHelper; } /** * This method will be called after instantiating the JobConnectorHelper to * initialize JobConnectorHelper with the properties needed to create JMX connector server * and initialize the lookup location needed to register and unregister the JMX connector servers. * * @param configProps Properties defined for this connector server as a map. * * @throws ConfigurationException If required properties for this connector server are missing. */ protected abstract void init(Map configProps); /** * Creates the JMX connector server based on the desired connector protocol. * * @return Returns the JMX connector server, null, if helper cannot create the connector server. */ public abstract JMXConnectorServer createConnectorServer(); /** * Registers the jmx connector server with the job name taken from job context in the desired lookup location. * * @param jobContext Context of the job is being run. * @param jmxConnectorServer JMX connector server where all the manager and monnitor mbeans are configured. * * @return Returns true, if it successfully registred in lookup location, false, otherwise. * * @throws IllegalArgumentException If job name paramter is null. */ public abstract boolean registerConnectorServer(JobContext jobContext,JMXConnectorServer connectorServer); /** * Unregisters the jmx connector server registered in desired lookup location with the * job name available in job context. * * @param jobContext Context of the job is being run. * * @return Returns true, if it successfully unregistered, false otherwise. * * @throws IllegalArgumentException If job name paramter is null. */ public abstract boolean unregisterConnectorServer(JobContext jobContext); } --- NEW FILE: JobManagerMBean.java --- package org.jmonks.batch.framework.management; import org.apache.log4j.Level; /** * <p> * This standard MBean interface provides the methods to instrument the * job controller and processors running under the job controller. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public interface JobManagerMBean { /** * Returns the status of the processor identified by the given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns the processor status. */ public ProcessorStatus getProcessorStatus(String processorID); /** * Controller should stop processing of the processor identified by the * given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor has been stopped, false otherwise. */ public boolean stop(String processorID); /** * Controller should suspend processing of the processor identified by the * give processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor has been suspended, false otherwise. */ public boolean suspend(String processorID); /** * Controller should resume processing of the processor identified by the * given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor can be resumed, false, otherwise. */ public boolean resume(String processorID); /** * Gets the log level of the given logger name. * * @param loggerName Name of the logger wants to find the log level. * * @return Returns the log level, null, if the given logger could not be found. */ public Level getLogLevel(String loggerName); /** * Changes the log level for the requested logger name with the given log level. * * @param loggerName Logger name needs to be modified. * @param newLogLevel new logging level. * * @return Returns true, if log level could be changed, false, otherwise. */ public boolean changeLogLevel(String loggerName,Level newLogLevel); } --- NEW FILE: JobNotification.java --- /* * JobNotification.java * * Created on April 5, 2006, 5:08 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.management; import java.util.Calendar; import javax.management.Notification; /** * Notification to send the specified job information and define the various * notification types. * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class JobNotification extends Notification { /** * Name of the job this notification belongs to. */ private String jobName=null; /** * Used to generate the sequence number. */ private static int sequence=1; /** * Notification type specifies that the job got finished. */ public static final String JOB_FINISHED = "job.status.finished"; /** * Creates a new instance of JobNotification */ public JobNotification(String jobName,String type,Object source,String message) { super(type,source,JobNotification.sequence++,Calendar.getInstance().getTimeInMillis(),message); if(jobName==null) throw new IllegalArgumentException("job name cannot be null to create the job notification."); this.jobName=jobName; } /** * Gets the job name; */ public String getJobName() { return this.jobName; } /** * <p> * Returns the string representation of JobNotification class in the format * <br> {JobNotification [jobName = value] [rest = value] } * </p> * * @return Returns the string representation of JobNotification. */ public String toString() { StringBuffer stringValue=new StringBuffer("{JobNotification "); stringValue.append("[jobName = " + this.jobName + "]"); stringValue.append("[rest = " + super.toString() + "]"); stringValue.append("}"); return stringValue.toString(); } } --- NEW FILE: JobManagementAgent.java --- package org.jmonks.batch.framework.management; import java.io.IOException; import javax.management.InstanceAlreadyExistsException; import javax.management.MBeanRegistrationException; import javax.management.MBeanServer; import javax.management.MBeanServerFactory; import javax.management.MalformedObjectNameException; import javax.management.NotCompliantMBeanException; import javax.management.ObjectName; import javax.management.remote.JMXConnectorServer; import org.apache.log4j.Logger; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.Main; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.controller.JobController; /** * <p> * JobManagementAgent defines the methods to start and stop the JMX Agent for this job. * Only one agent will be created for a job and this can be created only be the framework. * Agent service can be started and stopped only once during the life cycle of the job. This uses * the JobConnectorHelper to take the help in creating different JMX * connector servers and register and unregister jmx connector servers to be looked up * by the JMX connector clients. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class JobManagementAgent { /** * Mbean Server domain name. */ public static final String MBEAN_SERVER_DOMAIN_NAME = "EnterpriseBatchServer"; /** * Name of the bean that manages this job. */ public static final String JOB_MANAGER_MBEAN_NAME = "JobManagerMBean"; /** * Name of the bean monitors this job. */ public static final String JOB_MONITOR_MBEAN_NAME = "JobMonitorMBean"; /** * Singleton job management agent instance. */ private static final JobManagementAgent jobManagementAgent=new JobManagementAgent(); /** * JMX Connector server instance created for this agent. */ private JMXConnectorServer jmxConnectorServer=null; /** * Job connector helper instance. This defines the implementation * and lookup location of the JMX connector server. */ private JobConnectorHelper jobConnectorHelper=null; /** * Name of the job this agent configured for. */ private String jobName=null; /** * Tells whether the agent has been started or not. */ private boolean started=false; /** * Tells whether the agent has been stopped or not. */ private boolean stopped=false; private static Logger logger=Logger.getLogger(JobManagementAgent.class); /** * Initializes the job management agent. It make sure it would be able to get * the reference to job connector server helper class. * * @throws ConfigurationException If it couldnt get the job connector server helper class. */ private JobManagementAgent() { this.jobConnectorHelper=JobConnectorHelper.getJobConnectorHelper(); if(jobConnectorHelper==null) throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, "could not get the reference to job connector helper class."); } /** * Returns the job management agent instance. */ public static JobManagementAgent getJobManagementAgent() { return jobManagementAgent; } /** * <p> * This method start the job management agent with the given controller information. * Basicially, this does the following activities to start the management agent. * <br> * <ul> * <li>Creates the job manager and job monitor mbeans by passing the controller reference. * <li>Creates the MBeanServer and registers these two MBeans. * <li>Create the JMX Connector server based connector helper implementation. * <ii>Registers JMX Connector server in the lookup location defined by connector helper implementation. * <li>Marks the started flag as true to indicate that agent has been started. * </ul> * <br> * Agent can be started only once in the job life cycle. If it receives a request to start * when it is already started, it throws IllegalStateException. * </p> * * @param jobContext Context of the job is being run. * @param jobController Controller reference to be passed to the mbeans. * @param agentManager Main class reference as a agent manager. * * @return Returns true, if manager is started properly, false, otherwise. * * @throws SecurityException if an attempt is maded to start by any one other than Main(framework). * @throws IllegalArgumentException If input parameters job name or job controller is null. * @throws IllegalStateException If manager is already started or already stopped. */ public boolean start(JobContext jobContext, JobController jobController, Main agentManager) { logger.trace("Entering start"); if(!(agentManager instanceof Main)) throw new SecurityException("Not authorized to manage the agent."); if(this.started) throw new IllegalStateException("job management agent has been already started."); if(this.stopped) throw new IllegalStateException("job management agent has been already stopped."); if(jobContext==null) throw new IllegalArgumentException("job context cannot be null to start the job management agent."); if(jobController==null) throw new IllegalArgumentException("controller object cannot be null to start the job management agent."); this.jobName=jobContext.getJobName(); try { MBeanServer mbeanServer=MBeanServerFactory.createMBeanServer(JobManagementAgent.MBEAN_SERVER_DOMAIN_NAME); this.jmxConnectorServer=this.jobConnectorHelper.createConnectorServer(); if(this.jmxConnectorServer==null) { logger.error("Unable to get the jmx connector server from job connector helper implementation."); throw new ConfigurationException(ConfigurationException.JOB_CONNECTOR_CONFIG, "Unable to get the jmx connector server from job connector helper implementation."); } else { logger.debug("Got the jmx connector server from the implementation"); } /** * Create the mbeans and register them with this mbean server. */ JobMonitorMBean monitorBean=new JobMonitor(jobController); JobManagerMBean managerBean=new JobManager(jobController); ObjectName connectorServerObjectName= new ObjectName(JobManagementAgent.MBEAN_SERVER_DOMAIN_NAME+":jobName="+jobName+",objectType=JMXConnectorServer"); ObjectName monitorBeanObjectName= new ObjectName(JobManagementAgent.MBEAN_SERVER_DOMAIN_NAME+":jobName="+jobName+",objectType="+JobManagementAgent.JOB_MONITOR_MBEAN_NAME); ObjectName managerBeanObjectName= new ObjectName(JobManagementAgent.MBEAN_SERVER_DOMAIN_NAME+":jobName="+jobName+",objectType="+JobManagementAgent.JOB_MANAGER_MBEAN_NAME); mbeanServer.registerMBean(this.jmxConnectorServer, connectorServerObjectName); mbeanServer.registerMBean(monitorBean, monitorBeanObjectName); mbeanServer.registerMBean(managerBean, managerBeanObjectName); /** * Start the connector server and register it with the job name. */ this.jmxConnectorServer.start(); boolean registered=this.jobConnectorHelper.registerConnectorServer(jobContext,this.jmxConnectorServer); if(registered) { logger.debug("Successfully registered the connector server with the job name"); this.started=true; } else { logger.error("Unable to register the jmx connector server using job connector helper"); this.started=false; } } catch(MalformedObjectNameException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } catch(InstanceAlreadyExistsException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } catch(MBeanRegistrationException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } catch(NotCompliantMBeanException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } catch(IOException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); } logger.trace("Exiting start"); return this.started; } /** * Stops the JMX connector server, unregisters the MBeans and MBeanServer and * unregisters the jmx service url information from lookup location and marks * stopped flag as true to indicate that agent has been stopped. Given controller * exit status will be sent to all the JMX remote clients as a notification. * * @param jobContext Context of the job is being run. * @param exitCode errorCode returned by controller to the Main class. * @param agentManager Main class reference as a agent manager. * * @return Returns true, if agent could be stopped properly, false otherwise. * * @throws SecurityException if an attempt is maded to stop by any one other than Main(framework). * @throws IllegalArgumentException If input parameter status code is null. * @throws IllegalStateException If agent is already stopped or it not yet started. */ public boolean stop(JobContext jobContext, ErrorCode exitCode, Main agentManager) { logger.trace("Entering stop"); if(!(agentManager instanceof Main)) throw new SecurityException("Not authorized to manage the agent."); if(!this.started) throw new IllegalStateException("Job management agent has not been started."); if(this.stopped) throw new IllegalStateException("Job management agent has been stopped already."); if(jobContext==null) throw new IllegalArgumentException("Job context cannot be null to stop the server."); if(exitCode==null) throw new IllegalArgumentException("controller exit code cannot be null to stop the server."); try { JobNotification notification=new JobNotification(this.jobName, JobNotification.JOB_FINISHED, this, exitCode.toString()); this.jmxConnectorServer.sendNotification(notification); this.jmxConnectorServer.stop(); this.jobConnectorHelper.unregisterConnectorServer(jobContext); this.stopped=true; logger.debug("unregistered the jmx connector server from lookup location"); } catch(IOException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); this.stopped=false; } logger.trace("Exiting stop"); return this.stopped; } /** * Tells whether the agent is running or not. This state needs to be queried * before trying to stop the agent. * * @return Returns true if agent is running, false otherwise. */ public boolean isRunning() { return this.started; } } --- NEW FILE: JobMonitor.java --- /* * JobMonitor.java * * Created on March 25, 2006, 1:12 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.management; import org.jmonks.batch.framework.controller.JobController; /** * <p> * Implementation of job monitor mbean to instrument the job controller. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class JobMonitor implements JobMonitorMBean { private JobMonitorMBean jobMonitor=null; /** * Creates a new instance of JobMonitorMBean and initializes with job controller. * * @param jobController Job controller implements this interface. * * @throws IllegalArgumentException if input argument job controller is null. */ public JobMonitor(JobController jobController) { if(jobController==null) throw new IllegalArgumentException("job controller cannot be null to create the job monitor mbean."); this.jobMonitor=(JobMonitorMBean)jobController; } /** * @see org.jmonks.batch.framework.management.JobMonitorMBean#getExpectedRecordsCount */ public long getExpectedRecordsCount() { return this.jobMonitor.getExpectedRecordsCount(); } /** * @see org.jmonks.batch.framework.management.JobMonitorMBean#getProcessedRecordsCount */ public long getProcessedRecordsCount() { return this.jobMonitor.getProcessedRecordsCount(); } /** * @see org.jmonks.batch.framework.management.JobMonitorMBean#getProcessorIDList */ public String[] getProcessorIDList() { return this.jobMonitor.getProcessorIDList(); } /** * @see org.jmonks.batch.framework.management.JobMonitorMBean#getProcessorState(java.lang.String) */ public ProcessorState getProcessorState(String processorID) { return this.jobMonitor.getProcessorState(processorID); } } --- NEW FILE: JobManager.java --- /* * JobManager.java * * Created on March 25, 2006, 11:14 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.management; import javax.management.NotificationBroadcasterSupport; import org.apache.log4j.Level; import org.jmonks.batch.framework.controller.JobController; /** * <p> * JobManager instruments the job controller for management purposes. * </p> * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class JobManager extends NotificationBroadcasterSupport implements JobManagerMBean { private JobManagerMBean jobManager=null; /** * Creates a new instance of JobManagerMBean and initializes with job controller. * * @param jobController Job controller implements this interface. * * @throws IllegalArgumentException if input argument job controller is null. */ public JobManager(JobController jobController) { if(jobController==null) throw new IllegalArgumentException("job controller cannot be null to create the job manager mbean."); this.jobManager=(JobManagerMBean)jobController; } /** * Controller should stop processing of the processor identified by the * given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor has been stopped, false otherwise. */ public boolean stop(String processorID) { return this.jobManager.stop(processorID) ; } /** * Controller should suspend processing of the processor identified by the * give processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor has been suspended, false otherwise. */ public boolean suspend(String processorID) { return this.jobManager.suspend(processorID); } /** * Controller should resume processing of the processor identified by the * given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns true, if processor can be resumed, false, otherwise. */ public boolean resume(String processorID) { return this.jobManager.resume(processorID); } /** * Returns the status of the processor identified by the given processor ID. * * @param processorID ID assigned to each processor. * * @return Returns the processor status. */ public ProcessorStatus getProcessorStatus(String processorID) { return this.jobManager.getProcessorStatus(processorID); } /** * Gets the log level of the given logger name. * * @param loggerName Name of the logger wants to find the log level. * * @return Returns the log level, null, if the given logger could not be found. */ public Level getLogLevel(String loggerName) { return this.jobManager.getLogLevel(loggerName); } /** * Changes the log level for the requested logger name with the given log level. * * @param loggerName Logger name needs to be modified. * @param newLogLevel new logging level. * * @return Returns true, if log level could be changed, false, otherwise. */ public boolean changeLogLevel(String loggerName, Level newLogLevel) { return this.jobManager.changeLogLevel(loggerName, newLogLevel); } } --- NEW FILE: ProcessorState.java --- package org.jmonks.batch.framework.management; /** * <p> * This represents the state of the processor(thread) running under the controller. * This gives the information like processor ID, description and * processing information. * </p> * * @author Suresh pragada * @version 1.0 * @since 1.0 */ public class ProcessorState { /** * Holds the processor ID. */ private String processorID; /** * Holds the processor description. */ private String processorDescription; /** * Holds the processing information. This can be anything that client and server * can understand. */ private Object processingInfo; /** * Constructor initializes the object. */ public ProcessorState(String processorID,String processorDescription,Object processingInfo) { this.processorID=processorID; this.processorDescription=processorDescription; this.processingInfo=processingInfo; } /** * Returns the processor ID. */ public String getProcessorID() { return this.processorID; } /** * Returns the processor description. */ public String getProcessorDescription() { return this.processorDescription; } /** * Returns the processor processing information. */ public Object getProcessingInfo() { return this.processingInfo; } /** * <p> * Returns the string representation of ProcessorState class in the format * <br> {ProcessorState [ID = value] [desc = value] [processingInfo = value]} * </p> * * @return Returns the string representation of ProcessorState. */ public String toString() { StringBuffer stringValue=new StringBuffer("{ProcessorState "); stringValue.append("[ID = " + this.processorID + "]"); stringValue.append("[desc = " + this.processorDescription + "]"); stringValue.append("[processingInfo = " + this.processingInfo + "]"); stringValue.append("}"); return stringValue.toString(); } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:06:52
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/controller/pool In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv19925 Added Files: AbstractPoolJobLoader.java AbstractPoolJobProcessor.java CollectionJobPool.java JobPool.java PoolJobController.java PoolJobLoader.java PoolJobProcessor.java Log Message: no message --- NEW FILE: PoolJobLoader.java --- package org.jmonks.batch.framework.controller.pool; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * PoolJobLoader loads the job data into the pool to be processed by * PoolJobProcessor(s). Along with the methods to load job data into the * pool, it exposes the some other methods used by the management and * monitoring clients. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public interface PoolJobLoader { /** * <p> * Loads the job data that needs to be processed in to the job pool. * When finished loading of all the job data, loads the <i>null</i> into the pool * to singal the processor(s) that loading of all the jobs have been done. * Configuration defined for this loader in job configuration can be accessed * using job context reference. * <br> * Example loading the 100 integer objects into the pool. * <pre> * public class MyPoolJobLoader implements PoolJobLoader * { * public ErrorCode loadPool(JobContext jobContext, JobPool pool) * { * for(int i=0;i<100;i++) * pool.loadJobData(new Integer(i)); * pool.loadJobData(null); * * return ErrorCode.JOB_COMPLETED_SUCCESSFULLY * } * } * <pre> * </p> * * @param jobContext Context of the job being run. * @param pool Job Pool reference. * * @return Retrurns the final status of the loader. */ public ErrorCode loadPool(JobContext jobContext,JobPool pool); /** * Suspends the loader. * * @return Returns true if loader is suspended, false otherwise. */ public boolean suspend(); /** * Resumes the loader. * * @return Returns true if loader is resumed, false otherwise. */ public boolean resume(); /** * Stops the loader. * * @return Returns true if loader is stopped, false otherwise. */ public boolean stop(); /** * Gets the total records this loader is going to load. * * @return Returns the number of records this loader is going to load. */ public long getTotalJobDataCount(); /** * Gets the loader state as object which can be understan by the monitoring * client. Usually, this would be used for display purposes. * * @return Returns the displayable object representing the loader state. */ public Object getLoaderState(); /** * Gets the loader status. * * @return Returns the loader status. */ public ProcessorStatus getLoaderStatus(); } --- NEW FILE: JobPool.java --- package org.jmonks.batch.framework.controller.pool; import org.jmonks.batch.framework.JobContext; /** * <p> * JobPool pools all the job data being loaded by the loader and serves this data * when job processors requests for the processing. This interface enables the loader, * processor and controller to interact with the pool implementation. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public interface JobPool { /** * Controller calls this method to initialize the job pool. This will be called * before pool reference is being passed to the job loader and job processor. * All the configuration defined for the job pool can be retrieved using * job context reference. * * @param jobContext Context of the job being run. */ public void initialize(JobContext jobContext); /** * Gets the next available job data(piece of information) to be processed. * If it is not available, it will be waited until it gets the next job data. * If loader is done with loading the jobs, it returns null. At this time, * processor should quit its processing. * * @return Returns the next available job data to be process, * null if no job data is available. */ public Object getNextJobData(); /** * Job Loader will use this API to load the job data into the pool. * If job loader is done with the loading all the data to be processed, * it should load null indicating that loading of all the job data has been * done. * * @param jobData Job data needs to be processed. */ public boolean loadJobData(Object jobData); /** * Controller calls this method after job processing has been done. */ public void cleanup(); /** * Returns the number of job data objects being loaded into the pool. * * @return Returns the number of job data objects being loaded into the pool. */ public long getLoadedJobDataCount(); } --- NEW FILE: PoolJobProcessor.java --- package org.jmonks.batch.framework.controller.pool; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * PoolJobProcessor gets the job data to be processed from the job pool and * processes it. Along with the processing methods, it exposes some methods used * by management and monitoring clients. * </p> * * @author Suresh pragada * @version 1.0 * @since 1.0 */ public interface PoolJobProcessor { /** * <p> * Process the job data available in the job pool until the job loader done loading * of all the job data. JobContext reference provides the access to * many different resources in the framework. Following is an examples shows * the sample implementation of processPool method. * <br><br> * <pre> * public class MyPoolJobProcessor implements PoolJobProcessor * { * public ErrorCode processPool(JobContext jobContext, JobPool pool) * { * Object jobData=null; * while((jobData=pool.getNextJobData())!=null) * { * // Perform the business logic on jobData * } * * return ErrorCode.JOB_COMPLETED_SUCCESSFULLY; * } * } * </pre> * </p> * * @param jobContext Context the job is being run. * @param pool Reference to Job Pool. * * @return Returns the error code. */ public ErrorCode processPool(JobContext jobContext, JobPool pool); /** * Suspends the pool job processor. * * @return Returns true if processor is suspended, false otherwise. */ public boolean suspend(); /** * Resumes the pool job processor. * * @return Returns true if processor is resumed, false otherwise. */ public boolean resume(); /** * Stops the processor. * * @return Returns true if processor is stopped, false otherwise. */ public boolean stop(); /** * Gets the processor to be displyed or anaylyzed for the monitoring purposes. * * @return Returns an object understanble/displayable by monitoring client. */ public Object getProcessorState(); /** * Gets the processor status being used by the management clients. * * @return Returns the status of the job. */ public ProcessorStatus getProcessorStatus(); /** * Returns the number of job data objects this particular job processor * has finsihed. * * @return Returns the number of job data objects processed. */ public long getProcessedJobDataCount(); } --- NEW FILE: AbstractPoolJobProcessor.java --- package org.jmonks.batch.framework.controller.pool; import EDU.oswego.cs.dl.util.concurrent.Mutex; import org.apache.log4j.Logger; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * Abstract pool job processor implements some of the responsiblites defined by the * pool job processor and leaves the application specific functionality implementation * to the final processor. Following is an example class shows how to make use of the * AbstractPoolJobProcessor. * <br><br> * <pre> * public class MyPoolJobProcessor extends AbstractPoolJobProcessor * { * private Connection connection=null; * * public void initialize(JobContext jobContext) * { * // Perform the initialization for this instance of job processor. * // Good place to get any references to any resources to be used to * // to processing of all the job data. * connection=ConnectionManager.getDBConnection(); * } * * public ErrorCode process(Object jobData) * { * // Perform the business logic on the incoming jobData. * connection.performBusinessLogic(jobData); * * return ErrorCode.JOB_COMPLETED_SUCCESSFULLY; * } * * public void cleanup() * { * // Do some cleanup after all the jobData has been processed. * connection.close(); * } * } * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class AbstractPoolJobProcessor implements PoolJobProcessor { /** * Counts the number of job data objects processed. */ private long processedJobDataCount=0; /** * Signal to stop the processor. */ private boolean stopSignal=false; /** * Singal to suspend the processor. */ private boolean suspendSignal=false; /** * Mutex lock to suspend the processor. */ private Mutex suspendLock=new Mutex(); /** * Holds the processor status. */ protected ProcessorStatus processorStatus=ProcessorStatus.INSTANTIATED; /** * Holds the job data currently being processed. */ private Object processingJobData=null; private static Logger logger=Logger.getLogger(AbstractPoolJobProcessor.class); /** * <p> * Initializes the processor implementation by calling the <i>initialize</i> * method by passing job context reference, gets the job data from the pool * and passes that information to the processor implementation for * processing and cleans up the processor implementation by calling the <i>cleanup</i> * method. * </p> * * @param jobContext Context the job is being run. * @param pool Job pool reference where job data needs to be pulled. * * @return Returns the status code of this processor. */ public ErrorCode processPool(JobContext jobContext, JobPool pool) { logger.trace("Entering processPool"); ErrorCode returnCode=ErrorCode.JOB_COMPLETED_SUCCESSFULLY; /** * Calling the initializer on final processor implementation. */ try { this.processorStatus=ProcessorStatus.INITIALIZING; initialize(jobContext); this.processorStatus=ProcessorStatus.RUNNING; } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while calling initialize on processor " + exception.getMessage(), exception); } /** * Get the job data from pool and pass that on to the final processor implementation. */ Object jobData=null; while((jobData=pool.getNextJobData())!=null) { try { this.processingJobData=jobData; this.processedJobDataCount++; ErrorCode errorCode=this.process(jobData); returnCode=errorCode; } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while processing the job data " + exception.getMessage(),exception); returnCode=ErrorCode.POOL_JOB_PROCESSOR_EXCEPTION; } if(this.stopSignal) { logger.info("Received the stop signal.. Preparing to stop"); break; } if(this.suspendSignal) { try { logger.info("Received suspend signal... suspending the process"); this.processorStatus=ProcessorStatus.SUSPENDED; this.suspendLock.acquire(); this.processorStatus=ProcessorStatus.RUNNING; logger.info("Resuming the process"); } catch(InterruptedException exception) { exception.printStackTrace(); logger.error("Got exception while suspending..." + exception.getMessage() + " process is going to be resumed", exception); } } } try { this.processorStatus=ProcessorStatus.CLEANUP; cleanup(); this.processorStatus=ProcessorStatus.FINISHED; } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while calling cleanup on processor " + exception.getMessage(), exception); } logger.trace("Exiting processPool"); return ErrorCode.JOB_COMPLETED_SUCCESSFULLY; } /** * Suspends the processor. * * @return Returns true if the processor is suspended, false otherwise. */ public boolean suspend() { logger.trace("Entering suspend"); this.suspendSignal=true; logger.trace("Entering suspend"); return true; } /** * Resumes the processor. * * @return Retrurns true if the processor is resumed, false otherwise. */ public boolean resume() { logger.trace("Entering resume"); this.stopSignal=false; this.suspendSignal=false; this.suspendLock.release(); logger.trace("Exiting resume"); return true; } /** * Stops the processor. * * @return Retrurns true if the processor has stopped, false otherwise. */ public boolean stop() { logger.trace("Entering stop"); this.stopSignal=true; if(this.suspendSignal) { logger.info("Processor is in suspend status... resuming the processor"); this.suspendSignal=false; this.suspendLock.release(); } logger.trace("Exiting stop"); return true; } /** * @see org.jmonks.batch.framework.controller.pool.PoolJobProcessor#getProcessorState() */ public Object getProcessorState() { return this.processingJobData; } /** * @see org.jmonks.batch.framework.controller.pool.PoolJobProcessor#getProcessorStatus() */ public ProcessorStatus getProcessorStatus() { return this.processorStatus; } /** * @see org.jmonks.batch.framework.controller.pool.PoolJobProcessor#getProcessedJobDataCount() */ public long getProcessedJobDataCount() { return processedJobDataCount; } /** * Chance to initialize itself using the information provided through * job context. This will be called only once for each processor and before * it start processing job data using process method. * * @param jobContext Context the job is being run. */ public abstract void initialize(JobContext jobContext); /** * Execute the business logic on the given jobData and return the * appropriate error code. The format and type of jobData is depends * on the job loader configured for the same job. Usually, there will be * an understanding between the loader and processor on the type of jobData * being loaded into the pool. * * @param jobData Data to be processed. * * @return Returns the status of the processingn of this jobData. */ public abstract ErrorCode process(Object jobData); /** * Chance to do any cleanup at the end of the processing. Called once per * each processor at the end of processing of all the jobs. */ public abstract void cleanup(); } --- NEW FILE: AbstractPoolJobLoader.java --- /* * AbstractPoolJobLoader.java * * Created on May 16, 2006, 4:49 PM * * To change this template, choose Tools | Options and locate the template under * the Source Creation and Management node. Right-click the template and choose * Open. You can then make changes to the template in the Source Editor. */ package org.jmonks.batch.framework.controller.pool; import EDU.oswego.cs.dl.util.concurrent.Mutex; import org.apache.log4j.Logger; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * AbstractPoolJobLoader implements all the management and monitoring methods * and abstracts the user from the job pool. This allows the loader implementation * to concentrate on the business logic. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class AbstractPoolJobLoader implements PoolJobLoader { /** * Holds the status of the loader. */ protected ProcessorStatus loaderStatus=ProcessorStatus.INSTANTIATED; /** * Holds the pool reference passed by controller. */ private JobPool pool=null; /** * Holds the current job data that will be used for the monitoring purposes. */ private Object currentJobData=null; /** * Signal the loader to the stop the loading. */ private boolean stopSignal=false; /** * Signal the loader to suspend the loading. */ private boolean suspendSignal=false; /** * Mutex lock to suspend and resume the loader. */ private Mutex suspendLock=new Mutex(); private static Logger logger=Logger.getLogger(AbstractPoolJobLoader.class); /** * Abstracts the job pool details from the final loader by defining * other set of methods for the final loader and implements the management * and monitoring related methods. * * @param jobContext Context of the job being run. * @param pool Job Pool reference. * * @return Returns the final status of loading. */ public final ErrorCode loadPool(JobContext jobContext,JobPool pool) { logger.trace("Entering loadPool"); this.pool=pool; ErrorCode returnCode=ErrorCode.JOB_COMPLETED_SUCCESSFULLY; try { this.loaderStatus=ProcessorStatus.RUNNING; returnCode=this.loadPool(jobContext); this.loaderStatus=ProcessorStatus.FINISHED; } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while calling loaders loadPool = " + exception.getMessage(), exception); returnCode=ErrorCode.POOL_JOB_LOADER_EXCEPTION; } logger.trace("Exiting loadPool"); return returnCode; } /** * Loads the given job data into the job pool.This method is for the * final loaders to load the job data into the pool. * * @param jobData Job data object that needs to be processed. * * @return Returns true if the job data is loaded into the pool, false otherwise. */ protected final boolean loadJobData(Object jobData) { boolean loaded=false; if(jobData!=null) this.currentJobData=jobData; else logger.debug("Loading the null to signal the end of the pool for the processors(s)"); loaded=this.pool.loadJobData(jobData); if(this.suspendSignal) { try { this.loaderStatus=ProcessorStatus.SUSPENDED; this.suspendLock.acquire(); this.loaderStatus=ProcessorStatus.RUNNING; } catch(InterruptedException exception) { exception.printStackTrace(); logger.error("Exception while suspending the loader " + exception.getMessage(),exception); this.loaderStatus=ProcessorStatus.RUNNING; } } return loaded; } /** * Tells whether the loading of the jobs needs to be stopped or not. * Loader implementation should check for this flag before loading the jobs * into the pool. * * @return Returns true if loader to be stopped, false continue loading the jobs. */ protected final boolean stopLoading() { return this.stopSignal; } /** * Resumes the loading of the jobs. * * @return Returns true if loader is resumed, false otherwise. */ public boolean resume() { logger.trace("Entering resume"); this.stopSignal=false; this.suspendSignal=false; this.suspendLock.release(); logger.trace("Exiting resume"); return true; } /** * Stops the loading of the jobs into the pool. * * @return Returns true if loader is stopped, false otherwise. */ public boolean stop() { logger.trace("Entering stop"); this.stopSignal=true; if(this.suspendSignal) { logger.info("Loader is in suspend status... resuming the loader"); this.suspendSignal=false; this.suspendLock.release(); } logger.trace("Exiting stop"); return true; } /** * Suspends loading of the jobs into the pool. */ public boolean suspend() { logger.trace("Entering suspend"); this.suspendSignal=true; logger.trace("Exiting suspend"); return true; } /** * Returns the status of the loader. * * @return Returns the status of the loader. */ public ProcessorStatus getLoaderStatus() { return this.loaderStatus; } /** * Returns the processing state of the loader. * * @return Returns the job data that this loader is loading. */ public Object getLoaderState() { return currentJobData.toString(); } /** * <p> * Load the job data into the pool that needs to be processed by job processor(s). * Implementers can take the help of the <i>loadJobData(Object)</i> method defined * here to load the jobs into the pool. * <br> * <pre> * public class MyPoolJobLoader extends AbstractPoolJobLoader * { * public ErrorCode loadPool(JobContext jobContext) * { * for(int i=0;i<100;i++) * { * loadJobData(new Integer(i)); * if(super.stopLoading()) * { * doCleanup(); * super.loaderStatus=ProcessorStatus.STOPPED; * break; * } * } * loadJobData(null); * return ErrorCode.JOB_COMPLETED_SUCCESSFULLY; * } * } * </pre> * </p> * * @param jobContext Context of the job being run. * * @return Returns the final status of the loader. */ public abstract ErrorCode loadPool(JobContext jobContext); /** * Returns the number of job data objects that this loader is going to load. * * @return Return the number of job data object its going to load. */ public abstract long getTotalJobDataCount(); } --- NEW FILE: PoolJobController.java --- package org.jmonks.batch.framework.controller.pool; import EDU.oswego.cs.dl.util.concurrent.Callable; import EDU.oswego.cs.dl.util.concurrent.CountDown; import EDU.oswego.cs.dl.util.concurrent.FutureResult; import java.util.Calendar; import java.util.Hashtable; import java.util.Iterator; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobStatistics; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.config.PoolJobControllerConfig; import org.jmonks.batch.framework.controller.JobController; import org.jmonks.batch.framework.management.ProcessorState; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * PoolJobController provides the job architecture based on the pool concept, * where a loader is load all the information to be processed into the pool and * processors(>=1) retrieves the information from pool and process them. * It provides the abstract loader and processor classes to be overriden by job * implementation and comes with different implementation of pools. * <br><br> * To write any job using this controller, developer needs to write a loader class * extends AbstractPoolJobLoader, which loads the data needs to be processed into * the pool and write a processor class extending the AbstractPoolJobProcessor, * which process the data from the pool. Once implementation is done, * this job should be configured either using the XML configuration or DB configuration. * * <br><br> * <i>XML Configuration is as follows</i> <br><br> * <pre> * <job-config job-name="process_file_abc"> * <job-controller controller-class-name="org.jmonks.batch.framework.controller.pool.PoolJobController"> * <pool-job-loader pool-job-loader-class-name="com.mycompany.batch.processfileabc.AbcJobLoader"> * <property key="pool-job-loader-config1">pool-job-loader-value1</property> * </pool-job-loader> * <pool-job-processor pool-job-processor-class-name="com.mycompany.batch.processfileabc.AbcJobProcessor" thread-count="5"> * <property key="pool-job-processor-config1">pool-job-processor-value1</property> * </pool-job-processor> * <job-pool job-pool-class-name="org.jmonks.batch.framework.controller.pool.DefaultJobPool"> * <property key="pool-size">50000</property> * </job-pool> * <property key="pool-controller-config1">pool-controller-value1</property> * </job-controller> * <job-logging-config> * <logging-property-file>com.mycompany.batch.processfileabc.Logging</logging-property-file> * </job-logging-config> * </job-config> * </pre> * <br><br> * <i>DB Configuration is as follows</i> * <table border="1"> * <tr><td><b>TableName.ColumnName</b></td><td><b>Value</b></td></tr> * <tr><td>job_config.job_name</td><td>process_file_abc</td></tr> * <tr><td>job_config.job_status</td><td>1</td></tr> * <tr><td>job_config.job_controller_class_name</td><td>org.jmonks.batch.framework.controller.pool.PoolJobController</td></tr> * <tr><td>job_config.job_controller_props</td><td>pool-controller-config1=pool-controller-value1:pool-controller-config1=poo2-controller-value2</td></tr> * <tr><td>pool_job_controller_config.job_name</td><td>process_file_abc</td></tr> * <tr><td>pool_job_controller_config.pool_job_loader_class_name</td><td>com.mycompany.batch.processfileabc.AbcJobLoader</td></tr> * <tr><td>pool_job_controller_config.pool_job_loader_props</td><td>pool-job-loader-key1=loader-value1</td></tr> * <tr><td>pool_job_controller_config.pool_job_processor_class_name</td><td>com.mycompany.batch.processfileabc.AbcJobProcessor</td></tr> * <tr><td>pool_job_controller_config.pool_job_processor_props</td><td>pool-job-processor-key1=processor-value1</td></tr> * <tr><td>pool_job_controller_config.pool_job_processor_thread_cnt</td><td>5</td></tr> * <tr><td>pool_job_controller_config.job_pool_class_name</td><td>org.jmonks.batch.framework.controller.pool.CollectionJobPool</td></tr> * <tr><td>pool_job_controller_config.job_pool_props</td><td>job-pool-size=5000</td></tr> * </table> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class PoolJobController extends JobController { /** * Map holds all the pool job processors and loader being used for the given job * as values and name of the threads as keys. */ private Map jobProcessorsMap=new Hashtable(); /** * Map holds all the pool job processors and loader returned error codes as values and * name of the threads as keys. */ private Map jobProcessorsResultMap=new Hashtable(); /** * Job pool reference being used by the loader and processor. */ private JobPool pool=null; /** * Holds the statistics of the basic job controller. */ private JobStatistics jobStatistics=null; private static Logger logger=Logger.getLogger(PoolJobController.class); /** * Constructor enables the instantiation of the pool job controller instance. */ public PoolJobController() { } /** * <p> * Executes the job by creating the pool, loader and processor(s) based * on the configuration and have them work accordingly to process the job. * </p> * * @return Returns the processing error code for the job. */ public ErrorCode process() { logger.info("Entering process in pool job controller = " + super.jobContext.getJobName()); PoolJobControllerConfig poolJobControllerConfig=(PoolJobControllerConfig)super.jobContext.getJobConfig().getJobControllerConfig(); validateControllerConfiguration(poolJobControllerConfig); int processorCount=poolJobControllerConfig.getPoolJobProcessorThreadCount(); processorCount=(processorCount<1?1:processorCount); /** * Create and initialize the pool */ this.pool=(JobPool)this.getInstance(poolJobControllerConfig.getPoolClassName()); pool.initialize(super.jobContext); CountDown countDownLock=new CountDown(processorCount+1); /** * Create, initialize and spwan the loader */ PoolJobLoader jobLoader=(PoolJobLoader)this.getInstance(poolJobControllerConfig.getPoolJobLoaderClassName()); String jobLoaderName=super.jobContext.getJobName()+"_Loader"; FutureResult jobLoaderFutureResult=new FutureResult(); Thread jobLoaderThread=new Thread(jobLoaderFutureResult.setter(this.getCallableLoader( countDownLock,jobLoader,super.jobContext,pool)),jobLoaderName); jobLoaderThread.start(); this.jobProcessorsResultMap.put(jobLoaderName, jobLoaderFutureResult); this.jobProcessorsMap.put(jobLoaderName, jobLoader); this.jobStatistics=new JobStatistics(super.jobContext.getJobName()); this.jobStatistics.setStartTime(Calendar.getInstance().getTime()); /** * Create, initialize and spawn the processor(s). */ for(int i=0;i<processorCount;i++) { PoolJobProcessor jobProcessor=(PoolJobProcessor)this.getInstance(poolJobControllerConfig.getPoolJobProcessorClassName()); String jobProcessorName=super.jobContext.getJobName()+"_Processor_"+i; FutureResult jobProcessorFutureResult=new FutureResult(); Thread jobProcessorThread=new Thread(jobProcessorFutureResult.setter(this.getCallableProcessor( countDownLock,jobProcessor,super.jobContext,pool)),jobProcessorName); jobProcessorThread.start(); this.jobProcessorsResultMap.put(jobProcessorName, jobProcessorFutureResult); this.jobProcessorsMap.put(jobProcessorName, jobProcessor); } /** * Go to hydernate until all threads have finsihed their task. */ ErrorCode returnCode=hybernate(countDownLock); logger.info("Exiting process in pool job controller = " + super.jobContext.getJobName() + " with return code = " + returnCode); return returnCode; } /** * Returns the total number of records this job going to process. * This information will be obtained from the pool loader. * * @return Returns the number of records this job goint to process. */ public long getExpectedRecordsCount() { logger.trace("Entering getExpectedRecordsCount"); long expectedRecordsCount=0; String jobLoaderName=super.jobContext.getJobName()+"_Loader"; PoolJobLoader jobLoader=(PoolJobLoader)this.jobProcessorsMap.get(jobLoaderName); expectedRecordsCount=jobLoader.getTotalJobDataCount(); logger.trace("Exiting getExpectedRecordsCount = " + expectedRecordsCount); return expectedRecordsCount; } /** * Number of records got processed so far. * This information will be obtained from the pool. * * @return Returns the number of records processed so far. */ public long getProcessedRecordsCount() { logger.trace("Entering getProcessedRecordsCount"); long processedRecordsCount=0; for(Iterator iterator=this.jobProcessorsMap.values().iterator();iterator.hasNext();) { Object processor=iterator.next(); if(processor instanceof PoolJobProcessor) processedRecordsCount=processedRecordsCount+((PoolJobProcessor)processor).getProcessedJobDataCount(); } logger.trace("Exiting getProcessedRecordsCount = " + processedRecordsCount); return processedRecordsCount; } /** * Returns the IDs assigned to all the processors, loader as a string array. * * @return Returns the string array consist of all the processor(s) and loader IDs. */ public java.lang.String[] getProcessorIDList() { logger.trace("Entering getProcessorIDList"); String processorIDList[]=new String[this.jobProcessorsMap.size()]; int i=0; for(Iterator iterator=this.jobProcessorsMap.keySet().iterator();iterator.hasNext();i++) processorIDList[i]=(String)iterator.next(); logger.trace("Exiting getProcessorIDList"); return processorIDList; } /** * Returns the current state of the processor identified by the given processor * ID as the ProcessorState object. * * @param processorID processor ID identifies the processor or loader. * * @return Retuns the current state of the required processor. */ public ProcessorState getProcessorState(String processorID) { logger.trace("Exiting getProcessorState = " + processorID); ProcessorState state=null; Object processor=this.jobProcessorsMap.get(processorID); if(processor!=null) { if(processor instanceof PoolJobLoader) state=new ProcessorState(processorID, "Pool Job Loader", ((PoolJobLoader)processor).getLoaderState()); else if(processor instanceof PoolJobProcessor) state=new ProcessorState(processorID, "Pool Job Processor", ((PoolJobProcessor)processor).getProcessorState()); else { logger.error("What else it could be? " + processor.getClass().getName()); state=null; } } else state=null; logger.trace("Exiting getProcessorState = " + processorID + " state = " + state); return state; } /** * Returns the status of the processor identified by the given processor ID. * * @return Returns the status of the required processor. */ public ProcessorStatus getProcessorStatus(String processorID) { logger.trace("Exiting getProcessorStatus = " + processorID); ProcessorStatus status=null; Object processor=this.jobProcessorsMap.get(processorID); if(processor!=null) { if(processor instanceof PoolJobLoader) status=((PoolJobLoader)processor).getLoaderStatus(); else if(processor instanceof PoolJobProcessor) status=((PoolJobProcessor)processor).getProcessorStatus(); else { logger.error("What else it could be? " + processor.getClass().getName()); status=null; } } else status=null; logger.trace("Exiting getProcessorStatus = " + processorID + " status = " + status); return status; } /** * Stops the processor identified by the given processor ID. * * @return Returns true, if processor could be stopped, false otherwise. */ public boolean stop(String processorID) { logger.trace("Entering stop = " + processorID); boolean stopped=true; Object processor=this.jobProcessorsMap.get(processorID); if(processor!=null) { if(processor instanceof PoolJobLoader) stopped=((PoolJobLoader)processor).stop(); else if(processor instanceof PoolJobProcessor) stopped=((PoolJobProcessor)processor).stop(); else { logger.error("What else it could be? " + processor.getClass().getName()); stopped=false; } } else stopped=false; logger.trace("Exiting stop = " + processorID + " status = " + stopped); return stopped; } /** * Suspends the processor identified by the given processor ID. * * @return Returns true if it could suspend the processor, false otherwise. */ public boolean suspend(String processorID) { logger.trace("Entering suspend = " + processorID); boolean suspended=true; Object processor=this.jobProcessorsMap.get(processorID); if(processor!=null) { if(processor instanceof PoolJobLoader) suspended=((PoolJobLoader)processor).suspend(); else if(processor instanceof PoolJobProcessor) suspended=((PoolJobProcessor)processor).suspend(); else { logger.error("What else it could be? " + processor.getClass().getName()); suspended=false; } } else suspended=false; logger.trace("Exiting suspend = " + processorID + " status = " + suspended); return suspended; } /** * Resumes the processor identified by given processor ID. * * @return Returns true if processor is resumed, false otherwise. */ public boolean resume(String processorID) { logger.trace("Entering resume = " + processorID); boolean resumed=true; Object processor=this.jobProcessorsMap.get(processorID); if(processor!=null) { if(processor instanceof PoolJobLoader) resumed=((PoolJobLoader)processor).resume(); else if(processor instanceof PoolJobProcessor) resumed=((PoolJobProcessor)processor).resume(); else { logger.error("What else it could be? " + processor.getClass().getName()); resumed=false; } } else resumed=false; logger.trace("Exiting resume = " + processorID + " status = " + resumed); return resumed; } /** * <p> * Main thread will call this method and wait until all the processor(s) and loader have * finished their processing. Once they are finished, all the return codes * will be analyzed and one error code will be returned for this job. * </p * @param countDownLock CountDown lock being used by loader and all the processor(s). * * @return Returns the error code of the job. */ private ErrorCode hybernate(CountDown countDownLock) { logger.trace("Entering hybernate"); ErrorCode returnCode=ErrorCode.JOB_COMPLETED_SUCCESSFULLY; try { logger.debug("Going to wait until loader and all the processor(s) is gonna finish."); countDownLock.acquire(); logger.info("Loader and all processor(s) have finished their task."); /** As per the contract cleanup the pool. */ this.pool.cleanup(); } catch(InterruptedException exception) { exception.printStackTrace(); logger.error("Exception while waiting for loader and all the processor(s) = " + exception.getMessage(), exception); return ErrorCode.POOL_JOB_PROCESSOR_EXCEPTION; } this.jobStatistics.setEndTime(Calendar.getInstance().getTime()); this.jobStatistics.setMaxMemeoryUsage(Runtime.getRuntime().totalMemory()-Runtime.getRuntime().freeMemory()); this.jobStatistics.setRecordsProcessed(this.getProcessedRecordsCount()); for(Iterator iterator=this.jobProcessorsResultMap.values().iterator();iterator.hasNext();) { ErrorCode threadReturnCode=(ErrorCode)((FutureResult)iterator.next()).peek(); if(!ErrorCode.JOB_COMPLETED_SUCCESSFULLY.equals(threadReturnCode)) { returnCode=threadReturnCode; break; } } this.jobStatistics.setExitCode(returnCode); logger.trace("Exiting hybernate = " + returnCode); return returnCode; } /** * Wraps the job loader with Callable interface and return the callable * interface. * * @param countDownLock Count down to be released at the end of the processing. * @param jobLoader Job Loader that needs to be invoked. * @param jobContext Context of the job being run. * @param pool Job pool where the loader needs to load the jobs. * * @return Returns the callabel interface wrapped around the job loader. */ private Callable getCallableLoader(final CountDown countDownLock,final PoolJobLoader jobLoader, final JobContext jobContext, final JobPool pool) { logger.trace("Entering getCallableLoader"); Callable callable=new Callable(){ public Object call() { ErrorCode returnCode=null; try { logger.trace("Going to call the loadPool method"); returnCode=jobLoader.loadPool(jobContext,pool); logger.debug("Done calling the loadPool method"); } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while loading the job data into the pool = " + exception.getMessage(), exception); returnCode=ErrorCode.POOL_JOB_LOADER_EXCEPTION; } countDownLock.release(); logger.info(Thread.currentThread().getName() + " is exiting with the error code = " + returnCode); return returnCode; } }; logger.trace("Exiting getCallableLoader"); return callable; } /** * Wraps the job processor with Callable interface and return the callable * interface. * * @param countDownLock Count down to be released at the end of the processing. * @param jobProcessor Job processor that needs to be invoked. * @param jobContext Context of the job being run. * @param pool Job pool where the processor needs to pull the jobs. * * @return Returns the callabel interface wrapped around the job processor. */ private Callable getCallableProcessor(final CountDown countDownLock,final PoolJobProcessor jobProcessor, final JobContext jobContext, final JobPool pool) { logger.trace("Entering getCallableProcessor"); Callable callable=new Callable(){ public Object call() { ErrorCode returnCode=null; try { logger.trace("Going to call the processPool method"); returnCode=jobProcessor.processPool(jobContext,pool); logger.debug("Done calling the processPool method"); } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while processing the job data from the pool = " + exception.getMessage(), exception); returnCode=ErrorCode.POOL_JOB_PROCESSOR_EXCEPTION; } countDownLock.release(); logger.info(Thread.currentThread().getName() + " is exiting with the error code = " + returnCode); return returnCode; } }; logger.trace("Exiting getCallableProcessor"); return callable; } /** * Instantiates and returns the instance of the required class. * * @param className Class name of the required instance. * * @return Retrurns the instance of the required class. * * @throws ConfigurationException If class could not be instantiated. */ private Object getInstance(String className) { try { return Class.forName(className).newInstance(); } catch(Exception exception) { exception.printStackTrace(); logger.fatal(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, exception.getMessage()); } } /** * Validates the pool job controller configuration by instantiating and verifying * all the required loader, processor and pool classes. * * @param controllerConfig Controller configuration defined in job configuration. * * @throws ConfigurationException If any one required instances cannot be instantiated. */ private void validateControllerConfiguration(PoolJobControllerConfig controllerConfig) { Object jobPool=this.getInstance(controllerConfig.getPoolClassName()); if(jobPool instanceof JobPool) { logger.debug("Job pool is configured properly = " + controllerConfig.getPoolClassName()); } else { logger.fatal("Configured job pool class name " + controllerConfig.getPoolClassName() + " cannot be associated to JobPool"); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, "Configured job pool class name " + controllerConfig.getPoolClassName() + " cannot be associated to JobPool"); } Object jobLoader=this.getInstance(controllerConfig.getPoolJobLoaderClassName()); if(jobLoader instanceof PoolJobLoader) { logger.debug("Job loader is configured properly = " + controllerConfig.getPoolJobLoaderClassName()); } else { logger.fatal("Configured job loader class name " + controllerConfig.getPoolJobLoaderClassName() + " cannot be associated to PoolJobLoader"); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, "Configured job loader class name " + controllerConfig.getPoolJobLoaderClassName() + " cannot be associated to PoolJobLoader"); } Object jobProcessor=this.getInstance(controllerConfig.getPoolJobProcessorClassName()); if(jobProcessor instanceof PoolJobLoader) { logger.debug("Job Processor is configured properly = " + controllerConfig.getPoolJobProcessorClassName()); } else { logger.fatal("Configured job processor class name " + controllerConfig.getPoolJobProcessorClassName() + " cannot be associated to PoolJobProcessor"); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, "Configured job processor class name " + controllerConfig.getPoolJobProcessorClassName() + " cannot be associated to PoolJobProcessor"); } } /** * @see org.jmonks.batch.framework.controller.JobController#getJobStatistics() */ public JobStatistics getJobStatistics() { if(this.jobStatistics.getEndTime()!=null) return this.jobStatistics; else return null; } } --- NEW FILE: CollectionJobPool.java --- package org.jmonks.batch.framework.controller.pool; import EDU.oswego.cs.dl.util.concurrent.BoundedBuffer; import java.util.Hashtable; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.config.PoolJobControllerConfig; /** * <p> * Provides the implementation of job pool using java utility collections. * This job pool implementation can be configured for any given job using the * following configuration in job configuration.<br> * <pre> * <job-pool job-pool-class-name="org.jmonks.batch.framework.controller.pool.CollectionJobPool"> * <property key="job-pool-size">50000</property> * </job-pool> * </pre> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class CollectionJobPool implements JobPool { /** * Default collection pool size. This will be used when pool size is not * configured or problem in obtaining the pool size. */ public static final int DEFAULT_COLLECTION_POOL_SIZE = 1000; /** * Property name by which pool size will be associated with. */ protected static final String POOL_SIZE_PROPERTY_NAME = "job-pool-size"; /** * Map holds the configuration defined for the job pool. There might be properies * needed other than initialization. */ protected Map configProps=null; /** * Java utility collection holds the job data objects. */ protected BoundedBuffer pool=null; /** * Holds the number of job data objects have been loaded by job loader. */ protected long loadedJobsCount=0; /** * Object to be pushed to the bounded buffer to denote the end of the pool. */ private static final Object END_OF_POOL = new Object(); private static Logger logger=Logger.getLogger(CollectionJobPool.class); /** * Default constructor for the instantiation purposes. */ public CollectionJobPool() { } /** * @see org.jmonks.batch.framework.controller.pool.JobPool#loadJobData(Object) */ public boolean loadJobData(Object jobData) { boolean loaded=false; while(!loaded) { try { if(jobData!=null) { this.pool.put(jobData); /** END_OF_POOL will be loaded by getNextJobData API to wake up other processors */ if(jobData!=END_OF_POOL) this.loadedJobsCount++; } else { /** * null from loader indicates that he has done loading with the job data. * Push END_OF_POOL to the pool to let processors know that loader is done loading the jobs. */ this.pool.put(END_OF_POOL); } loaded=true; } catch(InterruptedException exception) { exception.printStackTrace(); logger.info("Got exception while loading the job data = " + jobData, exception); } } return loaded; } /** * @see org.jmonks.batch.framework.controller.pool.JobPool#getNextJobData */ public Object getNextJobData() { Object jobData=null; try { jobData=this.pool.take(); /** * If retrieved job data is END_OF_POOL object return null to caller as per * method contract and put EN_OF_POOL back to the pool to wake up other processors. */ if(jobData==END_OF_POOL) { this.pool.put(END_OF_POOL); jobData=null; } } catch(InterruptedException exception) { exception.printStackTrace(); logger.info("Got exception while getting the job data from pool", exception); } return jobData; } /** * Initializes the collection job pool using the configuration defined * in the job configuration. It gets the pool size from the defined configuration, * by looking for the property "job-pool-size", if it couldnt find it uses the default pool size "1000" * and initializes the collection to be used as the pool. * * @param jobContext Context of the job being run. */ public void initialize(JobContext jobContext) { PoolJobControllerConfig poolJobControllerConfig=(PoolJobControllerConfig)jobContext.getJobConfig().getJobControllerConfig(); this.configProps=new Hashtable(poolJobControllerConfig.getPoolConfigProperties()); /** * Try to get the pool size from configuration... If unable to get use default size. */ int poolSize=CollectionJobPool.DEFAULT_COLLECTION_POOL_SIZE; String poolSizePropertyValue=(String)this.configProps.get(CollectionJobPool.POOL_SIZE_PROPERTY_NAME); if(poolSizePropertyValue!=null) { logger.info("Received the pool size " + poolSizePropertyValue + " from configuration"); try { poolSize=Integer.parseInt(poolSizePropertyValue); if(poolSize<1) { poolSize=CollectionJobPool.DEFAULT_COLLECTION_POOL_SIZE; } } catch(Exception exception) { logger.info("Exception while obtaining the pool size from configuration... Using the default pool size"); } } else { logger.info("pool size has not been configured.. using the default pool size"); } this.pool=new BoundedBuffer(poolSize); } /** * Removes all the entries from collection. */ public void cleanup() { /** * Nothing to do the cleanup. */ logger.debug("Cleanup has been done"); } /** * @see org.jmonks.batch.framework.controller.pool.JobPool#getLoadedJobDataCount */ public long getLoadedJobDataCount() { return this.loadedJobsCount; } } |
|
From: Suresh <sur...@us...> - 2006-09-15 20:06:42
|
Update of /cvsroot/batchserver/batchserver/src/org/jmonks/batch/framework/controller/basic In directory sc8-pr-cvs3.sourceforge.net:/tmp/cvs-serv19904 Added Files: BasicJobController.java BasicJobProcessor.java Log Message: no message --- NEW FILE: BasicJobProcessor.java --- package org.jmonks.batch.framework.controller.basic; import org.apache.log4j.Logger; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.management.ProcessorStatus; /** * <p> * BasicJobProcessor lets job developers to implement their business logic * in the process method and this method will be executed by the BasicJobController. * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public abstract class BasicJobProcessor { /** * Holds the thread references this processor has been spawned. * This reference will be used to suspend, resume and stop the processor. */ protected Thread processorThread=null; /** * Indicates that thread has been registered or not. */ private boolean threadRegistered=false; /** * Indicates the status of this processor. */ protected ProcessorStatus processorStatus=ProcessorStatus.INSTANTIATED; private static Logger logger=Logger.getLogger(BasicJobProcessor.class); /** * Gets the thread reference this processor is going to be executed on. * If processorThread has already initialized, it ignores this request. * * @return Returns true if thread has been registered with the processor, false otherwise. */ final boolean registerThread() { logger.trace("Entering registerThread"); boolean registered=true; if(!threadRegistered) { this.processorThread=Thread.currentThread(); logger.debug("Registering the thread = " + this.processorThread.getName()); this.threadRegistered=true; } else { registered=false; } logger.trace("Exiting registerThread"); return registered; } /** * <p> * Basic job controller calls this method to suspend the job processor, when it receives * a <i>suspsend</i> message from the managemet interface. Default implementation * will suspend the job processor abruptly as soon as it receives the message. * Implementors can provide their own implementation by overriding this method * to provide better mechanism. * </p> * * @return Returns true if processor has suspended, false otherwise. */ public boolean suspend() { if(this.processorStatus!=ProcessorStatus.SUSPENDED) { this.processorThread.suspend(); this.processorStatus=ProcessorStatus.SUSPENDED; return true; } else return false; } /** * <p> * Basic job controller calls this method to resume the job processor, when it receives * a <i>resume</i> message from the managemet interface. Default implementation * will resume the job processor abruptly as soon as it receives the message. * Implementors can provide their own implementation by overriding this method * to provide better mechanism. * </p> * * @return Returns true if processor has resumed, false otherwise. */ public boolean resume() { if(this.processorStatus==ProcessorStatus.SUSPENDED) { this.processorThread.resume(); this.processorStatus=ProcessorStatus.RUNNING; return true; } else return false; } /** * <p> * Basic job controller calls this method to stop the job processor, when it receives * a <i>stop</i> message from the managemet interface. Default implementation * will stop the job processor abruptly as soon as it receives the message. * Implementors can provide their own implementation by overriding this method * to provide better mechanism. * </p> * * @return Returns true if processor has stopped, false otherwise. */ public boolean stop() { if(this.processorStatus!=ProcessorStatus.STOPPED || this.processorStatus!=ProcessorStatus.FINISHED) { this.processorThread.stop(); this.processorStatus=ProcessorStatus.STOPPED; return true; } else return false; } /** * <p> * Does the processing and return the appropriate error code. Properties * configured for this job processor can be retrieved from the JobContext. * <br> * <pre> * public class MyBasicJobProcessor extends BasicJobProcessor * { * public ErrorCode process(JobContext jobContext) * { * //Perform the business logic. * return ErrorCode.JOB_COMPLETED_SUCCESSFULLY; * } * } * </pre> * </p> * * @param jobContext Context of the job, where all the properties will be defined. * * @return Returns the appropriate error code needs to be passed to the invocation layer. */ public abstract ErrorCode process(JobContext jobContext); /** * This is for monitoring applications to know what the information (record) * this job processor is processing. Implementers should provide the * granular level of information that this processor is processing at this time * for better monitoring of this job. This returns object * which can be used for the purpose of display. * * @return Returns the information (record) this processor is processing. */ public abstract Object getProcessorState(); /** * Returns the status of the processor as a ProcessorStatus object, which gives * the information like whether the processor is running, suspended, resumed * or stopped. Defalt implementation of this method returns the status based * on the default implementation of suspend, resume and stop methods. * * @return Returns the processor status. */ public ProcessorStatus getProcessorStatus() { return this.processorStatus; } /** * This is for monitoring applications to know how many records this particular * (if there are multiple job processors) job processor is going to process. * * @return Returns the number of records/jobs this processor is going to process. */ public abstract long getTotalRecordsCount(); /** * This is for monitoring applications to know how many records this particular * (if there are multiple job processors) job processor has finished processing. * * @return Returns the number of records/jobs this processor has finished processing. */ public abstract long getProcessedRecordsCount(); } --- NEW FILE: BasicJobController.java --- package org.jmonks.batch.framework.controller.basic; import EDU.oswego.cs.dl.util.concurrent.Callable; import EDU.oswego.cs.dl.util.concurrent.CountDown; import EDU.oswego.cs.dl.util.concurrent.FutureResult; import java.util.Calendar; import java.util.Hashtable; import java.util.Iterator; import java.util.Map; import org.apache.log4j.Logger; import org.jmonks.batch.framework.JobStatistics; import org.jmonks.batch.framework.ErrorCode; import org.jmonks.batch.framework.JobContext; import org.jmonks.batch.framework.config.BasicJobControllerConfig; import org.jmonks.batch.framework.config.ConfigurationException; import org.jmonks.batch.framework.controller.JobController; import org.jmonks.batch.framework.management.ProcessorStatus; import org.jmonks.batch.framework.management.ProcessorState; /** * <p> * BasicJobController creates and executes the Basic Job Processor * class defined in job configuration. It creates the configured number * of instances and have each instance run in a seperate thread. * Once all the processors have finished their processing, returns the * appropriate return code. * </p> * * <p> * This controller is useful to write and execute a simple business logic * stands on its own. Allows the flexibility to run that code in a number * of instances. Following is an example configuration to configure the * batch job written based on BasicJobController. * <br><br> * <i>XML Configuration is as follows</i> <br><br> * <pre> * <job-config job-name="process_file_abc"> * <job-controller controller-class-name="org.jmonks.batch.framework.controller.basic.BasicJobController"> * <basic-job-processor basic-job-processor-class-name="com.mycompany.batch.processfileabc.AbcJobProcessor" thread-count="5"> * <property key="basic-job-processor-config1">basic-job-processor-value1</property> * </basic-job-processor> * <property key="basic-controller-config1">basic-controller-value1</property> * </job-controller> * <job-logging-config> * <logging-property-file>com.mycompany.batch.processfileabc.Logging</logging-property-file> * </job-logging-config> * </job-config> * </pre> * <br><br> * <i>DB Configuration is as follows</i> * <table border="1"> * <tr><td><b>TableName.ColumnName</b></td><td><b>Value</b></td></tr> * <tr><td>job_config.job_name</td><td>process_file_abc</td></tr> * <tr><td>job_config.job_status</td><td>1</td></tr> * <tr><td>job_config.job_controller_class_name</td><td>org.jmonks.batch.framework.controller.basic.BasicJobController</td></tr> * <tr><td>job_config.job_controller_props</td><td>basic-controller-config1=basic-controller-value1:basic-controller-config1=basic-controller-value2</td></tr> * <tr><td>basic_job_controller_config.job_name</td><td>process_file_abc</td></tr> * <tr><td>basic_job_controller_config.basic_job_processor_class_name</td><td>com.mycompany.batch.processfileabc.AbcJobProcessor</td></tr> * <tr><td>basic_job_controller_config.basic_job_processor_props</td><td>basic-job-processor-config1=basic-job-processor-value1</td></tr> * <tr><td>basic_job_controller_config.basic_job_processor_thread_cnt</td><td>1</td></tr> * </table> * </p> * * @author Suresh Pragada * @version 1.0 * @since 1.0 */ public class BasicJobController extends JobController { /** * Map holds all the basic job processors being used for the given job as values and * name of the threads as keys. */ private Map jobProcessorsMap=new Hashtable(); /** * Map holds all the basic job processors returned error codes as values and * name of the threads as keys. */ private Map jobProcessorsResultMap=new Hashtable(); /** * Holds the statistics of the basic job controller. */ private JobStatistics jobStatistics=null; private static Logger logger=Logger.getLogger(BasicJobController.class); /** * Constructor enables the instantiation of the basic job controller instance. */ public BasicJobController() { } /** * <p> * Executes the job by running the configured number of basic job processors * and returns the appropriate errorcode. It collects all the return codes returned * by job processors and return the first non successful return code it found in the * list. If it doesnt find any non successful return code, it will return successful * error code. * </p> * * @return Returns the processing error code for the job. */ public ErrorCode process() { logger.info("Entering process in basic job controller = " + super.jobContext.getJobName()); BasicJobControllerConfig basicJobControllerConfig=(BasicJobControllerConfig)super.jobContext.getJobConfig().getJobControllerConfig(); int threadCount=basicJobControllerConfig.getBasicJobProcessThreadCount(); /** * If thread count is not greater than zero.. set it as 0 */ threadCount=threadCount>0?threadCount:0; CountDown countDownLock=new CountDown(threadCount); logger.debug("Going to create " + threadCount + " basic job processor(s)"); this.jobStatistics=new JobStatistics(super.jobContext.getJobName()); this.jobStatistics.setStartTime(Calendar.getInstance().getTime()); for(int i=0;i<threadCount;i++) { String threadID=super.jobContext.getJobName()+"_"+(i+1); BasicJobProcessor jobProcessor=this.getBasicJobProcessor(basicJobControllerConfig.getBasicJobProcessorClassName()); FutureResult result=new FutureResult(); Thread processorThread=new Thread(result.setter(this.getCallableProcessor (countDownLock,jobProcessor,super.jobContext)), threadID); processorThread.start(); this.jobProcessorsMap.put(threadID,jobProcessor); this.jobProcessorsResultMap.put(threadID, result); logger.info(threadID + " basic job processor has been kicked off."); } ErrorCode returnCode=hybernate(countDownLock); logger.info("Exiting process in basic job controller = " + super.jobContext.getJobName() + " with return code = " + returnCode); return returnCode; } /** * Returns the total number of records this job going to process. This will be * the sum of the expected count from all the job processors. * * @return Returns the total number of records the job going to process. */ public long getExpectedRecordsCount() { logger.trace("Entering getExpectedRecordsCount"); long expectedRecordsCount=0; for(Iterator iterator=this.jobProcessorsMap.values().iterator();iterator.hasNext();) { BasicJobProcessor jobProcessor=(BasicJobProcessor)iterator.next(); expectedRecordsCount=expectedRecordsCount+jobProcessor.getTotalRecordsCount(); } logger.trace("Entering getExpectedRecordsCount = " + expectedRecordsCount); return expectedRecordsCount; } /** * Returns the number of records processed so far. This will be * the sum of the processed count from all the job processors. * * @return Returns the count of processed records. */ public long getProcessedRecordsCount() { logger.trace("Entering getProcessedRecordsCount"); long processedRecordsCount=0; for(Iterator iterator=this.jobProcessorsMap.values().iterator();iterator.hasNext();) { BasicJobProcessor jobProcessor=(BasicJobProcessor)iterator.next(); processedRecordsCount=processedRecordsCount+jobProcessor.getProcessedRecordsCount(); } logger.trace("Entering getProcessedRecordsCount = " + processedRecordsCount); return processedRecordsCount; } /** * Returns the IDs of all the processors as string array. * * @return Returns the string array consist of all the processor IDs. */ public String[] getProcessorIDList() { logger.trace("Entering getProcessorIDList"); String processorIDList[]=new String[this.jobProcessorsMap.size()]; int i=0; for(Iterator iterator=this.jobProcessorsMap.keySet().iterator();iterator.hasNext();i++) processorIDList[i]=(String)iterator.next(); logger.trace("Exiting getProcessorIDList"); return processorIDList; } /** * Returns the state of the requested processor as a ProcessorState object. * * @return Returns the processor state. */ public ProcessorState getProcessorState(String processorID) { logger.trace("Exiting getProcessorState = " + processorID); ProcessorState state=null; BasicJobProcessor jobProcessor=(BasicJobProcessor)this.jobProcessorsMap.get(processorID); if(jobProcessor!=null) state=new ProcessorState(processorID, "Basic Job Processor", jobProcessor.getProcessorState()); else state=null; logger.trace("Exiting getProcessorState = " + processorID + " state = " + state); return state; } /** * Returns the status of the processor identified by the given processor ID. * * @return Returns the ProcessorStatus object represents the status. */ public ProcessorStatus getProcessorStatus(String processorID) { logger.trace("Exiting getProcessorStatus = " + processorID); ProcessorStatus status=null; BasicJobProcessor jobProcessor=(BasicJobProcessor)this.jobProcessorsMap.get(processorID); if(jobProcessor!=null) status=jobProcessor.getProcessorStatus(); else status=null; logger.trace("Exiting getProcessorStatus = " + processorID + " state = " + status); return status; } /** * Stops the processor identified by the given processor ID. * * @return Returns true, if processor could be stopped, false otherwise. */ public boolean stop(String processorID) { logger.trace("Entering stop = " + processorID); boolean stopped=true; BasicJobProcessor jobProcessor=(BasicJobProcessor)this.jobProcessorsMap.get(processorID); if(jobProcessor!=null) stopped=jobProcessor.stop(); else stopped=false; logger.trace("Exiting stop = " + processorID + " status = " + stopped); return stopped; } /** * Suspends the processor identified by the given processor ID. * * @return Returns true if it could suspend the processor, false otherwise. */ public boolean suspend(String processorID) { logger.trace("Entering suspend = " + processorID); boolean suspended=true; BasicJobProcessor jobProcessor=(BasicJobProcessor)this.jobProcessorsMap.get(processorID); if(jobProcessor!=null) suspended=jobProcessor.suspend(); else suspended=false; logger.trace("Exiting suspend = " + processorID + " suspended = " + suspended); return suspended; } /** * Resumes the processor identified by given processor ID. * * @return Returns true if processor is resumed, false otherwise. */ public boolean resume(String processorID) { logger.trace("Entering resume = " + processorID); boolean resumed=true; BasicJobProcessor jobProcessor=(BasicJobProcessor)this.jobProcessorsMap.get(processorID); if(jobProcessor!=null) resumed=jobProcessor.resume(); else resumed=false; logger.trace("Exiting resume = " + processorID + " resumed = " + resumed); return resumed; } /** * Instantiates and returns the basic job processor using the given class name. * The given class name should implement the BasicJobProcessor interface. * * @param basicJobProcessorClassName Class name that implements the BasicJobProcessor interface. * * @return Returns the baic job processor instance. * * @throws ConfigurationException If it couldnt instantiate basic job processor instance. */ private BasicJobProcessor getBasicJobProcessor(String basicJobProcessorClassName) { logger.trace("Entering getBasicJobProcessor = " + basicJobProcessorClassName); BasicJobProcessor jobProcessor=null; try { jobProcessor=(BasicJobProcessor)Class.forName(basicJobProcessorClassName).newInstance(); } catch(ClassNotFoundException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, exception.getMessage()); } catch(InstantiationException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, exception.getMessage()); } catch(IllegalAccessException exception) { exception.printStackTrace(); logger.error(exception.getMessage(),exception); throw new ConfigurationException(ConfigurationException.JOB_CONTROLLER_CONFIG, exception.getMessage()); } logger.trace("Exiting getBasicJobProcessor"); return jobProcessor; } /** * <p> * Creates the runnable object by encapsulating the given job processor and * kick off the process method on the processor. Once processor has finsihed * its processing reports the error code to the controller by calling the "done" * method on the controller. * </p> * * @param countDownLock CountDown object to release when the processing has been done. * @param jobProcessor Initialized Basic job processor instance. * @param jobContext JobContext to be passed to the job processors. * * @return Returns the runnable instance. */ private Callable getCallableProcessor(final CountDown countDownLock,final BasicJobProcessor jobProcessor,final JobContext jobContext) { logger.trace("Entering getCallableProcessor"); Callable callable=new Callable(){ public Object call() { ErrorCode returnCode=null; try { boolean registered=jobProcessor.registerThread(); logger.debug("Status of registering thread with the processor = " + registered); logger.trace("Going to call the process method"); returnCode=jobProcessor.process(jobContext); logger.debug("Done calling the process method"); } catch(Throwable exception) { exception.printStackTrace(); logger.error("Exception while processing = " + exception.getMessage(), exception); returnCode=ErrorCode.BASIC_JOB_PROCESSOR_EXCEPTION; } countDownLock.release(); logger.info(Thread.currentThread().getName() + " is exiting with the error code = " + returnCode); return returnCode; } }; logger.trace("Exiting getCallableProcessor"); return callable; } /** * <p> * Main thread will call this method and wait until all the processors have * finished their processing. Once they are finished, all the return codes * will be analyzed and one error code will be returned for this job. * </p * @param countDownLock CountDown lock being used by all the processors. * * @return Returns the error code of the job. */ private ErrorCode hybernate(CountDown countDownLock) { logger.trace("Entering hybernate"); ErrorCode returnCode=ErrorCode.JOB_COMPLETED_SUCCESSFULLY; try { logger.debug("Going to wait until all the processors is gonna finish."); countDownLock.acquire(); logger.info("All processors have finished their task."); } catch(InterruptedException exception) { exception.printStackTrace(); logger.error("Exception while waiting for all the processors = " + exception.getMessage(), exception); return ErrorCode.BASIC_JOB_PROCESSOR_EXCEPTION; } this.jobStatistics.setEndTime(Calendar.getInstance().getTime()); this.jobStatistics.setMaxMemeoryUsage(Runtime.getRuntime().totalMemory()-Runtime.getRuntime().freeMemory()); this.jobStatistics.setRecordsProcessed(this.getProcessedRecordsCount()); for(Iterator iterator=this.jobProcessorsResultMap.values().iterator();iterator.hasNext();) { ErrorCode threadReturnCode=(ErrorCode)((FutureResult)iterator.next()).peek(); if(!ErrorCode.JOB_COMPLETED_SUCCESSFULLY.equals(threadReturnCode)) { returnCode=threadReturnCode; break; } } this.jobStatistics.setExitCode(returnCode); logger.trace("Exiting hybernate = " + returnCode); return returnCode; } /** * @see org.jmonks.batch.framework.controller.JobController#getJobStatistics() */ public JobStatistics getJobStatistics() { if(this.jobStatistics.getEndTime()!=null) return this.jobStatistics; else return null; } } |
