[Batchserver-coreframework] Re: Requirements Initiatvie

[Suresh P. <sur...@gm...>]

On 2/14/06, Suresh Pragada <sur...@gm...> wrote:
>
> Elaborated requirements.
>
>  1.1.1     Functional Requirements
>
> 1.      Should provide a solid architecture to write the batch jobs. This
> architecture needs to be easily understood, implement and use. This
> architecture should gives the flexibility to easily manageable and
> controllable and maintainable.
>
>
>
> 2.      Should support multithreading to execute jobs in required threads=
.
> Number of threads should be configurable in the batch configuration. When
> running in multiple threads, each thread should get a chance to initializ=
e
> its state itself and destroy the resources it has. Along with the individ=
ual
> initializations and destroy, there should be a global or common
> initialization and destroy point for all the threads. This is to accommod=
ate
> the scenarios of common initialization for all the threads and individual
> initializations for all the threads.
>
>
>
> 3.      Should provide a clean interface to enter/execute and exit the
> batch job. This means providing a nice interface to kickoff the batch job=
s.
> This enter interface should accept different kind of inputs to kick off t=
he
> batch job like job identifiers and job names and job configuration blocks=
.
> This interface should be responsible to identify the batch job needs to b=
e
> executed. This should provide clear exit information. It should define
> different kind of error codes and return them based on the state of the j=
ob.
> Generic error codes for different kind of errors are not acceptable.
>
>
>
> 4.      Performance statistics needs to be logged into the repository for
> analysis. These statistics not only includes the time to run the whole ba=
tch
> job, it should include time taken for each thread in case of multiple
> threads to understand the thread contention and thread usage. These
> statistics should also include the memory usage of the whole batch job an=
d
> each thread in case of multiple threads. *These statistics should also
> include the number of records the job has processed. *
>
>
>
> 5.      Should support different ways to configure the batch jobs i.e.,
> configuration should be able to provided in files and database as well.
> Configuration file formats should be defined for the flat and XML files a=
s
> well. Database Schema needs to be defined to configure the batch jobs in
> standard SQL. So that framework should be able to read the configuration
> from any database using standard JDBC drivers.
>
>
>
> 6.      Required information can be passed from the job configuration to
> batch jobs. Some components in the framework might need some extra
> configuration information of their own (may be like some name value pairs=
).
> Job configuration should provide a mechanism to defined the extra own
> configuration information.
>
>
>
> 7.      Configuration information given in configuration files should be
> overridden with the command line parameters. Configuration pertained to
> particular job can be overridden by passing the same kind of information
> through command line. Information passed through command line should take
> precedence over the configuration coming from the job configurations.
>
>
>
> 8.      Jobs can pass required information to other jobs and should able
> to read the information provided by other jobs. When a job wants to pass
> some data to other job that follows, it should be able to pass that
> information specifying the receiving job identifier (name) and data. Ther=
e
> should not be any constraints like what kind of data needs to be passed a=
nd
> format of data. The receiving job should able to understand this format.
> When a job passes some data to the job that follows, following job should
> able to read that information from the framework. When job passes the
> information to the next job, this data should override the earlier data. =
The
> receiving job should receive the latest data.
>
>
>
> 9.      Services like I/O, Transaction and Logging should be easily
> integrated with the framework. There should be placeholders for these
> services in the job configuration to integrate.
>
>
>
> 10.  Each job should provide enough information for the monitoring
> purposes. Framework should expose few standard interfaces to be implement=
ed
> by the batch jobs to provide standard monitoring information. This is lik=
e,
> if job is running in multiple threads, each thread should provide
> information what exactly it is processing. At the same time, framework al=
so
> needs to provide monitoring information like how far it is completed.
>
>
>
> 11.  Job should listen for the requests to do the soft kill. Framework
> should listen for the signals from server to stop, suspend, resume and
> restart. When it receives those signals, it should transmit them to the
> respective processing components in the framework to take necessary actio=
ns.
> In case of stop signal it should persist the state of the current job and
> quit. When it restarted again, it should get the persisted state back and
> resume the process. In case of suspend and resume signal, it doesn't have=
 to
> quit the job, it can wait until a defined time.
>
>
>
> 12.  Provide enough logging information. Log important events like data
> being transferred between different components and completion of processi=
ng
> of some data piece.
>
> Thank you,
> Suresh.
>
>
>  On 2/13/06, Suresh Pragada <sur...@gm...> wrote:
> >
> > Please see the following requirements. I am continue adding more
> > requirements and elaborating the existing requirements.
> >
> > 1.1.1     Functional Requirements
> >
> > 1.      Should provide a solid architecture to write the batch jobs.
> > This architecture needs to be easily understood, implement and use.
> >
> >
> >
> > 2.      Should support multithreading to execute jobs in required
> > threads.
> >
> >
> >
> > 3.      Should provide a clean interface to enter/execute and exit the
> > batch job. This means providing a nice interface to kickoff the batch j=
obs.
> >
> >
> >
> > 4.      Performance statistics needs to be logged into the repository
> > for analysis.
> >
> >
> >
> > 5.      Should support different ways to configure the batch jobs i.e.,
> > configuration should be able to provided in files and database.
> >
> >
> >
> > 6.      Required information can be passed from the job configuration t=
o
> > batch jobs.
> >
> >
> >
> > 7.      Configuration information given in configuration files should b=
e
> > overridden with the command line parameters.
> >
> >
> >
> > 8.      Jobs can pass required information to other jobs and should abl=
e
> > to read the information provided by other jobs.
> >
> >
> >
> > 9.      When running in a multithread mode, each worker (process
> > processing the data) should get a chance to initialize it self.
> >
> >
> >
> > 10.  When running in a multithread mode, each worker should get a chanc=
e
> > to close/destroy/release itself.
> >
> >
> >
> > 11.  When job is running in a multithread mode, there should be two
> > places executes only once (for all the threads) at the startup and in t=
he
> > end.
> >
> >
> >
> > 12.  Other services like I/O and Transaction should be easily integrate=
d
> > with the framework.
> >
> >
> >
> > 13.  Each job should provide enough information for the monitoring
> > purposes.
> >
> >
> >
> > 14.  Job should listen for the requests to do the soft kill.
> >
> >
> >
> > 15.  When a job is stopped in the middle, it should be restarted.
> >
> >
> >
> > 16.  Provide enough logging information.
> >
> >
> > Thank you,
> > Suresh.
> >
> >
> >
> > On 2/5/06, Suresh Pragada < sur...@gm...> wrote:
> > >
> > > Hey Anil & Sripal,
> > >  Do you guys have started anything on the detailed requirements of th=
e
> > > core framework & logging? If yes, please share your thoughts in this =
list.
> > >
> > >  I want to list a new requirement in the framework which is not liste=
d
> > > in the requirements document.
> > >
> > >  Req: Jobs should be able to share information.
> > >
> > > Thank you,
> > > Suresh.
> > >
> >
> >
>