[Batchserver-coreframework] Re: Requirements Initiatvie

[Suresh P. <sur...@gm...>]

Elaborated requirements.

 1.1.1     Functional Requirements

1.      Should provide a solid architecture to write the batch jobs. This
architecture needs to be easily understood, implement and use. This
architecture should gives the flexibility to easily manageable and
controllable and maintainable.



2.      Should support multithreading to execute jobs in required threads.
Number of threads should be configurable in the batch configuration. When
running in multiple threads, each thread should get a chance to initialize
its state itself and destroy the resources it has. Along with the individua=
l
initializations and destroy, there should be a global or common
initialization and destroy point for all the threads. This is to accommodat=
e
the scenarios of common initialization for all the threads and individual
initializations for all the threads.



3.      Should provide a clean interface to enter/execute and exit the batc=
h
job. This means providing a nice interface to kickoff the batch jobs. This
enter interface should accept different kind of inputs to kick off the batc=
h
job like job identifiers and job names and job configuration blocks. This
interface should be responsible to identify the batch job needs to be
executed. This should provide clear exit information. It should define
different kind of error codes and return them based on the state of the job=
.
Generic error codes for different kind of errors are not acceptable.



4.      Performance statistics needs to be logged into the repository for
analysis. These statistics not only includes the time to run the whole batc=
h
job, it should include time taken for each thread in case of multiple
threads to understand the thread contention and thread usage. These
statistics should also include the memory usage of the whole batch job and
each thread in case of multiple threads.



5.      Should support different ways to configure the batch jobs i.e.,
configuration should be able to provided in files and database as well.
Configuration file formats should be defined for the flat and XML files as
well. Database Schema needs to be defined to configure the batch jobs in
standard SQL. So that framework should be able to read the configuration
from any database using standard JDBC drivers.



6.      Required information can be passed from the job configuration to
batch jobs. Some components in the framework might need some extra
configuration information of their own (may be like some name value pairs).
Job configuration should provide a mechanism to defined the extra own
configuration information.



7.      Configuration information given in configuration files should be
overridden with the command line parameters. Configuration pertained to
particular job can be overridden by passing the same kind of information
through command line. Information passed through command line should take
precedence over the configuration coming from the job configurations.



8.      Jobs can pass required information to other jobs and should able to
read the information provided by other jobs. When a job wants to pass some
data to other job that follows, it should be able to pass that information
specifying the receiving job identifier (name) and data. There should not b=
e
any constraints like what kind of data needs to be passed and format of
data. The receiving job should able to understand this format. When a job
passes some data to the job that follows, following job should able to read
that information from the framework. When job passes the information to the
next job, this data should override the earlier data. The receiving job
should receive the latest data.



9.      Services like I/O, Transaction and Logging should be easily
integrated with the framework. There should be placeholders for these
services in the job configuration to integrate.



10.  Each job should provide enough information for the monitoring purposes=
.
Framework should expose few standard interfaces to be implemented by the
batch jobs to provide standard monitoring information. This is like, if job
is running in multiple threads, each thread should provide information what
exactly it is processing. At the same time, framework also needs to provide
monitoring information like how far it is completed.



11.  Job should listen for the requests to do the soft kill. Framework
should listen for the signals from server to stop, suspend, resume and
restart. When it receives those signals, it should transmit them to the
respective processing components in the framework to take necessary actions=
.
In case of stop signal it should persist the state of the current job and
quit. When it restarted again, it should get the persisted state back and
resume the process. In case of suspend and resume signal, it doesn't have t=
o
quit the job, it can wait until a defined time.



12.  Provide enough logging information. Log important events like data
being transferred between different components and completion of processing
of some data piece.

Thank you,
Suresh.


On 2/13/06, Suresh Pragada <sur...@gm...> wrote:
>
> Please see the following requirements. I am continue adding more
> requirements and elaborating the existing requirements.
>
> 1.1.1     Functional Requirements
>
> 1.      Should provide a solid architecture to write the batch jobs. This
> architecture needs to be easily understood, implement and use.
>
>
>
> 2.      Should support multithreading to execute jobs in required threads=
.
>
>
>
> 3.      Should provide a clean interface to enter/execute and exit the
> batch job. This means providing a nice interface to kickoff the batch job=
s.
>
>
>
> 4.      Performance statistics needs to be logged into the repository for
> analysis.
>
>
>
> 5.      Should support different ways to configure the batch jobs i.e.,
> configuration should be able to provided in files and database.
>
>
>
> 6.      Required information can be passed from the job configuration to
> batch jobs.
>
>
>
> 7.      Configuration information given in configuration files should be
> overridden with the command line parameters.
>
>
>
> 8.      Jobs can pass required information to other jobs and should able
> to read the information provided by other jobs.
>
>
>
> 9.      When running in a multithread mode, each worker (process
> processing the data) should get a chance to initialize it self.
>
>
>
> 10.  When running in a multithread mode, each worker should get a chance
> to close/destroy/release itself.
>
>
>
> 11.  When job is running in a multithread mode, there should be two place=
s
> executes only once (for all the threads) at the startup and in the end.
>
>
>
> 12.  Other services like I/O and Transaction should be easily integrated
> with the framework.
>
>
>
> 13.  Each job should provide enough information for the monitoring
> purposes.
>
>
>
> 14.  Job should listen for the requests to do the soft kill.
>
>
>
> 15.  When a job is stopped in the middle, it should be restarted.
>
>
>
> 16.  Provide enough logging information.
>
>
> Thank you,
> Suresh.
>
>
>
> On 2/5/06, Suresh Pragada <sur...@gm...> wrote:
> >
> > Hey Anil & Sripal,
> >  Do you guys have started anything on the detailed requirements of the
> > core framework & logging? If yes, please share your thoughts in this li=
st.
> >
> >  I want to list a new requirement in the framework which is not listed
> > in the requirements document.
> >
> >  Req: Jobs should be able to share information.
> >
> > Thank you,
> > Suresh.
> >
>
>