batchserver-coreframework — List to communication core framework information

Re: [Batchserver-coreframework] Framework Conceptual Design Document

[Anil V. P. <ani...@gm...>]

I am planning to have a group discussion on this .. sometime this evening
around 9pm (as we did it last time.)
So respond back to the group so that we can conf each other for the
discussion.


On 2/17/06, Suresh Pragada <sur...@gm...> wrote:
>
> Please find the attached document explains the conceptual design of
> framework. If anyone would like to go through document, please let me kno=
w,
> we can setup a call and discuss.
>
> Thank you,
> Suresh.
>
>


--
-- All our dreams can come true, If we have the courage to pursue them. -
Walt Disney

[Batchserver-coreframework] Re: Requirements Initiatvie

[Suresh P. <sur...@gm...>]

On 2/14/06, Suresh Pragada <sur...@gm...> wrote:
>
> Elaborated requirements.
>
>  1.1.1     Functional Requirements
>
> 1.      Should provide a solid architecture to write the batch jobs. This
> architecture needs to be easily understood, implement and use. This
> architecture should gives the flexibility to easily manageable and
> controllable and maintainable.
>
>
>
> 2.      Should support multithreading to execute jobs in required threads=
.
> Number of threads should be configurable in the batch configuration. When
> running in multiple threads, each thread should get a chance to initializ=
e
> its state itself and destroy the resources it has. Along with the individ=
ual
> initializations and destroy, there should be a global or common
> initialization and destroy point for all the threads. This is to accommod=
ate
> the scenarios of common initialization for all the threads and individual
> initializations for all the threads.
>
>
>
> 3.      Should provide a clean interface to enter/execute and exit the
> batch job. This means providing a nice interface to kickoff the batch job=
s.
> This enter interface should accept different kind of inputs to kick off t=
he
> batch job like job identifiers and job names and job configuration blocks=
.
> This interface should be responsible to identify the batch job needs to b=
e
> executed. This should provide clear exit information. It should define
> different kind of error codes and return them based on the state of the j=
ob.
> Generic error codes for different kind of errors are not acceptable.
>
>
>
> 4.      Performance statistics needs to be logged into the repository for
> analysis. These statistics not only includes the time to run the whole ba=
tch
> job, it should include time taken for each thread in case of multiple
> threads to understand the thread contention and thread usage. These
> statistics should also include the memory usage of the whole batch job an=
d
> each thread in case of multiple threads. *These statistics should also
> include the number of records the job has processed. *
>
>
>
> 5.      Should support different ways to configure the batch jobs i.e.,
> configuration should be able to provided in files and database as well.
> Configuration file formats should be defined for the flat and XML files a=
s
> well. Database Schema needs to be defined to configure the batch jobs in
> standard SQL. So that framework should be able to read the configuration
> from any database using standard JDBC drivers.
>
>
>
> 6.      Required information can be passed from the job configuration to
> batch jobs. Some components in the framework might need some extra
> configuration information of their own (may be like some name value pairs=
).
> Job configuration should provide a mechanism to defined the extra own
> configuration information.
>
>
>
> 7.      Configuration information given in configuration files should be
> overridden with the command line parameters. Configuration pertained to
> particular job can be overridden by passing the same kind of information
> through command line. Information passed through command line should take
> precedence over the configuration coming from the job configurations.
>
>
>
> 8.      Jobs can pass required information to other jobs and should able
> to read the information provided by other jobs. When a job wants to pass
> some data to other job that follows, it should be able to pass that
> information specifying the receiving job identifier (name) and data. Ther=
e
> should not be any constraints like what kind of data needs to be passed a=
nd
> format of data. The receiving job should able to understand this format.
> When a job passes some data to the job that follows, following job should
> able to read that information from the framework. When job passes the
> information to the next job, this data should override the earlier data. =
The
> receiving job should receive the latest data.
>
>
>
> 9.      Services like I/O, Transaction and Logging should be easily
> integrated with the framework. There should be placeholders for these
> services in the job configuration to integrate.
>
>
>
> 10.  Each job should provide enough information for the monitoring
> purposes. Framework should expose few standard interfaces to be implement=
ed
> by the batch jobs to provide standard monitoring information. This is lik=
e,
> if job is running in multiple threads, each thread should provide
> information what exactly it is processing. At the same time, framework al=
so
> needs to provide monitoring information like how far it is completed.
>
>
>
> 11.  Job should listen for the requests to do the soft kill. Framework
> should listen for the signals from server to stop, suspend, resume and
> restart. When it receives those signals, it should transmit them to the
> respective processing components in the framework to take necessary actio=
ns.
> In case of stop signal it should persist the state of the current job and
> quit. When it restarted again, it should get the persisted state back and
> resume the process. In case of suspend and resume signal, it doesn't have=
 to
> quit the job, it can wait until a defined time.
>
>
>
> 12.  Provide enough logging information. Log important events like data
> being transferred between different components and completion of processi=
ng
> of some data piece.
>
> Thank you,
> Suresh.
>
>
>  On 2/13/06, Suresh Pragada <sur...@gm...> wrote:
> >
> > Please see the following requirements. I am continue adding more
> > requirements and elaborating the existing requirements.
> >
> > 1.1.1     Functional Requirements
> >
> > 1.      Should provide a solid architecture to write the batch jobs.
> > This architecture needs to be easily understood, implement and use.
> >
> >
> >
> > 2.      Should support multithreading to execute jobs in required
> > threads.
> >
> >
> >
> > 3.      Should provide a clean interface to enter/execute and exit the
> > batch job. This means providing a nice interface to kickoff the batch j=
obs.
> >
> >
> >
> > 4.      Performance statistics needs to be logged into the repository
> > for analysis.
> >
> >
> >
> > 5.      Should support different ways to configure the batch jobs i.e.,
> > configuration should be able to provided in files and database.
> >
> >
> >
> > 6.      Required information can be passed from the job configuration t=
o
> > batch jobs.
> >
> >
> >
> > 7.      Configuration information given in configuration files should b=
e
> > overridden with the command line parameters.
> >
> >
> >
> > 8.      Jobs can pass required information to other jobs and should abl=
e
> > to read the information provided by other jobs.
> >
> >
> >
> > 9.      When running in a multithread mode, each worker (process
> > processing the data) should get a chance to initialize it self.
> >
> >
> >
> > 10.  When running in a multithread mode, each worker should get a chanc=
e
> > to close/destroy/release itself.
> >
> >
> >
> > 11.  When job is running in a multithread mode, there should be two
> > places executes only once (for all the threads) at the startup and in t=
he
> > end.
> >
> >
> >
> > 12.  Other services like I/O and Transaction should be easily integrate=
d
> > with the framework.
> >
> >
> >
> > 13.  Each job should provide enough information for the monitoring
> > purposes.
> >
> >
> >
> > 14.  Job should listen for the requests to do the soft kill.
> >
> >
> >
> > 15.  When a job is stopped in the middle, it should be restarted.
> >
> >
> >
> > 16.  Provide enough logging information.
> >
> >
> > Thank you,
> > Suresh.
> >
> >
> >
> > On 2/5/06, Suresh Pragada < sur...@gm...> wrote:
> > >
> > > Hey Anil & Sripal,
> > >  Do you guys have started anything on the detailed requirements of th=
e
> > > core framework & logging? If yes, please share your thoughts in this =
list.
> > >
> > >  I want to list a new requirement in the framework which is not liste=
d
> > > in the requirements document.
> > >
> > >  Req: Jobs should be able to share information.
> > >
> > > Thank you,
> > > Suresh.
> > >
> >
> >
>

[Batchserver-coreframework] Re: Requirements Initiatvie

[Suresh P. <sur...@gm...>]

Elaborated requirements.

 1.1.1     Functional Requirements

1.      Should provide a solid architecture to write the batch jobs. This
architecture needs to be easily understood, implement and use. This
architecture should gives the flexibility to easily manageable and
controllable and maintainable.



2.      Should support multithreading to execute jobs in required threads.
Number of threads should be configurable in the batch configuration. When
running in multiple threads, each thread should get a chance to initialize
its state itself and destroy the resources it has. Along with the individua=
l
initializations and destroy, there should be a global or common
initialization and destroy point for all the threads. This is to accommodat=
e
the scenarios of common initialization for all the threads and individual
initializations for all the threads.



3.      Should provide a clean interface to enter/execute and exit the batc=
h
job. This means providing a nice interface to kickoff the batch jobs. This
enter interface should accept different kind of inputs to kick off the batc=
h
job like job identifiers and job names and job configuration blocks. This
interface should be responsible to identify the batch job needs to be
executed. This should provide clear exit information. It should define
different kind of error codes and return them based on the state of the job=
.
Generic error codes for different kind of errors are not acceptable.



4.      Performance statistics needs to be logged into the repository for
analysis. These statistics not only includes the time to run the whole batc=
h
job, it should include time taken for each thread in case of multiple
threads to understand the thread contention and thread usage. These
statistics should also include the memory usage of the whole batch job and
each thread in case of multiple threads.



5.      Should support different ways to configure the batch jobs i.e.,
configuration should be able to provided in files and database as well.
Configuration file formats should be defined for the flat and XML files as
well. Database Schema needs to be defined to configure the batch jobs in
standard SQL. So that framework should be able to read the configuration
from any database using standard JDBC drivers.



6.      Required information can be passed from the job configuration to
batch jobs. Some components in the framework might need some extra
configuration information of their own (may be like some name value pairs).
Job configuration should provide a mechanism to defined the extra own
configuration information.



7.      Configuration information given in configuration files should be
overridden with the command line parameters. Configuration pertained to
particular job can be overridden by passing the same kind of information
through command line. Information passed through command line should take
precedence over the configuration coming from the job configurations.



8.      Jobs can pass required information to other jobs and should able to
read the information provided by other jobs. When a job wants to pass some
data to other job that follows, it should be able to pass that information
specifying the receiving job identifier (name) and data. There should not b=
e
any constraints like what kind of data needs to be passed and format of
data. The receiving job should able to understand this format. When a job
passes some data to the job that follows, following job should able to read
that information from the framework. When job passes the information to the
next job, this data should override the earlier data. The receiving job
should receive the latest data.



9.      Services like I/O, Transaction and Logging should be easily
integrated with the framework. There should be placeholders for these
services in the job configuration to integrate.



10.  Each job should provide enough information for the monitoring purposes=
.
Framework should expose few standard interfaces to be implemented by the
batch jobs to provide standard monitoring information. This is like, if job
is running in multiple threads, each thread should provide information what
exactly it is processing. At the same time, framework also needs to provide
monitoring information like how far it is completed.



11.  Job should listen for the requests to do the soft kill. Framework
should listen for the signals from server to stop, suspend, resume and
restart. When it receives those signals, it should transmit them to the
respective processing components in the framework to take necessary actions=
.
In case of stop signal it should persist the state of the current job and
quit. When it restarted again, it should get the persisted state back and
resume the process. In case of suspend and resume signal, it doesn't have t=
o
quit the job, it can wait until a defined time.



12.  Provide enough logging information. Log important events like data
being transferred between different components and completion of processing
of some data piece.

Thank you,
Suresh.


On 2/13/06, Suresh Pragada <sur...@gm...> wrote:
>
> Please see the following requirements. I am continue adding more
> requirements and elaborating the existing requirements.
>
> 1.1.1     Functional Requirements
>
> 1.      Should provide a solid architecture to write the batch jobs. This
> architecture needs to be easily understood, implement and use.
>
>
>
> 2.      Should support multithreading to execute jobs in required threads=
.
>
>
>
> 3.      Should provide a clean interface to enter/execute and exit the
> batch job. This means providing a nice interface to kickoff the batch job=
s.
>
>
>
> 4.      Performance statistics needs to be logged into the repository for
> analysis.
>
>
>
> 5.      Should support different ways to configure the batch jobs i.e.,
> configuration should be able to provided in files and database.
>
>
>
> 6.      Required information can be passed from the job configuration to
> batch jobs.
>
>
>
> 7.      Configuration information given in configuration files should be
> overridden with the command line parameters.
>
>
>
> 8.      Jobs can pass required information to other jobs and should able
> to read the information provided by other jobs.
>
>
>
> 9.      When running in a multithread mode, each worker (process
> processing the data) should get a chance to initialize it self.
>
>
>
> 10.  When running in a multithread mode, each worker should get a chance
> to close/destroy/release itself.
>
>
>
> 11.  When job is running in a multithread mode, there should be two place=
s
> executes only once (for all the threads) at the startup and in the end.
>
>
>
> 12.  Other services like I/O and Transaction should be easily integrated
> with the framework.
>
>
>
> 13.  Each job should provide enough information for the monitoring
> purposes.
>
>
>
> 14.  Job should listen for the requests to do the soft kill.
>
>
>
> 15.  When a job is stopped in the middle, it should be restarted.
>
>
>
> 16.  Provide enough logging information.
>
>
> Thank you,
> Suresh.
>
>
>
> On 2/5/06, Suresh Pragada <sur...@gm...> wrote:
> >
> > Hey Anil & Sripal,
> >  Do you guys have started anything on the detailed requirements of the
> > core framework & logging? If yes, please share your thoughts in this li=
st.
> >
> >  I want to list a new requirement in the framework which is not listed
> > in the requirements document.
> >
> >  Req: Jobs should be able to share information.
> >
> > Thank you,
> > Suresh.
> >
>
>

[Batchserver-coreframework] Re: Requirements Initiatvie

[Suresh P. <sur...@gm...>]

Please see the following requirements. I am continue adding more
requirements and elaborating the existing requirements.

1.1.1     Functional Requirements

1.      Should provide a solid architecture to write the batch jobs. This
architecture needs to be easily understood, implement and use.



2.      Should support multithreading to execute jobs in required threads.



3.      Should provide a clean interface to enter/execute and exit the batc=
h
job. This means providing a nice interface to kickoff the batch jobs.



4.      Performance statistics needs to be logged into the repository for
analysis.



5.      Should support different ways to configure the batch jobs i.e.,
configuration should be able to provided in files and database.



6.      Required information can be passed from the job configuration to
batch jobs.



7.      Configuration information given in configuration files should be
overridden with the command line parameters.



8.      Jobs can pass required information to other jobs and should able to
read the information provided by other jobs.



9.      When running in a multithread mode, each worker (process processing
the data) should get a chance to initialize it self.



10.  When running in a multithread mode, each worker should get a chance to
close/destroy/release itself.



11.  When job is running in a multithread mode, there should be two places
executes only once (for all the threads) at the startup and in the end.



12.  Other services like I/O and Transaction should be easily integrated
with the framework.



13.  Each job should provide enough information for the monitoring purposes=
.



14.  Job should listen for the requests to do the soft kill.



15.  When a job is stopped in the middle, it should be restarted.



16.  Provide enough logging information.


Thank you,
Suresh.



On 2/5/06, Suresh Pragada <sur...@gm...> wrote:
>
> Hey Anil & Sripal,
>  Do you guys have started anything on the detailed requirements of the
> core framework & logging? If yes, please share your thoughts in this list=
.
>
>  I want to list a new requirement in the framework which is not listed in
> the requirements document.
>
>  Req: Jobs should be able to share information.
>
> Thank you,
> Suresh.
>

[Batchserver-coreframework] Requirements Initiatvie

[Suresh P. <sur...@gm...>]

Hey Anil & Sripal,
 Do you guys have started anything on the detailed requirements of the core
framework & logging? If yes, please share your thoughts in this list.

 I want to list a new requirement in the framework which is not listed in
the requirements document.

 Req: Jobs should be able to share information.

Thank you,
Suresh.