parc-users — parc users mailing list

[Parc-users] Parc updates.

[McKinstry, P. (HQP) <Pet...@rh...>]

Neil passed along some good bug fixes for the JDBCInputAdapter. It was DB2
specific in the way that it expected to handle batches of records. I
commited his changes & added a new DB2JDBCInputAdapter that performs the old
way. For anyone using it that needs the existing functionality, you can just
prepend DB2 to the InputAdapter name in your config & you'll be good to go. 

-pgm

Pete McKinstry
IT Web Applications
Robert Half Intl
phone: 925.598.5422
email: pet...@rh...

[Parc-users] Fwd: new release .9.3.1

[Pete M. <pet...@ya...>]

This release is very minor. It provides the ability to
sleep between execution cycles. 
 
If you are using the InfiniteLoopStrategy or maybe the
ConditionalLoopStrategy, you may find it useful to
pause the interface for a bit between cycles. 

Enjoy! Let me know if you have any questions.

-pgm


__________________________________
Do you Yahoo!?
Protect your identity with Yahoo! Mail AddressGuard
http://antispam.yahoo.com/whatsnewfree

[Parc-users] Parc code updated.

[McKinstry, P. (HQP) <Pet...@rh...>]

The bug fix for CompositeProperties has been applied. (thanks Vaibhav). I
patched the .9.3 versions of the jar & source zip files last night. So if
you want to update, those files are the right ones. 

-pgm

Pete McKinstry
IT Web Applications
Robert Half Intl
phone: 925.598.5422
email: pet...@rh...

[Parc-users] update for error handling in thread pool

[Pete M. <pet...@ya...>]

I made a change to the threadpool impl & the output
records task to more properly handle uncaught
exceptions during output adapter processing. Thanks
Vaibhav for reporting this error. 

The output records task was changed to catch throwable
& report it as a fatal exception (just like
processingexception, auditexception, and
transactionexception) 

The threadpoolimpl was hanging when a worker thread
exited improperly. It was waiting in the join() method
for each worker thread to mark the work complete. If
the thread exited w/ a runtime or other uncaught
exception, the flag was never flipped. It's been
changed to check for work complete OR thread death. 

These fixes only affect improperly coded output
adapters. 

-pgm

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

[Parc-users] recent changes

[Pete M. <pet...@ya...>]

Vaibhav has added some support for nested property
files. This is especially useful when using multiple
datasources within a single parc application.

Take a look @ CompositeProperties.java along w/ it's
unit test to see how this works. Thanks Vaibhav!

On another note, to support multiple data sources,
some changes were required to the DataSourceFactory,
and the DBUtil class. The DataSourceFactory used to
take a Properties object that contained the
configuration data for a certain data source. It has
now changed to pass a data source name which keys to a
set of values in the configuration repositories. It is
nicer to use, but has a drawback. It breaks backward
compatibility w/ existing code. You now have to either
use DBUtil.getDataSource(), which retrieves the
default data source or DBUtil.getDataSource(String
name) which retrieves the named datasource. The
DataSource objects are also now loaded @ startup
instead of lazily upon first request. 

For users who got a DataSource by calling
DBUtil.getDataSource(), the required changes are
minimal. Basically you have to change your
configuration repository (properties file, database,
whatever) to prefix the datasource properties w/
DataSource.default.

For an example of the change, take a look @ the
DataSourceFactoryTest.properties. 

If users have been calling the DataSourceFactory
directly & not going through the DBUtil.java class,
the changes are more significant. They will need to
change the calls to pass a String ("default" for the
above example) instead of the actual properties
object. 

Anyway, I think the changes are a positive move, but I
wanted to get other peoples feedback on how painful
this change would be. Should we spend more time & find
a better compromise or is changing the property
prefixes minor enough that people are fine w/ it? 

Comments? 

-pgm

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

[Parc-users] RE: DataSourceFactory and DBUtil source

[Ranjangaonkar, V. (HQP) <Vai...@rh...>]

Yes, perfect ! first 2 sound reasonable and I can do that when I have access
to 
CVS source. But, first I need to build parc with the modified source in 
my sandbox. I will write a ConfigMixin implementation ( called
PropertiesConfigMixin )
and will send for review....
 
----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Thursday, August 21, 2003 3:28 PM
To: Ranjangaonkar, Vaibhav (HQP)
Cc: 'par...@li...'
Subject: RE: DataSourceFactory and DBUtil source



I would think we would want to change the BatchApplication class which loads
the properties instead of the PARCApplication. There are 2 useful subclasses
of BatchApplication: PARCApplication, which assumes a single
InputAdapter->{Filter Pipeline}->OutputAdapter set. And PBApplication, which
implements the GOF composite pattern and allows any number of
IA->Pipeline->OA sets (referred to as a "Process" -> see Process.java) to be
strung together. Both would benefit from the ability to nest properties
files. 
 
I guess the default parc jar has no ConfigMixin examples. That should be
fixed. The current example application that ships works because it passes a
System property (-D) on the cmdline telling PARC what properties file to
use. We should be providing a ConfigMixin implementation for loading from a
properties file as well. 
 
Anyway, I think we should do 3 things: 
1) Include your CompositeProperties class in the parc code base. 
2) Fix BatchApplication use the CompositeProperties class.
3) Write a ConfigMixin implementation that loads from a Properties file.
 
Sound reasonable? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 2:08 PM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Changed DataSourceFactory as per our discussion, please review this. But, to
use
CompositeProperties in the initilal part ( when application starts ), I had
to override 
getProperty( ) method in PARCApplication. Extended application code is
attached.
In order to add to PARC, we will have to change base class BatchApplication
itself.
 
Let me know your thoughts,
Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Thursday, August 21, 2003 10:24 AM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Let's do it. Make CompositeProperties resolve references as you've described
and we can fix PARC to use it. 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 10:16 AM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Yes, that's what we should do, actually, I misunderstood this part about 
the "#include" syntax before, I thught that with "#include" syntax, we are
deviating from 
the Properties file syntax and we will have to write whole parsing logic
ourselves.
But, now I see what you mean, the property file to be included should be
prefixed
with "include." and this reference will be resolved initially, so all the
modules will see
properties file as you have shown in 'after loading' part.
 
Yes, that's what we should do. If users prefer to enter all the values in
config.properties
instead of separating into another .properties file. And that's what is done
by
CompositeProperties class, after resolving references, it will put
properties into main file
as  datasourceA.db_url, the same way as you are thinking here. So, with the
use of
CompositeProperties, this can be supported easily. I guess, I didn't clarify
this part properly
in earlier emails.
 
So, what do you think, how should we go ahead from here ?
 
Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Thursday, August 21, 2003 10:03 AM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


I can certainly understand your position. 
 
I'm just hesitant to require extra property files for everyone. I'm
wondering if we can support both models? 
 
Perhaps via a prefix in the main property file we can identify named
DataSource parameters, but also allow the inclusion of other property files.
The property files would not need to require a prefix on the name if we
appended it when we loaded them. E.g. 
 
# config.properties
include.datasourceA=cia.properties
 
# cia.properties
db_url=xxx
username=scott
password=tiger
driver=com.oracle.xxx
 
after loading, becomes:
 
datasourceA.db_url-xxx
datasourceA.username=scott
datasourceA.password=tiger
datasourceA.driver=com.oracle.xxx
 
But, the user could also include the final syntax in the main properties
file if they so desire. 


What do you think? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 9:28 AM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


So, I think referencing other property files from the main one is like
#include only. 
It has additional advantage of being still a properties file, whereas, if we
add a syntax
like #include it will be a different config file and we will have to parse
it and still it will 
be similar to what is done in CompositeProperties. We can also use
CompositeProperties
to parse this main config.properties and then handover resolved Properties
object to all
the subsequent init() methods.


Keeping properties separate is my preferred approach as it keeps them
structured and
avoids copy-paste kind of reuse. For us, we will have these properties file
changing with
environements like, webdev.properties & ciadev.properties in sandbox,
webfix.properties
& ciafix.properties in mirror. If we copy-paste entries in config.properties
to avoid a separate
properties file, we need to change all the entries related to datasource
config in all the 
interface file as the environment changes. That's more work than having
separate properties 
file.
 
Then, this is not just the issue with DataSource configuration, same issue
might arise for
other subsystems of PARC like, logging or adapters. I am going to add
another set of 
properties files for specifying sql queries and other config things for
InputAdapter.
 
So, in general, I don't like having all the properties broken up in main
config.properties.
And, I don't see the difference between providing #include option and
mentioning a properties
file name as value for config key. That's my opinion, maybe I am missing
something here ...


Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Wednesday, August 20, 2003 6:02 PM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Do you think it's worth thinking about whether we could include these 2
property files inside the core config.properties? 
 
I'm just worried about proliferation of config files. If we could come up w/
a naming standard, there doesn't seem to be any reason we couldn't load the
DS specific props from the common file. We could also add a syntax to
#include extra config files in the common one. That would be a cool
addition. 
 
thoughts? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Wednesday, August 20, 2003 9:22 AM
To: McKinstry, Pete (HQP)
Subject: DataSourceFactory and DBUtil source


Hi Pete, 
 
Please take a look at these 2 changes for supporting multiple 
data sources, 
 
With this change, config.properties will look like this for 2 data sources -
 
DataSource.web=webdev.properties
DataSource.cia=ciadev.properties
 
and classes needing these will call DBUtil.getDataSource( "web" ).
Let me know is something is not correct or missing, mainly, if it goes
with other parts and overall design of the framework.
 
Thanks,
Vaibhav

[Parc-users] RE: DataSourceFactory and DBUtil source

[McKinstry, P. (HQP) <Pet...@rh...>]

I would think we would want to change the BatchApplication class which loads
the properties instead of the PARCApplication. There are 2 useful subclasses
of BatchApplication: PARCApplication, which assumes a single
InputAdapter->{Filter Pipeline}->OutputAdapter set. And PBApplication, which
implements the GOF composite pattern and allows any number of
IA->Pipeline->OA sets (referred to as a "Process" -> see Process.java) to be
strung together. Both would benefit from the ability to nest properties
files. 
 
I guess the default parc jar has no ConfigMixin examples. That should be
fixed. The current example application that ships works because it passes a
System property (-D) on the cmdline telling PARC what properties file to
use. We should be providing a ConfigMixin implementation for loading from a
properties file as well. 
 
Anyway, I think we should do 3 things: 
1) Include your CompositeProperties class in the parc code base. 
2) Fix BatchApplication use the CompositeProperties class.
3) Write a ConfigMixin implementation that loads from a Properties file.
 
Sound reasonable? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 2:08 PM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Changed DataSourceFactory as per our discussion, please review this. But, to
use
CompositeProperties in the initilal part ( when application starts ), I had
to override 
getProperty( ) method in PARCApplication. Extended application code is
attached.
In order to add to PARC, we will have to change base class BatchApplication
itself.
 
Let me know your thoughts,
Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Thursday, August 21, 2003 10:24 AM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Let's do it. Make CompositeProperties resolve references as you've described
and we can fix PARC to use it. 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 10:16 AM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Yes, that's what we should do, actually, I misunderstood this part about 
the "#include" syntax before, I thught that with "#include" syntax, we are
deviating from 
the Properties file syntax and we will have to write whole parsing logic
ourselves.
But, now I see what you mean, the property file to be included should be
prefixed
with "include." and this reference will be resolved initially, so all the
modules will see
properties file as you have shown in 'after loading' part.
 
Yes, that's what we should do. If users prefer to enter all the values in
config.properties
instead of separating into another .properties file. And that's what is done
by
CompositeProperties class, after resolving references, it will put
properties into main file
as  datasourceA.db_url, the same way as you are thinking here. So, with the
use of
CompositeProperties, this can be supported easily. I guess, I didn't clarify
this part properly
in earlier emails.
 
So, what do you think, how should we go ahead from here ?
 
Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Thursday, August 21, 2003 10:03 AM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


I can certainly understand your position. 
 
I'm just hesitant to require extra property files for everyone. I'm
wondering if we can support both models? 
 
Perhaps via a prefix in the main property file we can identify named
DataSource parameters, but also allow the inclusion of other property files.
The property files would not need to require a prefix on the name if we
appended it when we loaded them. E.g. 
 
# config.properties
include.datasourceA=cia.properties
 
# cia.properties
db_url=xxx
username=scott
password=tiger
driver=com.oracle.xxx
 
after loading, becomes:
 
datasourceA.db_url-xxx
datasourceA.username=scott
datasourceA.password=tiger
datasourceA.driver=com.oracle.xxx
 
But, the user could also include the final syntax in the main properties
file if they so desire. 


What do you think? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Thursday, August 21, 2003 9:28 AM
To: McKinstry, Pete (HQP)
Subject: RE: DataSourceFactory and DBUtil source


So, I think referencing other property files from the main one is like
#include only. 
It has additional advantage of being still a properties file, whereas, if we
add a syntax
like #include it will be a different config file and we will have to parse
it and still it will 
be similar to what is done in CompositeProperties. We can also use
CompositeProperties
to parse this main config.properties and then handover resolved Properties
object to all
the subsequent init() methods.


Keeping properties separate is my preferred approach as it keeps them
structured and
avoids copy-paste kind of reuse. For us, we will have these properties file
changing with
environements like, webdev.properties & ciadev.properties in sandbox,
webfix.properties
& ciafix.properties in mirror. If we copy-paste entries in config.properties
to avoid a separate
properties file, we need to change all the entries related to datasource
config in all the 
interface file as the environment changes. That's more work than having
separate properties 
file.
 
Then, this is not just the issue with DataSource configuration, same issue
might arise for
other subsystems of PARC like, logging or adapters. I am going to add
another set of 
properties files for specifying sql queries and other config things for
InputAdapter.
 
So, in general, I don't like having all the properties broken up in main
config.properties.
And, I don't see the difference between providing #include option and
mentioning a properties
file name as value for config key. That's my opinion, maybe I am missing
something here ...


Thanks,
Vaibhav

-----Original Message-----
From: McKinstry, Pete (HQP) 
Sent: Wednesday, August 20, 2003 6:02 PM
To: Ranjangaonkar, Vaibhav (HQP)
Subject: RE: DataSourceFactory and DBUtil source


Do you think it's worth thinking about whether we could include these 2
property files inside the core config.properties? 
 
I'm just worried about proliferation of config files. If we could come up w/
a naming standard, there doesn't seem to be any reason we couldn't load the
DS specific props from the common file. We could also add a syntax to
#include extra config files in the common one. That would be a cool
addition. 
 
thoughts? 
 
-pgm

-----Original Message-----
From: Ranjangaonkar, Vaibhav (HQP) 
Sent: Wednesday, August 20, 2003 9:22 AM
To: McKinstry, Pete (HQP)
Subject: DataSourceFactory and DBUtil source


Hi Pete, 
 
Please take a look at these 2 changes for supporting multiple 
data sources, 
 
With this change, config.properties will look like this for 2 data sources -
 
DataSource.web=webdev.properties
DataSource.cia=ciadev.properties
 
and classes needing these will call DBUtil.getDataSource( "web" ).
Let me know is something is not correct or missing, mainly, if it goes
with other parts and overall design of the framework.
 
Thanks,
Vaibhav

[Parc-users] test msg

[McKinstry, P. (HQP) <Pet...@rh...>]

Testing to make sure the lists are working....

-pgm

Pete McKinstry
IT Web Applications
Robert Half Int'l
*: 925.598.5422
*: pet...@rh...