...reduced the engine startup time today by a factor of 10.
This was done by postponing the initialization of the database in several phases.
1.)
everything was removed from the Database constructor, except:
if (Trace.TRACE) {
Trace.trace();
}
sName = name;
tTable = new Vector();
aAccess = new Access();
cChannel = new Vector();
hAlias = new Hashtable();
bReferentialIntegrity = true;
and a couple of new flags were introduced:
initialized
mounted
needsMount
2.)
The database is not initialized until the first connection attempt
the first line of Database.connect(...):
if (!initialized) initializeDatabase();
initialization consists of creating and registering the hidden "sys" user, opening the log (if not in-memory mode), determining if the database is mounted, and creating the "SA" user if not mounted.
The database is considered mounted only if, during initialization, the log is openend against an existing set of database files.
3.)
The Database's DatabaseInformation class is lazily loaded, based on the mounted and needsMount status.
This was done by wrapping all DInfo member use with a private getDatabaseInformation method call.
The only complexity was the fact that execute() set the DatabaseInformation dirty flag, hence the needsMount flag had to be introduced to avoid setting the dirty flag until after the first call to execute to create the "SA" user, in the case where the database is initialized but not mounted after the first connection attemtpt. i.e. setting the dirty flag aat this point would load the DatabaseInformation class at initialization time rather than mount time, undoing the reduction in startup time.
4.)
If, after 1, 2, and 3, the database is not mounted, mounting is delayed until the first externally generated (i.e. not the create "SA" user during intitialization of a new database) statement is issued against the database.
Mounting consists of granting on the default classes (Library, DIJdbcLibrary, and java.lang.Math) and granting select to PUBLIC on all the system tables.
--------------------------------------------------
Preliminary results (on DELL XPS PII 333, running Redhat 6.2)
time to get first memory/standalone jdbcConnection from DatabaseManager:
Eager initialization/mount ~3000+/-300 ms
Deferred initialization/mount ~300+/-30 ms
time to get first new Database(String name) (in test loop:
Eager initialization/mount ~3000+/-300 ms
Deferred initialization/mount ~40+/-2 ms
Average (not including first)
Eager initialization/mount ~866+/-2 ms
Deferred initialization/mount ~1+/-1 ms
Campbell
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2001-12-03
Needs a bit more work though, I think, as further testing indicates that although startup is now much faster (an order of magitude) against memory database mode (no log file) and new file-based instances, mounting an exisiting file actually takes significantly longer (175% of original) (:-(.
Hopefully, a little debugging will show where the problem lies so I don't have to drop this approach. But perhaps it's "back to the drawing board" time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Because it won't compile in the main tree, due to dependencies created by other changes I have made.
I'd have to rip out my other changes to submit the whole Database.java as a patch.
I'll post the snippets over on patch, however, so you can take a look and see if you can't fix or at least figure out the above mentioned problem (or perhaps even point out other errors in logic or refine the "solution").
I'd like to spend more time on this, but I'm really trying to get more focused on finishing the MetaData patch. The reason I got into this at all was that my code, compile, run/test cycle was getting a bit bogged down with the ~3-4 second first memory/file startup times I was experiencing (I have a tendency to do no more than a few lines between each compile/run/test...I'm a whittler). So as usual, where there is an itch to scratch...
PS
You may find that this does not significantly speed up vanilla 1.61 engine startup, as the vanilla 1.61b DatabaseInformation class is not that large and has no external dependencies (mine depends on classes DITableDefs, DISort, DIJdbcConst, DIReflect, DIScript and initializes a ResourceBundle backed by a properties file to bring in internationalized system table and system table column comments).
Campbell
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi all.
...reduced the engine startup time today by a factor of 10.
This was done by postponing the initialization of the database in several phases.
1.)
everything was removed from the Database constructor, except:
if (Trace.TRACE) {
Trace.trace();
}
sName = name;
tTable = new Vector();
aAccess = new Access();
cChannel = new Vector();
hAlias = new Hashtable();
bReferentialIntegrity = true;
and a couple of new flags were introduced:
initialized
mounted
needsMount
2.)
The database is not initialized until the first connection attempt
the first line of Database.connect(...):
if (!initialized) initializeDatabase();
initialization consists of creating and registering the hidden "sys" user, opening the log (if not in-memory mode), determining if the database is mounted, and creating the "SA" user if not mounted.
The database is considered mounted only if, during initialization, the log is openend against an existing set of database files.
3.)
The Database's DatabaseInformation class is lazily loaded, based on the mounted and needsMount status.
This was done by wrapping all DInfo member use with a private getDatabaseInformation method call.
The only complexity was the fact that execute() set the DatabaseInformation dirty flag, hence the needsMount flag had to be introduced to avoid setting the dirty flag until after the first call to execute to create the "SA" user, in the case where the database is initialized but not mounted after the first connection attemtpt. i.e. setting the dirty flag aat this point would load the DatabaseInformation class at initialization time rather than mount time, undoing the reduction in startup time.
4.)
If, after 1, 2, and 3, the database is not mounted, mounting is delayed until the first externally generated (i.e. not the create "SA" user during intitialization of a new database) statement is issued against the database.
Mounting consists of granting on the default classes (Library, DIJdbcLibrary, and java.lang.Math) and granting select to PUBLIC on all the system tables.
--------------------------------------------------
Preliminary results (on DELL XPS PII 333, running Redhat 6.2)
time to get first memory/standalone jdbcConnection from DatabaseManager:
Eager initialization/mount ~3000+/-300 ms
Deferred initialization/mount ~300+/-30 ms
time to get first new Database(String name) (in test loop:
Eager initialization/mount ~3000+/-300 ms
Deferred initialization/mount ~40+/-2 ms
Average (not including first)
Eager initialization/mount ~866+/-2 ms
Deferred initialization/mount ~1+/-1 ms
Campbell
Needs a bit more work though, I think, as further testing indicates that although startup is now much faster (an order of magitude) against memory database mode (no log file) and new file-based instances, mounting an exisiting file actually takes significantly longer (175% of original) (:-(.
Hopefully, a little debugging will show where the problem lies so I don't have to drop this approach. But perhaps it's "back to the drawing board" time.
Campbell, what about posting the code?
Because it won't compile in the main tree, due to dependencies created by other changes I have made.
I'd have to rip out my other changes to submit the whole Database.java as a patch.
I'll post the snippets over on patch, however, so you can take a look and see if you can't fix or at least figure out the above mentioned problem (or perhaps even point out other errors in logic or refine the "solution").
I'd like to spend more time on this, but I'm really trying to get more focused on finishing the MetaData patch. The reason I got into this at all was that my code, compile, run/test cycle was getting a bit bogged down with the ~3-4 second first memory/file startup times I was experiencing (I have a tendency to do no more than a few lines between each compile/run/test...I'm a whittler). So as usual, where there is an itch to scratch...
PS
You may find that this does not significantly speed up vanilla 1.61 engine startup, as the vanilla 1.61b DatabaseInformation class is not that large and has no external dependencies (mine depends on classes DITableDefs, DISort, DIJdbcConst, DIReflect, DIScript and initializes a ResourceBundle backed by a properties file to bring in internationalized system table and system table column comments).
Campbell
Please post it. It should be possible to isolate this patch from the rest of your changes.
posted at:
http://sourceforge.net/tracker/index.php?func=detail&aid=488537&group_id=23316&atid=378133