Menu
▾
▴
firebird-devel
|
From: Nikolay S. <nik...@re...> - 2008-07-19 02:21:43
|
Hello, Dear Firebird Team Members! After almost two years in development, four betas and four release candidates, Red Database 2.1 is finally released. Information about the release is present here: http://www.red-soft.biz/en/index.html We are not interested at all in forking basic engine functionality, so here are my thoughts on what major features Red Database engine has, and how they can me merged into Firebird: SuperClassic engine ~~~~~~~~~~~~~~ We're pretty much in sync with Firebird HEAD regarding this feature. Roman Simakov and our QA team collaborate very effectively with Dmitry Emanov. Nothing special needs to be done. Java Stored Procedures ~~~~~~~~~~~~~~~~ Implementation is based on the same original ideas as ExtEngines branch and earlier implementations, is simple and rugged. Can serve as proof of concept implementation. As ExtEngines work progresses parts of our infrastructure should become unnecessary and the feature should converge into single code base. We hope that Eugeny Putilin collaborates further with Firebird team, so Firebird project can gain from our experience (and maybe replicate our system tables structure). While testing we gained quite a bit of understanding of JRE behaviors in different circumstances. Full-Text Search ~~~~~~~~~~~~ From Firebird project point of view, this is very experimental feature, which shows how complete working full-text-search can be implemented for Firebird. I ask project members to review it from the architectural standpoint. It is implemented as ESPs in Java. When the feature becomes architecturally stable, we may want to migrate it to Lucene C. Again, Eugeny Putilin have done most of the work, and should be in a position to help. PSQL debugger for SP/triggers ~~~~~~~~~~~~~~~~~~~~~ Proof of concept implementation, which currently uses CORBA for communication with debugger on the client side, and server-side debugger engine implemented in Java. We have discussed with Eugeny that Y-Valve can be extended to accomodate communication with debugger via the variety of supported protocols (INET, XNET, etc). Server-side Java helper on the server side becomes unnecessary in this case. We're not planning to extend it much further from what is done now, but debugger API works, in can be re-worked and polished by the community and user tools developers. Security features ~~~~~~~~~~~ The feature set aims to ensure complete coverage of ISO 27000 standard requirements which can be implemented by the database engine. There are similar projects for PostgreSQL (see SE-PosgreSQL), MySQL and commercial DBMS, but we hope our approach is much more comprehensive. However item-by-item consideration needs to be given to the features for merging. For example, Trace API has been in use for years already, and is quite stable. Extensions for using it as the audit trail tool were simple and should not cause problems. RBAC rework (e.g. access checking for GENERATORS and DDL operations, cumulative force of roles assigned to the user) can be merged in a relatively straightforward fashion. Things related to cryptography and row-level security obviously need more work and architectural review before they can be merged into Firebird. Other ~~~~ There are also NBACKUP stabilization changes and Trace API feature, which in my opinion can make their way into Firebird in 2.5 time frame, subject to Roman Simakov (or somebody else) availability to perform the merge. Additionally we have designed "External Engine sandboxes" feature, using the same extensions as XNET/INET communication mechanism for debugger to communicate with ESP running in separate processes, but this work is unlikely to progress further in the nearest future. No code was written to implement this design to the best of my knowledge. All features have detailed specs, but in Russian only. Unfortunately, not all our specialists and contractors are proficient in English. Release notes describe all the features in still bad, but hopefully comprehensible English. Source code is included with the install packages. Internally, patches are kept in merge-friendly form in an SVN branch growing from hand-made Firebird baseline (currently based on two-month-old Firebird HEAD + patches). We hope that working implementation of some of the features targeted for Firebird 3.0 release can make it become available sooner. Thanks and Best Regards, -- Nikolay Samofatov, MBA Red Soft International +1 416 710 6854 |
|
From: Adriano d. S. F. <adr...@uo...> - 2008-07-21 11:44:28
|
Nikolay Samofatov escreveu: > PSQL debugger for SP/triggers > ~~~~~~~~~~~~~~~~~~~~~ > Proof of concept implementation, which currently uses CORBA for > communication with debugger on the client side, and server-side debugger > engine implemented in Java. We have discussed with Eugeny that Y-Valve > can be extended to accomodate communication with debugger via the > variety of supported protocols (INET, XNET, etc). Server-side Java > helper on the server side becomes unnecessary in this case. We're not > planning to extend it much further from what is done now, but debugger > API works, in can be re-worked and polished by the community and user > tools developers. I've very different ideas of how this feature can be made, and it worth discuss benefits and drawbacks. In the client side, I'd prefer to have set of procedures and functions (in a package) and debugger application controls the debugging attachment by another attachment. There will be procedures and functions to set breakpoints, inspect variables, etc. The debugging connection may post events that the debugger could wait for. In the server side, a new debug (or notification) plugin is created, as well as a DEBUG external engine used for the procedure and function declarations. It allows to use the existing infrastructure (SQL, PSQL, events) and allow easy use from applications (debuggers). Adriano |
|
From: Nikolay S. <nik...@re...> - 2008-07-21 18:07:25
|
Adriano,
Adriano dos Santos Fernandes wrote:
> Nikolay Samofatov escreveu:
>
>> PSQL debugger for SP/triggers
>> ~~~~~~~~~~~~~~~~~~~~~
>> Proof of concept implementation, which currently uses CORBA for
>> communication with debugger on the client side, and server-side debugger
>> engine implemented in Java. We have discussed with Eugeny that Y-Valve
>> can be extended to accomodate communication with debugger via the
>> variety of supported protocols (INET, XNET, etc). Server-side Java
>> helper on the server side becomes unnecessary in this case. We're not
>> planning to extend it much further from what is done now, but debugger
>> API works, in can be re-worked and polished by the community and user
>> tools developers.
>>
> I've very different ideas of how this feature can be made, and it worth
> discuss benefits and drawbacks.
>
> In the client side, I'd prefer to have set of procedures and functions
> (in a package) and debugger application controls the debugging
> attachment by another attachment. There will be procedures and functions
> to set breakpoints, inspect variables, etc. The debugging connection may
> post events that the debugger could wait for.
>
> In the server side, a new debug (or notification) plugin is created, as
> well as a DEBUG external engine used for the procedure and function
> declarations.
>
> It allows to use the existing infrastructure (SQL, PSQL, events) and
> allow easy use from applications (debuggers).
>
Yes, Andriano, this is the architecture (minus packages) we discussed
with Eugeny Putilin as the way to go for the future.
You do not cover important topic though. How debugger can execute
queries in the context of attachment/transaction being debugged?
Especially if you factor in multi-process multi-threaded nature of V3
engine?
In my previous letter by "extending Y-Valve to accomodate communication
with debugger" I meant the following.
- you set breakpoints via calling ESP from main debugger attachment
execute procedure rdb$set_breakpoint ('<object_name>', <line>, ...);
- when debugger receives event that breakpoint hits it can query
"breakpoint hits" virtual table (or ESP) in main debugger attachment
e.g.
rdb$breakpoint_hits (
rdb$breakpoint_id,
rdb$attachment_id,
rdb$transaction_id,
rdb$datetime_hit,
...
)
- note that breakpoints may hit multiple times in different
attachments/transactions on a live server
- to execute queries at the breakpoint execution context (with existing
temp tables, context variables, savepoints, etc) debugger needs to
attach using the following connection string (assuming TCP is the
connection protocol)
<host_name>:<database_name_or_alias>$<attachment_id> and execute queries
in the context of this attachment.
For this to work, when breakpoint hits worker process which handles the
attachment needs to start listening on XNET endpoint and wait for
debugger to attach.
Y-Valve "Redirection" feature can than be used to attach to this
endpoint remotely.
Some sort of API (SQL command will do) will also be necessary to
impersonate a breakpoint's transaction after debugger has connected to
debugged attachment.
These ideas are also cluster-friendly (you can have listener on one
host, and worker process on another).
The same Y-Valve infrastructure can be used for external engines
residing in separate processes ("ESP sandboxes") to communicate with the
engine worker processes.
Please let me know if you need any further clarification.
> Adriano
>
--
Nikolay Samofatov, MBA
Red Soft International
+1 416 710 6854
|
|
From: Adriano d. S. F. <adr...@uo...> - 2008-07-21 19:16:23
|
Nikolay Samofatov escreveu: > - note that breakpoints may hit multiple times in different > attachments/transactions on a live server > I think debugger should attach to specific database sessions (i.e., one attachment). It will debug one attachment per each debug attachment. Debugging the whole server or all attachments to the database is something that not work really, I think. Debugging external procedures (in Java, for example) works in this way and it's not appropriate for a database. > - to execute queries at the breakpoint execution context (with existing > temp tables, context variables, savepoints, etc) debugger needs to > attach using the following connection string (assuming TCP is the > connection protocol) > <host_name>:<database_name_or_alias>$<attachment_id> and execute queries > in the context of this attachment. > More or less, I agree. Also, ability to connect from different process (and not always from the debugger) is great, if the debugger is not good SQL tool. Debugger just show IDs and one can connect using the preferred tool. > For this to work, when breakpoint hits worker process which handles the > attachment needs to start listening on XNET endpoint and wait for > debugger to attach. > Y-Valve "Redirection" feature can than be used to attach to this > endpoint remotely. > ... > Some sort of API (SQL command will do) will also be necessary to > impersonate a breakpoint's transaction after debugger has connected to > debugged attachment. > SET TRANSACTION may be extended so you can pass ID of existing debugging transaction. > These ideas are also cluster-friendly (you can have listener on one > host, and worker process on another). I may have not understood what you mean "Y-Valve redirection" and cluster-friendly here. For me, it should not be handled at Y-Valve, but at engine level. It catch debug connection string and associate the new attachment with the debugged attachment. Adriano |
|
From: Nikolay S. <nik...@re...> - 2008-07-22 04:44:41
|
Adriano, Adriano dos Santos Fernandes wrote: > Nikolay Samofatov escreveu: > >> - note that breakpoints may hit multiple times in different >> attachments/transactions on a live server >> >> > I think debugger should attach to specific database sessions (i.e., one > attachment). It will debug one attachment per each debug attachment. > Debugging the whole server or all attachments to the database is > something that not work really, I think. My understanding is that the ability to set global breakpoint in some SP is pretty important requirement. At the moment I don't see how it can be easily dropped or worked around. Not always the developer can control the entire application stack, and connection pooling techniques popular today make identifying offending attachment problematic. Thus global breakpoints are necessary. >> Some sort of API (SQL command will do) will also be necessary to >> impersonate a breakpoint's transaction after debugger has connected to >> debugged attachment. >> >> > SET TRANSACTION may be extended so you can pass ID of existing debugging > transaction. > Yep, agree. TPB parameter is another good way. >> These ideas are also cluster-friendly (you can have listener on one >> host, and worker process on another). >> > I may have not understood what you mean "Y-Valve redirection" Here is the Y-Valve feature I'm talking about as it is described in firebird.conf: "# Ability to redirect requests to other servers was initially present # in Interbase, but was broken by Borland in Interbase 6.0, when # they added SQL dialects. Request redirection was fixed in firebird 2.0, # but today such behaviour (proxy) seems to be dangerous from security # point of view. Imagine, you have one carefully protected firebird server, # access to which is possible from global net. But in case when this server # has access to your internal LAN (may and should be restricted, # but often possible), it will work as a gateway for incoming requests like: # firebird.your.domain.com:internal_server:/private/database.fdb # It's enough to know name/IP of some internal server on your LAN, and for # this connection one even need not know login/password on external server. # Such gateway easily overrides firewall, installed to protect your LAN # from outside attack. # # DO NOT ENABLE THIS OPTION UNLESS YOU REALLY KNOW WHAT YOU ARE DOING. # # Type: boolean # #Redirection = 0" > and > cluster-friendly here. > By cluster friendliness I mean the V3 configuration when engine runs on multiple hosts coordinated via DLM. One or more listener nodes balance and redirect attachments to processor nodes. In this case, when debugger needs to attach to debugged attachment, Y-Valve at listener would redirect connection to appropriate processor node, but not using XNET, but some intra-cluster protocol (for example, INET). > For me, it should not be handled at Y-Valve, but at engine level. It > catch debug connection string and associate the new attachment with the > debugged attachment. > I don't understand how what you propose can work for Classic and Embedded server (even if we leave aside other possible V3 configurations). > Adriano > -- Nikolay Samofatov, MBA Red Soft International +1 416 710 6854 |
|
From: Alex P. <pes...@ya...> - 2008-07-28 13:56:55
|
On Saturday 19 July 2008 06:21, Nikolay Samofatov wrote: > Full-Text Search > ~~~~~~~~~~~~ > From Firebird project point of view, this is very experimental feature, > which shows how complete working full-text-search can be implemented for > Firebird. > I ask project members to review it from the architectural standpoint. It > is implemented as ESPs in Java. When the feature becomes architecturally > stable, we may want to migrate it to Lucene C. Can you provide a small example - what is needed to use FTS on (for example) column DOCCONTEXT of table DOCS? > Security features > ~~~~~~~~~~~ > The feature set aims to ensure complete coverage of ISO 27000 standard > requirements which can be implemented by the database engine. > There are similar projects for PostgreSQL (see SE-PosgreSQL), MySQL and > commercial DBMS, but we hope our approach is much more comprehensive. I know that you have much enhanced authentication process. What protocol number do you use now? How is it related with FB (and IB) protocol numbers? |
|
From: Roman S. <rom...@re...> - 2008-07-28 16:05:14
|
Hello, Alex
Alex Peshkov wrote:
> On Saturday 19 July 2008 06:21, Nikolay Samofatov wrote:
>
>> Full-Text Search
>> ~~~~~~~~~~~~
>> From Firebird project point of view, this is very experimental feature,
>> which shows how complete working full-text-search can be implemented for
>> Firebird.
>> I ask project members to review it from the architectural standpoint. It
>> is implemented as ESPs in Java. When the feature becomes architecturally
>> stable, we may want to migrate it to Lucene C.
>>
>
> Can you provide a small example - what is needed to use FTS on (for example)
> column DOCCONTEXT of table DOCS?
>
>
For using full text search feature a database should be all necessary
special objects. For automatic creation of these objects for new
databases it is necessary to specify path to an initialization script as
a value of parameter InitScript in a configuration file of a server.
Then this script will be carried out at creation of each new database.
By default the name of a initialization script is init.sql, it is
located in the root directory of a server. Certainly you can run this
script manually for needed database.
Example of FTS usage: Search on a one field of the one table.
In the first, it is necessary to create an index:
EXECUTE PROCEDURE FTS$CREATE_INDEX('TEST_INDEX', NULL, NULL);
In the second, it is necessary to add a field in an index an index:
EXECUTE PROCEDURE FTS$ADD_FIELD_TO_INDEX('TEST_INDEX',
'DOCS','DOCCONTEXT', 'Standard', NULL, NULL);
After change of structure of an index, it is necessary to update the
metadata of FTS:
EXECUTE PROCEDURE FTS$APPLY_METADATA_CHANGES;
Adding test data into the table:
INSERT INTO DOCS(DOCCONTEXT) VALUES('fts test text');
To update index data it is necessary to refresh index:
EXECUTE PROCEDURE FTS$REINDEX('TEST_INDEX');
For searching data in an index FTS$SEARCH procedure is used:
SELECT FTS$SCOPE, FTS$RELATION, FTS$HIGHLIGHT from
FTS$SEARCH('TEST_INDEX', NULL, 'fts_test_text', 100);
More detail description of FTS is located on
http://www.red-soft.biz/files/downloads/products/2.1_docs/fts_man.pdf
(now in Russian only) or in docs directory of installed Red Database.
>> Security features
>> ~~~~~~~~~~~
>> The feature set aims to ensure complete coverage of ISO 27000 standard
>> requirements which can be implemented by the database engine.
>> There are similar projects for PostgreSQL (see SE-PosgreSQL), MySQL and
>> commercial DBMS, but we hope our approach is much more comprehensive.
>>
>
> I know that you have much enhanced authentication process. What protocol
> number do you use now? How is it related with FB (and IB) protocol numbers?
>
The protocol number is 12 and is equal Firebird 2.5.Enhancing
authentication process consists in implementation multi factor
authentication. It's extention of Firebird trusted authentication
process. Presence of isc_dpb_multi_factor_auth or
isc_spb_multi_factor_auth (in case of service) initiates multifactor
authentication process. So, client library from Firebird should work
with Red Database 2.1. Client library from Red Database 2.1 also must
work with Firebird if will not use isc_dpb_multi_factor_auth clumplet
tag in isc_(attach | create)_database.
I'll describe shortly used authentication process:
After sending isc_dpb_multi_factor_auth client starts to transfer one by
one authentication factor data: OS security context, password (by
improvement than in Firebird protocol like CHAP), X.509 certificate.
Transferring of one factor may take more than one transferring
iteration, for example, in case with OS security context. However it's
like trusted authentication of Firebird.
There are so named secondary authentication factors such as retina scan
and finger print. They are encrypted by a session key while
transferring. Session key is generated while primary factors are
transferred.
I'm ready to explain it more detail.
--
Best Regards,
Roman Simakov
Red Soft Corp.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X