Thread: RE: [Firebird-devel] Embedded superserver

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Nando,

My 2 cents...

> - does it make sense from a logical POV?

Nope.  That functionality is know as a "full engine"

Sean

Nando,

> But, failing that, the "thing" should still be able to be driven by
> configuration files. WRT switches, perhaps the "start" function could
> accept a command line in input.

You have completely and utterly confused me.

You want an "embedded" server which:

  - is hosted in a single DLL/shared library

  - has "auto-start" functionality

  - can support remote connections

Is this correct?

Sean

Sean,

S> You have completely and utterly confused me.

It looks like you've got a pretty clear picture, instead.

S> You want an "embedded" server which:

S>   - is hosted in a single DLL/shared library

Yes. Think of it as a "loadable superserver", if "embedded" carries too
much of a meaning currently.

S>   - has "auto-start" functionality

Not necessary (not even desirable, according to Jim). An explicit start
exported function would be enough/better.

S>   - can support remote connections

Yes.

S> Is this correct?

Yes. And I have the feeling that once it's available people will start
to use it and wonder why haven't they thought about it before. ;-)

Ciao
-- 
Nando Dessena
mailto:na...@de...

> S>   - has "auto-start" functionality
>
> Not necessary (not even desirable, according to Jim). An explicit start
> exported function would be enough/better.
>
> S>   - can support remote connections
>
> Yes.
>
> S> Is this correct?
>
> Yes. And I have the feeling that once it's available people will start
> to use it and wonder why haven't they thought about it before. ;-)
>
> Ciao
> --
> Nando Dessena
> mailto:na...@de...

how would this 3rd party application now that the dll on this other platform
is loaded by this "other" application and ready to start by a call to it's
exported function?
Alan

Alan,

A> how would this 3rd party application now that the dll on this other platform
A> is loaded by this "other" application and ready to start by a call to it's
A> exported function?

Sorry, I don't follow you. Could you please elaborate?
Ciao
-- 
Nando Dessena
mailto:na...@de...

> S>   - has "auto-start" functionality
>
> Not necessary (not even desirable, according to Jim). An explicit start
> exported function would be enough/better.
>
> S>   - can support remote connections
>
> Yes.
>
> S> Is this correct?
>
> Yes. And I have the feeling that once it's available people will start
> to use it and wonder why haven't they thought about it before. ;-)
>
> Ciao
> --
> Nando Dessena

If the application hosting the dll desided to quit, what would happen to 3rd
party applications using the dll as it's means of serving data? The host
application would just close and drop the dll wouldn't it?
Alan

Alan,

A> If the application hosting the dll desided to quit, what would happen to 3rd
A> party applications using the dll as it's means of serving data? The host
A> application would just close and drop the dll wouldn't it?

Yes. Much like what happens when you shutdown an application that
hosts an embedded server (fbembed.dll) or the fbserver itself. Client
connections vanish.

For graceful shutdown a stop() exported function that pairs the
start() one would be useful, but anyway Firebird is able to
suffer termination without corrupting the data (in forced writes mode,
that is).

Ciao
-- 
Nando Dessena
mailto:na...@de...

> A> If the application hosting the dll desided to quit, what would
> happen to 3rd
> A> party applications using the dll as it's means of serving
> data? The host
> A> application would just close and drop the dll wouldn't it?
>
> Yes. Much like what happens when you shutdown an application that
> hosts an embedded server (fbembed.dll) or the fbserver itself. Client
> connections vanish.

but that's very rude isn't it? unlike a server which has a tendency to be
left running all the time, a host application can be started and stopped
frequently - you wouldn't get me connecting to a host app with little chance
of me being able to save my work before they decided to switch off and go
home for the night....

>
> For graceful shutdown a stop() exported function that pairs the
> start() one would be useful, but anyway Firebird is able to
> suffer termination without corrupting the data (in forced writes mode,
> that is).
>
> Ciao
> --
> Nando Dessena

That shoutdown would be fine if it were being called from the third party
application but if the host application called it, you would need a
mechanism to warn the 3rd party app to get ready for shutdown.

Alan

Alan,

>> A> If the application hosting the dll desided to quit, what would
>> happen to 3rd
>> A> party applications using the dll as it's means of serving
>> data? The host
>> A> application would just close and drop the dll wouldn't it?
>>
>> Yes. Much like what happens when you shutdown an application that
>> hosts an embedded server (fbembed.dll) or the fbserver itself. Client
>> connections vanish.

A> but that's very rude isn't it? unlike a server which has a tendency to be
A> left running all the time, a host application can be started and stopped
A> frequently -

my host application *is* a server and runs as a service. I'd think
that anyone intending to embed (or otherwise implement) a server open
to remote connections should take all the necessary precautions.
Again, this is not specific to Firebird. In this context Firebird is
just a component that I am using to build my server application. I
would like that component to be in the form of a DLL rather than the
current exe form.

A> you wouldn't get me connecting to a host app with little chance
A> of me being able to save my work before they decided to switch off and go
A> home for the night....

...which is also true for fbserver.exe running as an application.

I hope what I wrote above clears the misunderstanding. Otherwise
please feel free to repost.

Ciao
-- 
Nando Dessena
mailto:na...@de...

> my host application *is* a server and runs as a service. I'd
> think that anyone intending to embed (or otherwise implement)
> a server open to remote connections should take all the
> necessary precautions.

So what this is all about really is...

Rather than a two step install where you install
  1. Fb-Server as a service
  2. Your own application as a service

You would like to do it in one step
  1. Install your own app that also installs Fb-Server.

Other than installation what other advantages would such a
configuration offer?

I am guessing that, with an appropriate API, you may be able
to offer additional server management capabilities.  But then
again with the appropriate API these should be available even
with the normal configuration.

You can already integrate a FB install into your own install,
although the result is still a separate service.  So the
install advantage does not seem to be very great.

To be blunt;  What is the point?

-- 
Geoff Worboys
Telesis Computing

Geoff,

G> Rather than a two step install where you install
G>   1. Fb-Server as a service
G>   2. Your own application as a service

G> You would like to do it in one step
G>   1. Install your own app that also installs Fb-Server.

G> Other than installation what other advantages would such a
G> configuration offer?

I think I have briefly said that in a post to Dmitry. Here it is:
synchronizing startup and shutdown of two processes, or more generally
maintaining a host/child relation between two processes is problematic
in Windows without introducing a third "guardian" application.
Currently I start fbserver.exe on my app's startup and stop in on
termination, but it's not very reliable. Plus, fbserver.exe is shown
in the process list and could be terminated (since it runs as an
application) without my app noticing. I could setup (partial)
workarounds for these situations, but I find the proposal I am
advocating simpler, more straightforward, more robust and in general
architecturally sounder.

G> I am guessing that, with an appropriate API, you may be able
G> to offer additional server management capabilities.

That's one of the things my app does. Automated database upgrade, for
example, or alias listing, or additional security (the ability to
assign particular aliases to particular users). In short, I want to
augment the firebird server with features that are specific to my
application domain, and I don't necessariliy want to do it using a
three tier architecture (actually I currently can't).

G> But then
G> again with the appropriate API these should be available even
G> with the normal configuration.

There's nothing I plan to do that I can't do in a way or the other
now. Easier deployment and more robustness are the biggest advantages
of the proposal. Added functionality is not.

G> You can already integrate a FB install into your own install,
G> although the result is still a separate service.  So the
G> install advantage does not seem to be very great.

It is. I hope I have explained it better now.
Try to do it with the current tools and you'll soon run into small but
annoying brick walls.

Ciao
-- 
Nando Dessena
mailto:na...@de...

> A> but that's very rude isn't it? unlike a server which has a
> tendency to be
> A> left running all the time, a host application can be started
> and stopped
> A> frequently -
>
> my host application *is* a server and runs as a service. I'd think
> that anyone intending to embed (or otherwise implement) a server open
> to remote connections should take all the necessary precautions.
> Again, this is not specific to Firebird. In this context Firebird is
> just a component that I am using to build my server application. I
> would like that component to be in the form of a DLL rather than the
> current exe form.

but don't we already have this? it's called fbserver. You can write your
server application any way you wish and it can still talk to the server
service. Why do you need a dll to do the identical thing that the full
server does?

>
> A> you wouldn't get me connecting to a host app with little chance
> A> of me being able to save my work before they decided to switch
> off and go
> A> home for the night....
>
> ...which is also true for fbserver.exe running as an application.
>

nuh - I still don't understand what you want but I'll drop it now and watch
...

Alan

Alan,

A> Why do you need a dll to do the identical thing that the full
A> server does?

Why do you need a zip dll to do the identical thing that winzip does?

IIRC you are a Delphi developer. Why do you need a component on your
form or datamodule when you could just invoke an external executable
that does the same things?

See my post to Geoff. I seem to have lost the ability to translate
simple thoughts into understandable words - assuming I ever had that
ability. ;-)

Ciao
-- 
Nando Dessena
mailto:na...@de...

Nando Dessena wrote:

>G> I am guessing that, with an appropriate API, you may be able
>G> to offer additional server management capabilities.
>
>That's one of the things my app does. Automated database upgrade, for
>example, or alias listing, or additional security (the ability to
>assign particular aliases to particular users). In short, I want to
>augment the firebird server with features that are specific to my
>application domain, and I don't necessariliy want to do it using a
>three tier architecture (actually I currently can't).
>  
>
Let me start with some definition so we have a common vocabulary:

    * Engine -- callable code that manipulates one or more database files
    * Server -- a proxy that receives network or IPC data and makes API
      calls to an engine
    * Superserver -- a multi-threaded engine with exclusive access to
      database files
    * Classic -- an engine with shared access to database files
      coordinated with other classic engines with a lock manager.
    * Embedded -- an ambiguous reference to an engine
    * Single client server -- a server process started by inetd/xinetd
      to handle a single connection with a classic engine
    * Multi-client server -- a server that handles multiple incoming
      connections; may use either classic of superserver
    * (Vulcan engine -- an engine combining characteristics of
      superserver and classic)

The highest bandwidth path to the database is always a direct call.  
Applications that require the highest bandwidth access have a number of 
alternatives, among them:

   1. Exclusive embedded access with the existing Firebird embedded engine
   2. Shared embedded access with classic engine
   3. Adding specialized application code to database servers accessed
      by IPC or wire protocol
   4. Running database server threads in application process
   5. (Sharing database in classic mode with Vulcan multi-client server)

The system is architected so all should be possible, allowing the 
application designer to choose among performance, convenience, and 
availability tradeoffs.  The gold standard is probably #3, extending the 
existing server(s).  It's doable with the existing remote server code 
and easy with Vulcan, and the mechanisms are pretty much the same: 
create a master port object then start a server thread to listen on it.  
The current complexity is setting up the various environments that have 
nothing to do with the database service activity.

The historical goal of all database architectures is to move semantics 
closer to the disk platers.  This is the reason for high level language 
semantics (SQL vs. the earlier navigational interfaces), triggers, 
stored procedures, and plugins.  But the closer something gets to the 
disk, the more difficult it is to write, debug, and support.  For most 
applications requiring very high bandwidth database connections, outside 
the engine but inside the server is a very good compromise, and one that 
we should work to support.  I'm not suggesting (and would oppose) 
defining the internal architect of a database server process.  A clean, 
modular, class design is sufficient.

-- 

Jim Starkey
Netfrastructure, Inc.
978 526-1376

Jim,

J> Let me start with some definition so we have a common vocabulary:

Thanks; I think this could help simplifying a discussion that's going
a bit astray...

I agree with all the definitions, particularly this one:

J>     * Embedded -- an ambiguous reference to an engine

:-(

Perhaps the thread should change subject. I shouldn't have used a word
having such precise connotation in firebird parlance like "embedded".

J> The highest bandwidth path to the database is always a direct call.
J> Applications that require the highest bandwidth access have a number of
J> alternatives

This is not my requirement, although it would be nice not having to go
through TCP/IP to talk to an engine that lives in my own address space.

According to the vocabulary you have so gently set up, my requirement
is having a superserver engine + server loadable as a dynamic library,
with a function to start the server (possibly passing in a command
line or equivalent switches or parameter block) and a function to stop it.

J>    1. Exclusive embedded access with the existing Firebird embedded engine
J>    2. Shared embedded access with classic engine
J>    3. Adding specialized application code to database servers accessed
J>       by IPC or wire protocol
J>    4. Running database server threads in application process
J>    5. (Sharing database in classic mode with Vulcan multi-client server)

I think 4 is closest to my requirement, but I'm not sure whether
"running database server threads" is the same as "running an engine"
or not. If it is, then 4 is what I need. I am aware that 3 is much
more common and I am actually using it right now.
I am also using 1 for some things, BTW.

J> The historical goal of all database architectures is to move semantics
J> closer to the disk platers.  This is the reason for high level language
J> semantics (SQL vs. the earlier navigational interfaces), triggers, 
J> stored procedures, and plugins.  But the closer something gets to the
J> disk, the more difficult it is to write, debug, and support.  For most
J> applications requiring very high bandwidth database connections, outside
J> the engine but inside the server is a very good compromise, and one that
J> we should work to support.  I'm not suggesting (and would oppose) 
J> defining the internal architect of a database server process.  A clean,
J> modular, class design is sufficient.

Outside the engine, inside the server. I guess that sums it up.

Ciao
-- 
Nando Dessena
mailto:na...@de...

Hi Nando,

> Outside the engine, inside the server. I guess that sums it up.

I see the point now.  I even begin to see how I might want to
use it in the future :-)

Thanks for being patient with the explanations.

-- 
Geoff Worboys
Telesis Computing

Geoff,

>> Outside the engine, inside the server. I guess that sums it up.

G> I see the point now.  I even begin to see how I might want to
G> use it in the future :-)

Now the problem is finding someone willing to do it; I can't do
this myself.

Ciao
-- 
Nando Dessena
mailto:na...@de...

I wrote:

G>> I see the point now.  I even begin to see how I might want to
G>> use it in the future :-)

N> Now the problem is finding someone willing to do it; I can't do
N> this myself.

Following up, if anyone has experience with building Firebird, has
some spare time to do paid work on this experiment and is willing to
do it I can put together a requirements specification based on the
most precious comments made by people in this thread. Otherwise...
well, I'll prepare the requirements doc anyway, put it aside and retry
some day. :-)

BTW, I was thinking that, unlike what happens with fbembed, a special
build could not be needed. If fbserver.exe is transformed into a small
(20 lines of code at most) launcher that loads and starts fbserver.dll
then one can have the flexibility of using fbserver.dll without
fbserver.exe. Just an idea. Actually, even fbembed could be unified in
this scheme. Unless Vulcan comes and makes everything pointless, of
course. :-)

Thanks to everybody that has contributed thoughtful opinions to this
thread.

Ciao
-- 
Nando Dessena
mailto:na...@de...

Geoff,

> To be blunt;  What is the point?

I agree, I don't see it myself.

Sean

Sean,

>> To be blunt;  What is the point?
>
> I agree, I don't see it myself.

Consider following:

a) you have an application that needs the fastest access to the database.
Embedded engine is the best for you: you do not loose time neither in IPC
nor remote calls.
b) at the same time you need a possibility to provide a "not so fast"
access to other clients (either IPC or remote).
How are you going to achieve this without doing what Jim described or what
Nando wants?
Roman

Roman,

R> a) you have an application that needs the fastest access to the database.
R> Embedded engine is the best for you: you do not loose time neither in IPC
R> nor remote calls.
R> b) at the same time you need a possibility to provide a "not so fast"
R> access to other clients (either IPC or remote).
R> How are you going to achieve this without doing what Jim described or what
R> Nando wants?

Yours is a good case (for which it will be interesting to hear
replies) but it does not match mine 100%. I have not a) as a
requirement (although it would be a nice thing to have), and instead I
have

c) the same server process that serves database connections needs to
provide other services to remote clients, using different ports and
protocols, including but not limited to SOAP and HTTP. For those who
are concerned about the scalability of my all-in-one solution, don't
worry: The workload is never going to be heavy.

Ciao
-- 
Nando Dessena
mailto:na...@de...

> Yours is a good case (for which it will be interesting to hear
> replies) but it does not match mine 100%. I have not a) as a
> requirement (although it would be a nice thing to have), and instead I
> have
>
> c) the same server process that serves database connections needs to
> provide other services to remote clients, using different ports and
> protocols, including but not limited to SOAP and HTTP. For those who
> are concerned about the scalability of my all-in-one solution, don't
> worry: The workload is never going to be heavy.

It was not intended to be your case :) Since Sean is not buying c), I
created a case, pretty legal one, when Jim's solution is the only possible
one. Maybe he buys this one :)
Anyway, I do not think that anybody will do a release of such "embedded
superserver", and after Vulcan merge it will no longer be an issue. I'm
afraid you have no other choice then to build it yourself or to find some
other incentives to involve Dmitry for example in your project.
Roman

Roman,

R> It was not intended to be your case :) Since Sean is not buying c), I
R> created a case, pretty legal one, when Jim's solution is the only possible
R> one. Maybe he buys this one :)

Maybe. :-)

R> Anyway, I do not think that anybody will do a release of such "embedded
R> superserver", and after Vulcan merge it will no longer be an issue. I'm
R> afraid you have no other choice then to build it yourself or to find some
R> other incentives to involve Dmitry for example in your project.

Well, as I said I am willing to pay (or find a sponsor if I cannot
personally afford it) for it if it makes sense enough to be part of
the firebird project and a work/cost estimate is made - I'm not
enthusiastic about private builds/forks.

Ciao
-- 
Nando Dessena
mailto:na...@de...

Nando,

> G> Other than installation what other advantages would such a
> G> configuration offer?
>=20
> I think I have briefly said that in a post to Dmitry. Here it is:
> synchronizing startup and shutdown of two processes, or more generally
> maintaining a host/child relation between two processes is problematic
> in Windows without introducing a third "guardian" application.

Actually, if you investigate services further, you will find that you
can define dependencies which will ensure that FB is started in order
for your service to start (although the MS doc on dependencies is
'light').

Accordingly, starting your service will force Windows to ensure that FB
is started, and closing FB will close your service.

> Plus, fbserver.exe is shown
> in the process list and could be terminated (since it runs as an
> application) without my app noticing.

But what about the remote connections? =20

You don't seem to be concerned about the impact of the start/stop of
your application on their work.  Not a good thing.

> I could setup (partial)
> workarounds for these situations, but I find the proposal I am
> advocating simpler, more straightforward, more robust and in general
> architecturally sounder.

I wish I could agree with you...but I don't.

Sean

Sean,

>> I think I have briefly said that in a post to Dmitry. Here it is:
>> synchronizing startup and shutdown of two processes, or more generally
>> maintaining a host/child relation between two processes is problematic
>> in Windows without introducing a third "guardian" application.

S> Actually, if you investigate services further, you will find that you
S> can define dependencies which will ensure that FB is started in order
S> for your service to start (although the MS doc on dependencies is
S> 'light').

I don't want the firebird server to be available for anyone to stop it
or even see it. I don't want anyone perceiving that something called
"firebird" exist on the machine, except from the "powered by firebird"
logo in my application's about box. I want to run (as an application)
on machines that don't support services (Win98). I want to be able to
seamlessly install on machines which already have IB and/or Fb
installed and running, without causing harm to them. Plus I want all
the other things that I have already explained, of which the ability
to ensure that Fb get woken up when my service starts is but the tip
of the iceberg.

And, I do know how service dependencies work.

If anyone can come up with a different solution that allows all this
I'd be happy to hear.

>> Plus, fbserver.exe is shown
>> in the process list and could be terminated (since it runs as an
>> application) without my app noticing.

S> But what about the remote connections?  

S> You don't seem to be concerned about the impact of the start/stop of
S> your application on their work.  Not a good thing.

You are misquoting a paragraph. There I was talking about an issue
with my current setup that would not exist if I was able to load
firebird in my address space.

Sean, I am concerned about remote connections as much as you are and
then some. ;-) My application is a service; stopping a service while
it is servicing remote connections may cause harm; you don't do it
with the firebird server, you don't do it with my server, you don't do
it with *any* server at all. There is *nothing* specific about my
application in this regard. It is a service and you don't stop it
while users are working, period. The same applies to the firebird
service.

S> I wish I could agree with you...but I don't.

I think I have sound replies for all the issues you have put forward.
Wise people change their minds when confronted by facts. :-)

Ciao
-- 
Nando Dessena
mailto:na...@de...

Thread: RE: [Firebird-devel] Embedded superserver

A powerful, cross platform, SQL database system

firebird-devel