Thread: Re: [Firebird-devel] Planning the post v3 development (Page 3)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

29.04.2014 11:05, Vlad Khorsun wrote:
>   Practically - what generation of altered object
> user transaction should work with ?

   According to its isolation level, I'd say.

-- 
   WBR, SD.

29.04.2014 13:12, Dimitry Sibiryakov wrote:

> According to its isolation level, I'd say.

So user snapshot transaction should not seen changes done by itself (as 
user thinks, as it executes DDL in the same transaction)? Of course it 
can be documented, but I'd rather avoid such a tricky behavior.

Dmitry

29.04.2014 11:17, Dmitry Yemanov wrote:
> So user snapshot transaction should not seen changes done by itself (as
> user thinks, as it executes DDL in the same transaction)?

   That's why I think that executing DDL in separate autocommit transaction is a bad idea.

-- 
   WBR, SD.

29.04.2014 13:05, Vlad Khorsun wrote:

> Technically - it is possible. Practically - what generation of altered object
> user transaction should work with ?

Statements already having the object cached - the old version. 
Statements to be [re]prepared - the new version, as the metadata cache 
will be immediately (or later via AST) updated after DDL.

Dmitry

> -----Original Message-----
> From: Dmitry Yemanov [mailto:fir...@ya...] 
> Sent: Martes, 29 de Abril de 2014 3:44
> 
> 29.04.2014 01:21, Claudio Valderrama C. wrote:
> 
> > I know people will feel outraged with my opinion, but 
> anyway: make DDL
> > operations atomic and immediate.
> 
> Atomic and immediate means autocommitted or always executed in a 
> separate (e.g. system) transaction?

For me, autocommitted.

> > Uncommitted DDL and DML working in unison
> > with stable DB structure is a naive dream, period.
> 
> Maybe true for the current FB code, but not generally. Other 
> databases 
> can handle this reliably.

But they have a limited degree of data versioning, if any. I would like to
see a SqlServer DB with versioning activated and pending DDL changes (never
tried).

C.

29.04.2014 18:11, Claudio Valderrama C. wrote:

>> Maybe true for the current FB code, but not generally. Other
>> databases can handle this reliably.
>
> But they have a limited degree of data versioning, if any.

AFAIK, PostgreSQL can handle transactional DDL. And yes, it's pretty as 
much MGA as we are.

Dmitry

On 29/04/2014 12:04, Dmitry Yemanov wrote:
> 29.04.2014 18:11, Claudio Valderrama C. wrote:
>
>>> Maybe true for the current FB code, but not generally. Other
>>> databases can handle this reliably.
>> But they have a limited degree of data versioning, if any.
> AFAIK, PostgreSQL can handle transactional DDL. And yes, it's pretty as 
> much MGA as we are.
>
>
I think support for transactional DDL is not a big difficult if well
constrained and architected.

Main problems currently are:
- DDL must acquire exclusive locks and DML shared locks on objects. And
that must happen even in the same transaction, i.e., no DDL and DML
together.
- DFW must be remade: it should not query metadata (which is in a
different structure than in the moment of the actual DDL command) to
continue the tasks.
- Rollback of DDL: this includes revert the database and the in-memory
cache.

Note these problems exists even if you consider auto-commited DDL. They
are just hidden due to more simple interactions.

Adriano

On 4/28/2014 10:29 AM, Carlos H. Cantu wrote:
> 3) A way to schedule some tasks direct in FB, for example,
> running procures, sweeps, statistics recalcs, etc. at specific
> scheduled days/times. What would be accepted as a task could be
> limited to whatever you guys thinks that would be "safe" to implement:
> http://tracker.firebirdsql.org/browse/CORE-743
>
>

I put an internal scheduler in Netfrastructure that would kick off Java 
procedures using the embedded JVM.  It was a big hit, sufficiently big 
that I ended up with two schedulers, one for system tasks, the other for 
user tasks, so user tasks didn't block system tasks.  The system 
scheduler tended to get used for garbage collection, etc., and the user 
scheduler for reports and user mode replication.

The user mode replication was interesting.  A customer used a single 
multi-table (Java) trigger to write tables changes into a replication 
table.  When scheduled, the replication task uses a server to server 
communication facility to broadcast changes from the replication table.  
If the slave was down, the replication table just got bigger.

The resulting system was really slick and worked very well, but it used 
more than a fair bit more infrastructure than is presently in Firebird.

DY> Thanks, but I seemed to explicitly state that plain wishlists don't
DY> count.

It seems that I also didn't understand that you wanted separated
messages for each feature discussion :(

[]s
Carlos
http://www.firebirdnews.org
FireBase - http://www.FireBase.com.br

28.04.2014 20:30, Carlos H. Cantu wrote:

> It seems that I also didn't understand that you wanted separated
> messages for each feature discussion :(

Well, we may start with the features intermixed inside this thread and 
then separate them for a more detailed discussion, if necessary.

I just ask everybody to provide arguments why you think your suggestions 
worth more attention than something else.

Dmitry

> -----Original Message-----
> From: Carlos H. Cantu [mailto:li...@wa...] 
> Sent: Lunes, 28 de Abril de 2014 12:30
> 
> DY> Thanks, but I seemed to explicitly state that plain 
> wishlists don't
> DY> count.
> 
> It seems that I also didn't understand that you wanted separated
> messages for each feature discussion :(

Don't worry. If you go wrong, we will simply ask a volunteer to crucify you.
;-)

C.

Em 28/4/2014 18:21, Claudio Valderrama C. escreveu:
>> -----Original Message-----
>> From: Carlos H. Cantu [mailto:li...@wa...]
>> Sent: Lunes, 28 de Abril de 2014 12:30
>>
>> DY> Thanks, but I seemed to explicitly state that plain
>> wishlists don't
>> DY> count.
>>
>> It seems that I also didn't understand that you wanted separated
>> messages for each feature discussion :(
> Don't worry. If you go wrong, we will simply ask a volunteer to crucify you.
> ;-)
>
> C.
>
>

I am pretty near and will do it with pleasure ! :)

ABS> I am pretty near and will do it with pleasure ! :)

One less speaker at FDD so :P

[]s
Carlos
http://www.firebirdnews.org
FireBase - http://www.FireBase.com.br

ABS> Em 28/4/2014 18:21, Claudio Valderrama C. escreveu:
>>> -----Original Message-----
>>> From: Carlos H. Cantu [mailto:li...@wa...]
>>> Sent: Lunes, 28 de Abril de 2014 12:30
>>>
>>> DY> Thanks, but I seemed to explicitly state that plain
>>> wishlists don't
>>> DY> count.
>>>
>>> It seems that I also didn't understand that you wanted separated
>>> messages for each feature discussion :(
>> Don't worry. If you go wrong, we will simply ask a volunteer to crucify you.
>> ;-)
>>
>> C.

This is another one that I missed in my first list:

Alter domain depedence problem when changing VAR(CHAR) size
http://tracker.firebirdsql.org/browse/CORE-1427

Existing comments speaks for itself.

[]s
Carlos H. Cantu
www.FireBase.com.br - www.firebirdnews.org
www.warmboot.com.br - blog.firebase.com.br

On 04/28/2014 06:29 PM, Carlos H. Cantu wrote:

> 5) An embedded Firebird version for Android (even if only "basic"
> server features could be available):
> http://tracker.firebirdsql.org/browse/CORE-3885

Why do you treat it as post-v3? I plan to activate work with Android 
port after beta1.

 >> 5) An embedded Firebird version for Android (even if only "basic"
>> server features could be available):
>> http://tracker.firebirdsql.org/browse/CORE-3885

A> Why do you treat it as post-v3? I plan to activate work with Android 
A> port after beta1.

Really?! That's a great news!!!

[]s
Carlos
http://www.firebirdnews.org
FireBase - http://www.FireBase.com.br

28.04.2014 23:05, Carlos H. Cantu wrote:

> I think most of them needs basic asynchronous replication, covering
> single and multi-master scenarios. For those who needs more complex
> scenarios, there are third party comercial tools. Anyway, I'm not the
> right person to answer, since I didn't need replication in my projects
> so far, so I'll leave this for Jesus or any others to answer. If you
> are interested, I can run a poll at firebirdnews.org and firebase
> about what users would expect in "native" FB replication.

I believe that polls tend to show what users want, often mismatching 
with what they need ;-) But I have no objections to the poll, if 
questions are composed carefully.

> I asked Alexey and seems that most of the previous problems are
> fixed, but dropping tables with active connections still causes
> corruption. Maybe he can give more details.

Table used by active connections cannot be dropped. But maybe there are 
some race conditions yet unknown for us.

> Whatever the reason is, if it will not be done, it should not be left
> "open".

Time for a global tracker cleanup, huh? :-)

Dmitry

DY> 28.04.2014 23:05, Carlos H. Cantu wrote:

>> I think most of them needs basic asynchronous replication, covering
>> single and multi-master scenarios. For those who needs more complex
>> scenarios, there are third party comercial tools. Anyway, I'm not the
>> right person to answer, since I didn't need replication in my projects
>> so far, so I'll leave this for Jesus or any others to answer. If you
>> are interested, I can run a poll at firebirdnews.org and firebase
>> about what users would expect in "native" FB replication.

DY> I believe that polls tend to show what users want, often mismatching 
DY> with what they need ;-) But I have no objections to the poll, if 
DY> questions are composed carefully.

Give me the question(s) (or answers, if it would be a single choice
question) and I'll put it online.

>> I asked Alexey and seems that most of the previous problems are
>> fixed, but dropping tables with active connections still causes
>> corruption. Maybe he can give more details.

DY> Table used by active connections cannot be dropped. But maybe there are
DY> some race conditions yet unknown for us.

Please ask him for more details about this.

[]s
Carlos
http://www.firebirdnews.org
FireBase - http://www.FireBase.com.br

On 04/29/14 16:32, Carlos H. Cantu wrote:
> DY> 29.04.2014 12:09, Mark Rotteveel wrote:
>
>>> If you look at SQL Server, there jobs themselves are not defined for a
>>> specific database (although they may depend on one or more databases).
>>> AFAIK they are stored in the master database. Execution requires an Agent
>>> service to be running.
> DY> We neither have a master database, nor an agent service, nor database
> DY> links to access user dbs from a master one (or any other solution to
> DY> this issue). Shouldn't we think about something more realistic? Or at
> DY> least postpone scheduled tasks until we have a proper foundation built?
> DY> Dmitry
>
> I'm considering that "master database" would be used only to store the
> jobs information. A client/daemon would read it, create all the
> schedules and run the tasks at desired time (use of OS scheduler is an
> option too). Such "master database" (better call it
> jobs/tasks/scheduler database?) can be created and distributed with
> new installations (like the security2.fdb already is).
>
> If you don't worry about mixing things a bit, maybe even security2.fdb
> could be enhanced to store jobs data, avoiding the need to create a
> new pre-installed database.

We do not have single fixed security database in FB3. Therefore us of it 
as jobs storage is not good choice. Adding new one looks better to me. 
But once again embedded problem is here - up to the fact that embedded 
user may have no rights to start daemons.

Hi,

At April 29, 2014, 5:09 AM, Mark Rotteveel wrote:

> On Tue, 29 Apr 2014 11:58:57 +0400, Dmitry Yemanov <fir...@ya...>
> wrote:
>> Who should initiate that dedicated listener deamon/user after a server 
>> restart? Should the server attach all the databases itself to check 
>> whether one needs a startup? What about databases unknown to the server 
>> (not present in databases.conf)?

> If you look at SQL Server, there jobs themselves are not defined for a
> specific database (although they may depend on one or more databases).
> AFAIK they are stored in the master database. Execution requires an Agent
> service to be running.

And, even with the MS-SQL Agent service, it is no guarantee. We tried
setting up maintenance plans in MS-SQL 2012, and for some reason, the
maintenance plan file/record gets corrupted and will not work anymore,
and is no longer modifiable(and this is after creating multiple
maintenance plans, even after a full reinstall). So, we simply decided
to create a PowerShell script that will execute the maintenance tasks
and use the Windows Task Scheduler to schedule when the script will
run.

So, for Firebird, I think that it would be up to the user and/or the
software developer to decide how they would want to execute such
tasks(either creating their own service, or using a batch/script to
execute the commands scheduled via a task scheduler). As an example,
our application has a service(for an n-tier application) that runs on
the server, and we simply add the maintenance tasks in it, and
they are scheduled when they are to be executed.

So, for me, it is not something that I see as a priority for Firebird.
Since there is always another way of doing it.

-- 
Best regards,
 Daniel Rail
 Senior Software Developer
 ACCRA Solutions Inc. (www.accra.ca)
 ACCRA Med Software Inc. (www.filopto.com)

On 4/24/2014 6:19 AM, Dmitry Yemanov wrote:
> All,
>
> We're getting closer to the v3.0 feature freeze which is going to happen
> this summer. Everything roadmapped for v3 but not implemented before the
> deadline will be postponed. The next-after-v3 release is likely to
> incorporate most of the postponed features, but there may be new
> features as well. So it makes sense to start discussing what could and
> should be done in the next version(s). We have a few months to collect
> the proposals, discuss technical details and make estimates about the
> required efforts.
>
>
I suggest you consider the following:

 1. Switch internals to UTF-8 only.  Let the client handle mappings
    between national character sets onf UTF-8.
 2. Switch to an encoding based on actual value rather than declared
    value to achieve a packing density 30% less than run length encoding
    and to support strings of unbounded length and over-specified
    numeric types.  It might be necessary to extend the API to support
    unbounded string.  This would have a side effect of allowing 127
    character object names as per the standard.
 3. Implement schemas to support a two level name space.
 4.

Or,

Have a standard client/daemon such as gstat/gfix/gsec ship with firebird
that when started, it is provided with a config file or command line
specifying the database to use.

In this way, any database could be the master database as long as it has
the appropriate data structures in it.

The data structures could be setup via custom sql ddl commands, which would
also be used to create tasks.   In this way, a database including it's
tasks could be defined via an sql script.

I would call it something starting with "f" instead of 'g' as the project
has not been called 'grotten' for years.

Perhaps renaming the other utilities fb_sec, fb_isql, fb_stat, fb_fix,
fb_sched.......

just a thought.

On 29 April 2014 08:32, Carlos H. Cantu <li...@wa...> wrote:

> DY> 29.04.2014 12:09, Mark Rotteveel wrote:
>
> >> If you look at SQL Server, there jobs themselves are not defined for a
> >> specific database (although they may depend on one or more databases).
> >> AFAIK they are stored in the master database. Execution requires an
> Agent
> >> service to be running.
>
> DY> We neither have a master database, nor an agent service, nor database
> DY> links to access user dbs from a master one (or any other solution to
> DY> this issue). Shouldn't we think about something more realistic? Or at
> DY> least postpone scheduled tasks until we have a proper foundation built?
> DY> Dmitry
>
> I'm considering that "master database" would be used only to store the
> jobs information. A client/daemon would read it, create all the
> schedules and run the tasks at desired time (use of OS scheduler is an
> option too). Such "master database" (better call it
> jobs/tasks/scheduler database?) can be created and distributed with
> new installations (like the security2.fdb already is).
>
> If you don't worry about mixing things a bit, maybe even security2.fdb
> could be enhanced to store jobs data, avoiding the need to create a
> new pre-installed database.
>
> []s
> Carlos
> http://www.firebirdnews.org
> FireBase - http://www.FireBase.com.br
>
>
>
>
>
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.  Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
>

Den 2014-04-29 20:52 skrev Dalton Calford såhär:
> Have a standard client/daemon such as gstat/gfix/gsec ship with 
> firebird that when started, it is provided with a config file or 
> command line specifying the database to use.
>
> In this way, any database could be the master database as long as it 
> has the appropriate data structures in it.
>
> The data structures could be setup via custom sql ddl commands, which 
> would also be used to create tasks.   In this way, a database 
> including it's tasks could be defined via an sql script.

My thoughts exactly. This solution is so obvious I'm really surprised 
noone's mentioned until now. :-)

+1

Kjell

Thread: Re: [Firebird-devel] Planning the post v3 development (Page 3)

A powerful, cross platform, SQL database system

firebird-devel