Re: [Postgres-xc-developers] Regarding bug id 3607975

[Koichi S. <koi...@gm...>]

You're right.   Should we change parallel_schedule?

Regards;

----------
Koichi Suzuki


2013/6/26 Amit Khandekar <ami...@en...>

> Now I see how to reproduce the inherit.sql failure. It fails only in
> parallel schedule. And I just checked that it also failed 6 days ago
> with the same diff when I ran some regression test on the build farm.
> So it has nothing to do with the patch, and this test needs to be
> analyzed and fixed separately.
>
> On 26 June 2013 13:21, 鈴木 幸市 <ko...@in...> wrote:
> > It seems that the name of materialized table may be
> environment-dependent.
> > Any idea to make it environment-independent?
> >
> > Apparently, the result is correct but just does not match expected ones.
> >
> > Reagards;
> > ---
> > Koichi Suzuki
> >
> >
> >
> > On 2013/06/26, at 16:47, Koichi Suzuki <koi...@gm...>
> wrote:
> >
> > I tested this patch for the latest master (without any pending patches)
> and
> > found inherit fails in linker machine environment.
> >
> > PFA related files.
> >
> > Regards;
> >
> > ----------
> > Koichi Suzuki
> >
> >
> > 2013/6/26 Amit Khandekar <ami...@en...>
> >>
> >> On 26 June 2013 10:34, Amit Khandekar <ami...@en...>
> >> wrote:
> >> > On 26 June 2013 08:56, Ashutosh Bapat <
> ash...@en...>
> >> > wrote:
> >> >> Hi Amit,
> >> >> From a cursory look, this looks much more cleaner than Abbas's patch.
> >> >> So, it
> >> >> looks to be the approach we should take. BTW, you need to update the
> >> >> original outputs as well, instead of just the alternate expected
> >> >> outputs.
> >> >> Remember, the merge applies changes to the original expected outputs
> >> >> and not
> >> >> alternate ones (added in XC), thus we come to know about conflicts
> only
> >> >> when
> >> >> we apply changes to original expected outputs.
> >> >
> >> > Right. Will look into it.
> >>
> >> All of the expected files except inherit.sql are xc_* tests which
> >> don't have alternate files. I have made the same inherit_1.out changes
> >> onto inherit.out. Attached revised patch.
> >>
> >> Suzuki-san, I was looking at the following diff in the inherit.sql
> >> failure that you had attached from your local regression run:
> >>
> >> !    Hash Cond: (pg_temp_2.patest0.id = int4_tbl.f1)
> >> --- 1247,1253 ----
> >> !    Hash Cond: (pg_temp_7.patest0.id = int4_tbl.f1)
> >>
> >> I am not sure if this has started coming after you applied the patch.
> >> Can you please once again run the test after reinitializing the
> >> cluster ? I am not getting this diff although I ran using
> >> serial_schedule, not parallel; and the diff itself looks harmless. As
> >> I mentioned, the actual temp schema name may differ.
> >>
> >> >
> >> >>
> >> >> Regards to temporary namespaces, is it possible to schema-qualify the
> >> >> temporary namespaces as always pg_temp irrespective of the actual
> name?
> >> >
> >> > get_namespace_name() is used to deparse the schema name. In order to
> >> > keep the deparsing logic working for both local queries (i.e. for view
> >> > for e.g.) and remote queries, we need to push in the context that the
> >> > deparsing is being done for remote queries, and this needs to be done
> >> > all the way from the uppermost function (say pg_get_querydef) upto
> >> > get_namespace_name() which does not look good.
> >> >
> >> > We need to think about some solution in general for the existing issue
> >> > of deparsing temp table names. Not sure of any solution. May be, after
> >> > we resolve the bug id in subject, the fix may even not require the
> >> > schema qualification.
> >> >>
> >> >>
> >> >> On Tue, Jun 25, 2013 at 8:02 PM, Amit Khandekar
> >> >> <ami...@en...> wrote:
> >> >>>
> >> >>> On 25 June 2013 19:59, Amit Khandekar
> >> >>> <ami...@en...>
> >> >>> wrote:
> >> >>> > Attached is a patch that does schema qualification by overriding
> the
> >> >>> > search_path just before doing the deparse in deparse_query().
> >> >>> > PopOVerrideSearchPath() in the end pops out the temp path. Also,
> the
> >> >>> > transaction callback function already takes care of popping out
> such
> >> >>> > Push in case of transaction rollback.
> >> >>> >
> >> >>> > Unfortunately we cannot apply this solution to temp tables. The
> >> >>> > problem is, when a pg_temp schema is deparsed, it is deparsed into
> >> >>> > pg_temp_1, pg_temp_2 etc. and these names are specific to the
> node.
> >> >>> > An
> >> >>> > object in pg_temp_2 at coordinator may be present in pg_temp_1 at
> >> >>> > datanode. So the remote query generated may or may not work on
> >> >>> > datanode,  so totally unreliable.
> >> >>> >
> >> >>> > In fact, the issue with pg_temp_1 names in the deparsed remote
> query
> >> >>> > is present even currently.
> >> >>> >
> >> >>> > But wherever it is a correctness issue to *not* schema qualify
> temp
> >> >>> > object, I have kept the schema qualification.
> >> >>> > For e.g. user can set search_path to" s1, pg_temp",
> >> >>> > and have obj1 in both s1 and pg_temp, and want to refer to
> >> >>> > pg_temp.s1.
> >> >>> > In such case, the remote query should have pg_temp_[1-9].obj1,
> >> >>> > although it may cause errors because of the existing issue.
> >> >>> >
> >> >>> ---
> >> >>> > So, the prepare-execute with search_path would remain there for
> temp
> >> >>> > tables.
> >> >>> I mean, the prepare-exute issue with search_patch would remain there
> >> >>> for temp tables.
> >> >>>
> >> >>> >
> >> >>> > I tried to run the regression by extracting regression expected
> >> >>> > output
> >> >>> > files from Abbas's patch , and regression passes, including
> >> >>> > plancache.
> >> >>> >
> >> >>> > I think for this release, we should go ahead by keeping this issue
> >> >>> > open for temp tables. This solution is an improvement, and does
> not
> >> >>> > cause any new issues.
> >> >>> >
> >> >>> > Comments welcome.
> >> >>> >
> >> >>> >
> >> >>> > On 24 June 2013 13:00, Ashutosh Bapat
> >> >>> > <ash...@en...>
> >> >>> > wrote:
> >> >>> >> Hi Abbas,
> >> >>> >> We are changing a lot of PostgreSQL deparsing code, which would
> >> >>> >> create
> >> >>> >> problems in the future merges. Since this change is in query
> >> >>> >> deparsing
> >> >>> >> logic
> >> >>> >> any errors here would affect, EXPLAIN/ pg_dump etc. So, this
> patch
> >> >>> >> should
> >> >>> >> again be the last resort.
> >> >>> >>
> >> >>> >> Please take a look at how view definitions are dumped. That will
> >> >>> >> give a
> >> >>> >> good
> >> >>> >> idea as to how PG schema-qualifies (or not) objects. Here's how
> >> >>> >> view
> >> >>> >> definitions displayed changes with search path. Since the code to
> >> >>> >> dump
> >> >>> >> views
> >> >>> >> and display definitions is same, the view definition dumped also
> >> >>> >> changes
> >> >>> >> with the search path. Thus pg_dump must be using some trick to
> >> >>> >> always
> >> >>> >> dump a
> >> >>> >> consistent view definition (and hence a deparsed query). Thanks
> >> >>> >> Amit
> >> >>> >> for the
> >> >>> >> example.
> >> >>> >>
> >> >>> >> create table ttt (id int);
> >> >>> >>
> >> >>> >> postgres=# create domain dd int;
> >> >>> >> CREATE DOMAIN
> >> >>> >> postgres=# create view v2 as select id::dd from ttt;
> >> >>> >> CREATE VIEW
> >> >>> >> postgres=# set search_path TO '';
> >> >>> >> SET
> >> >>> >>
> >> >>> >> postgres=# \d+ public.v2
> >> >>> >>                     View "public.v2"
> >> >>> >>  Column |   Type    | Modifiers | Storage | Description
> >> >>> >> --------+-----------+---------
> >> >>> >> --+---------+-------------
> >> >>> >>  id     | public.dd |           | plain   |
> >> >>> >> View definition:
> >> >>> >>  SELECT ttt.id::public.dd AS id
> >> >>> >>    FROM public.ttt;
> >> >>> >>
> >> >>> >> postgres=# set search_path TO default ;
> >> >>> >> SET
> >> >>> >> postgres=# show search_path ;
> >> >>> >>   search_path
> >> >>> >> ----------------
> >> >>> >>  "$user",public
> >> >>> >> (1 row)
> >> >>> >>
> >> >>> >> postgres=# \d+ public.v2
> >> >>> >>                  View "public.v2"
> >> >>> >>  Column | Type | Modifiers | Storage | Description
> >> >>> >> --------+------+-----------+---------+-------------
> >> >>> >>  id     | dd   |           | plain   |
> >> >>> >> View definition:
> >> >>> >>  SELECT ttt.id::dd AS id
> >> >>> >>    FROM ttt;
> >> >>> >>
> >> >>> >> We need to leverage similar mechanism here to reduce PG
> footprint.
> >> >>> >>
> >> >>> >>
> >> >>> >> On Mon, Jun 24, 2013 at 8:12 AM, Abbas Butt
> >> >>> >> <abb...@en...>
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>> Hi,
> >> >>> >>> As discussed in the last F2F meeting, here is an updated patch
> >> >>> >>> that
> >> >>> >>> provides schema qualification of the following objects: Tables,
> >> >>> >>> Views,
> >> >>> >>> Functions, Types and Domains in case of remote queries.
> >> >>> >>> Sequence functions are never concerned with datanodes hence,
> >> >>> >>> schema
> >> >>> >>> qualification is not required in case of sequences.
> >> >>> >>> This solves plancache test case failure issue and does not
> >> >>> >>> introduce
> >> >>> >>> any
> >> >>> >>> more failures.
> >> >>> >>> I have also attached some tests with results to aid in review.
> >> >>> >>>
> >> >>> >>> Comments are welcome.
> >> >>> >>>
> >> >>> >>> Regards
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Mon, Jun 10, 2013 at 5:31 PM, Abbas Butt
> >> >>> >>> <abb...@en...>
> >> >>> >>> wrote:
> >> >>> >>>>
> >> >>> >>>> Hi,
> >> >>> >>>> Attached please find a WIP patch that provides the
> functionality
> >> >>> >>>> of
> >> >>> >>>> preparing the statement at the datanodes as soon as it is
> >> >>> >>>> prepared
> >> >>> >>>> on the coordinator.
> >> >>> >>>> This is to take care of a test case in plancache that makes
> sure
> >> >>> >>>> that
> >> >>> >>>> change of search_path is ignored by replans.
> >> >>> >>>> While the patch fixes this replan test case and the regression
> >> >>> >>>> works
> >> >>> >>>> fine
> >> >>> >>>> there are still these two problems I have to take care of.
> >> >>> >>>>
> >> >>> >>>> 1. This test case fails
> >> >>> >>>>
> >> >>> >>>>     CREATE TABLE xc_alter_table_3 (a int, b varchar(10))
> >> >>> >>>> DISTRIBUTE
> >> >>> >>>> BY
> >> >>> >>>> HASH(a);
> >> >>> >>>>     INSERT INTO xc_alter_table_3 VALUES (1, 'a');
> >> >>> >>>>     PREPARE d3 AS DELETE FROM xc_alter_table_3 WHERE a = $1; --
> >> >>> >>>> fails
> >> >>> >>>>
> >> >>> >>>>     test=# explain verbose DELETE FROM xc_alter_table_3 WHERE
> a =
> >> >>> >>>> 1;
> >> >>> >>>>                        QUERY PLAN
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> -------------------------------------------------------------------
> >> >>> >>>>      Delete on public.xc_alter_table_3  (cost=0.00..0.00
> >> >>> >>>> rows=1000
> >> >>> >>>> width=14)
> >> >>> >>>>        Node/s: data_node_1, data_node_2, data_node_3,
> data_node_4
> >> >>> >>>>        Remote query: DELETE FROM ONLY xc_alter_table_3 WHERE
> >> >>> >>>>                      ((xc_alter_table_3.ctid = $1) AND
> >> >>> >>>> (xc_alter_table_3.xc_node_id = $2))
> >> >>> >>>>        ->  Data Node Scan on xc_alter_table_3
> >> >>> >>>> "_REMOTE_TABLE_QUERY_"
> >> >>> >>>> (cost=0.00..0.00 rows=1000 width=14)
> >> >>> >>>>              Output: xc_alter_table_3.a, xc_alter_table_3.ctid,
> >> >>> >>>> xc_alter_table_3.xc_node_id
> >> >>> >>>>              Node/s: data_node_3
> >> >>> >>>>              Remote query: SELECT a, ctid, xc_node_id FROM ONLY
> >> >>> >>>> xc_alter_table_3 WHERE (a = 1)
> >> >>> >>>>     (7 rows)
> >> >>> >>>>
> >> >>> >>>>    The reason of the failure is that the select query is
> >> >>> >>>> selecting 3
> >> >>> >>>> items, the first of which is an int,
> >> >>> >>>>    whereas the delete query is comparing $1 with a ctid.
> >> >>> >>>>    I am not sure how this works without prepare, but it fails
> >> >>> >>>> when
> >> >>> >>>> used
> >> >>> >>>> with prepare.
> >> >>> >>>>
> >> >>> >>>>    The reason of this planning is this section of code in
> >> >>> >>>> function
> >> >>> >>>> pgxc_build_dml_statement
> >> >>> >>>>     else if (cmdtype == CMD_DELETE)
> >> >>> >>>>     {
> >> >>> >>>>         /*
> >> >>> >>>>          * Since there is no data to update, the first param is
> >> >>> >>>> going
> >> >>> >>>> to
> >> >>> >>>> be
> >> >>> >>>>          * ctid.
> >> >>> >>>>          */
> >> >>> >>>>         ctid_param_num = 1;
> >> >>> >>>>     }
> >> >>> >>>>
> >> >>> >>>>    Amit/Ashutosh can you suggest a fix for this problem?
> >> >>> >>>>    There are a number of possibilities.
> >> >>> >>>>    a) The select should not have selected column a.
> >> >>> >>>>    b) The DELETE should have referred to $2 and $3 for ctid and
> >> >>> >>>> xc_node_id respectively.
> >> >>> >>>>    c) Since the query works without PREPARE, we should make
> >> >>> >>>> PREPARE
> >> >>> >>>> work
> >> >>> >>>> the same way.
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> 2. This test case in plancache fails.
> >> >>> >>>>
> >> >>> >>>>     -- Try it with a view, which isn't directly used in the
> >> >>> >>>> resulting
> >> >>> >>>> plan
> >> >>> >>>>     -- but should trigger invalidation anyway
> >> >>> >>>>     create table tab33 (a int, b int);
> >> >>> >>>>     insert into tab33 values(1,2);
> >> >>> >>>>     CREATE VIEW v_tab33 AS SELECT * FROM tab33;
> >> >>> >>>>     PREPARE vprep AS SELECT * FROM v_tab33;
> >> >>> >>>>     EXECUTE vprep;
> >> >>> >>>>     CREATE OR REPLACE VIEW v_tab33 AS SELECT a, b/2 AS q2 FROM
> >> >>> >>>> tab33;
> >> >>> >>>>     -- does not cause plan invalidation because views are never
> >> >>> >>>> created
> >> >>> >>>> on datanodes
> >> >>> >>>>     EXECUTE vprep;
> >> >>> >>>>
> >> >>> >>>>    and the reason of the failure is that views are never
> created
> >> >>> >>>> on
> >> >>> >>>> the
> >> >>> >>>> datanodes hence plan invalidation is not triggered.
> >> >>> >>>>    This can be documented as an XC limitation.
> >> >>> >>>>
> >> >>> >>>> 3. I still have to add comments in the patch and some ifdefs
> may
> >> >>> >>>> be
> >> >>> >>>> missing too.
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> In addition to the patch I have also attached some example Java
> >> >>> >>>> programs
> >> >>> >>>> that test the some basic functionality through JDBC. I found
> that
> >> >>> >>>> these
> >> >>> >>>> programs are working fine after my patch.
> >> >>> >>>>
> >> >>> >>>> 1. Prepared.java : Issues parameterized delete, insert and
> update
> >> >>> >>>> through
> >> >>> >>>> JDBC. These are un-named prepared statements and works fine.
> >> >>> >>>> 2. NamedPrepared.java : Issues two named prepared statements
> >> >>> >>>> through
> >> >>> >>>> JDBC
> >> >>> >>>> and works fine.
> >> >>> >>>> 3. Retrieve.java : Runs a simple select to verify results.
> >> >>> >>>> The comments on top of the files explain their usage.
> >> >>> >>>>
> >> >>> >>>> Comments are welcome.
> >> >>> >>>>
> >> >>> >>>> Thanks
> >> >>> >>>> Regards
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> On Mon, Jun 3, 2013 at 10:54 AM, Ashutosh Bapat
> >> >>> >>>> <ash...@en...> wrote:
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> On Mon, Jun 3, 2013 at 10:51 AM, Abbas Butt
> >> >>> >>>>> <abb...@en...> wrote:
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> On Mon, Jun 3, 2013 at 8:43 AM, Ashutosh Bapat
> >> >>> >>>>>> <ash...@en...> wrote:
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> On Mon, Jun 3, 2013 at 7:40 AM, Abbas Butt
> >> >>> >>>>>>> <abb...@en...> wrote:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Attached please find updated patch to fix the bug. The
> patch
> >> >>> >>>>>>>> takes
> >> >>> >>>>>>>> care of the bug and the regression issues resulting from
> the
> >> >>> >>>>>>>> changes done in
> >> >>> >>>>>>>> the patch. Please note that the issue in test case
> plancache
> >> >>> >>>>>>>> still stands
> >> >>> >>>>>>>> unsolved because of the following test case (simplified but
> >> >>> >>>>>>>> taken
> >> >>> >>>>>>>> from
> >> >>> >>>>>>>> plancache.sql)
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> create schema s1 create table abc (f1 int);
> >> >>> >>>>>>>> create schema s2 create table abc (f1 int);
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> insert into s1.abc values(123);
> >> >>> >>>>>>>> insert into s2.abc values(456);
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> set search_path = s1;
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> prepare p1 as select f1 from abc;
> >> >>> >>>>>>>> execute p1; -- works fine, results in 123
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> set search_path = s2;
> >> >>> >>>>>>>> execute p1; -- works fine after the patch, results in 123
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> alter table s1.abc add column f2 float8;   -- force replan
> >> >>> >>>>>>>> execute p1; -- fails
> >> >>> >>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> Huh! The beast bit us.
> >> >>> >>>>>>>
> >> >>> >>>>>>> I think the right solution here is either of two
> >> >>> >>>>>>> 1. Take your previous patch to always use qualified names
> (but
> >> >>> >>>>>>> you
> >> >>> >>>>>>> need to improve it not to affect the view dumps)
> >> >>> >>>>>>> 2. Prepare the statements at the datanode at the time of
> >> >>> >>>>>>> prepare.
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> Is this test added new in 9.2?
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> No, it was added by commit
> >> >>> >>>>>> 547b6e537aa8bbae83a8a4c4d0d7f216390bdb9c
> >> >>> >>>>>> in
> >> >>> >>>>>> March 2007.
> >> >>> >>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> Why didn't we see this issue the first time prepare was
> >> >>> >>>>>>> implemented? I
> >> >>> >>>>>>> don't remember (but it was two years back).
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> I was unable to locate the exact reason but since statements
> >> >>> >>>>>> were
> >> >>> >>>>>> not
> >> >>> >>>>>> being prepared on datanodes due to a merge issue this issue
> >> >>> >>>>>> just
> >> >>> >>>>>> surfaced
> >> >>> >>>>>> up.
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> Well, even though statements were not getting prepared
> (actually
> >> >>> >>>>> prepared statements were not being used again and again) on
> >> >>> >>>>> datanodes, we
> >> >>> >>>>> never prepared them on datanode at the time of preparing the
> >> >>> >>>>> statement. So,
> >> >>> >>>>> this bug should have shown itself long back.
> >> >>> >>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> The last execute should result in 123, whereas it results
> in
> >> >>> >>>>>>>> 456.
> >> >>> >>>>>>>> The
> >> >>> >>>>>>>> reason is that the search path has already been changed at
> >> >>> >>>>>>>> the
> >> >>> >>>>>>>> datanode and
> >> >>> >>>>>>>> a replan would mean select from abc in s2.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> On Tue, May 28, 2013 at 7:17 PM, Ashutosh Bapat
> >> >>> >>>>>>>> <ash...@en...> wrote:
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> Hi Abbas,
> >> >>> >>>>>>>>> I think the fix is on the right track. There are couple of
> >> >>> >>>>>>>>> improvements that we need to do here (but you may not do
> >> >>> >>>>>>>>> those
> >> >>> >>>>>>>>> if the time
> >> >>> >>>>>>>>> doesn't permit).
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> 1. We should have a status in RemoteQuery node, as to
> >> >>> >>>>>>>>> whether
> >> >>> >>>>>>>>> the
> >> >>> >>>>>>>>> query in the node should use extended protocol or not,
> >> >>> >>>>>>>>> rather
> >> >>> >>>>>>>>> than relying
> >> >>> >>>>>>>>> on the presence of statement name and parameters etc. Amit
> >> >>> >>>>>>>>> has
> >> >>> >>>>>>>>> already added
> >> >>> >>>>>>>>> a status with that effect. We need to leverage it.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> On Tue, May 28, 2013 at 9:04 AM, Abbas Butt
> >> >>> >>>>>>>>> <abb...@en...> wrote:
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> The patch fixes the dead code issue, that I described
> >> >>> >>>>>>>>>> earlier.
> >> >>> >>>>>>>>>> The
> >> >>> >>>>>>>>>> code was dead because of two issues:
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> 1. The function CompleteCachedPlan was wrongly setting
> >> >>> >>>>>>>>>> stmt_name to
> >> >>> >>>>>>>>>> NULL and this was the main reason
> >> >>> >>>>>>>>>> ActivateDatanodeStatementOnNode was not
> >> >>> >>>>>>>>>> being called in the function
> >> >>> >>>>>>>>>> pgxc_start_command_on_connection.
> >> >>> >>>>>>>>>> 2. The function SetRemoteStatementName was wrongly
> assuming
> >> >>> >>>>>>>>>> that a
> >> >>> >>>>>>>>>> prepared statement must have some parameters.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> Fixing these two issues makes sure that the function
> >> >>> >>>>>>>>>> ActivateDatanodeStatementOnNode is now called and
> >> >>> >>>>>>>>>> statements
> >> >>> >>>>>>>>>> get prepared on
> >> >>> >>>>>>>>>> the datanode.
> >> >>> >>>>>>>>>> This patch would fix bug 3607975. It would however not
> fix
> >> >>> >>>>>>>>>> the
> >> >>> >>>>>>>>>> test
> >> >>> >>>>>>>>>> case I described in my previous email because of reasons
> I
> >> >>> >>>>>>>>>> described.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> On Tue, May 28, 2013 at 5:50 PM, Ashutosh Bapat
> >> >>> >>>>>>>>>> <ash...@en...> wrote:
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>> Can you please explain what this fix does? It would help
> >> >>> >>>>>>>>>>> to
> >> >>> >>>>>>>>>>> have
> >> >>> >>>>>>>>>>> an elaborate explanation with code snippets.
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>> On Sun, May 26, 2013 at 10:18 PM, Abbas Butt
> >> >>> >>>>>>>>>>> <abb...@en...> wrote:
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> On Fri, May 24, 2013 at 7:04 PM, Ashutosh Bapat
> >> >>> >>>>>>>>>>>> <ash...@en...> wrote:
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>> On Fri, May 24, 2013 at 9:01 AM, Abbas Butt
> >> >>> >>>>>>>>>>>>> <abb...@en...> wrote:
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> On Fri, May 24, 2013 at 7:22 AM, Ashutosh Bapat
> >> >>> >>>>>>>>>>>>>> <ash...@en...> wrote:
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>> On Thu, May 23, 2013 at 9:21 PM, Abbas Butt
> >> >>> >>>>>>>>>>>>>>> <abb...@en...> wrote:
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Hi,
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> While working on test case plancache it was brought
> >> >>> >>>>>>>>>>>>>>>> up as
> >> >>> >>>>>>>>>>>>>>>> a
> >> >>> >>>>>>>>>>>>>>>> review comment that solving bug id 3607975 should
> >> >>> >>>>>>>>>>>>>>>> solve
> >> >>> >>>>>>>>>>>>>>>> the problem of the
> >> >>> >>>>>>>>>>>>>>>> test case.
> >> >>> >>>>>>>>>>>>>>>> However there is some confusion in the statement of
> >> >>> >>>>>>>>>>>>>>>> bug
> >> >>> >>>>>>>>>>>>>>>> id
> >> >>> >>>>>>>>>>>>>>>> 3607975.
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> "When a user does and PREPARE and then EXECUTEs
> >> >>> >>>>>>>>>>>>>>>> multiple
> >> >>> >>>>>>>>>>>>>>>> times, the coordinator keeps on preparing and
> >> >>> >>>>>>>>>>>>>>>> executing
> >> >>> >>>>>>>>>>>>>>>> the query on
> >> >>> >>>>>>>>>>>>>>>> datanode al times, as against preparing once and
> >> >>> >>>>>>>>>>>>>>>> executing multiple times.
> >> >>> >>>>>>>>>>>>>>>> This is because somehow the remote query is being
> >> >>> >>>>>>>>>>>>>>>> prepared as an unnamed
> >> >>> >>>>>>>>>>>>>>>> statement."
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Consider this test case
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> A. create table abc(a int, b int);
> >> >>> >>>>>>>>>>>>>>>> B. insert into abc values(11, 22);
> >> >>> >>>>>>>>>>>>>>>> C. prepare p1 as select * from abc;
> >> >>> >>>>>>>>>>>>>>>> D. execute p1;
> >> >>> >>>>>>>>>>>>>>>> E. execute p1;
> >> >>> >>>>>>>>>>>>>>>> F. execute p1;
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Here are the confusions
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> 1. The coordinator never prepares on datanode in
> >> >>> >>>>>>>>>>>>>>>> response
> >> >>> >>>>>>>>>>>>>>>> to
> >> >>> >>>>>>>>>>>>>>>> a prepare issued by a user.
> >> >>> >>>>>>>>>>>>>>>>      In fact step C does nothing on the datanodes.
> >> >>> >>>>>>>>>>>>>>>>      Step D simply sends "SELECT a, b FROM abc" to
> >> >>> >>>>>>>>>>>>>>>> all
> >> >>> >>>>>>>>>>>>>>>> datanodes.
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> 2. In step D, ExecuteQuery calls BuildCachedPlan to
> >> >>> >>>>>>>>>>>>>>>> build
> >> >>> >>>>>>>>>>>>>>>> a
> >> >>> >>>>>>>>>>>>>>>> new generic plan,
> >> >>> >>>>>>>>>>>>>>>>     and steps E and F use the already built generic
> >> >>> >>>>>>>>>>>>>>>> plan.
> >> >>> >>>>>>>>>>>>>>>>     For details see function GetCachedPlan.
> >> >>> >>>>>>>>>>>>>>>>     This means that executing a prepared statement
> >> >>> >>>>>>>>>>>>>>>> again
> >> >>> >>>>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>>>> again does use cached plans
> >> >>> >>>>>>>>>>>>>>>>     and does not prepare again and again every time
> >> >>> >>>>>>>>>>>>>>>> we
> >> >>> >>>>>>>>>>>>>>>> issue
> >> >>> >>>>>>>>>>>>>>>> an execute.
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>> The problem is not here. The problem is in
> do_query()
> >> >>> >>>>>>>>>>>>>>> where
> >> >>> >>>>>>>>>>>>>>> somehow the name of prepared statement gets wiped
> out
> >> >>> >>>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>>> we keep on
> >> >>> >>>>>>>>>>>>>>> preparing unnamed statements at the datanode.
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> We never prepare any named/unnamed statements on the
> >> >>> >>>>>>>>>>>>>> datanode.
> >> >>> >>>>>>>>>>>>>> I spent time looking at the code written in do_query
> >> >>> >>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>> functions called
> >> >>> >>>>>>>>>>>>>> from with in do_query to handle prepared statements
> but
> >> >>> >>>>>>>>>>>>>> the
> >> >>> >>>>>>>>>>>>>> code written in
> >> >>> >>>>>>>>>>>>>> pgxc_start_command_on_connection to handle statements
> >> >>> >>>>>>>>>>>>>> prepared on datanodes
> >> >>> >>>>>>>>>>>>>> is dead as of now. It is never called during the
> >> >>> >>>>>>>>>>>>>> complete
> >> >>> >>>>>>>>>>>>>> regression run.
> >> >>> >>>>>>>>>>>>>> The function ActivateDatanodeStatementOnNode is never
> >> >>> >>>>>>>>>>>>>> called. The way
> >> >>> >>>>>>>>>>>>>> prepared statements are being handled now is the same
> >> >>> >>>>>>>>>>>>>> as I
> >> >>> >>>>>>>>>>>>>> described earlier
> >> >>> >>>>>>>>>>>>>> in the mail chain with the help of an example.
> >> >>> >>>>>>>>>>>>>> The code that is dead was originally added by Mason
> >> >>> >>>>>>>>>>>>>> through
> >> >>> >>>>>>>>>>>>>> commit d6d2d3d925f571b0b58ff6b4f6504d88e96bb342, back
> >> >>> >>>>>>>>>>>>>> in
> >> >>> >>>>>>>>>>>>>> December 2010. This
> >> >>> >>>>>>>>>>>>>> code has been changed a lot over the last two years.
> >> >>> >>>>>>>>>>>>>> This
> >> >>> >>>>>>>>>>>>>> commit does not
> >> >>> >>>>>>>>>>>>>> contain any test cases so I am not sure how did it
> use
> >> >>> >>>>>>>>>>>>>> to
> >> >>> >>>>>>>>>>>>>> work back then.
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>> This code wasn't dead, when I worked on prepared
> >> >>> >>>>>>>>>>>>> statements.
> >> >>> >>>>>>>>>>>>> So,
> >> >>> >>>>>>>>>>>>> something has gone wrong in-between. That's what we
> need
> >> >>> >>>>>>>>>>>>> to
> >> >>> >>>>>>>>>>>>> find out and
> >> >>> >>>>>>>>>>>>> fix. Not preparing statements on the datanode is not
> >> >>> >>>>>>>>>>>>> good
> >> >>> >>>>>>>>>>>>> for performance
> >> >>> >>>>>>>>>>>>> either.
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> I was able to find the reason why the code was dead and
> >> >>> >>>>>>>>>>>> the
> >> >>> >>>>>>>>>>>> attached patch (WIP) fixes the problem. This would now
> >> >>> >>>>>>>>>>>> ensure
> >> >>> >>>>>>>>>>>> that
> >> >>> >>>>>>>>>>>> statements are prepared on datanodes whenever required.
> >> >>> >>>>>>>>>>>> However there is a
> >> >>> >>>>>>>>>>>> problem in the way prepared statements are handled. The
> >> >>> >>>>>>>>>>>> problem is that
> >> >>> >>>>>>>>>>>> unless a prepared statement is executed it is never
> >> >>> >>>>>>>>>>>> prepared
> >> >>> >>>>>>>>>>>> on datanodes,
> >> >>> >>>>>>>>>>>> hence changing the path before executing the statement
> >> >>> >>>>>>>>>>>> gives
> >> >>> >>>>>>>>>>>> us incorrect
> >> >>> >>>>>>>>>>>> results. For Example
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>   create schema s1 create table abc (f1 int) distribute
> >> >>> >>>>>>>>>>>> by
> >> >>> >>>>>>>>>>>> replication;
> >> >>> >>>>>>>>>>>>   create schema s2 create table abc (f1 int) distribute
> >> >>> >>>>>>>>>>>> by
> >> >>> >>>>>>>>>>>> replication;
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>   insert into s1.abc values(123);
> >> >>> >>>>>>>>>>>>   insert into s2.abc values(456);
> >> >>> >>>>>>>>>>>>   set search_path = s2;
> >> >>> >>>>>>>>>>>>   prepare p1 as select f1 from abc;
> >> >>> >>>>>>>>>>>>   set search_path = s1;
> >> >>> >>>>>>>>>>>>   execute p1;
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> The last execute results in 123, where as it should
> have
> >> >>> >>>>>>>>>>>> resulted
> >> >>> >>>>>>>>>>>> in 456.
> >> >>> >>>>>>>>>>>> I can finalize the attached patch by fixing any
> >> >>> >>>>>>>>>>>> regression
> >> >>> >>>>>>>>>>>> issues
> >> >>> >>>>>>>>>>>> that may result and that would fix 3607975 and improve
> >> >>> >>>>>>>>>>>> performance however
> >> >>> >>>>>>>>>>>> the above test case would still fail.
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> My conclusion is that the bug ID 3607975 is not
> >> >>> >>>>>>>>>>>>>>>> reproducible.
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>> Did you verify it under the debugger? If that would
> >> >>> >>>>>>>>>>>>>>> have
> >> >>> >>>>>>>>>>>>>>> been
> >> >>> >>>>>>>>>>>>>>> the case, we would not have seen this problem if
> >> >>> >>>>>>>>>>>>>>> search_path changed in
> >> >>> >>>>>>>>>>>>>>> between steps D and E.
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> If search path is changed between steps D & E, the
> >> >>> >>>>>>>>>>>>>> problem
> >> >>> >>>>>>>>>>>>>> occurs because when the remote query node is created,
> >> >>> >>>>>>>>>>>>>> schema qualification
> >> >>> >>>>>>>>>>>>>> is not added in the sql statement to be sent to the
> >> >>> >>>>>>>>>>>>>> datanode, but changes in
> >> >>> >>>>>>>>>>>>>> search path do get communicated to the datanode. The
> >> >>> >>>>>>>>>>>>>> sql
> >> >>> >>>>>>>>>>>>>> statement is built
> >> >>> >>>>>>>>>>>>>> when execute is issued for the first time and is
> reused
> >> >>> >>>>>>>>>>>>>> on
> >> >>> >>>>>>>>>>>>>> subsequent
> >> >>> >>>>>>>>>>>>>> executes. The datanode is totally unaware that the
> >> >>> >>>>>>>>>>>>>> select
> >> >>> >>>>>>>>>>>>>> that it just
> >> >>> >>>>>>>>>>>>>> received is due to an execute of a prepared statement
> >> >>> >>>>>>>>>>>>>> that
> >> >>> >>>>>>>>>>>>>> was prepared when
> >> >>> >>>>>>>>>>>>>> search path was some thing else.
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>> Fixing the prepared statements the way I suggested,
> >> >>> >>>>>>>>>>>>> would
> >> >>> >>>>>>>>>>>>> fix
> >> >>> >>>>>>>>>>>>> the problem, since the statement will get prepared at
> >> >>> >>>>>>>>>>>>> the
> >> >>> >>>>>>>>>>>>> datanode, with the
> >> >>> >>>>>>>>>>>>> same search path settings, as it would on the
> >> >>> >>>>>>>>>>>>> coordinator.
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Comments are welcome.
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>>>>>> Abbas
> >> >>> >>>>>>>>>>>>>>>> Architect
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Ph: 92.334.5100153
> >> >>> >>>>>>>>>>>>>>>> Skype ID: gabbasb
> >> >>> >>>>>>>>>>>>>>>> www.enterprisedb.com
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Follow us on Twitter
> >> >>> >>>>>>>>>>>>>>>> @EnterpriseDB
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>> Visit EnterpriseDB for tutorials, webinars,
> >> >>> >>>>>>>>>>>>>>>> whitepapers
> >> >>> >>>>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>>>> more
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> ------------------------------------------------------------------------------
> >> >>> >>>>>>>>>>>>>>>> Try New Relic Now & We'll Send You this Cool Shirt
> >> >>> >>>>>>>>>>>>>>>> New Relic is the only SaaS-based application
> >> >>> >>>>>>>>>>>>>>>> performance
> >> >>> >>>>>>>>>>>>>>>> monitoring service
> >> >>> >>>>>>>>>>>>>>>> that delivers powerful full stack analytics.
> Optimize
> >> >>> >>>>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>>>> monitor your
> >> >>> >>>>>>>>>>>>>>>> browser, app, & servers with just a few lines of
> >> >>> >>>>>>>>>>>>>>>> code.
> >> >>> >>>>>>>>>>>>>>>> Try
> >> >>> >>>>>>>>>>>>>>>> New Relic
> >> >>> >>>>>>>>>>>>>>>> and get this awesome Nerd Life shirt!
> >> >>> >>>>>>>>>>>>>>>> http://p.sf.net/sfu/newrelic_d2d_may
> >> >>> >>>>>>>>>>>>>>>> _______________________________________________
> >> >>> >>>>>>>>>>>>>>>> Postgres-xc-developers mailing list
> >> >>> >>>>>>>>>>>>>>>> Pos...@li...
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >> >>> >>>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>>>>> Best Wishes,
> >> >>> >>>>>>>>>>>>>>> Ashutosh Bapat
> >> >>> >>>>>>>>>>>>>>> EntepriseDB Corporation
> >> >>> >>>>>>>>>>>>>>> The Postgres Database Company
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>>>> Abbas
> >> >>> >>>>>>>>>>>>>> Architect
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> Ph: 92.334.5100153
> >> >>> >>>>>>>>>>>>>> Skype ID: gabbasb
> >> >>> >>>>>>>>>>>>>> www.enterprisedb.com
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> Follow us on Twitter
> >> >>> >>>>>>>>>>>>>> @EnterpriseDB
> >> >>> >>>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>> Visit EnterpriseDB for tutorials, webinars,
> whitepapers
> >> >>> >>>>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>>>> more
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>>> Best Wishes,
> >> >>> >>>>>>>>>>>>> Ashutosh Bapat
> >> >>> >>>>>>>>>>>>> EntepriseDB Corporation
> >> >>> >>>>>>>>>>>>> The Postgres Database Company
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>> --
> >> >>> >>>>>>>>>>>> Abbas
> >> >>> >>>>>>>>>>>> Architect
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> Ph: 92.334.5100153
> >> >>> >>>>>>>>>>>> Skype ID: gabbasb
> >> >>> >>>>>>>>>>>> www.enterprisedb.com
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> Follow us on Twitter
> >> >>> >>>>>>>>>>>> @EnterpriseDB
> >> >>> >>>>>>>>>>>>
> >> >>> >>>>>>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers
> >> >>> >>>>>>>>>>>> and
> >> >>> >>>>>>>>>>>> more
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>>
> >> >>> >>>>>>>>>>> --
> >> >>> >>>>>>>>>>> Best Wishes,
> >> >>> >>>>>>>>>>> Ashutosh Bapat
> >> >>> >>>>>>>>>>> EntepriseDB Corporation
> >> >>> >>>>>>>>>>> The Postgres Database Company
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> --
> >> >>> >>>>>>>>>> --
> >> >>> >>>>>>>>>> Abbas
> >> >>> >>>>>>>>>> Architect
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> Ph: 92.334.5100153
> >> >>> >>>>>>>>>> Skype ID: gabbasb
> >> >>> >>>>>>>>>> www.enterprisedb.com
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> Follow us on Twitter
> >> >>> >>>>>>>>>> @EnterpriseDB
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers
> and
> >> >>> >>>>>>>>>> more
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> --
> >> >>> >>>>>>>>> Best Wishes,
> >> >>> >>>>>>>>> Ashutosh Bapat
> >> >>> >>>>>>>>> EntepriseDB Corporation
> >> >>> >>>>>>>>> The Postgres Database Company
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> --
> >> >>> >>>>>>>> --
> >> >>> >>>>>>>> Abbas
> >> >>> >>>>>>>> Architect
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Ph: 92.334.5100153
> >> >>> >>>>>>>> Skype ID: gabbasb
> >> >>> >>>>>>>> www.enterprisedb.com
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Follow us on Twitter
> >> >>> >>>>>>>> @EnterpriseDB
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers and
> >> >>> >>>>>>>> more
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> --
> >> >>> >>>>>>> Best Wishes,
> >> >>> >>>>>>> Ashutosh Bapat
> >> >>> >>>>>>> EntepriseDB Corporation
> >> >>> >>>>>>> The Postgres Database Company
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> --
> >> >>> >>>>>> --
> >> >>> >>>>>> Abbas
> >> >>> >>>>>> Architect
> >> >>> >>>>>>
> >> >>> >>>>>> Ph: 92.334.5100153
> >> >>> >>>>>> Skype ID: gabbasb
> >> >>> >>>>>> www.enterprisedb.com
> >> >>> >>>>>>
> >> >>> >>>>>> Follow us on Twitter
> >> >>> >>>>>> @EnterpriseDB
> >> >>> >>>>>>
> >> >>> >>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers and
> >> >>> >>>>>> more
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> --
> >> >>> >>>>> Best Wishes,
> >> >>> >>>>> Ashutosh Bapat
> >> >>> >>>>> EntepriseDB Corporation
> >> >>> >>>>> The Postgres Database Company
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> --
> >> >>> >>>> --
> >> >>> >>>> Abbas
> >> >>> >>>> Architect
> >> >>> >>>>
> >> >>> >>>> Ph: 92.334.5100153
> >> >>> >>>> Skype ID: gabbasb
> >> >>> >>>> www.enterprisedb.com
> >> >>> >>>>
> >> >>> >>>> Follow us on Twitter
> >> >>> >>>> @EnterpriseDB
> >> >>> >>>>
> >> >>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers and
> more
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> --
> >> >>> >>> --
> >> >>> >>> Abbas
> >> >>> >>> Architect
> >> >>> >>>
> >> >>> >>> Ph: 92.334.5100153
> >> >>> >>> Skype ID: gabbasb
> >> >>> >>> www.enterprisedb.com
> >> >>> >>>
> >> >>> >>> Follow us on Twitter
> >> >>> >>> @EnterpriseDB
> >> >>> >>>
> >> >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Best Wishes,
> >> >>> >> Ashutosh Bapat
> >> >>> >> EntepriseDB Corporation
> >> >>> >> The Postgres Database Company
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> ------------------------------------------------------------------------------
> >> >>> >> This SF.net email is sponsored by Windows:
> >> >>> >>
> >> >>> >> Build for Windows Store.
> >> >>> >>
> >> >>> >> http://p.sf.net/sfu/windows-dev2dev
> >> >>> >> _______________________________________________
> >> >>> >> Postgres-xc-developers mailing list
> >> >>> >> Pos...@li...
> >> >>> >>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >> >>> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best Wishes,
> >> >> Ashutosh Bapat
> >> >> EntepriseDB Corporation
> >> >> The Postgres Database Company
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> This SF.net email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev
> >> _______________________________________________
> >> Postgres-xc-developers mailing list
> >> Pos...@li...
> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >>
> >
> >
> <inherit.out><inherit.sql><regression.diffs><regression.out>------------------------------------------------------------------------------
> >
> > This SF.net email is sponsored by Windows:
> >
> > Build for Windows Store.
> >
> >
> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
> >
>