postgres-xc-developers — Postgres-XC hackers and developers

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 4 Cannot convert int to define

[Ashutosh B. <ash...@en...>]

We should move the Assert under #ifdef ASSERT_ENABLED as well the
definition of old_outEnd. We should remove #else defining old_outEnd to
handle->outEnd which is utterly wrong.


On Mon, Sep 30, 2013 at 12:15 PM, Koichi Suzuki <koi...@gm...>wrote:

> This change was made to eliminate a warning that old_outEnd is not used.
> To improve both, the patch would be as attached.
>
> Regards;
>
> ---
> Koichi Suzuki
>
>
> 2013/9/30 Abbas Butt <abb...@en...>
>
>> Hi,
>>
>> The commit ae29a966d3286d4e85b3adf162d00f9b1e8b8eae made this change
>> @@ -1132,7 +1132,11 @@ pgxc_node_send_parse(PGXCNodeHandle * handle, const
>> char* statement,
>>         /* message length */
>>         int                     msgLen;
>>         int                     cnt_params;
>> +#if 0
>>         size_t          old_outEnd = handle->outEnd;
>> +#else
>> +#define old_outEnd handle->outEnd
>> +#endif
>>
>> which is incorrect. The purpose of the declaration and assignment is to
>> save the current value of handle->outEnd which is later changed in the
>> function.
>> This change makes the following assert in function pgxc_node_send_parse
>>     Assert(old_outEnd + ntohl(msgLen) + 1 == handle->outEnd);
>> fail when the query
>> CREATE TABLE junk AS VALUES (1), (2);
>> is run.
>>
>> This patch is required on master as well.
>>
>> Regards
>>
>> --
>> *Abbas*
>>  Architect
>>
>> Ph: 92.334.5100153
>>  Skype ID: gabbasb
>> www.enterprisedb.co
>> <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>> *
>> Follow us on Twitter*
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars,
>> whitepapers<http://www.enterprisedb.com/resources-community>and
>> more<http://www.enterprisedb.com/resources-community>
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[鈴木 幸市 <ko...@in...>]

Hmmm.   I don't see clear background that CLEAN CONNECTION should have its event trigger either.
---
Koichi Suzuki

On 2013/09/30, at 18:53, Michael Paquier <mic...@gm...> wrote:

> On Mon, Sep 30, 2013 at 4:19 PM, 鈴木 幸市 <ko...@in...> wrote:
>> Do you agree not to support event triggers because it does support only a subset of CREATE/ALTER/DROP or XC architecture needs extra improvement?
> Supporting event triggers would be a nice thing, just I don't see why
> it would be necessary for CLEAN CONNECTION.
> -- 
> Michael
> 

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[Michael P. <mic...@gm...>]

On Mon, Sep 30, 2013 at 4:19 PM, 鈴木 幸市 <ko...@in...> wrote:
> Do you agree not to support event triggers because it does support only a subset of CREATE/ALTER/DROP or XC architecture needs extra improvement?
Supporting event triggers would be a nice thing, just I don't see why
it would be necessary for CLEAN CONNECTION.
-- 
Michael

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[鈴木 幸市 <ko...@in...>]

Do you agree not to support event triggers because it does support only a subset of CREATE/ALTER/DROP or XC architecture needs extra improvement?
---
Koichi Suzuki

On 2013/09/30, at 14:02, Michael Paquier <mic...@gm...> wrote:

> On Mon, Sep 30, 2013 at 11:23 AM, Abbas Butt
> <abb...@en...> wrote:
>> The current implementation of event triggers assumes that any statment
>> that is not handeled by the main switch-case in standard_ProcessUtility
>> supports event triggers.
>> CLEAN CONNECTION is a new statement added by XC and does not support
>> event triggers as of now.
> I would agree not to support event triggers at all for event triggers.
> You can have a look at the commands that work with event triggers
> here:
> http://www.postgresql.org/docs/9.3/static/event-trigger-matrix.html
> You would notice that only a subset of CREATE/ALTER/DROP commands is
> supported only. There is nothing like for example CLUSTER or VACUUM,
> etc.
> -- 
> Michael
> 
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[Michael P. <mic...@gm...>]

On Mon, Sep 30, 2013 at 11:23 AM, Abbas Butt
<abb...@en...> wrote:
> The current implementation of event triggers assumes that any statment
> that is not handeled by the main switch-case in standard_ProcessUtility
> supports event triggers.
> CLEAN CONNECTION is a new statement added by XC and does not support
> event triggers as of now.
I would agree not to support event triggers at all for event triggers.
You can have a look at the commands that work with event triggers
here:
http://www.postgresql.org/docs/9.3/static/event-trigger-matrix.html
You would notice that only a subset of CREATE/ALTER/DROP commands is
supported only. There is nothing like for example CLUSTER or VACUUM,
etc.
-- 
Michael

Re: [Postgres-xc-developers] Status of master_pg93_merge branch

[Michael P. <mic...@gm...>]

On Mon, Sep 30, 2013 at 1:59 PM, Michael Paquier
<mic...@gm...> wrote:
> That's at least for I did for 9.1 and 9.2 merges.
Sorry. I meant "What I did for 9.1 and 9.2 merges". Stupid typo.
-- 
Michael

Re: [Postgres-xc-developers] Status of master_pg93_merge branch

[Michael P. <mic...@gm...>]

On Mon, Sep 30, 2013 at 11:20 AM, Abbas Butt
<abb...@en...> wrote:
> I pulled the latest code and switched to master_pg93_merge branch.
> I then applied the following two patches and tried to run regression.
> 20130924_01.patch and fix_for_pooler_socket_20130926.patch
>
> I ran into multiple issues and would provide fixes for them one by one.
> However I noticed that both the above patches are not final as of now
> and contain various comments that need to be resolved.
>
> I suggest we make a list of all known issues in these two patches
> and plan to resolve them one by one.
Solving each issue one by one with a different commit instead of
applying some huge patches would bring visibility in the resolution
flow. That's at least for I did for 9.1 and 9.2 merges.
-- 
Michael

[Postgres-xc-developers] [master_pg93_merge] Patch # 4 Cannot convert int to define

[Abbas B. <abb...@en...>]

Hi,

The commit ae29a966d3286d4e85b3adf162d00f9b1e8b8eae made this change
@@ -1132,7 +1132,11 @@ pgxc_node_send_parse(PGXCNodeHandle * handle, const
char* statement,
        /* message length */
        int                     msgLen;
        int                     cnt_params;
+#if 0
        size_t          old_outEnd = handle->outEnd;
+#else
+#define old_outEnd handle->outEnd
+#endif

which is incorrect. The purpose of the declaration and assignment is to
save the current value of handle->outEnd which is later changed in the
function.
This change makes the following assert in function pgxc_node_send_parse
    Assert(old_outEnd + ntohl(msgLen) + 1 == handle->outEnd);
fail when the query
CREATE TABLE junk AS VALUES (1), (2);
is run.

This patch is required on master as well.

Regards

-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

Re: [Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[Abbas B. <abb...@en...>]

Here is the patch, forgot to attach it with the previous mail.
Regards



On Mon, Sep 30, 2013 at 7:23 AM, Abbas Butt <abb...@en...> wrote:
>
> Hi,
> The current implementation of event triggers assumes that any statment
> that is not handeled by the main switch-case in standard_ProcessUtility
> supports event triggers.
> CLEAN CONNECTION is a new statement added by XC and does not support
> event triggers as of now.
>
> This patch is intended for the master_pg93_merge branch and applies cleanly
> on top of the following two patches.
> 20130924_01.patch and fix_for_pooler_socket_20130926.patch
>
> Regards
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more




-- 
-- 
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more


On Mon, Sep 30, 2013 at 7:23 AM, Abbas Butt <abb...@en...> wrote:
>
> Hi,
> The current implementation of event triggers assumes that any statment
> that is not handeled by the main switch-case in standard_ProcessUtility
> supports event triggers.
> CLEAN CONNECTION is a new statement added by XC and does not support
> event triggers as of now.
>
> This patch is intended for the master_pg93_merge branch and applies cleanly
> on top of the following two patches.
> 20130924_01.patch and fix_for_pooler_socket_20130926.patch
>
> Regards
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more




-- 
-- 
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

[Postgres-xc-developers] [master_pg93_merge] Patch # 3 for CLEAN CONNECTION

[Abbas B. <abb...@en...>]

Hi,
The current implementation of event triggers assumes that any statment
that is not handeled by the main switch-case in standard_ProcessUtility
supports event triggers.
CLEAN CONNECTION is a new statement added by XC and does not support
event triggers as of now.

This patch is intended for the master_pg93_merge branch and applies cleanly
on top of the following two patches.
20130924_01.patch and fix_for_pooler_socket_20130926.patch

Regards
-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

[Postgres-xc-developers] Status of master_pg93_merge branch

[Abbas B. <abb...@en...>]

Hi,
I pulled the latest code and switched to master_pg93_merge branch.
I then applied the following two patches and tried to run regression.
20130924_01.patch and fix_for_pooler_socket_20130926.patch

I ran into multiple issues and would provide fixes for them one by one.
However I noticed that both the above patches are not final as of now
and contain various comments that need to be resolved.

I suggest we make a list of all known issues in these two patches
and plan to resolve them one by one.

Regards

-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

Re: [Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Tomonari K. <t.k...@gm...>]

Hi Ashotosh,

>    I never said "don't use functions". Here's what I mailed
>
Oh, Sorry for bothering you by my misunderstanding.

>>        What is better for this testcase?
>
>    May be we should just drop the testcase. Anyway, EXEC DIRECT is not
production feature, but debug feature and whosoever wrote this feature has
not added any testcases for it anyway.
>    If anyone else has objections to dropping the testcase, please suggest
Tomonari, a better to write it.
>
Past 2 days, any suggestions were not sent to me.
So it seems no objections to dropping the testcase.

regards,
----------------
NTT Software Corporation
Tomonari Katsumata



2013/9/26 Ashutosh Bapat <ash...@en...>

>
>
>
> On Thu, Sep 26, 2013 at 1:29 PM, Tomonari Katsumata <
> kat...@po...> wrote:
>
>> Hi Ashtosh,
>>
>> Thanks for your review!
>>
>> Previous patch(revised_fqs_r3.patch in [2013/08/14 19:55])
>> doesn't use a shell script.
>> Instead, It used test_execute_direct_all_xc_**nodes() function.
>>
>> I've got a comment "don't use functions" from you,
>> but I could not come up with good idea for the testcase.
>> The testcase needs to get all node_names to "EXECUTE DIRECT"
>> against all nodes.
>>
>>
> I never said "don't use functions". Here's what I mailed
>
> "In the test file, isn't there any way, you can add the offending statent
> without any wrapper function test_execute_direct_all_xc_
> nodes()? We should minimize the test-code to use only the minimal set of
> features. Since this statement no more gets into infinite loop, please do
> not use the statement timeout."
>
>
>> What is better for this testcase?
>>
>>
> May be we should just drop the testcase. Anyway, EXEC DIRECT is not
> production feature, but debug feature and whosoever wrote this feature has
> not added any testcases for it anyway.
>
> If anyone else has objections to dropping the testcase, please suggest
> Tomonari, a better to write it.
>
>
>> regards,
>>
>> ----------------
>> NTT Software Corporation
>> Tomonari Katsumata
>>
>> (2013/09/26 15:44), Ashutosh Bapat wrote:
>> > Hi Tomonary,
>> > The testcase you have written uses a shell script. This is not the
>> standard
>> > way to add testcases. Can you please see if the testcase can be added
>> > directly as SQL? Please send a revised patch ASAP, so that it can be
>> > committed. I would have liked to see more testcases for EXEC DIRECT, but
>> > since it's a debug only feature (not for production), it gets lesser
>> > priority in testing and thus we do not see testcases for it.
>> >
>> >
>> > On Thu, Sep 26, 2013 at 8:26 AM, Tomonari Katsumata <
>> > kat...@po....**jp <kat...@po...>>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> Does anyone check these patches?
>> >> If there are no comments, please commit it.
>> >>
>> >> regards,
>> >>
>> >> ----------------------
>> >> NTT Software Corporation
>> >> Tomonari Katsumata
>> >>
>> >> -------- Original Message --------
>> >> Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
>> >> Date:     Thu, 5 Sep 2013 21:34:32 +0900
>> >> From:     Tomonari Katsumata <t.k...@gm...>
>> >> To:     Tomonari Katsumata <katsumata.tomonari@po.ntts.****co.jp<http://co.jp>
>> <katsumata.tomonari@po.**ntts.co.jp <kat...@po...>>
>> >>>
>> >> CC:     Ashutosh Bapat <ashutosh.bapat@enterprisedb.****com<
>> ashutosh.bapat@**enterprisedb.com <ash...@en...>>>,
>> >> pgxc-hackers mailing list <postgres-xc-developers@lists.****
>> sourceforge.net<postgres-xc-**dev...@li...urceforge.**net<pos...@li...>
>> >
>>
>> >>>
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I'm sorry for long long leaving this thread alone.
>> >> I've devided the patch to source part and regression part.
>> >> (against f8e0f994bff1faefa9b1fc2f6e584a****1d3de70241)
>>
>> >>
>> >> I did not change any process on source part.
>> >> (revised_fqs_r4.patch)
>> >>
>> >> I added a script into src/test/regress/sql directory for
>> >> changing regression part.
>> >> (add_test_xc_misc.patch and execute_direct_regress.sh)
>> >> When regression test runs, the script is executed instead of
>> >> user defined function.
>> >>
>> >> Please check and commit it.
>> >>
>> >> regards,
>> >> ------------------------
>> >> NTT Software Corporation
>> >> Tomonari Katsumata
>> >>
>> >>
>> >> 2013/8/14 Tomonari Katsumata <katsumata.tomonari@po.ntts.****co.jp<http://co.jp>
>> <katsumata.tomonari@po.**ntts.co.jp <kat...@po...>>
>>
>> >>>
>> >>
>> >>> Hi Ashtosh,
>> >>>
>> >>> Thanks for your comment!
>> >>>
>> >>>
>> >>>> The code changes are fine, but, the comment in
>> >> pgxc_collect_RTE_walker()
>> >>>> seems to specific. I think it should read, "create a copy of query's
>> >>> range
>> >>>> table, so that it can be linked with other RTEs in the collector's
>> >>> context."
>> >>>>
>> >>> I've revised the comment as you suggested.
>> >>>
>> >>>
>> >>>> In the test file, isn't there any way, you can add the offending
>> >> statent
>> >>>> without any wrapper function test_execute_direct_all_xc_*****
>> *nodes()?
>>
>> >> We
>> >>
>> >>> should
>> >>>> minimize the test-code to use only the minimal set of features. Since
>> >>> this
>> >>>> statement no more gets into infinite loop, please do not use the
>> >>> statement
>> >>>> timeout.
>> >>>
>> >>> I've gotten rid of using statement timeout.
>> >>> But I couldn't come up any idea to testing more simply,
>> >>> so the function is remaining as same as before patch.
>> >>> Any ideas?
>> >>>
>> >>> Here is the new patch.
>> >>> (against dec40008b3d689911566514614c511******1c0a61327d)
>>
>> >>
>> >>>
>> >>>
>> >>> regards,
>> >>> -------------
>> >>> NTT Software Corporation
>> >>> Tomonari Katsumata
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >> ------------------------------**------------------------------**
>> ------------------
>> >> October Webinars: Code for Performance
>> >> Free Intel webinars can help you accelerate application performance.
>> >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>> most
>> >> from
>> >> the latest Intel processors and coprocessors. See abstracts and
>> register >
>> >> http://pubads.g.doubleclick.**net/gampad/clk?id=60133471&iu=**
>> /4140/ostg.clktrk<http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk>
>> >> ______________________________**_________________
>> >> Postgres-xc-developers mailing list
>> >> Postgres-xc-developers@lists.**sourceforge.net<Pos...@li...>
>> >> https://lists.sourceforge.net/**lists/listinfo/postgres-xc-**
>> developers<https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers>
>> >>
>> >>
>> >
>> >
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Re: [Postgres-xc-developers] [postgres-xc:bugs] #454 Update triggers on replicated tables may corrupt data

[Mason S. <ma...@st...>]

On Thu, Sep 26, 2013 at 9:51 PM, Amit Khandekar <
ami...@en...> wrote:

> http://sourceforge.net/p/postgres-xc/bugs/402/
>
> But this issue has nothing do with triggers. I think there should be some
> way to reproduce without triggers. Although, if any user triggers or unique
> constraint triggers makes our job of reproducing the issue easy, that would
> be good.
>

Under what other circumstances can one reproduce? When the statement cannot
be pushed down directly, like an UPDATE with volatile functions?

It was not hard to reproduce... insert a few hundred rows in a table.
Execute an UPDATE on one that has a trigger defined. I guess it could
depend on when each node's particular auto-vacuum was run, perhaps with
concurrent activity.

Anyway, I think this is a major issue.



>
>
> On 27 September 2013 09:33, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Hi Hackers,
>> While working on triggers me and Amit had thought of this problem and
>> there should be a mail from Amit in that regards. What stopped us from
>> working on this more, was a reproduction scenario where same row landed on
>> two different nodes with different CTID. We couldn't get that scenario even
>> with complex ingredients like vacuum analyze etc. If you are hitting this
>> problem, can one provide us a reproduction.
>>
>>
>> On Thu, Sep 26, 2013 at 11:30 PM, mason_s <ma...@us...> wrote:
>>
>>> ------------------------------
>>>
>>> * [bugs:#454] <http://sourceforge.net/p/postgres-xc/bugs/454/> Update
>>> triggers on replicated tables may corrupt data*
>>>
>>> *Status:* open
>>> *Created:* Thu Sep 26, 2013 06:00 PM UTC by mason_s
>>> *Last Updated:* Thu Sep 26, 2013 06:00 PM UTC
>>> *Owner:* nobody
>>>
>>> We noticed that when updating a single row on a replicated table that we
>>> were getting duplicate key violations, even though the primary key columns
>>> were not involved.
>>>
>>> We dug deeper and noticed that the mechanism uses ctid to identify what
>>> to update for update triggers that are non-"shippable" and executed on a
>>> coordinator. In the case of a replicated table, the ctid value may be
>>> different on different nodes for each tuple. It appears that Postgres-XC
>>> just uses one ctid value from one of the nodes and then sends down the same
>>> UPDATE statement to all of the individual nodes.
>>>
>>> It should either get each ctid for each node and update, or determine
>>> the corresponding unique key values and use that in the generated WHERE
>>> clause.
>>> ------------------------------
>>>
>>> Sent from sourceforge.net because you indicated interest in
>>> https://sourceforge.net/p/postgres-xc/bugs/454/
>>>
>>> To unsubscribe from further messages, please visit
>>> https://sourceforge.net/auth/subscriptions/
>>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Mason Sharp

StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Services

Re: [Postgres-xc-developers] [postgres-xc:bugs] #454 Update triggers on replicated tables may corrupt data

[Amit K. <ami...@en...>]

http://sourceforge.net/p/postgres-xc/bugs/402/

But this issue has nothing do with triggers. I think there should be some
way to reproduce without triggers. Although, if any user triggers or unique
constraint triggers makes our job of reproducing the issue easy, that would
be good.


On 27 September 2013 09:33, Ashutosh Bapat
<ash...@en...>wrote:

> Hi Hackers,
> While working on triggers me and Amit had thought of this problem and
> there should be a mail from Amit in that regards. What stopped us from
> working on this more, was a reproduction scenario where same row landed on
> two different nodes with different CTID. We couldn't get that scenario even
> with complex ingredients like vacuum analyze etc. If you are hitting this
> problem, can one provide us a reproduction.
>
>
> On Thu, Sep 26, 2013 at 11:30 PM, mason_s <ma...@us...> wrote:
>
>> ------------------------------
>>
>> * [bugs:#454] <http://sourceforge.net/p/postgres-xc/bugs/454/> Update
>> triggers on replicated tables may corrupt data*
>>
>> *Status:* open
>> *Created:* Thu Sep 26, 2013 06:00 PM UTC by mason_s
>> *Last Updated:* Thu Sep 26, 2013 06:00 PM UTC
>> *Owner:* nobody
>>
>> We noticed that when updating a single row on a replicated table that we
>> were getting duplicate key violations, even though the primary key columns
>> were not involved.
>>
>> We dug deeper and noticed that the mechanism uses ctid to identify what
>> to update for update triggers that are non-"shippable" and executed on a
>> coordinator. In the case of a replicated table, the ctid value may be
>> different on different nodes for each tuple. It appears that Postgres-XC
>> just uses one ctid value from one of the nodes and then sends down the same
>> UPDATE statement to all of the individual nodes.
>>
>> It should either get each ctid for each node and update, or determine the
>> corresponding unique key values and use that in the generated WHERE clause.
>> ------------------------------
>>
>> Sent from sourceforge.net because you indicated interest in
>> https://sourceforge.net/p/postgres-xc/bugs/454/
>>
>> To unsubscribe from further messages, please visit
>> https://sourceforge.net/auth/subscriptions/
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Re: [Postgres-xc-developers] [postgres-xc:bugs] #454 Update triggers on replicated tables may corrupt data

[Ashutosh B. <ash...@en...>]

Hi Hackers,
While working on triggers me and Amit had thought of this problem and there
should be a mail from Amit in that regards. What stopped us from working on
this more, was a reproduction scenario where same row landed on two
different nodes with different CTID. We couldn't get that scenario even
with complex ingredients like vacuum analyze etc. If you are hitting this
problem, can one provide us a reproduction.


On Thu, Sep 26, 2013 at 11:30 PM, mason_s <ma...@us...> wrote:

> ------------------------------
>
> * [bugs:#454] <http://sourceforge.net/p/postgres-xc/bugs/454/> Update
> triggers on replicated tables may corrupt data*
>
> *Status:* open
> *Created:* Thu Sep 26, 2013 06:00 PM UTC by mason_s
> *Last Updated:* Thu Sep 26, 2013 06:00 PM UTC
> *Owner:* nobody
>
> We noticed that when updating a single row on a replicated table that we
> were getting duplicate key violations, even though the primary key columns
> were not involved.
>
> We dug deeper and noticed that the mechanism uses ctid to identify what to
> update for update triggers that are non-"shippable" and executed on a
> coordinator. In the case of a replicated table, the ctid value may be
> different on different nodes for each tuple. It appears that Postgres-XC
> just uses one ctid value from one of the nodes and then sends down the same
> UPDATE statement to all of the individual nodes.
>
> It should either get each ctid for each node and update, or determine the
> corresponding unique key values and use that in the generated WHERE clause.
> ------------------------------
>
> Sent from sourceforge.net because you indicated interest in
> https://sourceforge.net/p/postgres-xc/bugs/454/
>
> To unsubscribe from further messages, please visit
> https://sourceforge.net/auth/subscriptions/
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: [Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Ashutosh B. <ash...@en...>]

On Thu, Sep 26, 2013 at 1:29 PM, Tomonari Katsumata <
kat...@po...> wrote:

> Hi Ashtosh,
>
> Thanks for your review!
>
> Previous patch(revised_fqs_r3.patch in [2013/08/14 19:55])
> doesn't use a shell script.
> Instead, It used test_execute_direct_all_xc_**nodes() function.
>
> I've got a comment "don't use functions" from you,
> but I could not come up with good idea for the testcase.
> The testcase needs to get all node_names to "EXECUTE DIRECT"
> against all nodes.
>
>
I never said "don't use functions". Here's what I mailed
"In the test file, isn't there any way, you can add the offending statent
without any wrapper function test_execute_direct_all_xc_
nodes()? We should minimize the test-code to use only the minimal set of
features. Since this statement no more gets into infinite loop, please do
not use the statement timeout."


> What is better for this testcase?
>
>
May be we should just drop the testcase. Anyway, EXEC DIRECT is not
production feature, but debug feature and whosoever wrote this feature has
not added any testcases for it anyway.

If anyone else has objections to dropping the testcase, please suggest
Tomonari, a better to write it.


> regards,
>
> ----------------
> NTT Software Corporation
> Tomonari Katsumata
>
> (2013/09/26 15:44), Ashutosh Bapat wrote:
> > Hi Tomonary,
> > The testcase you have written uses a shell script. This is not the
> standard
> > way to add testcases. Can you please see if the testcase can be added
> > directly as SQL? Please send a revised patch ASAP, so that it can be
> > committed. I would have liked to see more testcases for EXEC DIRECT, but
> > since it's a debug only feature (not for production), it gets lesser
> > priority in testing and thus we do not see testcases for it.
> >
> >
> > On Thu, Sep 26, 2013 at 8:26 AM, Tomonari Katsumata <
> > kat...@po....**jp <kat...@po...>>
> wrote:
> >
> >> Hi,
> >>
> >> Does anyone check these patches?
> >> If there are no comments, please commit it.
> >>
> >> regards,
> >>
> >> ----------------------
> >> NTT Software Corporation
> >> Tomonari Katsumata
> >>
> >> -------- Original Message --------
> >> Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
> >> Date:     Thu, 5 Sep 2013 21:34:32 +0900
> >> From:     Tomonari Katsumata <t.k...@gm...>
> >> To:     Tomonari Katsumata <katsumata.tomonari@po.ntts.****co.jp<http://co.jp>
> <katsumata.tomonari@po.**ntts.co.jp <kat...@po...>>
> >>>
> >> CC:     Ashutosh Bapat <ashutosh.bapat@enterprisedb.****com<
> ashutosh.bapat@**enterprisedb.com <ash...@en...>>>,
> >> pgxc-hackers mailing list <postgres-xc-developers@lists.****
> sourceforge.net<postgres-xc-**dev...@li...urceforge.**net<pos...@li...>
> >
>
> >>>
> >>
> >>
> >> Hi,
> >>
> >> I'm sorry for long long leaving this thread alone.
> >> I've devided the patch to source part and regression part.
> >> (against f8e0f994bff1faefa9b1fc2f6e584a****1d3de70241)
>
> >>
> >> I did not change any process on source part.
> >> (revised_fqs_r4.patch)
> >>
> >> I added a script into src/test/regress/sql directory for
> >> changing regression part.
> >> (add_test_xc_misc.patch and execute_direct_regress.sh)
> >> When regression test runs, the script is executed instead of
> >> user defined function.
> >>
> >> Please check and commit it.
> >>
> >> regards,
> >> ------------------------
> >> NTT Software Corporation
> >> Tomonari Katsumata
> >>
> >>
> >> 2013/8/14 Tomonari Katsumata <katsumata.tomonari@po.ntts.****co.jp<http://co.jp>
> <katsumata.tomonari@po.**ntts.co.jp <kat...@po...>>
>
> >>>
> >>
> >>> Hi Ashtosh,
> >>>
> >>> Thanks for your comment!
> >>>
> >>>
> >>>> The code changes are fine, but, the comment in
> >> pgxc_collect_RTE_walker()
> >>>> seems to specific. I think it should read, "create a copy of query's
> >>> range
> >>>> table, so that it can be linked with other RTEs in the collector's
> >>> context."
> >>>>
> >>> I've revised the comment as you suggested.
> >>>
> >>>
> >>>> In the test file, isn't there any way, you can add the offending
> >> statent
> >>>> without any wrapper function test_execute_direct_all_xc_*****
> *nodes()?
>
> >> We
> >>
> >>> should
> >>>> minimize the test-code to use only the minimal set of features. Since
> >>> this
> >>>> statement no more gets into infinite loop, please do not use the
> >>> statement
> >>>> timeout.
> >>>
> >>> I've gotten rid of using statement timeout.
> >>> But I couldn't come up any idea to testing more simply,
> >>> so the function is remaining as same as before patch.
> >>> Any ideas?
> >>>
> >>> Here is the new patch.
> >>> (against dec40008b3d689911566514614c511******1c0a61327d)
>
> >>
> >>>
> >>>
> >>> regards,
> >>> -------------
> >>> NTT Software Corporation
> >>> Tomonari Katsumata
> >>>
> >>>
> >>
> >>
> >>
> >>
> >> ------------------------------**------------------------------**
> ------------------
> >> October Webinars: Code for Performance
> >> Free Intel webinars can help you accelerate application performance.
> >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> >> from
> >> the latest Intel processors and coprocessors. See abstracts and
> register >
> >> http://pubads.g.doubleclick.**net/gampad/clk?id=60133471&iu=**
> /4140/ostg.clktrk<http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk>
> >> ______________________________**_________________
> >> Postgres-xc-developers mailing list
> >> Postgres-xc-developers@lists.**sourceforge.net<Pos...@li...>
> >> https://lists.sourceforge.net/**lists/listinfo/postgres-xc-**developers<https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers>
> >>
> >>
> >
> >
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: [Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Tomonari K. <kat...@po...>]

Hi Ashtosh,

Thanks for your review!

Previous patch(revised_fqs_r3.patch in [2013/08/14 19:55])
doesn't use a shell script.
Instead, It used test_execute_direct_all_xc_nodes() function.

I've got a comment "don't use functions" from you,
but I could not come up with good idea for the testcase.
The testcase needs to get all node_names to "EXECUTE DIRECT"
against all nodes.

What is better for this testcase?

regards,
----------------
NTT Software Corporation
Tomonari Katsumata

(2013/09/26 15:44), Ashutosh Bapat wrote:
 > Hi Tomonary,
 > The testcase you have written uses a shell script. This is not the 
standard
 > way to add testcases. Can you please see if the testcase can be added
 > directly as SQL? Please send a revised patch ASAP, so that it can be
 > committed. I would have liked to see more testcases for EXEC DIRECT, but
 > since it's a debug only feature (not for production), it gets lesser
 > priority in testing and thus we do not see testcases for it.
 >
 >
 > On Thu, Sep 26, 2013 at 8:26 AM, Tomonari Katsumata <
 > kat...@po...> wrote:
 >
 >> Hi,
 >>
 >> Does anyone check these patches?
 >> If there are no comments, please commit it.
 >>
 >> regards,
 >>
 >> ----------------------
 >> NTT Software Corporation
 >> Tomonari Katsumata
 >>
 >> -------- Original Message --------
 >> Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
 >> Date:     Thu, 5 Sep 2013 21:34:32 +0900
 >> From:     Tomonari Katsumata <t.k...@gm...>
 >> To:     Tomonari Katsumata 
<katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
 >>>
 >> CC:     Ashutosh Bapat 
<ashutosh.bapat@enterprisedb.**com<ash...@en...>>,
 >> pgxc-hackers mailing list 
<postgres-xc-developers@lists.**sourceforge.net<pos...@li...>
 >>>
 >>
 >>
 >> Hi,
 >>
 >> I'm sorry for long long leaving this thread alone.
 >> I've devided the patch to source part and regression part.
 >> (against f8e0f994bff1faefa9b1fc2f6e584a**1d3de70241)
 >>
 >> I did not change any process on source part.
 >> (revised_fqs_r4.patch)
 >>
 >> I added a script into src/test/regress/sql directory for
 >> changing regression part.
 >> (add_test_xc_misc.patch and execute_direct_regress.sh)
 >> When regression test runs, the script is executed instead of
 >> user defined function.
 >>
 >> Please check and commit it.
 >>
 >> regards,
 >> ------------------------
 >> NTT Software Corporation
 >> Tomonari Katsumata
 >>
 >>
 >> 2013/8/14 Tomonari Katsumata 
<katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
 >>>
 >>
 >>> Hi Ashtosh,
 >>>
 >>> Thanks for your comment!
 >>>
 >>>
 >>>> The code changes are fine, but, the comment in
 >> pgxc_collect_RTE_walker()
 >>>> seems to specific. I think it should read, "create a copy of query's
 >>> range
 >>>> table, so that it can be linked with other RTEs in the collector's
 >>> context."
 >>>>
 >>> I've revised the comment as you suggested.
 >>>
 >>>
 >>>> In the test file, isn't there any way, you can add the offending
 >> statent
 >>>> without any wrapper function test_execute_direct_all_xc_****nodes()?
 >> We
 >>
 >>> should
 >>>> minimize the test-code to use only the minimal set of features. Since
 >>> this
 >>>> statement no more gets into infinite loop, please do not use the
 >>> statement
 >>>> timeout.
 >>>
 >>> I've gotten rid of using statement timeout.
 >>> But I couldn't come up any idea to testing more simply,
 >>> so the function is remaining as same as before patch.
 >>> Any ideas?
 >>>
 >>> Here is the new patch.
 >>> (against dec40008b3d689911566514614c511****1c0a61327d)
 >>
 >>>
 >>>
 >>> regards,
 >>> -------------
 >>> NTT Software Corporation
 >>> Tomonari Katsumata
 >>>
 >>>
 >>
 >>
 >>
 >>
 >> 
------------------------------------------------------------------------------
 >> October Webinars: Code for Performance
 >> Free Intel webinars can help you accelerate application performance.
 >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
 >> from
 >> the latest Intel processors and coprocessors. See abstracts and 
register >
 >> 
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
 >> _______________________________________________
 >> Postgres-xc-developers mailing list
 >> Pos...@li...
 >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
 >>
 >>
 >
 >

Re: [Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Ashutosh B. <ash...@en...>]

Hi Tomonary,
The testcase you have written uses a shell script. This is not the standard
way to add testcases. Can you please see if the testcase can be added
directly as SQL? Please send a revised patch ASAP, so that it can be
committed. I would have liked to see more testcases for EXEC DIRECT, but
since it's a debug only feature (not for production), it gets lesser
priority in testing and thus we do not see testcases for it.


On Thu, Sep 26, 2013 at 8:26 AM, Tomonari Katsumata <
kat...@po...> wrote:

> Hi,
>
> Does anyone check these patches?
> If there are no comments, please commit it.
>
> regards,
>
> ----------------------
> NTT Software Corporation
> Tomonari Katsumata
>
> -------- Original Message --------
> Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
> Date:     Thu, 5 Sep 2013 21:34:32 +0900
> From:     Tomonari Katsumata <t.k...@gm...>
> To:     Tomonari Katsumata <katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
> >
> CC:     Ashutosh Bapat <ashutosh.bapat@enterprisedb.**com<ash...@en...>>,
> pgxc-hackers mailing list <postgres-xc-developers@lists.**sourceforge.net<pos...@li...>
> >
>
>
> Hi,
>
> I'm sorry for long long leaving this thread alone.
> I've devided the patch to source part and regression part.
> (against f8e0f994bff1faefa9b1fc2f6e584a**1d3de70241)
>
> I did not change any process on source part.
> (revised_fqs_r4.patch)
>
> I added a script into src/test/regress/sql directory for
> changing regression part.
> (add_test_xc_misc.patch and execute_direct_regress.sh)
> When regression test runs, the script is executed instead of
> user defined function.
>
> Please check and commit it.
>
> regards,
> ------------------------
> NTT Software Corporation
> Tomonari Katsumata
>
>
> 2013/8/14 Tomonari Katsumata <katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
> >
>
> > Hi Ashtosh,
> >
> > Thanks for your comment!
> >
> >
> > > The code changes are fine, but, the comment in
> pgxc_collect_RTE_walker()
> > > seems to specific. I think it should read, "create a copy of query's
> > range
> > > table, so that it can be linked with other RTEs in the collector's
> > context."
> > >
> > I've revised the comment as you suggested.
> >
> >
> > > In the test file, isn't there any way, you can add the offending
> statent
> > > without any wrapper function test_execute_direct_all_xc_****nodes()?
> We
>
> > should
> > > minimize the test-code to use only the minimal set of features. Since
> > this
> > > statement no more gets into infinite loop, please do not use the
> > statement
> > > timeout.
> >
> > I've gotten rid of using statement timeout.
> > But I couldn't come up any idea to testing more simply,
> > so the function is remaining as same as before patch.
> > Any ideas?
> >
> > Here is the new patch.
> > (against dec40008b3d689911566514614c511****1c0a61327d)
>
> >
> >
> > regards,
> > -------------
> > NTT Software Corporation
> > Tomonari Katsumata
> >
> >
>
>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: [Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Koichi S. <koi...@gm...>]

This patch has been in process long and it seems that Tomonari provided
sufficient materials.

I'd like to know if there's any more issues on his work.

Regards;

---
Koichi Suzuki


2013/9/26 Tomonari Katsumata <kat...@po...>

> Hi,
>
> Does anyone check these patches?
> If there are no comments, please commit it.
>
> regards,
>
> ----------------------
> NTT Software Corporation
> Tomonari Katsumata
>
> -------- Original Message --------
> Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
> Date:     Thu, 5 Sep 2013 21:34:32 +0900
> From:     Tomonari Katsumata <t.k...@gm...>
> To:     Tomonari Katsumata <katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
> >
> CC:     Ashutosh Bapat <ashutosh.bapat@enterprisedb.**com<ash...@en...>>,
> pgxc-hackers mailing list <postgres-xc-developers@lists.**sourceforge.net<pos...@li...>
> >
>
>
> Hi,
>
> I'm sorry for long long leaving this thread alone.
> I've devided the patch to source part and regression part.
> (against f8e0f994bff1faefa9b1fc2f6e584a**1d3de70241)
>
> I did not change any process on source part.
> (revised_fqs_r4.patch)
>
> I added a script into src/test/regress/sql directory for
> changing regression part.
> (add_test_xc_misc.patch and execute_direct_regress.sh)
> When regression test runs, the script is executed instead of
> user defined function.
>
> Please check and commit it.
>
> regards,
> ------------------------
> NTT Software Corporation
> Tomonari Katsumata
>
>
> 2013/8/14 Tomonari Katsumata <katsumata.tomonari@po.ntts.**co.jp<kat...@po...>
> >
>
> > Hi Ashtosh,
> >
> > Thanks for your comment!
> >
> >
> > > The code changes are fine, but, the comment in
> pgxc_collect_RTE_walker()
> > > seems to specific. I think it should read, "create a copy of query's
> > range
> > > table, so that it can be linked with other RTEs in the collector's
> > context."
> > >
> > I've revised the comment as you suggested.
> >
> >
> > > In the test file, isn't there any way, you can add the offending
> statent
> > > without any wrapper function test_execute_direct_all_xc_****nodes()?
> We
>
> > should
> > > minimize the test-code to use only the minimal set of features. Since
> > this
> > > statement no more gets into infinite loop, please do not use the
> > statement
> > > timeout.
> >
> > I've gotten rid of using statement timeout.
> > But I couldn't come up any idea to testing more simply,
> > so the function is remaining as same as before patch.
> > Any ideas?
> >
> > Here is the new patch.
> > (against dec40008b3d689911566514614c511****1c0a61327d)
>
> >
> >
> > regards,
> > -------------
> > NTT Software Corporation
> > Tomonari Katsumata
> >
> >
>
>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

[Postgres-xc-developers] Fwd: Re: Query Planning bug ?

[Tomonari K. <kat...@po...>]

Hi,

Does anyone check these patches?
If there are no comments, please commit it.

regards,
----------------------
NTT Software Corporation
Tomonari Katsumata

-------- Original Message --------
Subject:     Re: [Postgres-xc-developers] Query Planning bug ?
Date:     Thu, 5 Sep 2013 21:34:32 +0900
From:     Tomonari Katsumata <t.k...@gm...>
To:     Tomonari Katsumata <kat...@po...>
CC:     Ashutosh Bapat <ash...@en...>, pgxc-hackers 
mailing list <pos...@li...>


Hi,

I'm sorry for long long leaving this thread alone.
I've devided the patch to source part and regression part.
(against f8e0f994bff1faefa9b1fc2f6e584a1d3de70241)

I did not change any process on source part.
(revised_fqs_r4.patch)

I added a script into src/test/regress/sql directory for
changing regression part.
(add_test_xc_misc.patch and execute_direct_regress.sh)
When regression test runs, the script is executed instead of
user defined function.

Please check and commit it.

regards,
------------------------
NTT Software Corporation
Tomonari Katsumata


2013/8/14 Tomonari Katsumata <kat...@po...>

 > Hi Ashtosh,
 >
 > Thanks for your comment!
 >
 >
 > > The code changes are fine, but, the comment in 
pgxc_collect_RTE_walker()
 > > seems to specific. I think it should read, "create a copy of query's
 > range
 > > table, so that it can be linked with other RTEs in the collector's
 > context."
 > >
 > I've revised the comment as you suggested.
 >
 >
 > > In the test file, isn't there any way, you can add the offending 
statent
 > > without any wrapper function test_execute_direct_all_xc_**nodes()? We
 > should
 > > minimize the test-code to use only the minimal set of features. Since
 > this
 > > statement no more gets into infinite loop, please do not use the
 > statement
 > > timeout.
 >
 > I've gotten rid of using statement timeout.
 > But I couldn't come up any idea to testing more simply,
 > so the function is remaining as same as before patch.
 > Any ideas?
 >
 > Here is the new patch.
 > (against dec40008b3d689911566514614c511**1c0a61327d)
 >
 >
 > regards,
 > -------------
 > NTT Software Corporation
 > Tomonari Katsumata
 >
 >

Re: [Postgres-xc-developers] Issues with GTM Standby Syncup

[Koichi S. <koi...@gm...>]

Hi, Nikhil.

Sorry for the late response.   I appreciate for this report.   Such
corner-case report is what I expected.   The patch looks reasonable but
please let me take a close look.

Best;

---
Koichi Suzuki


2013/9/16 Nikhil Sontakke <ni...@st...>

> Hi,
>
> There's an issue with GTM Standby syncup when it's joining a busy GTM
> server. This issue will not occur if the GTM has just been started and the
> GTM standby follows immediately when there are no backends asking for GXIDs.
>
> When a standby connects and asks for the GXID list, the GTM goes through
> the gt_transactions_array and sends only those structures which are in use
> (gti_in_use == true). So it's possible that slots
>
> 1, 2, 3 are in use. Then 4 and 5 might be unused and 6 might be used.
>
> So, GTM will send 4 such structures back to the Standby. Now the Standby
> naively assigns these structures to slots 1, 2, 3 and 4 (see
> gtm_standby_restore_gxid()). This messes up the assignment of handles.
>
> The fix is obviously to assign the structure to the specific gxi_handle
> index that has been retrieved from the GTM.
>
> PFA, patch which applies against master, rel11 and rel10.
>
> Regards,
> Nikhils
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>
>
> ------------------------------------------------------------------------------
> LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
> 1,500+ hours of tutorials including VisualStudio 2012, Windows 8,
> SharePoint
> 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack
> includes
> Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13.
> http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Re: [Postgres-xc-developers] GTM standby issue: Cannot assign the transaction, already in use

[Koichi S. <koi...@gm...>]

I've got the issue.   Yes, this is what I was concerning about and I
appreciate for finding the issue and possible cause.

How much you're in a hurry?   I'm now in Chicago for Postgres Open and
might not have a time to look into it until I finish my presentation
tomorrow.

Regards;

---
Koichi Suzuki


2013/9/16 Nikhil Sontakke <ni...@st...>

> Hi,
>
> One of our customers reported the following issue on the Standby:
>
> 1:140638715778816:2013-09-13 07:08:11.351 EDT -ERROR:  GTM_TransactionInfo
> already in use.  Cannot assign the transaction: handle (3745).
>
> I investigated this and thought I had fixed this with the patch that I
> have provided on the thread with $SUBJECT (Issues with GTM Standby Syncup)
>
> That patch reduced the number of these messages, but yet, some still found
> their way into the Standby logs.
>
> What's happening is that GTM Standby as part of the initial syncup with
> the GTM, sends gtm_standby_end_backup() after which the main GTM activates
> all the threads which had been paused earlier for the syncup.
>
> In the time between Standby calls the above function and actually starts
> listening for incoming data, some packets from the GTM are lost. Remember
> this will be the case when the GTM is already very highly loaded and lots
> of threads are running.
>
> If these packets contain commit info for those GXIDs that were added in
> the initial syncup, then they are lost and as a result these slots do not
> get cleaned up at the standby causing the above error messages.
>
> I think the best fix would be to delay and send gtm_standby_end_backup()
> inside the Serverloop, JUST before the Standby is going to do the select()
> call. Am trying a patch on similar lines but getting some issues, will get
> back with a patch soon.
>
> Regards,
> Nikhils
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>
>
> ------------------------------------------------------------------------------
> LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
> 1,500+ hours of tutorials including VisualStudio 2012, Windows 8,
> SharePoint
> 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack
> includes
> Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13.
> http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

[Postgres-xc-developers] GTM standby issue: Cannot assign the transaction, already in use

[Nikhil S. <ni...@st...>]

Hi,

One of our customers reported the following issue on the Standby:

1:140638715778816:2013-09-13 07:08:11.351 EDT -ERROR:  GTM_TransactionInfo
already in use.  Cannot assign the transaction: handle (3745).

I investigated this and thought I had fixed this with the patch that I have
provided on the thread with $SUBJECT (Issues with GTM Standby Syncup)

That patch reduced the number of these messages, but yet, some still found
their way into the Standby logs.

What's happening is that GTM Standby as part of the initial syncup with the
GTM, sends gtm_standby_end_backup() after which the main GTM activates all
the threads which had been paused earlier for the syncup.

In the time between Standby calls the above function and actually starts
listening for incoming data, some packets from the GTM are lost. Remember
this will be the case when the GTM is already very highly loaded and lots
of threads are running.

If these packets contain commit info for those GXIDs that were added in the
initial syncup, then they are lost and as a result these slots do not get
cleaned up at the standby causing the above error messages.

I think the best fix would be to delay and send gtm_standby_end_backup()
inside the Serverloop, JUST before the Standby is going to do the select()
call. Am trying a patch on similar lines but getting some issues, will get
back with a patch soon.

Regards,
Nikhils
-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-developers] hash by UUID

[Ashutosh B. <ash...@en...>]

On Tue, Sep 10, 2013 at 5:37 PM, Mason Sharp <ma...@st...> wrote:

>
>
>
> On Tue, Sep 10, 2013 at 4:20 AM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Unrelated to your patch, I think, we should now change the way we are
>> handling hash distribution. Instead of having switch cases to get the hash
>> functions, to decide whether a certain column is distributable, we should
>> really be looking up the pg_type catalogs. This has following advantages
>> 1. hashing for distribution would be inline with rest of the hashing
>> (hash joins for example), in the server
>> 2. We don't need to keep adding datatypes to switch cases. Any data type
>> with a hash function would be hash-distributable. Only thing is we
>> shouldn't allow the hash function of a data type to be changed if there is
>> a distributed table based on this hash function.
>> 3. Tables can be distributed by user defined types which have hash
>> functions.
>>
>> This was serious flaw in the previous design, which needs to be corrected
>> before we can add new types to distribution.
>>
>
> I would put other things at a higher priority right now to help increase
> the user base rather than redesigning this.  Some past reworking of
> portions of the code broke things or disabled optimizations that were
> previously working that took a long time before they were working again, so
> I think we should be careful before undertaking changes.
>

The more code we add over badly designed code, we will have more of those
instances and in later versions of Postgres-XC, which means harder it
becomes to fix things.


> In particular, if newly proposed changes increase the diff and complexity
> for merging with vanilla PostgreSQL, we should be cautious.
>
>
This change will be only in the XC specific code, so there won't be
conflicts with PG merge.


> Anyway, this is needed by a StormDB support customer who is using
> Postgres-XC (not StormDB). We are hoping to keep them as close to the
> REL1_1_STABLE branch as possible.  For 1.0, we have had to maintain a
> specially patched version for customers to ensure a stable version and
> still merge in important bug fixes.
>
> Regards,
>
> Mason
>
>
>
>> Same would, then be applicable for modulo distribution where existence of
>> % operator with integer input and integer output can be used.
>>
>>
>
>
>
>
>>
>> On Tue, Sep 10, 2013 at 1:00 PM, Andrei Martsinchyk <
>> and...@gm...> wrote:
>>
>>>  Hello,
>>>
>>> PostgresXC does not support hash distribution by UUID data type.
>>> Attached a small patch that enables the feature.
>>>
>>> --
>>> Andrei Martsinchyk
>>>
>>> StormDB - http://www.stormdb.com
>>> The Database Cloud
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> How ServiceNow helps IT people transform IT departments:
>>> 1. Consolidate legacy IT systems to a single system of record for IT
>>> 2. Standardize and globalize service processes across IT
>>> 3. Implement zero-touch automation to replace manual, redundant tasks
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EnterpriseDB Corporation
>> The Postgres Database Company
>>
>>
>> ------------------------------------------------------------------------------
>> How ServiceNow helps IT people transform IT departments:
>> 1. Consolidate legacy IT systems to a single system of record for IT
>> 2. Standardize and globalize service processes across IT
>> 3. Implement zero-touch automation to replace manual, redundant tasks
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Mason Sharp
>
>
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Services
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: [Postgres-xc-developers] hash by UUID

[Mason S. <ma...@st...>]

On Tue, Sep 10, 2013 at 4:20 AM, Ashutosh Bapat <
ash...@en...> wrote:

> Unrelated to your patch, I think, we should now change the way we are
> handling hash distribution. Instead of having switch cases to get the hash
> functions, to decide whether a certain column is distributable, we should
> really be looking up the pg_type catalogs. This has following advantages
> 1. hashing for distribution would be inline with rest of the hashing (hash
> joins for example), in the server
> 2. We don't need to keep adding datatypes to switch cases. Any data type
> with a hash function would be hash-distributable. Only thing is we
> shouldn't allow the hash function of a data type to be changed if there is
> a distributed table based on this hash function.
> 3. Tables can be distributed by user defined types which have hash
> functions.
>
> This was serious flaw in the previous design, which needs to be corrected
> before we can add new types to distribution.
>

I would put other things at a higher priority right now to help increase
the user base rather than redesigning this.  Some past reworking of
portions of the code broke things or disabled optimizations that were
previously working that took a long time before they were working again, so
I think we should be careful before undertaking changes. In particular, if
newly proposed changes increase the diff and complexity for merging with
vanilla PostgreSQL, we should be cautious.

Anyway, this is needed by a StormDB support customer who is using
Postgres-XC (not StormDB). We are hoping to keep them as close to the
REL1_1_STABLE branch as possible.  For 1.0, we have had to maintain a
specially patched version for customers to ensure a stable version and
still merge in important bug fixes.

Regards,

Mason



> Same would, then be applicable for modulo distribution where existence of
> % operator with integer input and integer output can be used.
>
>




>
> On Tue, Sep 10, 2013 at 1:00 PM, Andrei Martsinchyk <
> and...@gm...> wrote:
>
>>  Hello,
>>
>> PostgresXC does not support hash distribution by UUID data type.
>> Attached a small patch that enables the feature.
>>
>> --
>> Andrei Martsinchyk
>>
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
>>
>>
>> ------------------------------------------------------------------------------
>> How ServiceNow helps IT people transform IT departments:
>> 1. Consolidate legacy IT systems to a single system of record for IT
>> 2. Standardize and globalize service processes across IT
>> 3. Implement zero-touch automation to replace manual, redundant tasks
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Mason Sharp

StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Services