Menu
▾
▴
postgres-xc-developers — Postgres-XC hackers and developers
You can subscribe to this list here.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
(10) |
May
(17) |
Jun
(3) |
Jul
|
Aug
|
Sep
(8) |
Oct
(18) |
Nov
(51) |
Dec
(74) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(47) |
Feb
(44) |
Mar
(44) |
Apr
(102) |
May
(35) |
Jun
(25) |
Jul
(56) |
Aug
(69) |
Sep
(32) |
Oct
(37) |
Nov
(31) |
Dec
(16) |
| 2012 |
Jan
(34) |
Feb
(127) |
Mar
(218) |
Apr
(252) |
May
(80) |
Jun
(137) |
Jul
(205) |
Aug
(159) |
Sep
(35) |
Oct
(50) |
Nov
(82) |
Dec
(52) |
| 2013 |
Jan
(107) |
Feb
(159) |
Mar
(118) |
Apr
(163) |
May
(151) |
Jun
(89) |
Jul
(106) |
Aug
(177) |
Sep
(49) |
Oct
(63) |
Nov
(46) |
Dec
(7) |
| 2014 |
Jan
(65) |
Feb
(128) |
Mar
(40) |
Apr
(11) |
May
(4) |
Jun
(8) |
Jul
(16) |
Aug
(11) |
Sep
(4) |
Oct
(1) |
Nov
(5) |
Dec
(16) |
| 2015 |
Jan
(5) |
Feb
|
Mar
(2) |
Apr
(5) |
May
(4) |
Jun
(12) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
| 2019 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Ashutosh B. <ash...@en...> - 2013-05-28 05:47:50
|
No, not for this random issue. Adding alternate expected outputs adds maintenance burdens. So, we have to think ten times before adding an alternate output file. On Tue, May 28, 2013 at 12:48 AM, 鈴木 幸市 <ko...@in...> wrote: > Can we have alternate files to accept this as "success"? > --- > Koichi Suzuki > > > > On 2013/05/23, at 13:37, Abbas Butt <abb...@en...> wrote: > > Its a random failure occurring due to > ! ERROR: failed to send COMMIT command to node > We can neglect it for now. > > > On Thu, May 23, 2013 at 1:17 AM, Ahsan Hadi <ahs...@en...>wrote: > >> Abbas, >> Can you take a look, the plpgsql test case is failing again in BF >> regression from couple of days ago... >> >> >> On Tue, May 7, 2013 at 6:43 AM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Ok, this is good to go then. >>> >>> >>> On Tue, Apr 30, 2013 at 6:00 PM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> >>>> >>>> On Tue, Apr 30, 2013 at 5:04 PM, Ashutosh Bapat < >>>> ash...@en...> wrote: >>>> >>>>> Hi Abbas, >>>>> This change looks confusing. The statements where you have added order >>>>> by clause do not correspond to the statements where you have changed the >>>>> output. Can you please explain this? >>>>> >>>> >>>> The output of the test cases are changed because there was an error in >>>> the expected output files of these test cases. These outputs have to be >>>> modified (and the modified output is correct) because of your check in >>>> 2608af3d5288607e8245fa10e74e7307bb1bb23d. >>>> >>>> The order by clause is added in test cases where the expected output >>>> was already in order, but results were coming out to be in different order >>>> depending on node configuration. >>>> >>>> >>>>> >>>>> Also, whenever you change .sql and need to change the .out for that, >>>>> please change all the alt. expected output files. >>>>> >>>> >>>> Sure >>>> >>>> >>>>> >>>>> >>>>> On Thu, Apr 25, 2013 at 4:58 PM, Abbas Butt < >>>>> abb...@en...> wrote: >>>>> >>>>>> Hi, >>>>>> Attached please find patch to fix test case plpgsql. The test was >>>>>> failing for some order by issues, Also there were some errors in expected >>>>>> output. >>>>>> >>>>>> -- >>>>>> *Abbas* >>>>>> Architect >>>>>> >>>>>> Ph: 92.334.5100153 >>>>>> Skype ID: gabbasb >>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>> * >>>>>> Follow us on Twitter* >>>>>> @EnterpriseDB >>>>>> >>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Try New Relic Now & We'll Send You this Cool Shirt >>>>>> New Relic is the only SaaS-based application performance monitoring >>>>>> service >>>>>> that delivers powerful full stack analytics. Optimize and monitor your >>>>>> browser, app, & servers with just a few lines of code. Try New Relic >>>>>> and get this awesome Nerd Life shirt! >>>>>> http://p.sf.net/sfu/newrelic_d2d_apr >>>>>> _______________________________________________ >>>>>> Postgres-xc-developers mailing list >>>>>> Pos...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Wishes, >>>>> Ashutosh Bapat >>>>> EntepriseDB Corporation >>>>> The Postgres Database Company >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >>> >>> ------------------------------------------------------------------------------ >>> Learn Graph Databases - Download FREE O'Reilly Book >>> "Graph Databases" is the definitive new guide to graph databases and >>> their applications. This 200-page book is written by three acclaimed >>> leaders in the field. The early access version is available now. >>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>> >>> _______________________________________________ >>> Postgres-xc-developers mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>> >>> >> >> >> -- >> Ahsan Hadi >> Snr Director Product Development >> EnterpriseDB Corporation >> The Enterprise Postgres Company >> >> Phone: +92-51-8358874 >> Mobile: +92-333-5162114 >> >> Website: www.enterprisedb.com >> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >> Follow us on Twitter: http://www.twitter.com/enterprisedb >> >> This e-mail message (and any attachment) is intended for the use of the >> individual or entity to whom it is addressed. This message contains >> information from EnterpriseDB Corporation that may be privileged, >> confidential, or exempt from disclosure under applicable law. If you are >> not the intended recipient or authorized to receive this for the intended >> recipient, any use, dissemination, distribution, retention, archiving, or >> copying of this communication is strictly prohibited. If you have received >> this e-mail in error, please notify the sender immediately by reply e-mail >> and delete this message. >> > > > > -- > -- > *Abbas* > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> > * > Follow us on Twitter* > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring > service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! > http://p.sf.net/sfu/newrelic_d2d_may_______________________________________________ > > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Koichi S. <koi...@gm...> - 2013-05-28 05:10:12
|
I have a question on the patch. Why we close GTM connection here? Why shouldn't we keep the connection open? Regards; ---------- Koichi Suzuki 2013/5/28 Koichi Suzuki <koi...@gm...> > Thank you Andrei for the patch. I took a glance at it and will review it > before commit. > > Best; > > ---------- > Koichi Suzuki > > > 2013/5/27 Andrei Martsinchyk <and...@gm...> > >> We noticed that transaction handles are not released after direct >> connections to datanodes, if they are connecting to GTM through GTM proxy. >> So if Datanode is periodically connected directly (ex. for monitoring) >> GTM eventually starts throwing error "Max transaction limit reached". >> Please find fix attached. >> >> -- >> Andrei Martsinchyk >> >> StormDB - http://www.stormdb.com >> The Database Cloud >> >> >> >> ------------------------------------------------------------------------------ >> Try New Relic Now & We'll Send You this Cool Shirt >> New Relic is the only SaaS-based application performance monitoring >> service >> that delivers powerful full stack analytics. Optimize and monitor your >> browser, app, & servers with just a few lines of code. Try New Relic >> and get this awesome Nerd Life shirt! >> http://p.sf.net/sfu/newrelic_d2d_may >> _______________________________________________ >> Postgres-xc-developers mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> > |
|
From: 鈴木 幸市 <ko...@in...> - 2013-05-28 04:48:18
|
Can we have alternate files to accept this as "success"? --- Koichi Suzuki On 2013/05/23, at 13:37, Abbas Butt <abb...@en...> wrote: > Its a random failure occurring due to > ! ERROR: failed to send COMMIT command to node > We can neglect it for now. > > > On Thu, May 23, 2013 at 1:17 AM, Ahsan Hadi <ahs...@en...> wrote: > Abbas, > Can you take a look, the plpgsql test case is failing again in BF regression from couple of days ago... > > > On Tue, May 7, 2013 at 6:43 AM, Ashutosh Bapat <ash...@en...> wrote: > Ok, this is good to go then. > > > On Tue, Apr 30, 2013 at 6:00 PM, Abbas Butt <abb...@en...> wrote: > > > On Tue, Apr 30, 2013 at 5:04 PM, Ashutosh Bapat <ash...@en...> wrote: > Hi Abbas, > This change looks confusing. The statements where you have added order by clause do not correspond to the statements where you have changed the output. Can you please explain this? > > The output of the test cases are changed because there was an error in the expected output files of these test cases. These outputs have to be modified (and the modified output is correct) because of your check in 2608af3d5288607e8245fa10e74e7307bb1bb23d. > > The order by clause is added in test cases where the expected output was already in order, but results were coming out to be in different order depending on node configuration. > > > Also, whenever you change .sql and need to change the .out for that, please change all the alt. expected output files. > > Sure > > > > On Thu, Apr 25, 2013 at 4:58 PM, Abbas Butt <abb...@en...> wrote: > Hi, > Attached please find patch to fix test case plpgsql. The test was failing for some order by issues, Also there were some errors in expected output. > > -- > Abbas > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.com > > Follow us on Twitter > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers and more > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > > > -- > -- > Abbas > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.com > > Follow us on Twitter > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers and more > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and > their applications. This 200-page book is written by three acclaimed > leaders in the field. The early access version is available now. > Download your free book today! http://p.sf.net/sfu/neotech_d2d_may > > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > > > > -- > Ahsan Hadi > Snr Director Product Development > EnterpriseDB Corporation > The Enterprise Postgres Company > > Phone: +92-51-8358874 > Mobile: +92-333-5162114 > > Website: www.enterprisedb.com > EnterpriseDB Blog: http://blogs.enterprisedb.com/ > Follow us on Twitter: http://www.twitter.com/enterprisedb > > This e-mail message (and any attachment) is intended for the use of the individual or entity to whom it is addressed. This message contains information from EnterpriseDB Corporation that may be privileged, confidential, or exempt from disclosure under applicable law. If you are not the intended recipient or authorized to receive this for the intended recipient, any use, dissemination, distribution, retention, archiving, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and delete this message. > > > > -- > -- > Abbas > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.com > > Follow us on Twitter > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers and more > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may_______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers |
|
From: Koichi S. <koi...@gm...> - 2013-05-27 22:41:31
|
Thank you Andrei for the patch. I took a glance at it and will review it before commit. Best; ---------- Koichi Suzuki 2013/5/27 Andrei Martsinchyk <and...@gm...> > We noticed that transaction handles are not released after direct > connections to datanodes, if they are connecting to GTM through GTM proxy. > So if Datanode is periodically connected directly (ex. for monitoring) > GTM eventually starts throwing error "Max transaction limit reached". > Please find fix attached. > > -- > Andrei Martsinchyk > > StormDB - http://www.stormdb.com > The Database Cloud > > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > |
|
From: Abbas B. <abb...@en...> - 2013-05-27 02:53:20
|
On Fri, May 24, 2013 at 7:54 AM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Thu, May 23, 2013 at 10:23 PM, Ashutosh Bapat < > ash...@en...> wrote: > >> >> >> >> On Thu, May 23, 2013 at 10:19 PM, Abbas Butt <abb...@en... >> > wrote: >> >>> >>> >>> On Fri, May 17, 2013 at 1:58 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> >>>> >>>> >>>> On Fri, May 17, 2013 at 2:23 PM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> >>>>> >>>>> On Thu, May 16, 2013 at 3:13 PM, Ashutosh Bapat < >>>>> ash...@en...> wrote: >>>>> >>>>>> Hi Abbas, >>>>>> I am also seeing a lot of changes in the expected output where the >>>>>> rows output have changed. What are these changes? >>>>>> >>>>> >>>>> These changes are a result of blocking partition column updates >>>>> >>>> >>>> Are those in sync with PG expected output? >>>> >>> >>> No, in PG the update does not fail, in XC it fails. >>> >>> >>>> Why did we change the original expected output in first place? >>>> >>> >>> Do you mean that the changes in expected output due to blocking of >>> partition column updates should only be done in alternate expected output >>> file? >>> >>> >> >> yes, of course. >> >> > > This response can be confusing. If you are talking about the changing of > table distribution, then that has to be changed everywhere. But, I do not > understand, why should we see those many changes. The original output file > must have preserved the correct output, right? > Unfortunately the results in original output were incorrect, especially for the cases where it was possible to update partition column using WITH syntax. The patch fixes two issues 1. Block partition column updates using WITH syntax 2. WITH query that updates a table in the main query and inserts a row in the same table in the WITH query Hence there are more changes in the output files. > > >> >>>> >>>>> and changing the distribution of tables to replication. >>>>> >>>>> >>>> >>>> That's acceptable. >>>> >>>> >>>>> >>>>>> >>>>>> On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat < >>>>>> ash...@en...> wrote: >>>>>> >>>>>>> Hi Abbas, >>>>>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is >>>>>>> it possible to traverse the Query in validate_part_col_updatable() >>>>>>> recursively to find UPDATE statements and apply partition column check? >>>>>>> That would cover all the possibilities, I guess. That also saves us much >>>>>>> effort in case we come to support distribution column updation. >>>>>>> >>>>>>> I think, we need a generic solution to solve this command id issue, >>>>>>> e.g. punching command id always and efficiently. But for now this suffices. >>>>>>> Please log a bug/feature and put it in 1.2 bucket. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>>>>> abb...@en...> wrote: >>>>>>> >>>>>>>> Adding developers mailing list. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>>>>> abb...@en...> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Attached please find a patch to fix test case with. >>>>>>>>> There were two issues making the test to fail. >>>>>>>>> 1. Updates to partition column were possible using syntax like >>>>>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>>>>> The patch blocks this syntax. >>>>>>>>> >>>>>>>>> 2. For a WITH query that updates a table in the main query and >>>>>>>>> inserts a row in the same table in the WITH query we need to >>>>>>>>> use >>>>>>>>> command ID communication to remote nodes in order to >>>>>>>>> maintain global data visibility. >>>>>>>>> For example >>>>>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>>>>> REPLICATION; >>>>>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id >>>>>>>>> AS newid) >>>>>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>>>>> The last query gets translated into the following >>>>>>>>> multi-statement >>>>>>>>> transaction on the primary datanode >>>>>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ >>>>>>>>> WRITE >>>>>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>>>>> (42,'new)' >>>>>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>>>>> (43,(0,1)] >>>>>>>>> (e) COMMIT TRANSACTION >>>>>>>>> The command id of the select in step (c), should be such that >>>>>>>>> it does not see the insert of step (b) >>>>>>>>> >>>>>>>>> Comments are welcome. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Abbas* >>>>>>>>> Architect >>>>>>>>> >>>>>>>>> Ph: 92.334.5100153 >>>>>>>>> Skype ID: gabbasb >>>>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>>>> * >>>>>>>>> Follow us on Twitter* >>>>>>>>> @EnterpriseDB >>>>>>>>> >>>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> *Abbas* >>>>>>>> Architect >>>>>>>> >>>>>>>> Ph: 92.334.5100153 >>>>>>>> Skype ID: gabbasb >>>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>>> * >>>>>>>> Follow us on Twitter* >>>>>>>> @EnterpriseDB >>>>>>>> >>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> AlienVault Unified Security Management (USM) platform delivers >>>>>>>> complete >>>>>>>> security visibility with the essential security capabilities. >>>>>>>> Easily and >>>>>>>> efficiently configure, manage, and operate all of your security >>>>>>>> controls >>>>>>>> from a single console and one unified framework. Download a free >>>>>>>> trial. >>>>>>>> http://p.sf.net/sfu/alienvault_d2d >>>>>>>> _______________________________________________ >>>>>>>> Postgres-xc-core mailing list >>>>>>>> Pos...@li... >>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Wishes, >>>>>>> Ashutosh Bapat >>>>>>> EntepriseDB Corporation >>>>>>> The Postgres Database Company >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Wishes, >>>>>> Ashutosh Bapat >>>>>> EntepriseDB Corporation >>>>>> The Postgres Database Company >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> AlienVault Unified Security Management (USM) platform delivers complete >>>> security visibility with the essential security capabilities. Easily and >>>> efficiently configure, manage, and operate all of your security controls >>>> from a single console and one unified framework. Download a free trial. >>>> http://p.sf.net/sfu/alienvault_d2d >>>> _______________________________________________ >>>> Postgres-xc-core mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>> >>>> >>> >>> >>> -- >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Abbas B. <abb...@en...> - 2013-05-27 02:18:34
|
On Fri, May 24, 2013 at 7:04 PM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Fri, May 24, 2013 at 9:01 AM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Fri, May 24, 2013 at 7:22 AM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> >>> >>> >>> On Thu, May 23, 2013 at 9:21 PM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> Hi, >>>> >>>> While working on test case plancache it was brought up as a review >>>> comment that solving bug id 3607975 should solve the problem of the test >>>> case. >>>> However there is some confusion in the statement of bug id 3607975. >>>> >>>> "When a user does and PREPARE and then EXECUTEs multiple times, the >>>> coordinator keeps on preparing and executing the query on datanode al >>>> times, as against preparing once and executing multiple times. This is >>>> because somehow the remote query is being prepared as an unnamed statement." >>>> >>>> Consider this test case >>>> >>>> A. create table abc(a int, b int); >>>> B. insert into abc values(11, 22); >>>> C. prepare p1 as select * from abc; >>>> D. execute p1; >>>> E. execute p1; >>>> F. execute p1; >>>> >>>> Here are the confusions >>>> >>>> 1. The coordinator never prepares on datanode in response to a prepare >>>> issued by a user. >>>> In fact step C does nothing on the datanodes. >>>> Step D simply sends "SELECT a, b FROM abc" to all datanodes. >>>> >>>> 2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic >>>> plan, >>>> and steps E and F use the already built generic plan. >>>> For details see function GetCachedPlan. >>>> This means that executing a prepared statement again and again does >>>> use cached plans >>>> and does not prepare again and again every time we issue an execute. >>>> >>>> >>> The problem is not here. The problem is in do_query() where somehow the >>> name of prepared statement gets wiped out and we keep on preparing unnamed >>> statements at the datanode. >>> >> >> We never prepare any named/unnamed statements on the datanode. I spent >> time looking at the code written in do_query and functions called from with >> in do_query to handle prepared statements but the code written in >> pgxc_start_command_on_connection to handle statements prepared on datanodes >> is dead as of now. It is never called during the complete regression run. >> The function ActivateDatanodeStatementOnNode is never called. The way >> prepared statements are being handled now is the same as I described >> earlier in the mail chain with the help of an example. >> The code that is dead was originally added by Mason through commit >> d6d2d3d925f571b0b58ff6b4f6504d88e96bb342, back in December 2010. This code >> has been changed a lot over the last two years. This commit does not >> contain any test cases so I am not sure how did it use to work back then. >> >> > > This code wasn't dead, when I worked on prepared statements. So, something > has gone wrong in-between. That's what we need to find out and fix. Not > preparing statements on the datanode is not good for performance either. > I was able to find the reason why the code was dead and the attached patch (WIP) fixes the problem. This would now ensure that statements are prepared on datanodes whenever required. However there is a problem in the way prepared statements are handled. The problem is that unless a prepared statement is executed it is never prepared on datanodes, hence changing the path before executing the statement gives us incorrect results. For Example create schema s1 create table abc (f1 int) distribute by replication; create schema s2 create table abc (f1 int) distribute by replication; insert into s1.abc values(123); insert into s2.abc values(456); set search_path = s2; prepare p1 as select f1 from abc; set search_path = s1; execute p1; The last execute results in 123, where as it should have resulted in 456. I can finalize the attached patch by fixing any regression issues that may result and that would fix 3607975 and improve performance however the above test case would still fail. > > >> >>> >>>> My conclusion is that the bug ID 3607975 is not reproducible. >>>> >>>> >>> Did you verify it under the debugger? If that would have been the case, >>> we would not have seen this problem if search_path changed in between steps >>> D and E. >>> >> >> If search path is changed between steps D & E, the problem occurs because >> when the remote query node is created, schema qualification is not added in >> the sql statement to be sent to the datanode, but changes in search path do >> get communicated to the datanode. The sql statement is built when execute >> is issued for the first time and is reused on subsequent executes. The >> datanode is totally unaware that the select that it just received is due to >> an execute of a prepared statement that was prepared when search path was >> some thing else. >> >> > Fixing the prepared statements the way I suggested, would fix the problem, > since the statement will get prepared at the datanode, with the same search > path settings, as it would on the coordinator. > > >> >> >>> >>> >>>> Comments are welcome. >>>> >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try New Relic Now & We'll Send You this Cool Shirt >>>> New Relic is the only SaaS-based application performance monitoring >>>> service >>>> that delivers powerful full stack analytics. Optimize and monitor your >>>> browser, app, & servers with just a few lines of code. Try New Relic >>>> and get this awesome Nerd Life shirt! >>>> http://p.sf.net/sfu/newrelic_d2d_may >>>> _______________________________________________ >>>> Postgres-xc-developers mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Ashutosh B. <ash...@en...> - 2013-05-24 14:04:49
|
On Fri, May 24, 2013 at 9:01 AM, Abbas Butt <abb...@en...>wrote: > > > On Fri, May 24, 2013 at 7:22 AM, Ashutosh Bapat < > ash...@en...> wrote: > >> >> >> >> On Thu, May 23, 2013 at 9:21 PM, Abbas Butt <abb...@en...>wrote: >> >>> Hi, >>> >>> While working on test case plancache it was brought up as a review >>> comment that solving bug id 3607975 should solve the problem of the test >>> case. >>> However there is some confusion in the statement of bug id 3607975. >>> >>> "When a user does and PREPARE and then EXECUTEs multiple times, the >>> coordinator keeps on preparing and executing the query on datanode al >>> times, as against preparing once and executing multiple times. This is >>> because somehow the remote query is being prepared as an unnamed statement." >>> >>> Consider this test case >>> >>> A. create table abc(a int, b int); >>> B. insert into abc values(11, 22); >>> C. prepare p1 as select * from abc; >>> D. execute p1; >>> E. execute p1; >>> F. execute p1; >>> >>> Here are the confusions >>> >>> 1. The coordinator never prepares on datanode in response to a prepare >>> issued by a user. >>> In fact step C does nothing on the datanodes. >>> Step D simply sends "SELECT a, b FROM abc" to all datanodes. >>> >>> 2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic >>> plan, >>> and steps E and F use the already built generic plan. >>> For details see function GetCachedPlan. >>> This means that executing a prepared statement again and again does >>> use cached plans >>> and does not prepare again and again every time we issue an execute. >>> >>> >> The problem is not here. The problem is in do_query() where somehow the >> name of prepared statement gets wiped out and we keep on preparing unnamed >> statements at the datanode. >> > > We never prepare any named/unnamed statements on the datanode. I spent > time looking at the code written in do_query and functions called from with > in do_query to handle prepared statements but the code written in > pgxc_start_command_on_connection to handle statements prepared on datanodes > is dead as of now. It is never called during the complete regression run. > The function ActivateDatanodeStatementOnNode is never called. The way > prepared statements are being handled now is the same as I described > earlier in the mail chain with the help of an example. > The code that is dead was originally added by Mason through commit > d6d2d3d925f571b0b58ff6b4f6504d88e96bb342, back in December 2010. This code > has been changed a lot over the last two years. This commit does not > contain any test cases so I am not sure how did it use to work back then. > > This code wasn't dead, when I worked on prepared statements. So, something has gone wrong in-between. That's what we need to find out and fix. Not preparing statements on the datanode is not good for performance either. > >> >>> My conclusion is that the bug ID 3607975 is not reproducible. >>> >>> >> Did you verify it under the debugger? If that would have been the case, >> we would not have seen this problem if search_path changed in between steps >> D and E. >> > > If search path is changed between steps D & E, the problem occurs because > when the remote query node is created, schema qualification is not added in > the sql statement to be sent to the datanode, but changes in search path do > get communicated to the datanode. The sql statement is built when execute > is issued for the first time and is reused on subsequent executes. The > datanode is totally unaware that the select that it just received is due to > an execute of a prepared statement that was prepared when search path was > some thing else. > > Fixing the prepared statements the way I suggested, would fix the problem, since the statement will get prepared at the datanode, with the same search path settings, as it would on the coordinator. > > >> >> >>> Comments are welcome. >>> >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >>> >>> ------------------------------------------------------------------------------ >>> Try New Relic Now & We'll Send You this Cool Shirt >>> New Relic is the only SaaS-based application performance monitoring >>> service >>> that delivers powerful full stack analytics. Optimize and monitor your >>> browser, app, & servers with just a few lines of code. Try New Relic >>> and get this awesome Nerd Life shirt! >>> http://p.sf.net/sfu/newrelic_d2d_may >>> _______________________________________________ >>> Postgres-xc-developers mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>> >>> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> > > > > -- > -- > *Abbas* > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> > * > Follow us on Twitter* > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Koichi S. <koi...@gm...> - 2013-05-24 13:30:53
|
I didn't realize this idea while listening to Robert's talk. Yes, it will be nice. I think we should test how it works. I understand this is a very good candidate for the problem. Regards; ---------- Koichi Suzuki 2013/5/23 Ashutosh Bapat <ash...@en...> > Hi All, > While I hear Robert's presentation on query planning go wrong, I heard > (first time) about MergeAppend plan. It looks like, we have something > better to do with the way we are pushing the ORDER BY clauses and doing > merge at the coordinator. Right now we are using Sort node and in ExecSort > we start merging the rows. Instead, I think we should be using MergeAppend > node. The only bad thing about this approach is MergeAppend node expects > separate plans for each run of the sorted data. To do this, we need to > replicate the RemoteQuery node as many times as there are nodes and put > them in the list to the MergeAppend plan. I think that's going to solve the > problem with materialisation (see mail thread "Using remote sorting for > merge-join") and thus improve performance. It also takes away the need to > have xc_node_id in the tupleslot of tuplestore. Obviously, simplifying the > code. > > Any comments? > > Created 3613815 > <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>and > assigned to me. > <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232> > > On Mon, Dec 3, 2012 at 3:44 AM, Ashutosh Bapat < > ash...@en...> wrote: > >> On Fri, Nov 30, 2012 at 9:37 AM, Michael Paquier >> <mic...@gm...> wrote: >> > Hi Ashutosh, >> > >> > Thanks for the patch. >> > I found for the time being the following problems: >> > 1) The patch has some whitespaces >> > /home/michael/download/xc_sort.patch:16: trailing whitespace. >> > >> > /home/michael/download/xc_sort.patch:34: trailing whitespace. >> > >> > /home/michael/download/xc_sort.patch:307: trailing whitespace. >> > return local_plan_left; >> > /home/michael/download/xc_sort.patch:356: trailing whitespace. >> > >> > /home/michael/download/xc_sort.patch:360: trailing whitespace. >> > >> >> Fixed. >> >> > error: patch failed: src/include/pgxc/execRemote.h:95 >> > 2) the patch does not apply cleanly even by using "patch -p1"-like >> commands >> >> When I sent the patch (more than a week before you replied) it was >> based on the HEAD, but then there were some commits which created the >> conflicts. The one attached should apply. >> >> > 3) No documentation about the new option enable_remotesort. >> > 4) enable_remotesort is not listed in postgresql.conf. >> > Please provide both documentation and reference in >> postgresql.conf.sample >> > for transparency of the code. As an open source project it is important >> to >> > classify all the options available, for both the user and the >> developers. >> >> Added although, I still don't approve of it myself. >> >> > 5) What is cruelly missing with this patch are numbers. By how much this >> > feature improves performance? >> > Simple tests like a ORDER BY on a table with millions of rows would be >> > enough, but as this is a performance feature, you should provide data >> about >> > how much those cases are improved. >> > 6) A complain that I say for each planner-related patch in this >> project.... >> > This patch has a size of 200KB, 65% being updates on the EXPLAIN >> queries of >> > the XC query planning regression tests. This makes the commit tree >> > unreadable and the evolution of the regression tests realy hard to >> follow >> > because they need to be modified each time a planner optimization is >> > introduced. We should move to a set of tests which is far more >> > code-modification proof! >> >> This is planner code, so it's going to affect the plans that it >> creates. In the test files we add EXPLAIN (with proper switches) to >> check 1. whether the plans are coming as expected at that stage of >> development, 2. Whether the clauses expected to be pushed down are >> being indeed pushed down 3. the queries to be sent to the datanodes >> are being crafted correctly. For every planner change, all the three >> things above change, for the operation being planned and hence the >> expected output changes are unavoidable. In fact, if you look at how >> the expected output changing you get a idea of how correctly/wrong the >> planner changes are made. Those diffs if go wrong, are great help to >> me in finding problems earlier. During merge as well, if those tests >> fail because of EXPLAIN outputs, that's a way to unwanted planner >> changes early enough and correct them. Sorry, for the huge change, but >> it's unavoidable. The testcases are intentionally prone to (planner) >> code modification. >> >> > >> > Regards, >> > >> > On Mon, Nov 26, 2012 at 8:42 PM, Ashutosh Bapat >> > <ash...@en...> wrote: >> >> >> >> Hi All, >> >> PFA patch which enables pushing down the ORDER BY clause or sorting to >> the >> >> Datanode/s through standard_planner(). >> >> >> >> This patch leverages the already existing merge sort mechanism for >> >> RemoteQuery but now outside RemoteQuery using Sort plans. >> >> >> >> For a Sort plan it checks if all the expressions on which the result >> needs >> >> to be sorted are shippable and that the underlying tree allows to push >> ORDER >> >> BY to the Datanodes. If so, it includes the ORDER BY clause in the >> query to >> >> be sent to the Datanodes. If the number of Datanodes which participate >> in >> >> RemoteQuery is not 1, it adds a covering Sort node to merge the >> results from >> >> the Datanodes, otherwise, there is no covering Sort plan. >> >> >> >> The merge sort mechanism treats every Datanode as a tape and prefetches >> >> the results from the Datanode corresponding to a dwindling tape. This >> is >> >> enabled by setting the node from where to pull the next row before >> calling >> >> ExecProcNode on RemoteQueryState. >> >> >> >> The same mechanism is being used for Sort plan node under Group or Agg >> >> nodes. >> >> >> >> The patch adds test xc_sort.sql for testing this functionality. It also >> >> removes the now useless members sort and tuplesortstate from >> RemoteQuery and >> >> RemoteQueryState resp. >> >> -- >> >> Best Wishes, >> >> Ashutosh Bapat >> >> EntepriseDB Corporation >> >> The Enterprise Postgres Company >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Monitor your physical, virtual and cloud infrastructure from a single >> >> web console. Get in-depth insight into apps, servers, databases, >> vmware, >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial. >> >> Pricing starts from $795 for 25 servers or applications! >> >> http://p.sf.net/sfu/zoho_dev2dev_nov >> >> _______________________________________________ >> >> Postgres-xc-developers mailing list >> >> Pos...@li... >> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> >> > >> > >> > >> > -- >> > Michael Paquier >> > http://michael.otacoo.com >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Enterprise Postgres Company >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > |
|
From: Abbas B. <abb...@en...> - 2013-05-24 13:01:51
|
On Fri, May 24, 2013 at 7:22 AM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Thu, May 23, 2013 at 9:21 PM, Abbas Butt <abb...@en...>wrote: > >> Hi, >> >> While working on test case plancache it was brought up as a review >> comment that solving bug id 3607975 should solve the problem of the test >> case. >> However there is some confusion in the statement of bug id 3607975. >> >> "When a user does and PREPARE and then EXECUTEs multiple times, the >> coordinator keeps on preparing and executing the query on datanode al >> times, as against preparing once and executing multiple times. This is >> because somehow the remote query is being prepared as an unnamed statement." >> >> Consider this test case >> >> A. create table abc(a int, b int); >> B. insert into abc values(11, 22); >> C. prepare p1 as select * from abc; >> D. execute p1; >> E. execute p1; >> F. execute p1; >> >> Here are the confusions >> >> 1. The coordinator never prepares on datanode in response to a prepare >> issued by a user. >> In fact step C does nothing on the datanodes. >> Step D simply sends "SELECT a, b FROM abc" to all datanodes. >> >> 2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic >> plan, >> and steps E and F use the already built generic plan. >> For details see function GetCachedPlan. >> This means that executing a prepared statement again and again does >> use cached plans >> and does not prepare again and again every time we issue an execute. >> >> > The problem is not here. The problem is in do_query() where somehow the > name of prepared statement gets wiped out and we keep on preparing unnamed > statements at the datanode. > We never prepare any named/unnamed statements on the datanode. I spent time looking at the code written in do_query and functions called from with in do_query to handle prepared statements but the code written in pgxc_start_command_on_connection to handle statements prepared on datanodes is dead as of now. It is never called during the complete regression run. The function ActivateDatanodeStatementOnNode is never called. The way prepared statements are being handled now is the same as I described earlier in the mail chain with the help of an example. The code that is dead was originally added by Mason through commit d6d2d3d925f571b0b58ff6b4f6504d88e96bb342, back in December 2010. This code has been changed a lot over the last two years. This commit does not contain any test cases so I am not sure how did it use to work back then. > > >> My conclusion is that the bug ID 3607975 is not reproducible. >> >> > Did you verify it under the debugger? If that would have been the case, we > would not have seen this problem if search_path changed in between steps D > and E. > If search path is changed between steps D & E, the problem occurs because when the remote query node is created, schema qualification is not added in the sql statement to be sent to the datanode, but changes in search path do get communicated to the datanode. The sql statement is built when execute is issued for the first time and is reused on subsequent executes. The datanode is totally unaware that the select that it just received is due to an execute of a prepared statement that was prepared when search path was some thing else. > > >> Comments are welcome. >> >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> >> >> ------------------------------------------------------------------------------ >> Try New Relic Now & We'll Send You this Cool Shirt >> New Relic is the only SaaS-based application performance monitoring >> service >> that delivers powerful full stack analytics. Optimize and monitor your >> browser, app, & servers with just a few lines of code. Try New Relic >> and get this awesome Nerd Life shirt! >> http://p.sf.net/sfu/newrelic_d2d_may >> _______________________________________________ >> Postgres-xc-developers mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Ashutosh B. <ash...@en...> - 2013-05-24 02:54:58
|
On Thu, May 23, 2013 at 10:23 PM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Thu, May 23, 2013 at 10:19 PM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Fri, May 17, 2013 at 1:58 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> >>> >>> >>> On Fri, May 17, 2013 at 2:23 PM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> >>>> >>>> On Thu, May 16, 2013 at 3:13 PM, Ashutosh Bapat < >>>> ash...@en...> wrote: >>>> >>>>> Hi Abbas, >>>>> I am also seeing a lot of changes in the expected output where the >>>>> rows output have changed. What are these changes? >>>>> >>>> >>>> These changes are a result of blocking partition column updates >>>> >>> >>> Are those in sync with PG expected output? >>> >> >> No, in PG the update does not fail, in XC it fails. >> >> >>> Why did we change the original expected output in first place? >>> >> >> Do you mean that the changes in expected output due to blocking of >> partition column updates should only be done in alternate expected output >> file? >> >> > > yes, of course. > > This response can be confusing. If you are talking about the changing of table distribution, then that has to be changed everywhere. But, I do not understand, why should we see those many changes. The original output file must have preserved the correct output, right? > >>> >>>> and changing the distribution of tables to replication. >>>> >>>> >>> >>> That's acceptable. >>> >>> >>>> >>>>> >>>>> On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat < >>>>> ash...@en...> wrote: >>>>> >>>>>> Hi Abbas, >>>>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is >>>>>> it possible to traverse the Query in validate_part_col_updatable() >>>>>> recursively to find UPDATE statements and apply partition column check? >>>>>> That would cover all the possibilities, I guess. That also saves us much >>>>>> effort in case we come to support distribution column updation. >>>>>> >>>>>> I think, we need a generic solution to solve this command id issue, >>>>>> e.g. punching command id always and efficiently. But for now this suffices. >>>>>> Please log a bug/feature and put it in 1.2 bucket. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>>>> abb...@en...> wrote: >>>>>> >>>>>>> Adding developers mailing list. >>>>>>> >>>>>>> >>>>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>>>> abb...@en...> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Attached please find a patch to fix test case with. >>>>>>>> There were two issues making the test to fail. >>>>>>>> 1. Updates to partition column were possible using syntax like >>>>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>>>> The patch blocks this syntax. >>>>>>>> >>>>>>>> 2. For a WITH query that updates a table in the main query and >>>>>>>> inserts a row in the same table in the WITH query we need to >>>>>>>> use >>>>>>>> command ID communication to remote nodes in order to >>>>>>>> maintain global data visibility. >>>>>>>> For example >>>>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>>>> REPLICATION; >>>>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>>>>> newid) >>>>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>>>> The last query gets translated into the following >>>>>>>> multi-statement >>>>>>>> transaction on the primary datanode >>>>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ >>>>>>>> WRITE >>>>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>>>> (42,'new)' >>>>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>>>> (43,(0,1)] >>>>>>>> (e) COMMIT TRANSACTION >>>>>>>> The command id of the select in step (c), should be such that >>>>>>>> it does not see the insert of step (b) >>>>>>>> >>>>>>>> Comments are welcome. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> -- >>>>>>>> *Abbas* >>>>>>>> Architect >>>>>>>> >>>>>>>> Ph: 92.334.5100153 >>>>>>>> Skype ID: gabbasb >>>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>>> * >>>>>>>> Follow us on Twitter* >>>>>>>> @EnterpriseDB >>>>>>>> >>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> *Abbas* >>>>>>> Architect >>>>>>> >>>>>>> Ph: 92.334.5100153 >>>>>>> Skype ID: gabbasb >>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>> * >>>>>>> Follow us on Twitter* >>>>>>> @EnterpriseDB >>>>>>> >>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> AlienVault Unified Security Management (USM) platform delivers >>>>>>> complete >>>>>>> security visibility with the essential security capabilities. Easily >>>>>>> and >>>>>>> efficiently configure, manage, and operate all of your security >>>>>>> controls >>>>>>> from a single console and one unified framework. Download a free >>>>>>> trial. >>>>>>> http://p.sf.net/sfu/alienvault_d2d >>>>>>> _______________________________________________ >>>>>>> Postgres-xc-core mailing list >>>>>>> Pos...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Wishes, >>>>>> Ashutosh Bapat >>>>>> EntepriseDB Corporation >>>>>> The Postgres Database Company >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Wishes, >>>>> Ashutosh Bapat >>>>> EntepriseDB Corporation >>>>> The Postgres Database Company >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >>> >>> ------------------------------------------------------------------------------ >>> AlienVault Unified Security Management (USM) platform delivers complete >>> security visibility with the essential security capabilities. Easily and >>> efficiently configure, manage, and operate all of your security controls >>> from a single console and one unified framework. Download a free trial. >>> http://p.sf.net/sfu/alienvault_d2d >>> _______________________________________________ >>> Postgres-xc-core mailing list >>> Pos...@li... >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>> >>> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Abbas B. <abb...@en...> - 2013-05-24 02:45:54
|
On Fri, May 17, 2013 at 2:26 PM, Ashutosh Bapat < ash...@en...> wrote: > This looks good. Are there other ways where we can have UPDATE statement > somewhere in the query tree list? > I checked the UPDATE syntax and skimmed through the test cases in regression, I did not find any other variation of UPDATE we need to take care of. > Do we need to worry about such cases. > > > On Fri, May 17, 2013 at 2:22 PM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Thu, May 16, 2013 at 2:25 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Hi Abbas, >>> Instead of fixing the first issue in pgxc_build_dml_statement(), is it >>> possible to traverse the Query in validate_part_col_updatable() recursively >>> to find UPDATE statements and apply partition column check? >>> >> >> Yes. I have attached that patch for your feedback. If you think its ok I >> can send the updated patch including the rest of the changes. >> >> >>> That would cover all the possibilities, I guess. That also saves us >>> much effort in case we come to support distribution column updation. >>> >>> I think, we need a generic solution to solve this command id issue, e.g. >>> punching command id always and efficiently. But for now this suffices. >>> Please log a bug/feature and put it in 1.2 bucket. >>> >> >> Done. >> (Artifact 3613498<https://sourceforge.net/tracker/?func=detail&aid=3613498&group_id=311227&atid=1310235> >> ) >> >>> >>> >>> >>> >>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> Adding developers mailing list. >>>> >>>> >>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> Hi, >>>>> Attached please find a patch to fix test case with. >>>>> There were two issues making the test to fail. >>>>> 1. Updates to partition column were possible using syntax like >>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>> The patch blocks this syntax. >>>>> >>>>> 2. For a WITH query that updates a table in the main query and >>>>> inserts a row in the same table in the WITH query we need to use >>>>> command ID communication to remote nodes in order to >>>>> maintain global data visibility. >>>>> For example >>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY REPLICATION; >>>>> INSERT INTO tab VALUES (1,'p1'); >>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>> newid) >>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>> The last query gets translated into the following multi-statement >>>>> transaction on the primary datanode >>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE >>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>> (42,'new)' >>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>> (43,(0,1)] >>>>> (e) COMMIT TRANSACTION >>>>> The command id of the select in step (c), should be such that >>>>> it does not see the insert of step (b) >>>>> >>>>> Comments are welcome. >>>>> >>>>> Regards >>>>> >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> AlienVault Unified Security Management (USM) platform delivers complete >>>> security visibility with the essential security capabilities. Easily and >>>> efficiently configure, manage, and operate all of your security controls >>>> from a single console and one unified framework. Download a free trial. >>>> http://p.sf.net/sfu/alienvault_d2d >>>> _______________________________________________ >>>> Postgres-xc-core mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Ashutosh B. <ash...@en...> - 2013-05-24 02:23:28
|
On Thu, May 23, 2013 at 10:19 PM, Abbas Butt <abb...@en...>wrote: > > > On Fri, May 17, 2013 at 1:58 PM, Ashutosh Bapat < > ash...@en...> wrote: > >> >> >> >> On Fri, May 17, 2013 at 2:23 PM, Abbas Butt <abb...@en...>wrote: >> >>> >>> >>> On Thu, May 16, 2013 at 3:13 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> Hi Abbas, >>>> I am also seeing a lot of changes in the expected output where the rows >>>> output have changed. What are these changes? >>>> >>> >>> These changes are a result of blocking partition column updates >>> >> >> Are those in sync with PG expected output? >> > > No, in PG the update does not fail, in XC it fails. > > >> Why did we change the original expected output in first place? >> > > Do you mean that the changes in expected output due to blocking of > partition column updates should only be done in alternate expected output > file? > > yes, of course. > >> >>> and changing the distribution of tables to replication. >>> >>> >> >> That's acceptable. >> >> >>> >>>> >>>> On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat < >>>> ash...@en...> wrote: >>>> >>>>> Hi Abbas, >>>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is it >>>>> possible to traverse the Query in validate_part_col_updatable() recursively >>>>> to find UPDATE statements and apply partition column check? That would >>>>> cover all the possibilities, I guess. That also saves us much effort in >>>>> case we come to support distribution column updation. >>>>> >>>>> I think, we need a generic solution to solve this command id issue, >>>>> e.g. punching command id always and efficiently. But for now this suffices. >>>>> Please log a bug/feature and put it in 1.2 bucket. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>>> abb...@en...> wrote: >>>>> >>>>>> Adding developers mailing list. >>>>>> >>>>>> >>>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>>> abb...@en...> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Attached please find a patch to fix test case with. >>>>>>> There were two issues making the test to fail. >>>>>>> 1. Updates to partition column were possible using syntax like >>>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>>> The patch blocks this syntax. >>>>>>> >>>>>>> 2. For a WITH query that updates a table in the main query and >>>>>>> inserts a row in the same table in the WITH query we need to use >>>>>>> command ID communication to remote nodes in order to >>>>>>> maintain global data visibility. >>>>>>> For example >>>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>>> REPLICATION; >>>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>>>> newid) >>>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>>> The last query gets translated into the following >>>>>>> multi-statement >>>>>>> transaction on the primary datanode >>>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE >>>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>>> (42,'new)' >>>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>>> (43,(0,1)] >>>>>>> (e) COMMIT TRANSACTION >>>>>>> The command id of the select in step (c), should be such that >>>>>>> it does not see the insert of step (b) >>>>>>> >>>>>>> Comments are welcome. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> -- >>>>>>> *Abbas* >>>>>>> Architect >>>>>>> >>>>>>> Ph: 92.334.5100153 >>>>>>> Skype ID: gabbasb >>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>> * >>>>>>> Follow us on Twitter* >>>>>>> @EnterpriseDB >>>>>>> >>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> *Abbas* >>>>>> Architect >>>>>> >>>>>> Ph: 92.334.5100153 >>>>>> Skype ID: gabbasb >>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>> * >>>>>> Follow us on Twitter* >>>>>> @EnterpriseDB >>>>>> >>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> AlienVault Unified Security Management (USM) platform delivers >>>>>> complete >>>>>> security visibility with the essential security capabilities. Easily >>>>>> and >>>>>> efficiently configure, manage, and operate all of your security >>>>>> controls >>>>>> from a single console and one unified framework. Download a free >>>>>> trial. >>>>>> http://p.sf.net/sfu/alienvault_d2d >>>>>> _______________________________________________ >>>>>> Postgres-xc-core mailing list >>>>>> Pos...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Wishes, >>>>> Ashutosh Bapat >>>>> EntepriseDB Corporation >>>>> The Postgres Database Company >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>> >>> >>> >>> -- >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> >> >> ------------------------------------------------------------------------------ >> AlienVault Unified Security Management (USM) platform delivers complete >> security visibility with the essential security capabilities. Easily and >> efficiently configure, manage, and operate all of your security controls >> from a single console and one unified framework. Download a free trial. >> http://p.sf.net/sfu/alienvault_d2d >> _______________________________________________ >> Postgres-xc-core mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >> >> > > > -- > -- > *Abbas* > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> > * > Follow us on Twitter* > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Ashutosh B. <ash...@en...> - 2013-05-24 02:22:30
|
On Thu, May 23, 2013 at 9:21 PM, Abbas Butt <abb...@en...>wrote: > Hi, > > While working on test case plancache it was brought up as a review comment > that solving bug id 3607975 should solve the problem of the test case. > However there is some confusion in the statement of bug id 3607975. > > "When a user does and PREPARE and then EXECUTEs multiple times, the > coordinator keeps on preparing and executing the query on datanode al > times, as against preparing once and executing multiple times. This is > because somehow the remote query is being prepared as an unnamed statement." > > Consider this test case > > A. create table abc(a int, b int); > B. insert into abc values(11, 22); > C. prepare p1 as select * from abc; > D. execute p1; > E. execute p1; > F. execute p1; > > Here are the confusions > > 1. The coordinator never prepares on datanode in response to a prepare > issued by a user. > In fact step C does nothing on the datanodes. > Step D simply sends "SELECT a, b FROM abc" to all datanodes. > > 2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic > plan, > and steps E and F use the already built generic plan. > For details see function GetCachedPlan. > This means that executing a prepared statement again and again does > use cached plans > and does not prepare again and again every time we issue an execute. > > The problem is not here. The problem is in do_query() where somehow the name of prepared statement gets wiped out and we keep on preparing unnamed statements at the datanode. > My conclusion is that the bug ID 3607975 is not reproducible. > > Did you verify it under the debugger? If that would have been the case, we would not have seen this problem if search_path changed in between steps D and E. > Comments are welcome. > > -- > *Abbas* > Architect > > Ph: 92.334.5100153 > Skype ID: gabbasb > www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> > * > Follow us on Twitter* > @EnterpriseDB > > Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Abbas B. <abb...@en...> - 2013-05-24 02:19:39
|
On Fri, May 17, 2013 at 1:58 PM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Fri, May 17, 2013 at 2:23 PM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Thu, May 16, 2013 at 3:13 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Hi Abbas, >>> I am also seeing a lot of changes in the expected output where the rows >>> output have changed. What are these changes? >>> >> >> These changes are a result of blocking partition column updates >> > > Are those in sync with PG expected output? > No, in PG the update does not fail, in XC it fails. > Why did we change the original expected output in first place? > Do you mean that the changes in expected output due to blocking of partition column updates should only be done in alternate expected output file? > > >> and changing the distribution of tables to replication. >> >> > > That's acceptable. > > >> >>> >>> On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> Hi Abbas, >>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is it >>>> possible to traverse the Query in validate_part_col_updatable() recursively >>>> to find UPDATE statements and apply partition column check? That would >>>> cover all the possibilities, I guess. That also saves us much effort in >>>> case we come to support distribution column updation. >>>> >>>> I think, we need a generic solution to solve this command id issue, >>>> e.g. punching command id always and efficiently. But for now this suffices. >>>> Please log a bug/feature and put it in 1.2 bucket. >>>> >>>> >>>> >>>> >>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> Adding developers mailing list. >>>>> >>>>> >>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>> abb...@en...> wrote: >>>>> >>>>>> Hi, >>>>>> Attached please find a patch to fix test case with. >>>>>> There were two issues making the test to fail. >>>>>> 1. Updates to partition column were possible using syntax like >>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>> The patch blocks this syntax. >>>>>> >>>>>> 2. For a WITH query that updates a table in the main query and >>>>>> inserts a row in the same table in the WITH query we need to use >>>>>> command ID communication to remote nodes in order to >>>>>> maintain global data visibility. >>>>>> For example >>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>> REPLICATION; >>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>>> newid) >>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>> The last query gets translated into the following multi-statement >>>>>> transaction on the primary datanode >>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE >>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>> (42,'new)' >>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>> (43,(0,1)] >>>>>> (e) COMMIT TRANSACTION >>>>>> The command id of the select in step (c), should be such that >>>>>> it does not see the insert of step (b) >>>>>> >>>>>> Comments are welcome. >>>>>> >>>>>> Regards >>>>>> >>>>>> -- >>>>>> *Abbas* >>>>>> Architect >>>>>> >>>>>> Ph: 92.334.5100153 >>>>>> Skype ID: gabbasb >>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>> * >>>>>> Follow us on Twitter* >>>>>> @EnterpriseDB >>>>>> >>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> AlienVault Unified Security Management (USM) platform delivers complete >>>>> security visibility with the essential security capabilities. Easily >>>>> and >>>>> efficiently configure, manage, and operate all of your security >>>>> controls >>>>> from a single console and one unified framework. Download a free trial. >>>>> http://p.sf.net/sfu/alienvault_d2d >>>>> _______________________________________________ >>>>> Postgres-xc-core mailing list >>>>> Pos...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > > ------------------------------------------------------------------------------ > AlienVault Unified Security Management (USM) platform delivers complete > security visibility with the essential security capabilities. Easily and > efficiently configure, manage, and operate all of your security controls > from a single console and one unified framework. Download a free trial. > http://p.sf.net/sfu/alienvault_d2d > _______________________________________________ > Postgres-xc-core mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-core > > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Abbas B. <abb...@en...> - 2013-05-24 01:21:27
|
Hi,
While working on test case plancache it was brought up as a review comment
that solving bug id 3607975 should solve the problem of the test case.
However there is some confusion in the statement of bug id 3607975.
"When a user does and PREPARE and then EXECUTEs multiple times, the
coordinator keeps on preparing and executing the query on datanode al
times, as against preparing once and executing multiple times. This is
because somehow the remote query is being prepared as an unnamed statement."
Consider this test case
A. create table abc(a int, b int);
B. insert into abc values(11, 22);
C. prepare p1 as select * from abc;
D. execute p1;
E. execute p1;
F. execute p1;
Here are the confusions
1. The coordinator never prepares on datanode in response to a prepare
issued by a user.
In fact step C does nothing on the datanodes.
Step D simply sends "SELECT a, b FROM abc" to all datanodes.
2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic
plan,
and steps E and F use the already built generic plan.
For details see function GetCachedPlan.
This means that executing a prepared statement again and again does use
cached plans
and does not prepare again and again every time we issue an execute.
My conclusion is that the bug ID 3607975 is not reproducible.
Comments are welcome.
--
*Abbas*
Architect
Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB
Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>
|
|
From: Ashutosh B. <ash...@en...> - 2013-05-23 20:00:44
|
Hi All, While I hear Robert's presentation on query planning go wrong, I heard (first time) about MergeAppend plan. It looks like, we have something better to do with the way we are pushing the ORDER BY clauses and doing merge at the coordinator. Right now we are using Sort node and in ExecSort we start merging the rows. Instead, I think we should be using MergeAppend node. The only bad thing about this approach is MergeAppend node expects separate plans for each run of the sorted data. To do this, we need to replicate the RemoteQuery node as many times as there are nodes and put them in the list to the MergeAppend plan. I think that's going to solve the problem with materialisation (see mail thread "Using remote sorting for merge-join") and thus improve performance. It also takes away the need to have xc_node_id in the tupleslot of tuplestore. Obviously, simplifying the code. Any comments? Created 3613815 <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>and assigned to me. <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232> On Mon, Dec 3, 2012 at 3:44 AM, Ashutosh Bapat < ash...@en...> wrote: > On Fri, Nov 30, 2012 at 9:37 AM, Michael Paquier > <mic...@gm...> wrote: > > Hi Ashutosh, > > > > Thanks for the patch. > > I found for the time being the following problems: > > 1) The patch has some whitespaces > > /home/michael/download/xc_sort.patch:16: trailing whitespace. > > > > /home/michael/download/xc_sort.patch:34: trailing whitespace. > > > > /home/michael/download/xc_sort.patch:307: trailing whitespace. > > return local_plan_left; > > /home/michael/download/xc_sort.patch:356: trailing whitespace. > > > > /home/michael/download/xc_sort.patch:360: trailing whitespace. > > > > Fixed. > > > error: patch failed: src/include/pgxc/execRemote.h:95 > > 2) the patch does not apply cleanly even by using "patch -p1"-like > commands > > When I sent the patch (more than a week before you replied) it was > based on the HEAD, but then there were some commits which created the > conflicts. The one attached should apply. > > > 3) No documentation about the new option enable_remotesort. > > 4) enable_remotesort is not listed in postgresql.conf. > > Please provide both documentation and reference in postgresql.conf.sample > > for transparency of the code. As an open source project it is important > to > > classify all the options available, for both the user and the developers. > > Added although, I still don't approve of it myself. > > > 5) What is cruelly missing with this patch are numbers. By how much this > > feature improves performance? > > Simple tests like a ORDER BY on a table with millions of rows would be > > enough, but as this is a performance feature, you should provide data > about > > how much those cases are improved. > > 6) A complain that I say for each planner-related patch in this > project.... > > This patch has a size of 200KB, 65% being updates on the EXPLAIN queries > of > > the XC query planning regression tests. This makes the commit tree > > unreadable and the evolution of the regression tests realy hard to follow > > because they need to be modified each time a planner optimization is > > introduced. We should move to a set of tests which is far more > > code-modification proof! > > This is planner code, so it's going to affect the plans that it > creates. In the test files we add EXPLAIN (with proper switches) to > check 1. whether the plans are coming as expected at that stage of > development, 2. Whether the clauses expected to be pushed down are > being indeed pushed down 3. the queries to be sent to the datanodes > are being crafted correctly. For every planner change, all the three > things above change, for the operation being planned and hence the > expected output changes are unavoidable. In fact, if you look at how > the expected output changing you get a idea of how correctly/wrong the > planner changes are made. Those diffs if go wrong, are great help to > me in finding problems earlier. During merge as well, if those tests > fail because of EXPLAIN outputs, that's a way to unwanted planner > changes early enough and correct them. Sorry, for the huge change, but > it's unavoidable. The testcases are intentionally prone to (planner) > code modification. > > > > > Regards, > > > > On Mon, Nov 26, 2012 at 8:42 PM, Ashutosh Bapat > > <ash...@en...> wrote: > >> > >> Hi All, > >> PFA patch which enables pushing down the ORDER BY clause or sorting to > the > >> Datanode/s through standard_planner(). > >> > >> This patch leverages the already existing merge sort mechanism for > >> RemoteQuery but now outside RemoteQuery using Sort plans. > >> > >> For a Sort plan it checks if all the expressions on which the result > needs > >> to be sorted are shippable and that the underlying tree allows to push > ORDER > >> BY to the Datanodes. If so, it includes the ORDER BY clause in the > query to > >> be sent to the Datanodes. If the number of Datanodes which participate > in > >> RemoteQuery is not 1, it adds a covering Sort node to merge the results > from > >> the Datanodes, otherwise, there is no covering Sort plan. > >> > >> The merge sort mechanism treats every Datanode as a tape and prefetches > >> the results from the Datanode corresponding to a dwindling tape. This is > >> enabled by setting the node from where to pull the next row before > calling > >> ExecProcNode on RemoteQueryState. > >> > >> The same mechanism is being used for Sort plan node under Group or Agg > >> nodes. > >> > >> The patch adds test xc_sort.sql for testing this functionality. It also > >> removes the now useless members sort and tuplesortstate from > RemoteQuery and > >> RemoteQueryState resp. > >> -- > >> Best Wishes, > >> Ashutosh Bapat > >> EntepriseDB Corporation > >> The Enterprise Postgres Company > >> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Monitor your physical, virtual and cloud infrastructure from a single > >> web console. Get in-depth insight into apps, servers, databases, vmware, > >> SAP, cloud infrastructure, etc. Download 30-day Free Trial. > >> Pricing starts from $795 for 25 servers or applications! > >> http://p.sf.net/sfu/zoho_dev2dev_nov > >> _______________________________________________ > >> Postgres-xc-developers mailing list > >> Pos...@li... > >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > >> > > > > > > > > -- > > Michael Paquier > > http://michael.otacoo.com > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Enterprise Postgres Company > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Abbas B. <abb...@en...> - 2013-05-23 04:56:56
|
On Thu, May 23, 2013 at 1:06 AM, Ahsan Hadi <ahs...@en...>wrote: > Abbas, > Are you planning to commit this soon? > No, This patch is dependent on the patch I sent in the email with subject "Patch to change name of a couple of variables to more general names" that is under review. Also in this email chain, Ashutosh has asked a few questions that I need to reply. > > > On Fri, May 17, 2013 at 5:26 AM, Ashutosh Bapat < > ash...@en...> wrote: > >> This looks good. Are there other ways where we can have UPDATE statement >> somewhere in the query tree list? Do we need to worry about such cases. >> >> >> On Fri, May 17, 2013 at 2:22 PM, Abbas Butt <abb...@en...>wrote: >> >>> >>> >>> On Thu, May 16, 2013 at 2:25 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> Hi Abbas, >>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is it >>>> possible to traverse the Query in validate_part_col_updatable() recursively >>>> to find UPDATE statements and apply partition column check? >>>> >>> >>> Yes. I have attached that patch for your feedback. If you think its ok I >>> can send the updated patch including the rest of the changes. >>> >>> >>>> That would cover all the possibilities, I guess. That also saves us >>>> much effort in case we come to support distribution column updation. >>>> >>>> I think, we need a generic solution to solve this command id issue, >>>> e.g. punching command id always and efficiently. But for now this suffices. >>>> Please log a bug/feature and put it in 1.2 bucket. >>>> >>> >>> Done. >>> (Artifact 3613498<https://sourceforge.net/tracker/?func=detail&aid=3613498&group_id=311227&atid=1310235> >>> ) >>> >>>> >>>> >>>> >>>> >>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> Adding developers mailing list. >>>>> >>>>> >>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>> abb...@en...> wrote: >>>>> >>>>>> Hi, >>>>>> Attached please find a patch to fix test case with. >>>>>> There were two issues making the test to fail. >>>>>> 1. Updates to partition column were possible using syntax like >>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>> The patch blocks this syntax. >>>>>> >>>>>> 2. For a WITH query that updates a table in the main query and >>>>>> inserts a row in the same table in the WITH query we need to use >>>>>> command ID communication to remote nodes in order to >>>>>> maintain global data visibility. >>>>>> For example >>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>> REPLICATION; >>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>>> newid) >>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>> The last query gets translated into the following multi-statement >>>>>> transaction on the primary datanode >>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE >>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>> (42,'new)' >>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>> (43,(0,1)] >>>>>> (e) COMMIT TRANSACTION >>>>>> The command id of the select in step (c), should be such that >>>>>> it does not see the insert of step (b) >>>>>> >>>>>> Comments are welcome. >>>>>> >>>>>> Regards >>>>>> >>>>>> -- >>>>>> *Abbas* >>>>>> Architect >>>>>> >>>>>> Ph: 92.334.5100153 >>>>>> Skype ID: gabbasb >>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>> * >>>>>> Follow us on Twitter* >>>>>> @EnterpriseDB >>>>>> >>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> AlienVault Unified Security Management (USM) platform delivers complete >>>>> security visibility with the essential security capabilities. Easily >>>>> and >>>>> efficiently configure, manage, and operate all of your security >>>>> controls >>>>> from a single console and one unified framework. Download a free trial. >>>>> http://p.sf.net/sfu/alienvault_d2d >>>>> _______________________________________________ >>>>> Postgres-xc-core mailing list >>>>> Pos...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>> >>> >>> >>> -- >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> >> >> ------------------------------------------------------------------------------ >> AlienVault Unified Security Management (USM) platform delivers complete >> security visibility with the essential security capabilities. Easily and >> efficiently configure, manage, and operate all of your security controls >> from a single console and one unified framework. Download a free trial. >> http://p.sf.net/sfu/alienvault_d2d >> _______________________________________________ >> Postgres-xc-core mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >> >> > > > -- > Ahsan Hadi > Snr Director Product Development > EnterpriseDB Corporation > The Enterprise Postgres Company > > Phone: +92-51-8358874 > Mobile: +92-333-5162114 > > Website: www.enterprisedb.com > EnterpriseDB Blog: http://blogs.enterprisedb.com/ > Follow us on Twitter: http://www.twitter.com/enterprisedb > > This e-mail message (and any attachment) is intended for the use of the > individual or entity to whom it is addressed. This message contains > information from EnterpriseDB Corporation that may be privileged, > confidential, or exempt from disclosure under applicable law. If you are > not the intended recipient or authorized to receive this for the intended > recipient, any use, dissemination, distribution, retention, archiving, or > copying of this communication is strictly prohibited. If you have received > this e-mail in error, please notify the sender immediately by reply e-mail > and delete this message. > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Abbas B. <abb...@en...> - 2013-05-23 04:41:50
|
I will try to spare some time for this over the weekend. On Thu, May 23, 2013 at 1:09 AM, Ahsan Hadi <ahs...@en...>wrote: > Abbas, > Can you please review this patch this week? > > > On Tue, May 21, 2013 at 3:55 AM, Amit Khandekar < > ami...@en...> wrote: > >> Currently the number of tuples processed is updated in both >> HandleCommandComplete and ExecInsert/Update/Delete. >> >> In HandleCommandComplete() it gets it from the command tag returned from >> the datanode i.e. INSERT 0 2, UPDATE 5 and likewise. And then it updates >> estate->es_processed. But it does this only for FQS. For non-FQS, in >> ExecInsert/Update, it is just incremented by 1. So if a trigger function >> skips one row on datanode, the command tag returned from datanode is INSERT >> 0 0. But still in ExecInsert() increments the row count. >> >> I have added a new field RemoteQueryState->rqs_processed, which is >> updated in HandleCommandComplete(). Then it is used in >> ExecInsert/Update/Delete() for non-FQS, and in RemoteQueryNext() for FQS. >> >> While fixing this issue, I see that there seem to be some issue with >> combiner->command_complete_count. Currently it checks for consistency of >> number of tuples returned for replicated tables, but it does that only for >> FQS. Need to completely remove the dependency on whether it's an FQS or >> non-FQS DML query. For this, command_complete_count needs to be better >> handled. I felt it needs some refactoring which I did not feel good to do >> in this release. Currently this field is being updated for each iteration >> of FetchTuple by re-using the same combiner for each iteration, whereas it >> seems it should be updated only for each node execution, not for each tuple >> fetched. I haven't touched this part, but added a TODO, and opened 3613645. >> >> Added some testcases in existing tests xc_trigship and xc_returning. >> >> >> >> >> ------------------------------------------------------------------------------ >> Try New Relic Now & We'll Send You this Cool Shirt >> New Relic is the only SaaS-based application performance monitoring >> service >> that delivers powerful full stack analytics. Optimize and monitor your >> browser, app, & servers with just a few lines of code. Try New Relic >> and get this awesome Nerd Life shirt! >> http://p.sf.net/sfu/newrelic_d2d_may >> _______________________________________________ >> Postgres-xc-developers mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> > > > -- > Ahsan Hadi > Snr Director Product Development > EnterpriseDB Corporation > The Enterprise Postgres Company > > Phone: +92-51-8358874 > Mobile: +92-333-5162114 > > Website: www.enterprisedb.com > EnterpriseDB Blog: http://blogs.enterprisedb.com/ > Follow us on Twitter: http://www.twitter.com/enterprisedb > > This e-mail message (and any attachment) is intended for the use of the > individual or entity to whom it is addressed. This message contains > information from EnterpriseDB Corporation that may be privileged, > confidential, or exempt from disclosure under applicable law. If you are > not the intended recipient or authorized to receive this for the intended > recipient, any use, dissemination, distribution, retention, archiving, or > copying of this communication is strictly prohibited. If you have received > this e-mail in error, please notify the sender immediately by reply e-mail > and delete this message. > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Abbas B. <abb...@en...> - 2013-05-23 04:37:12
|
Its a random failure occurring due to ! ERROR: failed to send COMMIT command to node We can neglect it for now. On Thu, May 23, 2013 at 1:17 AM, Ahsan Hadi <ahs...@en...>wrote: > Abbas, > Can you take a look, the plpgsql test case is failing again in BF > regression from couple of days ago... > > > On Tue, May 7, 2013 at 6:43 AM, Ashutosh Bapat < > ash...@en...> wrote: > >> Ok, this is good to go then. >> >> >> On Tue, Apr 30, 2013 at 6:00 PM, Abbas Butt <abb...@en...>wrote: >> >>> >>> >>> On Tue, Apr 30, 2013 at 5:04 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> Hi Abbas, >>>> This change looks confusing. The statements where you have added order >>>> by clause do not correspond to the statements where you have changed the >>>> output. Can you please explain this? >>>> >>> >>> The output of the test cases are changed because there was an error in >>> the expected output files of these test cases. These outputs have to be >>> modified (and the modified output is correct) because of your check in >>> 2608af3d5288607e8245fa10e74e7307bb1bb23d. >>> >>> The order by clause is added in test cases where the expected output was >>> already in order, but results were coming out to be in different order >>> depending on node configuration. >>> >>> >>>> >>>> Also, whenever you change .sql and need to change the .out for that, >>>> please change all the alt. expected output files. >>>> >>> >>> Sure >>> >>> >>>> >>>> >>>> On Thu, Apr 25, 2013 at 4:58 PM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> Hi, >>>>> Attached please find patch to fix test case plpgsql. The test was >>>>> failing for some order by issues, Also there were some errors in expected >>>>> output. >>>>> >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Try New Relic Now & We'll Send You this Cool Shirt >>>>> New Relic is the only SaaS-based application performance monitoring >>>>> service >>>>> that delivers powerful full stack analytics. Optimize and monitor your >>>>> browser, app, & servers with just a few lines of code. Try New Relic >>>>> and get this awesome Nerd Life shirt! >>>>> http://p.sf.net/sfu/newrelic_d2d_apr >>>>> _______________________________________________ >>>>> Postgres-xc-developers mailing list >>>>> Pos...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>> >>> >>> >>> -- >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> >> >> ------------------------------------------------------------------------------ >> Learn Graph Databases - Download FREE O'Reilly Book >> "Graph Databases" is the definitive new guide to graph databases and >> their applications. This 200-page book is written by three acclaimed >> leaders in the field. The early access version is available now. >> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >> >> _______________________________________________ >> Postgres-xc-developers mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> > > > -- > Ahsan Hadi > Snr Director Product Development > EnterpriseDB Corporation > The Enterprise Postgres Company > > Phone: +92-51-8358874 > Mobile: +92-333-5162114 > > Website: www.enterprisedb.com > EnterpriseDB Blog: http://blogs.enterprisedb.com/ > Follow us on Twitter: http://www.twitter.com/enterprisedb > > This e-mail message (and any attachment) is intended for the use of the > individual or entity to whom it is addressed. This message contains > information from EnterpriseDB Corporation that may be privileged, > confidential, or exempt from disclosure under applicable law. If you are > not the intended recipient or authorized to receive this for the intended > recipient, any use, dissemination, distribution, retention, archiving, or > copying of this communication is strictly prohibited. If you have received > this e-mail in error, please notify the sender immediately by reply e-mail > and delete this message. > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Ahsan H. <ahs...@en...> - 2013-05-22 20:18:02
|
Abbas, Can you take a look, the plpgsql test case is failing again in BF regression from couple of days ago... On Tue, May 7, 2013 at 6:43 AM, Ashutosh Bapat < ash...@en...> wrote: > Ok, this is good to go then. > > > On Tue, Apr 30, 2013 at 6:00 PM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Tue, Apr 30, 2013 at 5:04 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Hi Abbas, >>> This change looks confusing. The statements where you have added order >>> by clause do not correspond to the statements where you have changed the >>> output. Can you please explain this? >>> >> >> The output of the test cases are changed because there was an error in >> the expected output files of these test cases. These outputs have to be >> modified (and the modified output is correct) because of your check in >> 2608af3d5288607e8245fa10e74e7307bb1bb23d. >> >> The order by clause is added in test cases where the expected output was >> already in order, but results were coming out to be in different order >> depending on node configuration. >> >> >>> >>> Also, whenever you change .sql and need to change the .out for that, >>> please change all the alt. expected output files. >>> >> >> Sure >> >> >>> >>> >>> On Thu, Apr 25, 2013 at 4:58 PM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> Hi, >>>> Attached please find patch to fix test case plpgsql. The test was >>>> failing for some order by issues, Also there were some errors in expected >>>> output. >>>> >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try New Relic Now & We'll Send You this Cool Shirt >>>> New Relic is the only SaaS-based application performance monitoring >>>> service >>>> that delivers powerful full stack analytics. Optimize and monitor your >>>> browser, app, & servers with just a few lines of code. Try New Relic >>>> and get this awesome Nerd Life shirt! >>>> http://p.sf.net/sfu/newrelic_d2d_apr >>>> _______________________________________________ >>>> Postgres-xc-developers mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and > their applications. This 200-page book is written by three acclaimed > leaders in the field. The early access version is available now. > Download your free book today! http://p.sf.net/sfu/neotech_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > -- Ahsan Hadi Snr Director Product Development EnterpriseDB Corporation The Enterprise Postgres Company Phone: +92-51-8358874 Mobile: +92-333-5162114 Website: www.enterprisedb.com EnterpriseDB Blog: http://blogs.enterprisedb.com/ Follow us on Twitter: http://www.twitter.com/enterprisedb This e-mail message (and any attachment) is intended for the use of the individual or entity to whom it is addressed. This message contains information from EnterpriseDB Corporation that may be privileged, confidential, or exempt from disclosure under applicable law. If you are not the intended recipient or authorized to receive this for the intended recipient, any use, dissemination, distribution, retention, archiving, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and delete this message. |
|
From: Ahsan H. <ahs...@en...> - 2013-05-22 20:09:45
|
Abbas, Can you please review this patch this week? On Tue, May 21, 2013 at 3:55 AM, Amit Khandekar < ami...@en...> wrote: > Currently the number of tuples processed is updated in both > HandleCommandComplete and ExecInsert/Update/Delete. > > In HandleCommandComplete() it gets it from the command tag returned from > the datanode i.e. INSERT 0 2, UPDATE 5 and likewise. And then it updates > estate->es_processed. But it does this only for FQS. For non-FQS, in > ExecInsert/Update, it is just incremented by 1. So if a trigger function > skips one row on datanode, the command tag returned from datanode is INSERT > 0 0. But still in ExecInsert() increments the row count. > > I have added a new field RemoteQueryState->rqs_processed, which is updated > in HandleCommandComplete(). Then it is used in ExecInsert/Update/Delete() > for non-FQS, and in RemoteQueryNext() for FQS. > > While fixing this issue, I see that there seem to be some issue with > combiner->command_complete_count. Currently it checks for consistency of > number of tuples returned for replicated tables, but it does that only for > FQS. Need to completely remove the dependency on whether it's an FQS or > non-FQS DML query. For this, command_complete_count needs to be better > handled. I felt it needs some refactoring which I did not feel good to do > in this release. Currently this field is being updated for each iteration > of FetchTuple by re-using the same combiner for each iteration, whereas it > seems it should be updated only for each node execution, not for each tuple > fetched. I haven't touched this part, but added a TODO, and opened 3613645. > > Added some testcases in existing tests xc_trigship and xc_returning. > > > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > -- Ahsan Hadi Snr Director Product Development EnterpriseDB Corporation The Enterprise Postgres Company Phone: +92-51-8358874 Mobile: +92-333-5162114 Website: www.enterprisedb.com EnterpriseDB Blog: http://blogs.enterprisedb.com/ Follow us on Twitter: http://www.twitter.com/enterprisedb This e-mail message (and any attachment) is intended for the use of the individual or entity to whom it is addressed. This message contains information from EnterpriseDB Corporation that may be privileged, confidential, or exempt from disclosure under applicable law. If you are not the intended recipient or authorized to receive this for the intended recipient, any use, dissemination, distribution, retention, archiving, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and delete this message. |
|
From: Ahsan H. <ahs...@en...> - 2013-05-22 20:06:46
|
Abbas, Are you planning to commit this soon? On Fri, May 17, 2013 at 5:26 AM, Ashutosh Bapat < ash...@en...> wrote: > This looks good. Are there other ways where we can have UPDATE statement > somewhere in the query tree list? Do we need to worry about such cases. > > > On Fri, May 17, 2013 at 2:22 PM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Thu, May 16, 2013 at 2:25 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Hi Abbas, >>> Instead of fixing the first issue in pgxc_build_dml_statement(), is it >>> possible to traverse the Query in validate_part_col_updatable() recursively >>> to find UPDATE statements and apply partition column check? >>> >> >> Yes. I have attached that patch for your feedback. If you think its ok I >> can send the updated patch including the rest of the changes. >> >> >>> That would cover all the possibilities, I guess. That also saves us >>> much effort in case we come to support distribution column updation. >>> >>> I think, we need a generic solution to solve this command id issue, e.g. >>> punching command id always and efficiently. But for now this suffices. >>> Please log a bug/feature and put it in 1.2 bucket. >>> >> >> Done. >> (Artifact 3613498<https://sourceforge.net/tracker/?func=detail&aid=3613498&group_id=311227&atid=1310235> >> ) >> >>> >>> >>> >>> >>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> Adding developers mailing list. >>>> >>>> >>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> Hi, >>>>> Attached please find a patch to fix test case with. >>>>> There were two issues making the test to fail. >>>>> 1. Updates to partition column were possible using syntax like >>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>> The patch blocks this syntax. >>>>> >>>>> 2. For a WITH query that updates a table in the main query and >>>>> inserts a row in the same table in the WITH query we need to use >>>>> command ID communication to remote nodes in order to >>>>> maintain global data visibility. >>>>> For example >>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY REPLICATION; >>>>> INSERT INTO tab VALUES (1,'p1'); >>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS >>>>> newid) >>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>> The last query gets translated into the following multi-statement >>>>> transaction on the primary datanode >>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE >>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>> (42,'new)' >>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>> (43,(0,1)] >>>>> (e) COMMIT TRANSACTION >>>>> The command id of the select in step (c), should be such that >>>>> it does not see the insert of step (b) >>>>> >>>>> Comments are welcome. >>>>> >>>>> Regards >>>>> >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> AlienVault Unified Security Management (USM) platform delivers complete >>>> security visibility with the essential security capabilities. Easily and >>>> efficiently configure, manage, and operate all of your security controls >>>> from a single console and one unified framework. Download a free trial. >>>> http://p.sf.net/sfu/alienvault_d2d >>>> _______________________________________________ >>>> Postgres-xc-core mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > > > ------------------------------------------------------------------------------ > AlienVault Unified Security Management (USM) platform delivers complete > security visibility with the essential security capabilities. Easily and > efficiently configure, manage, and operate all of your security controls > from a single console and one unified framework. Download a free trial. > http://p.sf.net/sfu/alienvault_d2d > _______________________________________________ > Postgres-xc-core mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-core > > -- Ahsan Hadi Snr Director Product Development EnterpriseDB Corporation The Enterprise Postgres Company Phone: +92-51-8358874 Mobile: +92-333-5162114 Website: www.enterprisedb.com EnterpriseDB Blog: http://blogs.enterprisedb.com/ Follow us on Twitter: http://www.twitter.com/enterprisedb This e-mail message (and any attachment) is intended for the use of the individual or entity to whom it is addressed. This message contains information from EnterpriseDB Corporation that may be privileged, confidential, or exempt from disclosure under applicable law. If you are not the intended recipient or authorized to receive this for the intended recipient, any use, dissemination, distribution, retention, archiving, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and delete this message. |
|
From: Amit K. <ami...@en...> - 2013-05-20 06:01:07
|
On 17 May 2013 17:16, Ashutosh Bapat <ash...@en...>wrote: > > > > On Fri, May 17, 2013 at 4:15 PM, Amit Khandekar < > ami...@en...> wrote: > >> >> >> On 15 May 2013 12:16, Ashutosh Bapat <ash...@en...>wrote: >> >>> Hi Amit, >>> Here are comment on trig_shippability patch. >>> >>> 1. The function pgxc_trigevent_quickfind() needs a better name like >>> pgxc_has_trigger_for_event() to convey the functionality clearly. >>> 2. The prologue of function pgxc_should_exec_triggers() has all the >>> necessary content, but it needs to be written in a better order. The >>> prologue talks about a single trigger (to be executed) but in reality the >>> function is executed for checking whether all the triggers matching the >>> given criteria and belonging to a given relation are firable or not. The >>> prologue needs to be corrected. Also, first specify why is such a check >>> needed (trigger order being alphabetical makes it necessary that all or >>> none of the triggers execute on the same node). >>> 3. pgxc_is_inttrigger_firable(), the function name needs to change a >>> bit. The int in inttrigger can be easily associated with integer instead of >>> internal. Can you please change the name to use internal_ or intern_ >>> instead of int. If the name becomes too long, please use suitable prefix. >>> >> >> Done all the above. >> >> >>> 4. RemoteQueryNext() is called multiple times; the after statement >>> triggers too will be called those many times. Right now this does not >>> happen since there is no way the function will be called multiple times for >>> a DML. But in future, we will start supporting RETURNING with FQS, in which >>> case RemoteQueryNext will be called multiple times. >>> >> >> True. Right now it works but once RemoteQueryNExt would be called >> multiple times for FQS, this won't work. I have added a check of >> TupIsNull() before firing AS trigger. BS trigger was ok. >> >> >> > This looks fine. > > >> Also, I have removed the relation_access_type field check for statement >> triggers. Instead, for FQS updated the remote_query field with the Query >> structure, and used it's command tag. >> >> > Good. > > >> >> There was one more redundancy discovered. When we have non-shippable AR >> triggers but BR triggers don't even exist, we are currently using all >> columns in the SET clause in the remote update statement. This is not >> necessary. The function should_exec_br_triggers() is used to determine >> whether we should do this. But this function returns true even if there are >> no BR triggers and one or more non-shippable AR triggers exist. In the >> updated patch, I have added a check to see if the BR triggers exist in the >> first place, and if not, return false. >> >> > Ok, thanks for noticing this one. > > -- > >> Before checking in the patch, I want to check one more thing. The >> with.sql shows a different output for statements like: >> WITH wcte AS ( INSERT INTO child1 VALUES ( 42, 'new' ) RETURNING id AS >> newid ) >> >> Although the current results also fail for the same statements, my patch >> results in further diffs for the same statements, so I am not sure what's >> going on. When Abbas checks in the with.sql , I will re-run the regression. >> Abbas, do you have any rough patch ready in order for me to test that my >> patch does not create any new with.sql failures ? Just one that passes >> with.sql should suffice. >> >> I verified using Abbas's patch that the with.sql passes with both mine and Abbas's patch applied. I am committing my patch. > > Once you check this, it's fine to commit the patch. > > >> >>> test xc_trigship >>> >>> 1. we are using same name xc_auditlog for table and function. Please use >>> different names. >>> 2. For every object (table/function etc) that the test creates, please >>> mention the purpose of that object. That helps validating the object >>> definition there and there itself. >>> >> >> Done. >> >> >>> 3. I couldn't understand, how does the test check where the trigger is >>> being fired? >>> >> >> The last column of xc_auditlog is node type. >> >> >>> >>> >>> >>> On Mon, May 13, 2013 at 2:36 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> Hi Amit, >>>> We now have Query structure in RemoteQuery. This corresponds to the >>>> query being fired on the datanodes. THe commandType here should tell you >>>> whether it's DELETE or not. >>>> >>>> >>>> On Mon, May 13, 2013 at 1:14 PM, Amit Khandekar < >>>> ami...@en...> wrote: >>>> >>>>> In case of multiple triggers of the same type for a given table, the >>>>> requirement is that they should be fired in alphabetical order. To do so, >>>>> we need to fire either all of them on coordinator, or all of them on >>>>> datanode. The main changes in the attached patch are related to this >>>>> requirement. >>>>> >>>>> All of the Exec*Trigger() functions now execute should_exec_trigger*() >>>>> functions that return true if the node on which they are run is the right >>>>> node to execute those type of triggers. >>>>> >>>>> Also for BR triggers, the additional requirement is that they should >>>>> be run on coordinator if AR triggers are not shippable, regardless of >>>>> whether BR themselves are shippable or not. This is because, if we fire BR >>>>> triggers on datanode, they might change the final row updated, and so we >>>>> need to again fetch the new row back to the coordinator. Instead, if we >>>>> fire them on coordinator, we already know what's the final row. Also, we >>>>> would have required additional changes to add RETURNING to the remote query >>>>> to fetch the final updated row. >>>>> >>>>> Constraint triggers are exception; we need to fire them always on >>>>> datanode. Once we support global constraints, these need not be specially >>>>> handled. >>>>> >>>>> I was trying to avoid the should_exec_trigger*() calls for each of the >>>>> Exec*Trigger() functions by doing this in TriggerEnabled() because it is a >>>>> common function called for all triggers. But the issue is, for AFTER >>>>> triggers, it is called with a different set of event type values than the >>>>> usual TRIGGER_TYPE* values. For AFTER triggers, TRIGGER_EVENT_* values are >>>>> used. Anyways, TriggerEnabled() is called for each of the trigger. >>>>> >>>>> The trigger shippability helper functions are now completely changed. >>>>> pgxc_find_nonshippable_row_trig() is the key function. The comments in >>>>> these functions should give a fair idea of their functionality. >>>>> >>>>> >>>>> For stmt shippability, the trigger functions are now explicitly called >>>>> if it's not an internally generated query. rq_internal_params field is used >>>>> to know that this DML is user-supplied query (that is, it would be FQS). >>>>> The functions are called in RemoteQueryNext(). >>>>> >>>>> >>>>> THere is another patch (relaccess_type.patch) that you need to first >>>>> apply before the main patch (trig_shippability.patch) is applied. I had to >>>>> add another RELATION_ACCESS_DELETE in ExecNodes. To handle the stmt >>>>> triggers, I had to know whether the statement is DELETE or INSERT or UPDATE. >>>>> >>>>> The new test xc_trigship is added. >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Learn Graph Databases - Download FREE O'Reilly Book >>>>> "Graph Databases" is the definitive new guide to graph databases and >>>>> their applications. This 200-page book is written by three acclaimed >>>>> leaders in the field. The early access version is available now. >>>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>>> _______________________________________________ >>>>> Postgres-xc-developers mailing list >>>>> Pos...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > |
|
From: Ashutosh B. <ash...@en...> - 2013-05-17 11:46:34
|
On Fri, May 17, 2013 at 4:15 PM, Amit Khandekar < ami...@en...> wrote: > > > On 15 May 2013 12:16, Ashutosh Bapat <ash...@en...>wrote: > >> Hi Amit, >> Here are comment on trig_shippability patch. >> >> 1. The function pgxc_trigevent_quickfind() needs a better name like >> pgxc_has_trigger_for_event() to convey the functionality clearly. >> 2. The prologue of function pgxc_should_exec_triggers() has all the >> necessary content, but it needs to be written in a better order. The >> prologue talks about a single trigger (to be executed) but in reality the >> function is executed for checking whether all the triggers matching the >> given criteria and belonging to a given relation are firable or not. The >> prologue needs to be corrected. Also, first specify why is such a check >> needed (trigger order being alphabetical makes it necessary that all or >> none of the triggers execute on the same node). >> 3. pgxc_is_inttrigger_firable(), the function name needs to change a bit. >> The int in inttrigger can be easily associated with integer instead of >> internal. Can you please change the name to use internal_ or intern_ >> instead of int. If the name becomes too long, please use suitable prefix. >> > > Done all the above. > > >> 4. RemoteQueryNext() is called multiple times; the after statement >> triggers too will be called those many times. Right now this does not >> happen since there is no way the function will be called multiple times for >> a DML. But in future, we will start supporting RETURNING with FQS, in which >> case RemoteQueryNext will be called multiple times. >> > > True. Right now it works but once RemoteQueryNExt would be called multiple > times for FQS, this won't work. I have added a check of TupIsNull() before > firing AS trigger. BS trigger was ok. > > > This looks fine. > Also, I have removed the relation_access_type field check for statement > triggers. Instead, for FQS updated the remote_query field with the Query > structure, and used it's command tag. > > Good. > > There was one more redundancy discovered. When we have non-shippable AR > triggers but BR triggers don't even exist, we are currently using all > columns in the SET clause in the remote update statement. This is not > necessary. The function should_exec_br_triggers() is used to determine > whether we should do this. But this function returns true even if there are > no BR triggers and one or more non-shippable AR triggers exist. In the > updated patch, I have added a check to see if the BR triggers exist in the > first place, and if not, return false. > > Ok, thanks for noticing this one. > > Before checking in the patch, I want to check one more thing. The with.sql > shows a different output for statements like: > WITH wcte AS ( INSERT INTO child1 VALUES ( 42, 'new' ) RETURNING id AS > newid ) > > Although the current results also fail for the same statements, my patch > results in further diffs for the same statements, so I am not sure what's > going on. When Abbas checks in the with.sql , I will re-run the regression. > Abbas, do you have any rough patch ready in order for me to test that my > patch does not create any new with.sql failures ? Just one that passes > with.sql should suffice. > > Once you check this, it's fine to commit the patch. > >> test xc_trigship >> >> 1. we are using same name xc_auditlog for table and function. Please use >> different names. >> 2. For every object (table/function etc) that the test creates, please >> mention the purpose of that object. That helps validating the object >> definition there and there itself. >> > > Done. > > >> 3. I couldn't understand, how does the test check where the trigger is >> being fired? >> > > The last column of xc_auditlog is node type. > > >> >> >> >> On Mon, May 13, 2013 at 2:36 PM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> Hi Amit, >>> We now have Query structure in RemoteQuery. This corresponds to the >>> query being fired on the datanodes. THe commandType here should tell you >>> whether it's DELETE or not. >>> >>> >>> On Mon, May 13, 2013 at 1:14 PM, Amit Khandekar < >>> ami...@en...> wrote: >>> >>>> In case of multiple triggers of the same type for a given table, the >>>> requirement is that they should be fired in alphabetical order. To do so, >>>> we need to fire either all of them on coordinator, or all of them on >>>> datanode. The main changes in the attached patch are related to this >>>> requirement. >>>> >>>> All of the Exec*Trigger() functions now execute should_exec_trigger*() >>>> functions that return true if the node on which they are run is the right >>>> node to execute those type of triggers. >>>> >>>> Also for BR triggers, the additional requirement is that they should be >>>> run on coordinator if AR triggers are not shippable, regardless of whether >>>> BR themselves are shippable or not. This is because, if we fire BR triggers >>>> on datanode, they might change the final row updated, and so we need to >>>> again fetch the new row back to the coordinator. Instead, if we fire them >>>> on coordinator, we already know what's the final row. Also, we would have >>>> required additional changes to add RETURNING to the remote query to fetch >>>> the final updated row. >>>> >>>> Constraint triggers are exception; we need to fire them always on >>>> datanode. Once we support global constraints, these need not be specially >>>> handled. >>>> >>>> I was trying to avoid the should_exec_trigger*() calls for each of the >>>> Exec*Trigger() functions by doing this in TriggerEnabled() because it is a >>>> common function called for all triggers. But the issue is, for AFTER >>>> triggers, it is called with a different set of event type values than the >>>> usual TRIGGER_TYPE* values. For AFTER triggers, TRIGGER_EVENT_* values are >>>> used. Anyways, TriggerEnabled() is called for each of the trigger. >>>> >>>> The trigger shippability helper functions are now completely changed. >>>> pgxc_find_nonshippable_row_trig() is the key function. The comments in >>>> these functions should give a fair idea of their functionality. >>>> >>>> >>>> For stmt shippability, the trigger functions are now explicitly called >>>> if it's not an internally generated query. rq_internal_params field is used >>>> to know that this DML is user-supplied query (that is, it would be FQS). >>>> The functions are called in RemoteQueryNext(). >>>> >>>> >>>> THere is another patch (relaccess_type.patch) that you need to first >>>> apply before the main patch (trig_shippability.patch) is applied. I had to >>>> add another RELATION_ACCESS_DELETE in ExecNodes. To handle the stmt >>>> triggers, I had to know whether the statement is DELETE or INSERT or UPDATE. >>>> >>>> The new test xc_trigship is added. >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Learn Graph Databases - Download FREE O'Reilly Book >>>> "Graph Databases" is the definitive new guide to graph databases and >>>> their applications. This 200-page book is written by three acclaimed >>>> leaders in the field. The early access version is available now. >>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>> _______________________________________________ >>>> Postgres-xc-developers mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> > > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Postgres Database Company |
|
From: Abbas B. <abb...@en...> - 2013-05-17 10:56:30
|
Yah it was failing for me. On Fri, May 17, 2013 at 2:18 PM, Ashutosh Bapat < ash...@en...> wrote: > Is this failing for you? > > I do not see it failing on my machine, neither on build-farm. > > > On Fri, May 17, 2013 at 2:01 PM, Abbas Butt <abb...@en...>wrote: > >> Hi, >> Attached please find a patch to add some missing order by clauses in >> truncate test case. >> >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> >> >> ------------------------------------------------------------------------------ >> AlienVault Unified Security Management (USM) platform delivers complete >> security visibility with the essential security capabilities. Easily and >> efficiently configure, manage, and operate all of your security controls >> from a single console and one unified framework. Download a free trial. >> http://p.sf.net/sfu/alienvault_d2d >> _______________________________________________ >> Postgres-xc-developers mailing list >> Pos...@li... >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >> >> > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
261 messages has been excluded from this view by a project administrator.
