Menu
▾
▴
Re: [Postgres-xc-developers] Patch to fix pg_dump to generate TO NODE clause in the dump [Feature ID 3608375]
|
From: Ashutosh B. <ash...@en...> - 2013-04-15 06:04:36
|
It seems that you wrote an infrastructure to test dump in general. I thought, PG already has tests to test pg_dump and pg_restore functionality. I don't think, we should add infrastructure to test the dump functionality. It has to be done in PG and then incorporated in XC. On Sun, Apr 14, 2013 at 7:15 AM, Abbas Butt <abb...@en...>wrote: > Hi, > Attached please find a patch that adds support to test this feature. > To test this feature we had to add Support For Shell Scripts In > Regression Tests. > > For this purpose a new keyword "script" is added in schedule files. > The folder "scripts_input" is supposed to contain shell scripts in which > the following place holders can be used (in addition to the ones already > there). > > @abs_bin_dir@ which is replaced by the installation bin directory. > @database_name@ which is replaced by the database name used to run > regression tests. > The schedule file can have a command like this for running a shell script > script: script_name.sh > > In oder to test TO NODE clause in the CREATE TABLE statements the > following scheme is used. > > 1. Run test xc_1_to_node.sql > It creates some tables in a certain schema on some specific nodes > > 2. Create a script xc_to_node.source with place holders in the folder > "scripts_input". This script contains a command like this: > > @abs_bin_dir@/pg_dump -c --include-nodes --schema=test_dump_restore > -s > @database_name@ --file=@abs_srcdir@/sql/xc_2_to_node.sql > > 3. The function convert_sourcefiles is changed to accommodate script > files > After the place holders are replaced the source file in folder > "scripts_input" gets copied into folder "scripts", > with the name xc_to_node.sh > > /usr/local/pgsql/bin/pg_dump -c --include-nodes > --schema=test_dump_restore -s > regression > --file=/home/edb/pgxc/postgres-xc/src/test/regress/sql/xc_2_to_node.sql > > 4. In the schedule file the regression system is asked to run this > script using script keyword > > script: xc_to_node.sh > > 5. The function run_schedule is changed to accommodate the script > keyword. > It does not support running more than one scripts in parallel. > When the function encounters a script keyword, it calls a function > shell_command passing it the name of the test which is script name > itself. The function shell_command is new, added to run a shell > script. > > 6. When the shell script runs it generates a dump which gets copied into > the sql folder as a xc_2_to_node.sql. > It will be run next to restore the dump. > > 7. Run xc_2_to_node.sql, to restore the dump, use ignore keyword because > the order of objects might change in the dump. > > 8. Run xc_3_to_node.sql, which tests that the tables got created on the > required nodes. > > Comments are welcome. > > > > On Sat, Mar 30, 2013 at 11:49 PM, Abbas Butt <abb...@en...>wrote: > >> Hi, >> Attached please find revised patch that provides the --include-nodes >> option in both pg_dump and pg_dumpall. Please note that this patch applies >> on top of the one sent for feature ID 3608376. >> >> >> On Wed, Mar 27, 2013 at 5:05 PM, Abbas Butt <abb...@en...>wrote: >> >>> Feature ID 3608375 >>> >>> On Tue, Mar 5, 2013 at 1:45 PM, Abbas Butt <abb...@en...>wrote: >>> >>>> The attached patch changes the name of the option to --include-nodes. >>>> >>>> >>>> On Mon, Mar 4, 2013 at 2:41 PM, Abbas Butt <abb...@en... >>>> > wrote: >>>> >>>>> >>>>> >>>>> On Mon, Mar 4, 2013 at 2:09 PM, Ashutosh Bapat < >>>>> ash...@en...> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 4, 2013 at 1:51 PM, Abbas Butt < >>>>>> abb...@en...> wrote: >>>>>> >>>>>>> What I had in mind was to have pg_dump, when run with include-node, >>>>>>> emit CREATE NODE/ CREATE NODE GROUP commands only and nothing else. Those >>>>>>> commands will be used to create existing nodes/groups on the new >>>>>>> coordinator to be added. So it does make sense to use this option >>>>>>> independently, in fact it is supposed to be used independently. >>>>>>> >>>>>>> >>>>>> Ok, got it. But then include-node is really a misnomer. We should use >>>>>> --dump-nodes or something like that. >>>>>> >>>>> >>>>> In that case we can use include-nodes here. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Mon, Mar 4, 2013 at 11:21 AM, Ashutosh Bapat < >>>>>>> ash...@en...> wrote: >>>>>>> >>>>>>>> Dumping TO NODE clause only makes sense if we dump CREATE NODE/ >>>>>>>> CREATE NODE GROUP. Dumping CREATE NODE/CREATE NODE GROUP may make sense >>>>>>>> independently, but might be useless without dumping TO NODE clause. >>>>>>>> >>>>>>>> BTW, OTOH, dumping CREATE NODE/CREATE NODE GROUP clause wouldn't >>>>>>>> create the nodes on all the coordinators, >>>>>>> >>>>>>> >>>>>>> All the coordinators already have the nodes information. >>>>>>> >>>>>>> >>>>>>>> but only the coordinator where dump will be restored. That's >>>>>>>> another thing you will need to consider OR are you going to fix that as >>>>>>>> well? >>>>>>> >>>>>>> >>>>>>> As a first step I am only listing the manual steps required to add a >>>>>>> new node, that might say run this command on all the existing coordinators >>>>>>> by connecting to them one by one manually. We can decide to automate these >>>>>>> steps later. >>>>>>> >>>>>>> >>>>>> ok >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 4, 2013 at 11:41 AM, Abbas Butt < >>>>>>>> abb...@en...> wrote: >>>>>>>> >>>>>>>>> I was thinking of using include-nodes to dump CREATE NODE / CREATE >>>>>>>>> NODE GROUP, that is required as one of the missing links in adding a new >>>>>>>>> node. How do you think about that? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 4, 2013 at 9:02 AM, Ashutosh Bapat < >>>>>>>>> ash...@en...> wrote: >>>>>>>>> >>>>>>>>>> Hi Abbas, >>>>>>>>>> Please take a look at >>>>>>>>>> http://www.postgresql.org/docs/9.2/static/app-pgdump.html, which >>>>>>>>>> gives all the command line options for pg_dump. instead of >>>>>>>>>> include-to-node-clause, just include-nodes would suffice, I guess. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Mar 1, 2013 at 8:36 PM, Abbas Butt < >>>>>>>>>> abb...@en...> wrote: >>>>>>>>>> >>>>>>>>>>> PFA a updated patch that provides a command line argument called >>>>>>>>>>> --include-to-node-clause to let pg_dump know that the created dump is >>>>>>>>>>> supposed to emit TO NODE clause in the CREATE TABLE command. >>>>>>>>>>> If the argument is provided while taking the dump from a >>>>>>>>>>> datanode, it does not show TO NODE clause in the dump since the catalog >>>>>>>>>>> table is empty in this case. >>>>>>>>>>> The documentation of pg_dump is updated accordingly. >>>>>>>>>>> The rest of the functionality stays the same as before. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 25, 2013 at 10:29 AM, Ashutosh Bapat < >>>>>>>>>>> ash...@en...> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think we should always dump DISTRIBUTE BY. >>>>>>>>>>>> >>>>>>>>>>>> PG does not stop dumping (or provide an option to do so) newer >>>>>>>>>>>> syntax so that the dump will work on older versions. On similar lines, an >>>>>>>>>>>> XC dump can not be used against PG without modification (removing >>>>>>>>>>>> DISTRIBUTE BY). There can be more serious problems like exceeding table >>>>>>>>>>>> size limits if an XC dump is tried to be restored in PG. >>>>>>>>>>>> >>>>>>>>>>>> As to TO NODE clause, I agree, that one can restore the dump on >>>>>>>>>>>> a cluster with different configuration, so giving an option to dump TO NODE >>>>>>>>>>>> clause will help. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 25, 2013 at 6:42 AM, Michael Paquier < >>>>>>>>>>>> mic...@gm...> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 25, 2013 at 4:17 AM, Abbas Butt < >>>>>>>>>>>>> abb...@en...> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Feb 24, 2013 at 5:33 PM, Michael Paquier < >>>>>>>>>>>>>> mic...@gm...> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sun, Feb 24, 2013 at 7:04 PM, Abbas Butt < >>>>>>>>>>>>>>> abb...@en...> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sun, Feb 24, 2013 at 1:44 PM, Michael Paquier < >>>>>>>>>>>>>>>> mic...@gm...> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sun, Feb 24, 2013 at 3:51 PM, Abbas Butt < >>>>>>>>>>>>>>>>> abb...@en...> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> PFA a patch to fix pg_dump to generate TO NODE clause in >>>>>>>>>>>>>>>>>> the dump. >>>>>>>>>>>>>>>>>> This is required because otherwise all tables get created >>>>>>>>>>>>>>>>>> on all nodes after a dump-restore cycle. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Not sure this is good if you take a dump of an XC cluster >>>>>>>>>>>>>>>>> to restore that to a vanilla Postgres cluster. >>>>>>>>>>>>>>>>> Why not adding a new option that would control the >>>>>>>>>>>>>>>>> generation of this clause instead of forcing it? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think you can use the pg_dump that comes with vanilla PG >>>>>>>>>>>>>>>> to do that, can't you? But I am open to adding a control option if every >>>>>>>>>>>>>>>> body thinks so. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sure you can, this is just to simplify the life of users a >>>>>>>>>>>>>>> maximum by not having multiple pg_dump binaries in their serves. >>>>>>>>>>>>>>> Saying that, I think that there is no option to choose if >>>>>>>>>>>>>>> DISTRIBUTE BY is printed in the dump or not... >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yah if we choose to have an option we will put both >>>>>>>>>>>>>> DISTRIBUTE BY and TO NODE under it. >>>>>>>>>>>>>> >>>>>>>>>>>>> Why not an option for DISTRIBUTE BY, and another for TO NODE? >>>>>>>>>>>>> This would bring more flexibility to the way dumps are >>>>>>>>>>>>> generated. >>>>>>>>>>>>> -- >>>>>>>>>>>>> Michael >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>>> Everyone hates slow websites. So do we. >>>>>>>>>>>>> Make your web apps faster with AppDynamics >>>>>>>>>>>>> Download AppDynamics Lite for free today: >>>>>>>>>>>>> http://p.sf.net/sfu/appdyn_d2d_feb >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Postgres-xc-developers mailing list >>>>>>>>>>>>> Pos...@li... >>>>>>>>>>>>> >>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best Wishes, >>>>>>>>>>>> Ashutosh Bapat >>>>>>>>>>>> EntepriseDB Corporation >>>>>>>>>>>> The Enterprise Postgres Company >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> -- >>>>>>>>>>> Abbas >>>>>>>>>>> Architect >>>>>>>>>>> EnterpriseDB Corporation >>>>>>>>>>> The Enterprise PostgreSQL Company >>>>>>>>>>> >>>>>>>>>>> Phone: 92-334-5100153 >>>>>>>>>>> >>>>>>>>>>> Website: www.enterprisedb.com >>>>>>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>>>>>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>>>>>>>>>> >>>>>>>>>>> This e-mail message (and any attachment) is intended for the use >>>>>>>>>>> of >>>>>>>>>>> the individual or entity to whom it is addressed. This message >>>>>>>>>>> contains information from EnterpriseDB Corporation that may be >>>>>>>>>>> privileged, confidential, or exempt from disclosure under >>>>>>>>>>> applicable >>>>>>>>>>> law. If you are not the intended recipient or authorized to >>>>>>>>>>> receive >>>>>>>>>>> this for the intended recipient, any use, dissemination, >>>>>>>>>>> distribution, >>>>>>>>>>> retention, archiving, or copying of this communication is >>>>>>>>>>> strictly >>>>>>>>>>> prohibited. If you have received this e-mail in error, please >>>>>>>>>>> notify >>>>>>>>>>> the sender immediately by reply e-mail and delete this message. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Wishes, >>>>>>>>>> Ashutosh Bapat >>>>>>>>>> EntepriseDB Corporation >>>>>>>>>> The Enterprise Postgres Company >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> Abbas >>>>>>>>> Architect >>>>>>>>> EnterpriseDB Corporation >>>>>>>>> The Enterprise PostgreSQL Company >>>>>>>>> >>>>>>>>> Phone: 92-334-5100153 >>>>>>>>> >>>>>>>>> Website: www.enterprisedb.com >>>>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>>>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>>>>>>>> >>>>>>>>> This e-mail message (and any attachment) is intended for the use of >>>>>>>>> the individual or entity to whom it is addressed. This message >>>>>>>>> contains information from EnterpriseDB Corporation that may be >>>>>>>>> privileged, confidential, or exempt from disclosure under >>>>>>>>> applicable >>>>>>>>> law. If you are not the intended recipient or authorized to receive >>>>>>>>> this for the intended recipient, any use, dissemination, >>>>>>>>> distribution, >>>>>>>>> retention, archiving, or copying of this communication is strictly >>>>>>>>> prohibited. If you have received this e-mail in error, please >>>>>>>>> notify >>>>>>>>> the sender immediately by reply e-mail and delete this message. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Wishes, >>>>>>>> Ashutosh Bapat >>>>>>>> EntepriseDB Corporation >>>>>>>> The Enterprise Postgres Company >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Abbas >>>>>>> Architect >>>>>>> EnterpriseDB Corporation >>>>>>> The Enterprise PostgreSQL Company >>>>>>> >>>>>>> Phone: 92-334-5100153 >>>>>>> >>>>>>> Website: www.enterprisedb.com >>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>>>>>> >>>>>>> This e-mail message (and any attachment) is intended for the use of >>>>>>> the individual or entity to whom it is addressed. This message >>>>>>> contains information from EnterpriseDB Corporation that may be >>>>>>> privileged, confidential, or exempt from disclosure under applicable >>>>>>> law. If you are not the intended recipient or authorized to receive >>>>>>> this for the intended recipient, any use, dissemination, >>>>>>> distribution, >>>>>>> retention, archiving, or copying of this communication is strictly >>>>>>> prohibited. If you have received this e-mail in error, please notify >>>>>>> the sender immediately by reply e-mail and delete this message. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Wishes, >>>>>> Ashutosh Bapat >>>>>> EntepriseDB Corporation >>>>>> The Enterprise Postgres Company >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Abbas >>>>> Architect >>>>> EnterpriseDB Corporation >>>>> The Enterprise PostgreSQL Company >>>>> >>>>> Phone: 92-334-5100153 >>>>> >>>>> Website: www.enterprisedb.com >>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>>>> >>>>> This e-mail message (and any attachment) is intended for the use of >>>>> the individual or entity to whom it is addressed. This message >>>>> contains information from EnterpriseDB Corporation that may be >>>>> privileged, confidential, or exempt from disclosure under applicable >>>>> law. If you are not the intended recipient or authorized to receive >>>>> this for the intended recipient, any use, dissemination, distribution, >>>>> retention, archiving, or copying of this communication is strictly >>>>> prohibited. If you have received this e-mail in error, please notify >>>>> the sender immediately by reply e-mail and delete this message. >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> Abbas >>>> Architect >>>> EnterpriseDB Corporation >>>> The Enterprise PostgreSQL Company >>>> >>>> Phone: 92-334-5100153 >>>> >>>> Website: www.enterprisedb.com >>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>>> >>>> This e-mail message (and any attachment) is intended for the use of >>>> the individual or entity to whom it is addressed. This message >>>> contains information from EnterpriseDB Corporation that may be >>>> privileged, confidential, or exempt from disclosure under applicable >>>> law. If you are not the intended recipient or authorized to receive >>>> this for the intended recipient, any use, dissemination, distribution, >>>> retention, archiving, or copying of this communication is strictly >>>> prohibited. If you have received this e-mail in error, please notify >>>> the sender immediately by reply e-mail and delete this message. >>>> >>> >>> >>> >>> -- >>> -- >>> Abbas >>> Architect >>> EnterpriseDB Corporation >>> The Enterprise PostgreSQL Company >>> >>> Phone: 92-334-5100153 >>> >>> Website: www.enterprisedb.com >>> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >>> Follow us on Twitter: http://www.twitter.com/enterprisedb >>> >>> This e-mail message (and any attachment) is intended for the use of >>> the individual or entity to whom it is addressed. This message >>> contains information from EnterpriseDB Corporation that may be >>> privileged, confidential, or exempt from disclosure under applicable >>> law. If you are not the intended recipient or authorized to receive >>> this for the intended recipient, any use, dissemination, distribution, >>> retention, archiving, or copying of this communication is strictly >>> prohibited. If you have received this e-mail in error, please notify >>> the sender immediately by reply e-mail and delete this message. >> >> >> >> >> -- >> -- >> Abbas >> Architect >> EnterpriseDB Corporation >> The Enterprise PostgreSQL Company >> >> Phone: 92-334-5100153 >> >> Website: www.enterprisedb.com >> EnterpriseDB Blog: http://blogs.enterprisedb.com/ >> Follow us on Twitter: http://www.twitter.com/enterprisedb >> >> This e-mail message (and any attachment) is intended for the use of >> the individual or entity to whom it is addressed. This message >> contains information from EnterpriseDB Corporation that may be >> privileged, confidential, or exempt from disclosure under applicable >> law. If you are not the intended recipient or authorized to receive >> this for the intended recipient, any use, dissemination, distribution, >> retention, archiving, or copying of this communication is strictly >> prohibited. If you have received this e-mail in error, please notify >> the sender immediately by reply e-mail and delete this message. >> > > > > -- > -- > Abbas > Architect > EnterpriseDB Corporation > The Enterprise PostgreSQL Company > > Phone: 92-334-5100153 > > Website: www.enterprisedb.com > EnterpriseDB Blog: http://blogs.enterprisedb.com/ > Follow us on Twitter: http://www.twitter.com/enterprisedb > > This e-mail message (and any attachment) is intended for the use of > the individual or entity to whom it is addressed. This message > contains information from EnterpriseDB Corporation that may be > privileged, confidential, or exempt from disclosure under applicable > law. If you are not the intended recipient or authorized to receive > this for the intended recipient, any use, dissemination, distribution, > retention, archiving, or copying of this communication is strictly > prohibited. If you have received this e-mail in error, please notify > the sender immediately by reply e-mail and delete this message. > -- Best Wishes, Ashutosh Bapat EntepriseDB Corporation The Enterprise Postgres Company |
