postgres-xc-bugs — Bug report

Re: [Postgres-xc-bugs] Corrupt Objects

[West, W. <ww...@uc...>]

Sorry forget the version in my last response:

psql (PGXC 1.1, based on PG 9.2.4)


Regards,

bw

On 11/13/13, 3:39 PM, "West, William" <ww...@uc...> wrote:

>I am deploying a database to a production server that was developed
>without a problem on my dev platform. There are a couple of schemas and 4
>tables. I can create and populate the tables but once I apply a primary
>key to one of the tables, it is no longer available for any DML operation.
>Apparently it cannot find the primary key wherever it is looking for it.
>If I recreate the table under a new name and populate and apply the
>primary key it works fine but now I am stuck with two useless objects
>which will interfere with maintenance routines down the line.
>
>My production environment is fairly unique in that I am deploying this
>instance on a new supercomputer architecture and this could very possible
>be the source of this instability (since this all works on my dev cluster
>which is just a couple of linux VM nodes). However, I don¹t know where to
>look at the metadata or logs to find out what is wrong.
>
>I have reinitialized and rebuilt the database a number of time but this
>won¹t be a real option when the project goes live (I hope) since it takes
>quite a bit of time even when doing a restore. Is there any manual way to
>remove these objects (limited to the table and primary key)from metadata
>tables and therefore eliminate them from the instance?
>
>Thanks,
>
>bw
>
>On 11/13/13, 3:27 PM, "Michael Paquier" <mic...@gm...> wrote:
>
>>On Thu, Nov 14, 2013 at 5:34 AM, West, William <ww...@uc...> wrote:
>>> drop table gene.mutect cascade;
>>> ERROR: cache lookup failed for relation 24594
>>> It doesn¹t seemed based on what I read that there is a mechanism to
>>>drop
>>> corrupted objects. I hope this isn¹t the case since being unable to
>>>remove
>>> invalid objects would make this product unsuitable for production work.
>>This is clearly a bug. This error appears because even if this session
>>has taken a lock on this relation that should guarantee its presence,
>>well it is not here, hence the error.
>>
>>Telling first what is the version used, and then giving an explicit
>>test case would help the dev team to solve this issue.
>>Regards,
>>-- 
>>Michael
>

Re: [Postgres-xc-bugs] Corrupt Objects

[West, W. <ww...@uc...>]

I am deploying a database to a production server that was developed
without a problem on my dev platform. There are a couple of schemas and 4
tables. I can create and populate the tables but once I apply a primary
key to one of the tables, it is no longer available for any DML operation.
Apparently it cannot find the primary key wherever it is looking for it.
If I recreate the table under a new name and populate and apply the
primary key it works fine but now I am stuck with two useless objects
which will interfere with maintenance routines down the line.

My production environment is fairly unique in that I am deploying this
instance on a new supercomputer architecture and this could very possible
be the source of this instability (since this all works on my dev cluster
which is just a couple of linux VM nodes). However, I don¹t know where to
look at the metadata or logs to find out what is wrong.

I have reinitialized and rebuilt the database a number of time but this
won¹t be a real option when the project goes live (I hope) since it takes
quite a bit of time even when doing a restore. Is there any manual way to
remove these objects (limited to the table and primary key)from metadata
tables and therefore eliminate them from the instance?

Thanks,

bw

On 11/13/13, 3:27 PM, "Michael Paquier" <mic...@gm...> wrote:

>On Thu, Nov 14, 2013 at 5:34 AM, West, William <ww...@uc...> wrote:
>> drop table gene.mutect cascade;
>> ERROR: cache lookup failed for relation 24594
>> It doesn¹t seemed based on what I read that there is a mechanism to drop
>> corrupted objects. I hope this isn¹t the case since being unable to
>>remove
>> invalid objects would make this product unsuitable for production work.
>This is clearly a bug. This error appears because even if this session
>has taken a lock on this relation that should guarantee its presence,
>well it is not here, hence the error.
>
>Telling first what is the version used, and then giving an explicit
>test case would help the dev team to solve this issue.
>Regards,
>-- 
>Michael

Re: [Postgres-xc-bugs] Corrupt Objects

[Michael P. <mic...@gm...>]

On Thu, Nov 14, 2013 at 5:34 AM, West, William <ww...@uc...> wrote:
> drop table gene.mutect cascade;
> ERROR: cache lookup failed for relation 24594
> It doesn’t seemed based on what I read that there is a mechanism to drop
> corrupted objects. I hope this isn’t the case since being unable to remove
> invalid objects would make this product unsuitable for production work.
This is clearly a bug. This error appears because even if this session
has taken a lock on this relation that should guarantee its presence,
well it is not here, hence the error.

Telling first what is the version used, and then giving an explicit
test case would help the dev team to solve this issue.
Regards,
-- 
Michael

[Postgres-xc-bugs] Corrupt Objects

[West, W. <ww...@uc...>]

What is the best way do deal with this situation:

drop table gene.mutect cascade;


ERROR:  cache lookup failed for relation 24594



********** Error **********


ERROR: cache lookup failed for relation 24594

SQL state: XX000


It doesn’t seemed based on what I read that there is a mechanism to drop corrupted objects. I hope this isn’t the case since being unable to remove invalid objects would make this product unsuitable for production work.


Thanks,


Bill West

Re: [Postgres-xc-bugs] Incomplete transaction clean up during pgxc_lock_for_backup violation

[Koichi S. <koi...@gm...>]

If you would like to cleanup outstanding 2PCs, you should do the following:

1. Visit all the nodes (both coordinators and datanodes) and get what nodes
are involved in such 2PC TXN.   You can do this using EXECUTE DIRECT and
issue SELECT command to pg_prepared_xacts.   The information you've got is
the status at the coordinator you connected.

2. Connect direct to each of the node using psql as a superuser.   Isuue
"set xc_maintenance_mode to on" and then issue "rollback prepared".   This
is how to clean up such transactions.

If you're interested in, contrib includes pgxc_clean module, which will do
similar cleanup, although I've not tested it in your situation.   The above
method is more direct and could be helpful.

Regards;

---
Koichi Suzuki


2013/11/12 Víctor Daniel M. O. <vda...@gm...>

> Greetings to everyone, I'm new to the mail list,
>
> I was testing master branch of postgres-xc, and I think exist a strange
> behavior during transaction redo when Postgres-XC is violated trying to
> create a database while pgxc_lock_for_backup is active.
>
> GIT commit:
>
> $ git log | tail | grep ^commit
> commit d31084e9d1118b25fd16580d9d8c2924b5740dff
> $ git branch -vv
> * master a50bdd3 [origin/master] Replace strndup calls by strdup in
> makesgml
>
> $ uname -a
> Linux eva-01-nerv-seele 3.11.7-200.fc19.x86_64 #1 SMP Mon Nov 4 14:09:03
> UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>  ~/.opt/pgxc/bin/initgtm -D ~/.opt/pgxc/var/lib/pgsql/data_gtm
>  ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_datanode2
> --nodename datanode2
>  ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_datanode1
> --nodename datanode1
>  ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_coord1
> --nodename coord1
>
> ~/.opt/pgxc/bin/gtm -D ~/.opt/pgxc/var/lib/pgsql/data_gtm
> ~/.opt/pgxc/bin/postgres --coordinator -D
> ~/.opt/pgxc/var/lib/pgsql/data_coord1
> ~/.opt/pgxc/bin/postgres --datanode -D
> ~/.opt/pgxc/var/lib/pgsql/data_datanode1 -p 15432
>
>
> $ ~/.opt/pgxc/bin/psql postgres
> psql (PGXC 1.1devel, based on PG 9.2beta2)
> Type "help" for help.
>
> postgres=# CREATE NODE datanode1 WITH (TYPE = 'datanode', PORT = 15432);
> CREATE NODE
> postgres=# select pgxc_pool_reload();
>  pgxc_pool_reload
> ------------------
>  t
> (1 row)
>
> postgres=# select * from pg_prepared_xacts ;
>  transaction | gid | prepared | owner | database
> -------------+-----+----------+-------+----------
> (0 rows)
>
> postgres=# select pgxc_lock_for_backup();
> INFO:  please do not close this session until you are done adding the new
> node
>  pgxc_lock_for_backup
> ----------------------
>  t
> (1 row)
>
> postgres=# select txid_current();
>  txid_current
> --------------
>         10011
> (1 row)
>
> postgres=# create database test;
> PANIC:  we seem to have dropped a bit somewhere
> PANIC:  we seem to have dropped a bit somewhere
> The connection to the server was lost. Attempting reset: Failed.
> !> \q
>
> --- LOGS IN DIRECTOR ---
> LOG:  server process (PID 23684) was terminated by signal 6: Aborted
> DETAIL:  Failed process was running: create database test;
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of another server process
> DETAIL:  The postmaster has commanded this server process to roll back the
> current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
> HINT:  In a moment you should be able to reconnect to the database and
> repeat your command.
> FATAL:  the database system is in recovery mode
> LOG:  all server processes terminated; reinitializing
> LOG:  database system was interrupted; last known up at 2013-11-11
> 21:38:21 COT
> LOG:  database system was not properly shut down; automatic recovery in
> progress
> LOG:  redo starts at 0/189C940
> LOG:  record with zero length at 0/189CB18
> LOG:  redo done at 0/189C940
> LOG:  recovering prepared transaction 10012
> LOG:  database system is ready to accept connections
> LOG:  autovacuum launcher started
> ---
>
> $ ~/.opt/pgxc/bin/psql postgres
> psql (PGXC 1.1devel, based on PG 9.2beta2)
> Type "help" for help.
>
> postgres=# select * from pg_prepared_xacts ;
>  transaction |  gid   |           prepared            | owner | database
> -------------+--------+-------------------------------+-------+----------
>        10012 | T10012 | 2013-11-11 21:38:22.246206-05 | vdmo  | postgres
> (1 row)
>
> postgres=# rollback prepared 'T10012';
> ERROR:  prepared transaction with identifier "T10012" does not exist
>
> Unfortunately from here not is possible to rollback or commit the
> transaction, and not allow run again pgxc_lock_for_backup because exists a
> transaction in prepare stage.
>
> Excuse my ignorance but is this a problem or normal expected behavior?.
>
> Thks,
>
> Daniel.
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models.
> Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-bugs mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-bugs
>
>

[Postgres-xc-bugs] Incomplete transaction clean up during pgxc_lock_for_backup violation

[Víctor D. M. O. <vda...@gm...>]

Greetings to everyone, I'm new to the mail list,

I was testing master branch of postgres-xc, and I think exist a strange
behavior during transaction redo when Postgres-XC is violated trying to
create a database while pgxc_lock_for_backup is active.

GIT commit:

$ git log | tail | grep ^commit
commit d31084e9d1118b25fd16580d9d8c2924b5740dff
$ git branch -vv
* master a50bdd3 [origin/master] Replace strndup calls by strdup in makesgml

$ uname -a
Linux eva-01-nerv-seele 3.11.7-200.fc19.x86_64 #1 SMP Mon Nov 4 14:09:03
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

 ~/.opt/pgxc/bin/initgtm -D ~/.opt/pgxc/var/lib/pgsql/data_gtm
 ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_datanode2
--nodename datanode2
 ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_datanode1
--nodename datanode1
 ~/.opt/pgxc/bin/initdb -D ~/.opt/pgxc/var/lib/pgsql/data_coord1 --nodename
coord1

~/.opt/pgxc/bin/gtm -D ~/.opt/pgxc/var/lib/pgsql/data_gtm
~/.opt/pgxc/bin/postgres --coordinator -D
~/.opt/pgxc/var/lib/pgsql/data_coord1
~/.opt/pgxc/bin/postgres --datanode -D
~/.opt/pgxc/var/lib/pgsql/data_datanode1 -p 15432


$ ~/.opt/pgxc/bin/psql postgres
psql (PGXC 1.1devel, based on PG 9.2beta2)
Type "help" for help.

postgres=# CREATE NODE datanode1 WITH (TYPE = 'datanode', PORT = 15432);
CREATE NODE
postgres=# select pgxc_pool_reload();
 pgxc_pool_reload
------------------
 t
(1 row)

postgres=# select * from pg_prepared_xacts ;
 transaction | gid | prepared | owner | database
-------------+-----+----------+-------+----------
(0 rows)

postgres=# select pgxc_lock_for_backup();
INFO:  please do not close this session until you are done adding the new
node
 pgxc_lock_for_backup
----------------------
 t
(1 row)

postgres=# select txid_current();
 txid_current
--------------
        10011
(1 row)

postgres=# create database test;
PANIC:  we seem to have dropped a bit somewhere
PANIC:  we seem to have dropped a bit somewhere
The connection to the server was lost. Attempting reset: Failed.
!> \q

--- LOGS IN DIRECTOR ---
LOG:  server process (PID 23684) was terminated by signal 6: Aborted
DETAIL:  Failed process was running: create database test;
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and
repeat your command.
FATAL:  the database system is in recovery mode
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2013-11-11 21:38:21
COT
LOG:  database system was not properly shut down; automatic recovery in
progress
LOG:  redo starts at 0/189C940
LOG:  record with zero length at 0/189CB18
LOG:  redo done at 0/189C940
LOG:  recovering prepared transaction 10012
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
---

$ ~/.opt/pgxc/bin/psql postgres
psql (PGXC 1.1devel, based on PG 9.2beta2)
Type "help" for help.

postgres=# select * from pg_prepared_xacts ;
 transaction |  gid   |           prepared            | owner | database
-------------+--------+-------------------------------+-------+----------
       10012 | T10012 | 2013-11-11 21:38:22.246206-05 | vdmo  | postgres
(1 row)

postgres=# rollback prepared 'T10012';
ERROR:  prepared transaction with identifier "T10012" does not exist

Unfortunately from here not is possible to rollback or commit the
transaction, and not allow run again pgxc_lock_for_backup because exists a
transaction in prepare stage.

Excuse my ignorance but is this a problem or normal expected behavior?.

Thks,

Daniel.

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Michael P. <mic...@gm...>]

On Fri, Nov 8, 2013 at 5:49 AM, DIMI <nic...@un...> wrote:
> I assume that the release tarballs are those that can be downloaded at http://sourceforge.net/apps/mediawiki/postgres-xc/index.php?title=Download.
Yeah.

> I don’t want to appear pedantic, but the archive for version 1.0.3 includes the sgml files inside doc-xc/src/sgml; the archive for version 1.1 does not. I can ‘make world’ without Jade in the former case, but I cannot in the latter.
I didn't know. In this case it is definitely an error from the
distribution tarball of 1.1. Normally, a tarball that is distributed
inside source forge should be build with "make distprep", and it
builds sgml, man and html docs such as people can run make world by
default without a dependency on jade. That being said...
-- 
Michael

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[DIMI <nic...@un...>]

On 06/nov/2013, at 23:19, Michael Paquier <mic...@gm...> wrote:

> On Wed, Nov 6, 2013 at 11:29 PM, DIMI <nic...@un...> wrote:
>> The above looks like a good compromise to me. But sgml files are missing from the v1.1. tarball. Is that intentional?
> sgml files, html documentation and man pages should be incuded in
> release tarballs :)

I assume that the release tarballs are those that can be downloaded at http://sourceforge.net/apps/mediawiki/postgres-xc/index.php?title=Download.
I don’t want to appear pedantic, but the archive for version 1.0.3 includes the sgml files inside doc-xc/src/sgml; the archive for version 1.1 does not. I can ‘make world’ without Jade in the former case, but I cannot in the latter.

Nicola

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Michael P. <mic...@gm...>]

On Wed, Nov 6, 2013 at 11:29 PM, DIMI <nic...@un...> wrote:
> The above looks like a good compromise to me. But sgml files are missing from the v1.1. tarball. Is that intentional?
sgml files, html documentation and man pages should be incuded in
release tarballs :)
-- 
Michael

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[DIMI <nic...@un...>]

On 06/nov/2013, at 13:46, Michael Paquier <mic...@gm...> wrote:

> On Wed, Nov 6, 2013 at 7:32 PM, Nikhil Sontakke <ni...@st...> wrote:
>> Nicola,
>> 
>> You are asking it to "make the world". So, it will try to generate the
>> documentation, for which it will need [open]jade. I don't think you can do
>> away with it, if you want "install-world".
> Agreed. make world => docs => jade.

Ok, that makes sense. In fact, I should have said “skipping jade” rather than “removing jade dependency”. Which was this commit’s purpose:

https://github.com/postgres-xc/postgres-xc/commit/1e42775468b013c840a3c170108fd439716c9c04

The above looks like a good compromise to me. But sgml files are missing from the v1.1. tarball. Is that intentional?

Nicola

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Michael P. <mic...@gm...>]

On Wed, Nov 6, 2013 at 7:32 PM, Nikhil Sontakke <ni...@st...> wrote:
> Nicola,
>
> You are asking it to "make the world". So, it will try to generate the
> documentation, for which it will need [open]jade. I don't think you can do
> away with it, if you want "install-world".
Agreed. make world => docs => jade.
-- 
Michael

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Nikhil S. <ni...@st...>]

Nicola,

You are asking it to "make the world". So, it will try to generate the
documentation, for which it will need [open]jade. I don't think you can do
away with it, if you want "install-world".

Regards,
Nikhils


On Wed, Nov 6, 2013 at 2:35 PM, DIMI <nic...@un...> wrote:

> On 05/nov/2013, at 14:44, Michael Paquier <mic...@gm...>
> wrote:
>
> > On Tue, Nov 5, 2013 at 5:06 AM, Nikhil Sontakke <ni...@st...>
> wrote:
> >> Hi Michael,
> >>
> >> IIRC, setting up docbook, openjade was a little bit of a pain when I
> did it
> >> a while ago. However try the below and see if it helps.
> >>
> >> port install docbook-sgml-4.2 openjade
> > So that's macports, thanks. I'm just using brew, and there is no
> > formula for jade there :)
>
> The ./configure script correctly detects whether jade & co. are installed:
>
> […]
> checking for onsgmls... no
> checking for nsgmls... no
> checking for openjade... no
> checking for jade... no
> checking for DocBook V4.2... no
> checking for DocBook stylesheets... no
> checking for collateindex.pl... /opt/local/bin/collateindex.pl
> checking for xsltproc... xsltproc
> checking for osx... no
> checking for sgml2xml... no
> checking for sx... no
> […]
>
> so I think that 'make install-world' should not attempt to call (open)jade
> when one is not available.
>
> Jade is not in Homebrew, so to upgrade Postgres-XC I would have to change
> the formula to use ‘make install’ instead of ‘make install-world’, which is
> not ideal. Since this problem has occurred in the past (see
> http://sourceforge.net/mailarchive/message.php?msg_id=29327650), and it
> was fixed, I am wondering if the jade dependency can be removed.
>
> Nicola




-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[DIMI <nic...@un...>]

On 05/nov/2013, at 14:44, Michael Paquier <mic...@gm...> wrote:

> On Tue, Nov 5, 2013 at 5:06 AM, Nikhil Sontakke <ni...@st...> wrote:
>> Hi Michael,
>> 
>> IIRC, setting up docbook, openjade was a little bit of a pain when I did it
>> a while ago. However try the below and see if it helps.
>> 
>> port install docbook-sgml-4.2 openjade
> So that's macports, thanks. I'm just using brew, and there is no
> formula for jade there :)

The ./configure script correctly detects whether jade & co. are installed:

[…]
checking for onsgmls... no
checking for nsgmls... no
checking for openjade... no
checking for jade... no
checking for DocBook V4.2... no
checking for DocBook stylesheets... no
checking for collateindex.pl... /opt/local/bin/collateindex.pl
checking for xsltproc... xsltproc
checking for osx... no
checking for sgml2xml... no
checking for sx... no
[…]

so I think that 'make install-world' should not attempt to call (open)jade when one is not available.

Jade is not in Homebrew, so to upgrade Postgres-XC I would have to change the formula to use ‘make install’ instead of ‘make install-world’, which is not ideal. Since this problem has occurred in the past (see http://sourceforge.net/mailarchive/message.php?msg_id=29327650), and it was fixed, I am wondering if the jade dependency can be removed.

Nicola

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Michael P. <mic...@gm...>]

On Tue, Nov 5, 2013 at 5:06 AM, Nikhil Sontakke <ni...@st...> wrote:
> Hi Michael,
>
> IIRC, setting up docbook, openjade was a little bit of a pain when I did it
> a while ago. However try the below and see if it helps.
>
> port install docbook-sgml-4.2 openjade
So that's macports, thanks. I'm just using brew, and there is no
formula for jade there :)
-- 
Michael

Re: [Postgres-xc-bugs] [Postgres-xc-developers] Repairing Corrupt Objects

[鈴木 幸市 <ko...@in...>]

There could be several different cause of the problem.

First of all, to ensure that there's no more outstanding 2PCs, could you visit all the nodes (coordinators and datanodes) directly with psql and see if there's no running 2PCs in any of the nodes?

If you find any, you should rollback all of them using the same technique.

If not, could you restart the whole cluster?    I advise to use -m immediate or -m fast option to stop them with pg_ctl.     Then restart the cluster and see what's going on.

I believe only 2PC-repated resources are carried over when nodes are restarted and all the others are cleared up.

I do hope you a allowed to do so.

Regards;
---
Koichi Suzuki

On 2013/11/05, at 12:15, "West, William" <ww...@uc...<mailto:ww...@uc...>> wrote:

Koichi,

Yes it is related. I used your solution and it did rollback the long running prepared statement. I assumed this might have been causing an unregistered block on the table I was attempting to run queries against or vacuum. Unfortunately though this did not appear to alleviate the situation below. I have a single table affected by this. The other object in the database seem to work as expected. If I try to run a query against this table or vacuum it creates 2 pids like so:

505149   19636  0.0  0.0 192248  6824 ?        Ss   18:26   0:00 postgres: bigonc_prd bigonc 72.220.234.184(49302) VACUUM
505149   19889  0.0  0.0 191872  4992 ?        Ss   19:03   0:00 postgres: bigonc_prd bigonc 127.0.0.1(57250) VACUUM waiting

To me this looks like a process is pawned from the client while another process is started on the server. The client process seems to be causing a blocking like against the server pid. This is the output from PG_LOCKS for the client process:

“virtualxid" "3/324” "3/324” 19636 “ExclusiveLock” t t
“transactionid" “” 63030 "3/324” 19636 “ExclusiveLock” t f

Thanks for your help with the other issue as well,

Bill West

From: 鈴木 幸市 <ko...@in...<mailto:ko...@in...>>
Date: Monday, November 4, 2013 at 6:36 PM
To: William West <ww...@uc...<mailto:ww...@uc...>>
Cc: "pos...@li...<mailto:pos...@li...>" <pos...@li...<mailto:pos...@li...>>, "pos...@li...<mailto:pos...@li...>" <pos...@li...<mailto:pos...@li...>>
Subject: Re: [Postgres-xc-developers] Repairing Corrupt Objects

Is this related to the outstanding 2PC three days ago, which I responded with a solution?   If not, could you identify corrupt database object?

Regards;
---
Koichi Suzuki

On 2013/11/02, at 7:34, "West, William" <ww...@uc...<mailto:ww...@uc...>> wrote:

All,

Sorry for the multiple posting but the first emails seemed to get garbled. Hopefully this will post with the full text. Please ignore the other posts with this topic.

I have a database set up but it seems fairly unstable. I have a number of objects in the running database but I have one object that is corrupt. I know this because doing any query or dml statement on this table stalls the client. I also notice that when I run statements that encounter the invalid object. It spawns 2 process ID like below:

505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00 postgres:  postgresql bigonc [local] VACUUM
505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres: postgresql bigonc 127.0.0.1(46305) VACUUM waiting

These processes never complete and seem like they are locking one another. However there is nothing showing in pg_locks. Does anyone recognize this behavior. Is there anyway to repair or drop a corrupt object in the database?

Thanks,
Bill West
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Nikhil S. <ni...@st...>]

Hi Michael,

IIRC, setting up docbook, openjade was a little bit of a pain when I did it
a while ago. However try the below and see if it helps.

port install docbook-sgml-4.2 openjade

Regards,
Nikhils


On Tue, Nov 5, 2013 at 6:58 AM, Michael Paquier
<mic...@gm...>wrote:

> On Mon, Nov 4, 2013 at 2:05 PM, Nikhil Sontakke <ni...@st...>
> wrote:
> > completes fine for me on Maverick. I can see openjade being used in my
> case
> > instead of jade. So you might have to install openjade.
> Just wondering: where are you fetching your version of jade/openjade.
> It is not in homebrew. macports perhaps or directly xcode? I got the
> developer tools installed but it doesn't seem there is any jade
> package.
> --
> Michael
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-bugs] [Postgres-xc-developers] Repairing Corrupt Objects

[West, W. <ww...@uc...>]

Koichi,

Yes it is related. I used your solution and it did rollback the long running prepared statement. I assumed this might have been causing an unregistered block on the table I was attempting to run queries against or vacuum. Unfortunately though this did not appear to alleviate the situation below. I have a single table affected by this. The other object in the database seem to work as expected. If I try to run a query against this table or vacuum it creates 2 pids like so:

505149   19636  0.0  0.0 192248  6824 ?        Ss   18:26   0:00 postgres: bigonc_prd bigonc 72.220.234.184(49302) VACUUM
505149   19889  0.0  0.0 191872  4992 ?        Ss   19:03   0:00 postgres: bigonc_prd bigonc 127.0.0.1(57250) VACUUM waiting

To me this looks like a process is pawned from the client while another process is started on the server. The client process seems to be causing a blocking like against the server pid. This is the output from PG_LOCKS for the client process:

“virtualxid" "3/324” "3/324” 19636 “ExclusiveLock” t t
“transactionid" “” 63030 "3/324” 19636 “ExclusiveLock” t f

Thanks for your help with the other issue as well,

Bill West

From: 鈴木 幸市 <ko...@in...<mailto:ko...@in...>>
Date: Monday, November 4, 2013 at 6:36 PM
To: William West <ww...@uc...<mailto:ww...@uc...>>
Cc: "pos...@li...<mailto:pos...@li...>" <pos...@li...<mailto:pos...@li...>>, "pos...@li...<mailto:pos...@li...>" <pos...@li...<mailto:pos...@li...>>
Subject: Re: [Postgres-xc-developers] Repairing Corrupt Objects

Is this related to the outstanding 2PC three days ago, which I responded with a solution?   If not, could you identify corrupt database object?

Regards;
---
Koichi Suzuki

On 2013/11/02, at 7:34, "West, William" <ww...@uc...<mailto:ww...@uc...>> wrote:

All,

Sorry for the multiple posting but the first emails seemed to get garbled. Hopefully this will post with the full text. Please ignore the other posts with this topic.

I have a database set up but it seems fairly unstable. I have a number of objects in the running database but I have one object that is corrupt. I know this because doing any query or dml statement on this table stalls the client. I also notice that when I run statements that encounter the invalid object. It spawns 2 process ID like below:

505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00 postgres:  postgresql bigonc [local] VACUUM
505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres: postgresql bigonc 127.0.0.1(46305) VACUUM waiting

These processes never complete and seem like they are locking one another. However there is nothing showing in pg_locks. Does anyone recognize this behavior. Is there anyway to repair or drop a corrupt object in the database?

Thanks,
Bill West
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Re: [Postgres-xc-bugs] [Postgres-xc-developers] Repairing Corrupt Objects

[鈴木 幸市 <ko...@in...>]

Is this related to the outstanding 2PC three days ago, which I responded with a solution?   If not, could you identify corrupt database object?

Regards;
---
Koichi Suzuki

On 2013/11/02, at 7:34, "West, William" <ww...@uc...<mailto:ww...@uc...>> wrote:

All,

Sorry for the multiple posting but the first emails seemed to get garbled. Hopefully this will post with the full text. Please ignore the other posts with this topic.

I have a database set up but it seems fairly unstable. I have a number of objects in the running database but I have one object that is corrupt. I know this because doing any query or dml statement on this table stalls the client. I also notice that when I run statements that encounter the invalid object. It spawns 2 process ID like below:

505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00 postgres:  postgresql bigonc [local] VACUUM
505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres: postgresql bigonc 127.0.0.1(46305) VACUUM waiting

These processes never complete and seem like they are locking one another. However there is nothing showing in pg_locks. Does anyone recognize this behavior. Is there anyway to repair or drop a corrupt object in the database?

Thanks,
Bill West
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Michael P. <mic...@gm...>]

On Mon, Nov 4, 2013 at 2:05 PM, Nikhil Sontakke <ni...@st...> wrote:
> completes fine for me on Maverick. I can see openjade being used in my case
> instead of jade. So you might have to install openjade.
Just wondering: where are you fetching your version of jade/openjade.
It is not in homebrew. macports perhaps or directly xcode? I got the
developer tools installed but it doesn't seem there is any jade
package.
-- 
Michael

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Koichi S. <koi...@gm...>]

Also please check if you have sufficient resource to handle docbook.   It
is a separate resource from open jade.

Regards;

---
Koichi Suzuki


2013/11/4 Nikhil Sontakke <ni...@st...>

> Hi Nicola,
>
> make install-world
>
> completes fine for me on Maverick. I can see openjade being used in my
> case instead of jade. So you might have to install openjade.
>
> HTH,
> Nikhils
>
>
>
>
> On Sun, Nov 3, 2013 at 6:26 PM, DIMI <nic...@un...> wrote:
>
>> Hi,
>> I’m trying to update Homebrew’s formula for Postgres-XC to build v1.1.
>> After following the suggestion from Nikhil at
>> http://sourceforge.net/mailarchive/forum.php?thread_name=82E44F89-543A-44F2-8AF8-F6909B5DC561%40uniud.it&forum_name=postgres-xc-bugs,
>> I can get a successful build if I use:
>>
>> ./configure <args>
>> make
>> make install
>>
>> but the build still fails if I use
>>
>> ./configure <args>
>> make install-world
>>
>> (which is what the formula does currently). When I try manually I get:
>>
>> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C doc-xc install
>> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C src install
>> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C sgml install
>> { \
>>           echo "<!ENTITY version \"1.1\">"; \
>>           echo "<!ENTITY majorversion \"9.2\">"; \
>>         } > version.sgml
>> '/usr/bin/perl' ./mk_feature_tables.pl YES
>> ../../../src/backend/catalog/sql_feature_packages.txt
>> ../../../src/backend/catalog/sql_features.txt > features-supported.sgml
>> '/usr/bin/perl' ./mk_feature_tables.pl NO
>> ../../../src/backend/catalog/sql_feature_packages.txt
>> ../../../src/backend/catalog/sql_features.txt > features-unsupported.sgml
>> '/usr/bin/perl' ./generate-errcodes-table.pl../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
>> jade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D .  -d
>> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
>> make[3]: jade: No such file or directory
>> make[3]: *** [HTML.index] Error 1
>> make[2]: *** [install] Error 2
>> make[1]: *** [install] Error 2
>> make: *** [install-world-doc-xc-recurse] Error 2
>>
>> I thought this issue with Jade was fixed with 1.0. Is this a regression?
>>
>> Nicola
>>
>> ------------------------------------------------------------------------------
>> Android is increasing in popularity, but the open development platform
>> that
>> developers love is also attractive to malware creators. Download this
>> white
>> paper to learn more about secure code signing practices that can help keep
>> Android apps secure.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-bugs mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-bugs
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-bugs mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-bugs
>
>

Re: [Postgres-xc-bugs] Repairing Corrupt Objects

[Koichi S. <koi...@gm...>]

Thanks for this detailed info.

Could you try to run pgxc_clean to clean-up such locally-existing 2PC?   If
you'd like to abort such outstanding 2PCs, you can do the following:

1. Connect to the node where you outstanding 2PC exists as a super-user.
2. Set 'xc_maintenance_mode' to "on".   This enables local maintenance
work.   Please note that you cannot set this to ON in your postgresql.conf
file.   You can set this to ON only within a session.
3. Issue ROLLBACK PREPARED command.

I hope this works.   This is basically what pgxc_clean is doing but you can
do it manually too.

Pgxc_clean will be found in contrib directory.

Good luck;

---
Koichi Suzuki


2013/11/2 West, William <ww...@uc...>

>  I think I may have found the source of my problem but not the solution.
> There seems to be a prepared transaction running for days in my XC
> instance. I tried to roll it back but got the following:
>
>  bigonc=# select * from pg_prepared_xacts;
>  transaction |  gid   |           prepared            |   owner    |
> database
>
> -------------+--------+-------------------------------+------------+----------
>        11248 | T11248 | 2013-10-25 14:35:14.467445-07 | bigonc_prd | bigonc
> (1 row)
>
>  bigonc=# rollback prepared '11248';
> ERROR:  prepared transaction with identifier "11248" does not exist
> STATEMENT:  rollback prepared '11248';
> ERROR:  prepared transaction with identifier "11248" does not exist
>
>  It cannot find the transaction id. I am assuming it is the first field
> in the table but since it doesn’t have the work ID in it, I could be wrong.
> Stopping and starting the server has no impact on this transaction. What
> other methods might be available to me short of reinitializing the database
> to clear this impediment.
>
>  Thanks,
>
>  Bill West
>
>   From: <West>, William West <ww...@uc...>
> Date: Friday, November 1, 2013 at 10:56 AM
> To: "pos...@li..." <
> pos...@li...>
> Subject: Repairing Corrupt Objects
>
>   All,
>
>  I have a database set up but it seems fairly unstable. I have a number
> of objects in the running database but I have one object that is corrupt. I
> know this because doing any query or dml statement on this table stalls the
> client. I also notice that when I run statements that encounter the invalid
> object. It spawns 2 process ID like below:
>
>  505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00
> postgres:  postgresql bigonc [local] VACUUM
> 505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres:
> postgresql bigonc 127.0.0.1(46305) VACUUM waiting
>
>  These processes never complete and seem like they are locking one
> another. However there is nothing showing in pg_locks. Does anyone
> recognize this behavior. Is there anyway to repair or drop a corrupt
> object in the database?
>
>  Thanks,
> Bill West
>
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-bugs mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-bugs
>
>

Re: [Postgres-xc-bugs] Building on OS X Mavericks

[Nikhil S. <ni...@st...>]

Hi Nicola,

make install-world

completes fine for me on Maverick. I can see openjade being used in my case
instead of jade. So you might have to install openjade.

HTH,
Nikhils




On Sun, Nov 3, 2013 at 6:26 PM, DIMI <nic...@un...> wrote:

> Hi,
> I’m trying to update Homebrew’s formula for Postgres-XC to build v1.1.
> After following the suggestion from Nikhil at
> http://sourceforge.net/mailarchive/forum.php?thread_name=82E44F89-543A-44F2-8AF8-F6909B5DC561%40uniud.it&forum_name=postgres-xc-bugs,
> I can get a successful build if I use:
>
> ./configure <args>
> make
> make install
>
> but the build still fails if I use
>
> ./configure <args>
> make install-world
>
> (which is what the formula does currently). When I try manually I get:
>
> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C doc-xc install
> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C src install
> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C sgml install
> { \
>           echo "<!ENTITY version \"1.1\">"; \
>           echo "<!ENTITY majorversion \"9.2\">"; \
>         } > version.sgml
> '/usr/bin/perl' ./mk_feature_tables.pl YES
> ../../../src/backend/catalog/sql_feature_packages.txt
> ../../../src/backend/catalog/sql_features.txt > features-supported.sgml
> '/usr/bin/perl' ./mk_feature_tables.pl NO
> ../../../src/backend/catalog/sql_feature_packages.txt
> ../../../src/backend/catalog/sql_features.txt > features-unsupported.sgml
> '/usr/bin/perl' ./generate-errcodes-table.pl../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
> jade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D .  -d
> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
> make[3]: jade: No such file or directory
> make[3]: *** [HTML.index] Error 1
> make[2]: *** [install] Error 2
> make[1]: *** [install] Error 2
> make: *** [install-world-doc-xc-recurse] Error 2
>
> I thought this issue with Jade was fixed with 1.0. Is this a regression?
>
> Nicola
>
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-bugs mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-bugs
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

[Postgres-xc-bugs] Building on OS X Mavericks

[DIMI <nic...@un...>]

Hi,
I’m trying to update Homebrew’s formula for Postgres-XC to build v1.1. After following the suggestion from Nikhil at http://sourceforge.net/mailarchive/forum.php?thread_name=82E44F89-543A-44F2-8AF8-F6909B5DC561%40uniud.it&forum_name=postgres-xc-bugs, I can get a successful build if I use:

./configure <args>
make
make install

but the build still fails if I use

./configure <args>
make install-world

(which is what the formula does currently). When I try manually I get:

/Applications/Xcode.app/Contents/Developer/usr/bin/make -C doc-xc install
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src install
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C sgml install
{ \
	  echo "<!ENTITY version \"1.1\">"; \
	  echo "<!ENTITY majorversion \"9.2\">"; \
	} > version.sgml
'/usr/bin/perl' ./mk_feature_tables.pl YES ../../../src/backend/catalog/sql_feature_packages.txt ../../../src/backend/catalog/sql_features.txt > features-supported.sgml
'/usr/bin/perl' ./mk_feature_tables.pl NO ../../../src/backend/catalog/sql_feature_packages.txt ../../../src/backend/catalog/sql_features.txt > features-unsupported.sgml
'/usr/bin/perl' ./generate-errcodes-table.pl ../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
jade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D .  -d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
make[3]: jade: No such file or directory
make[3]: *** [HTML.index] Error 1
make[2]: *** [install] Error 2
make[1]: *** [install] Error 2
make: *** [install-world-doc-xc-recurse] Error 2

I thought this issue with Jade was fixed with 1.0. Is this a regression?

Nicola

[Postgres-xc-bugs] Repairing Corrupt Objects

[West, W. <ww...@uc...>]

I think I may have found the source of my problem but not the solution. There seems to be a prepared transaction running for days in my XC instance. I tried to roll it back but got the following:

bigonc=# select * from pg_prepared_xacts;
 transaction |  gid   |           prepared            |   owner    | database
-------------+--------+-------------------------------+------------+----------
       11248 | T11248 | 2013-10-25 14:35:14.467445-07 | bigonc_prd | bigonc
(1 row)

bigonc=# rollback prepared '11248';
ERROR:  prepared transaction with identifier "11248" does not exist
STATEMENT:  rollback prepared '11248';
ERROR:  prepared transaction with identifier "11248" does not exist

It cannot find the transaction id. I am assuming it is the first field in the table but since it doesn’t have the work ID in it, I could be wrong. Stopping and starting the server has no impact on this transaction. What other methods might be available to me short of reinitializing the database to clear this impediment.

Thanks,

Bill West

From: <West>, William West <ww...@uc...<mailto:ww...@uc...>>
Date: Friday, November 1, 2013 at 10:56 AM
To: "pos...@li...<mailto:pos...@li...>" <pos...@li...<mailto:pos...@li...>>
Subject: Repairing Corrupt Objects

All,

I have a database set up but it seems fairly unstable. I have a number of objects in the running database but I have one object that is corrupt. I know this because doing any query or dml statement on this table stalls the client. I also notice that when I run statements that encounter the invalid object. It spawns 2 process ID like below:

505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00 postgres:  postgresql bigonc [local] VACUUM
505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres: postgresql bigonc 127.0.0.1(46305) VACUUM waiting

These processes never complete and seem like they are locking one another. However there is nothing showing in pg_locks. Does anyone recognize this behavior. Is there anyway to repair or drop a corrupt object in the database?

Thanks,
Bill West

[Postgres-xc-bugs] Repairing Corrupt Objects

[West, W. <ww...@uc...>]

All,

Sorry for the multiple posting but the first emails seemed to get garbled. Hopefully this will post with the full text. Please ignore the other posts with this topic.

I have a database set up but it seems fairly unstable. I have a number of objects in the running database but I have one object that is corrupt. I know this because doing any query or dml statement on this table stalls the client. I also notice that when I run statements that encounter the invalid object. It spawns 2 process ID like below:

505149   19373  0.0  0.0 191768  5908 ?        Ss   10:30   0:00 postgres:  postgresql bigonc [local] VACUUM
505149   19379  1.2  0.0 191972 24776 ?        Ss   10:30   0:00 postgres: postgresql bigonc 127.0.0.1(46305) VACUUM waiting

These processes never complete and seem like they are locking one another. However there is nothing showing in pg_locks. Does anyone recognize this behavior. Is there anyway to repair or drop a corrupt object in the database?

Thanks,
Bill West