Menu
▾
▴
Re: [Postgres-xc-developers] PGXC hangs when run with concurrent inserts
|
From: Ashutosh B. <ash...@en...> - 2013-04-12 03:25:55
|
Are the FK table and Refering table on the same node? Foreign key
constraints can not be implemented if they involve rows across nodes. In
fact, global constraints (constraints that involve data across various
nodes) is not supported by 1.0. Neither it will be supported in the next
version.
On Thu, Apr 11, 2013 at 6:51 AM, Venky Kandaswamy <ve...@ad...> wrote:
> We are processing inserts/updates using multiple threads. Here is the
> trace log of the actual statements that are hung. The scenario shows the
> statements on the coordinator and 2 datanodes. The scenario is similar
> across all the datanodes.
> The same data updates did not cause Postgres 9.1.2 to hang. This could be
> related to an application problem, although we could not reproduce it on
> Postgres 9.1.2.
>
> At a high level, there is an update on the 'feature' table that is holding
> an exclusive lock on the row. The inserts are inserting to another table
> that has a foreign key that references the row being locked by the update.
> Pid 7174 and 7179 are waiting to complete and they are also similar
> inserts. The only thing in common seems to be that the update is locking
> the feature row that is referenced in a foreign key in the other inserts.
> This should not cause a deadlock, I believe.
>
> The question in my mind is whether pids 7181 and 7186 should have been
> granted exclusive access to a tuple while others were granted share access.
> This might cause a race condition.
>
> This causes PGXC to hang. Obviously, the update is in turn waiting for
> something (which we cannot figure out from the logs) and therefore not
> committing the update.
>
> [postgres@sv4-pgxc-db01 pgxc]$ ps -ef | grep adchemy1234
> <COORDINATOR>
> postgres 7169 7113 0 16:41 ? 00:00:02 postgres: adchemy
> adchemy1234 192.168.51.73(49186) INSERT
> postgres 7170 7113 0 16:41 ? 00:00:02 postgres: adchemy
> adchemy1234 192.168.51.73(49187) INSERT
> postgres 7171 7113 0 16:41 ? 00:00:02 postgres: adchemy
> adchemy1234 192.168.51.73(49188) UPDATE
> postgres 7172 7113 0 16:41 ? 00:00:02 postgres: adchemy
> adchemy1234 192.168.51.73(49189) INSERT
> postgres 7173 7113 0 16:41 ? 00:00:02 postgres: adchemy
> adchemy1234 192.168.51.73(49190) INSERT
> <COORDINATOR>
>
> <DATANODE1>
> postgres 7174 7127 0 16:41 ? 00:00:01 postgres: adchemy
> adchemy1234 172.17.28.61(51909) idle in transaction
> postgres 7175 7127 0 16:41 ? 00:00:01 postgres: adchemy
> adchemy1234 172.17.28.61(51910) INSERT waiting
> postgres 7181 7127 0 16:41 ? 00:00:01 postgres: adchemy
> adchemy1234 172.17.28.61(51924) UPDATE waiting
> postgres 7182 7127 0 16:41 ? 00:00:01 postgres: adchemy
> adchemy1234 172.17.28.61(51925) INSERT waiting
> postgres 7183 7127 0 16:41 ? 00:00:01 postgres: adchemy
> adchemy1234 172.17.28.61(51926) INSERT waiting
> <DATANODE1>
>
> <DATANODE2>
> postgres 7179 7140 0 16:41 ? 00:00:00 postgres: adchemy
> adchemy1234 172.17.28.61(48957) idle in transaction
> postgres 7180 7140 0 16:41 ? 00:00:00 postgres: adchemy
> adchemy1234 172.17.28.61(48962) INSERT waiting
> postgres 7184 7140 0 16:41 ? 00:00:00 postgres: adchemy
> adchemy1234 172.17.28.61(48970) INSERT waiting
> postgres 7185 7140 0 16:41 ? 00:00:00 postgres: adchemy
> adchemy1234 172.17.28.61(48975) INSERT waiting
> postgres 7186 7140 0 16:41 ? 00:00:00 postgres: adchemy
> adchemy1234 172.17.28.61(48980) UPDATE waiting
> <DATANODE2>
>
> -----LOGS----- formatted %t %u %p
>
> 2013-04-10 16:42:16 PDT adchemy 7169 LOG: execute S_1: BEGIN
> 2013-04-10 16:42:16 PDT adchemy 7169 LOG: execute <unnamed>: select
> nextval ('hibernate_sequence')
> 2013-04-10 16:42:16 PDT adchemy 7169 LOG: execute <unnamed>: insert into
> biods.product_feature (category_id, category_semid, created_ts, feature_id,
> feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id,
> prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7,
> $8, $9, $10, $11, $12)
> 2013-04-10 16:42:16 PDT adchemy 7169 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 =
> 'description', $6 = '46105', $7 = 'description,Give your riches the
> designer treatment with Mcms leather heritage wallet. The logo-stamped
> little number stores your essentials in luxe vintage style.', $8 = NULL, $9
> = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 =
> '46455'
>
> 2013-04-10 16:42:16 PDT adchemy 7170 LOG: execute S_1: BEGIN
> 2013-04-10 16:42:16 PDT adchemy 7170 LOG: execute <unnamed>: select
> nextval ('hibernate_sequence')
> 2013-04-10 16:42:16 PDT adchemy 7170 LOG: execute <unnamed>: insert into
> biods.product_feature (category_id, category_semid, created_ts, feature_id,
> feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id,
> prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7,
> $8, $9, $10, $11, $12)
> 2013-04-10 16:42:16 PDT adchemy 7170 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 =
> 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up
> with a bold logo-stamped plaque in this utility chic wallet from Marc By
> Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 =
> '2013-04-10 15:02:43.413-07', $12 = '46445'
>
> 2013-04-10 16:42:15 PDT adchemy 7171 LOG: execute S_1: BEGIN
> 2013-04-10 16:42:15 PDT adchemy 7171 LOG: execute <unnamed>: select
> feature0_.feature_id as feature1_8_1_, feature0_.created_ts as
> created2_8_1_, feature0_.feature_name as feature3_8_1_,
> feature0_.feature_semid as feature4_8_1_, feature0_.modified_by as
> modified5_8_1_, feature0_.source_msg_ts as source6_8_1_,
> feature0_.updated_ts as updated7_8_1_, featureval1_.feature_id as
> feature9_8_3_, featureval1_.feature_value_id as feature1_14_3_,
> featureval1_.feature_value_id as feature1_14_0_, featureval1_.created_ts as
> created2_14_0_, featureval1_.feature_id as feature9_14_0_,
> featureval1_.feature_semid as feature3_14_0_, featureval1_.feature_value as
> feature4_14_0_, featureval1_.feature_value_semid as feature5_14_0_,
> featureval1_.modified_by as modified6_14_0_, featureval1_.source_msg_ts as
> source7_14_0_, featureval1_.updated_ts as updated8_14_0_ from biods.feature
> feature0_ left outer join biods.feature_value featureval1_ on
> feature0_.feature_id=featureval1_.feature_id where feature0_.feature_id=$1
> 2013-04-10 16:42:15 PDT adchemy 7171 DETAIL: parameters: $1 = '42318'
> 2013-04-10 16:42:15 PDT adchemy 7171 LOG: execute <unnamed>: update
> biods.feature set created_ts=$1, feature_name=$2, feature_semid=$3,
> modified_by=$4, source_msg_ts=$5, updated_ts=$6 where feature_id=$7
> 2013-04-10 16:42:15 PDT adchemy 7171 DETAIL: parameters: $1 = '2013-04-10
> 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 =
> '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 =
> '42318'
>
> 2013-04-10 16:42:17 PDT adchemy 7172 LOG: execute S_1: BEGIN
> 2013-04-10 16:42:17 PDT adchemy 7172 LOG: execute <unnamed>: select
> nextval ('hibernate_sequence')
> 2013-04-10 16:42:17 PDT adchemy 7172 LOG: execute <unnamed>: insert into
> biods.product_feature (category_id, category_semid, created_ts, feature_id,
> feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id,
> prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7,
> $8, $9, $10, $11, $12)
> 2013-04-10 16:42:17 PDT adchemy 7172 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 =
> 'description', $6 = '44831', $7 = 'description,A chic logo-detailed
> cosmetic case for the contemporary girl from Tory Burch. Exclusive to
> Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 =
> '2013-04-10 15:02:42.003-07', $12 = '46460'
>
> 2013-04-10 16:42:15 PDT adchemy 7173 LOG: execute S_1: BEGIN
> 2013-04-10 16:42:15 PDT adchemy 7173 LOG: execute <unnamed>: select
> nextval ('hibernate_sequence')
> 2013-04-10 16:42:15 PDT adchemy 7173 LOG: execute <unnamed>: insert into
> biods.product_feature (category_id, category_semid, created_ts, feature_id,
> feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id,
> prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7,
> $8, $9, $10, $11, $12)
> 2013-04-10 16:42:15 PDT adchemy 7173 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 =
> 'description', $6 = '46154', $7 = 'description,Keep the essentials close
> with LeSportsacs crossbody bag in matte black nylon practical interior zip
> compartments make those daily errands a little bit easier.', $8 = NULL, $9
> = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 =
> '46435'
>
> 2013-04-10 16:42:15 PDT adchemy 7174 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:15 PDT adchemy 7174 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:15 PDT adchemy 7174 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 =
> 'description', $6 = '46154', $7 = 'description,Keep the essentials close
> with LeSportsacs crossbody bag in matte black nylon practical interior zip
> compartments make those daily errands a little bit easier.', $8 = NULL, $9
> = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 =
> '46435'
>
> 2013-04-10 16:42:16 PDT adchemy 7175 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:16 PDT adchemy 7175 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:16 PDT adchemy 7175 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 =
> 'description', $6 = '46105', $7 = 'description,Give your riches the
> designer treatment with Mcms leather heritage wallet. The logo-stamped
> little number stores your essentials in luxe vintage style.', $8 = NULL, $9
> = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 =
> '46455'
>
> 2013-04-10 16:42:15 PDT adchemy 7179 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:15 PDT adchemy 7179 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:15 PDT adchemy 7179 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 =
> 'description', $6 = '46154', $7 = 'description,Keep the essentials close
> with LeSportsacs crossbody bag in matte black nylon practical interior zip
> compartments make those daily errands a little bit easier.', $8 = NULL, $9
> = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 =
> '46435'
>
> 2013-04-10 16:42:16 PDT adchemy 7180 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:16 PDT adchemy 7180 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:16 PDT adchemy 7180 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 =
> 'description', $6 = '46105', $7 = 'description,Give your riches the
> designer treatment with Mcms leather heritage wallet. The logo-stamped
> little number stores your essentials in luxe vintage style.', $8 = NULL, $9
> = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 =
> '46455'
>
> 2013-04-10 16:42:15 PDT adchemy 7181 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:15 PDT adchemy 7181 LOG: execute <unnamed>: UPDATE
> biods.feature SET feature_semid = $3, feature_name = $2, created_ts = $1,
> updated_ts = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id =
> $7)
> 2013-04-10 16:42:15 PDT adchemy 7181 DETAIL: parameters: $1 = '2013-04-10
> 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 =
> '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 =
> '42318'
>
> 2013-04-10 16:42:17 PDT adchemy 7182 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:17 PDT adchemy 7182 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:17 PDT adchemy 7182 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 =
> 'description', $6 = '44831', $7 = 'description,A chic logo-detailed
> cosmetic case for the contemporary girl from Tory Burch. Exclusive to
> Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 =
> '2013-04-10 15:02:42.003-07', $12 = '46460'
>
> 2013-04-10 16:42:16 PDT adchemy 7183 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:16 PDT adchemy 7183 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:16 PDT adchemy 7183 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 =
> 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up
> with a bold logo-stamped plaque in this utility chic wallet from Marc By
> Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 =
> '2013-04-10 15:02:43.413-07', $12 = '46445'
>
> 2013-04-10 16:42:17 PDT adchemy 7184 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:17 PDT adchemy 7184 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:17 PDT adchemy 7184 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 =
> 'description', $6 = '44831', $7 = 'description,A chic logo-detailed
> cosmetic case for the contemporary girl from Tory Burch. Exclusive to
> Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 =
> '2013-04-10 15:02:42.003-07', $12 = '46460'
>
> 2013-04-10 16:42:16 PDT adchemy 7185 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:16 PDT adchemy 7185 LOG: execute <unnamed>: INSERT INTO
> biods.product_feature (prd_feature_id, category_id, prd_id, feature_id,
> feature_value_id, category_semid, prd_semid, feature_semid,
> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
> 2013-04-10 16:42:16 PDT adchemy 7185 DETAIL: parameters: $1 = '42302', $2
> = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 =
> 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up
> with a bold logo-stamped plaque in this utility chic wallet from Marc By
> Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 =
> '2013-04-10 15:02:43.413-07', $12 = '46445'
>
> 2013-04-10 16:42:15 PDT adchemy 7186 LOG: statement: START TRANSACTION
> ISOLATION LEVEL read committed READ WRITE
> 2013-04-10 16:42:15 PDT adchemy 7186 LOG: execute <unnamed>: UPDATE
> biods.feature SET feature_semid = $3, feature_name = $2, created_ts = $1,
> updated_ts = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id =
> $7)
> 2013-04-10 16:42:15 PDT adchemy 7186 DETAIL: parameters: $1 = '2013-04-10
> 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 =
> '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 =
> '42318'
>
> LOCKS ON COORDINATOR:
>
> [venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5432 -U postgres -d
> adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks,
> pg_class where relation=oid and relname not like 'pg_%' order by mode;"
> pid | relname | locktype | mode | granted
> ------+--------------------+----------+------------------+---------
> 7169 | hibernate_sequence | relation | AccessShareLock | t
> 7173 | hibernate_sequence | relation | AccessShareLock | t
> 7172 | hibernate_sequence | relation | AccessShareLock | t
> 7171 | feature_value | relation | AccessShareLock | t
> 7171 | feature | relation | AccessShareLock | t
> 7170 | hibernate_sequence | relation | AccessShareLock | t
> 7171 | feature | relation | RowExclusiveLock | t
> 7172 | product_feature | relation | RowExclusiveLock | t
> 7170 | product_feature | relation | RowExclusiveLock | t
> 7173 | product_feature | relation | RowExclusiveLock | t
> 7169 | product_feature | relation | RowExclusiveLock | t
> (11 rows)
>
> LOCKS ON DATANODE1:
>
> [venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5433 -U postgres -d
> adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks,
> pg_class where relation=oid and relname not like 'pg_%' order by mode;"
> pid | relname | locktype | mode |
> granted
>
> ------+-----------------------+----------+--------------------------+---------
> 7174 | prd_id | relation | AccessShareLock | t
> 7182 | feature_id | relation | AccessShareLock | t
> 7174 | feature_value_id | relation | AccessShareLock | t
> 7183 | feature_id | relation | AccessShareLock | t
> 7174 | feature_id | relation | AccessShareLock | t
> 7175 | feature_id | relation | AccessShareLock | t
> 7181 | feature | tuple | ExclusiveLock | t
> 7181 | feature_semid | relation | RowExclusiveLock | t
> 7181 | feature_id | relation | RowExclusiveLock | t
> 7181 | feature | relation | RowExclusiveLock | t
> 7175 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7183 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7183 | prd_feature_id | relation | RowExclusiveLock | t
> 7183 | product_feature | relation | RowExclusiveLock | t
> 7182 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7182 | prd_feature_id | relation | RowExclusiveLock | t
> 7182 | product_feature | relation | RowExclusiveLock | t
> 7175 | prd_feature_id | relation | RowExclusiveLock | t
> 7175 | product_feature | relation | RowExclusiveLock | t
> 7174 | product_feature | relation | RowExclusiveLock | t
> 7206 | feature_semid | relation | RowExclusiveLock | t
> 7206 | feature_id | relation | RowExclusiveLock | t
> 7174 | product | relation | RowShareLock | t
> 7182 | feature | relation | RowShareLock | t
> 7174 | feature_value | relation | RowShareLock | t
> 7183 | category | relation | RowShareLock | t
> 7174 | feature | relation | RowShareLock | t
> 7174 | category | relation | RowShareLock | t
> 7175 | category | relation | RowShareLock | t
> 7175 | feature | relation | RowShareLock | t
> 7183 | feature | relation | RowShareLock | t
> 7182 | category | relation | RowShareLock | t
> 7182 | feature | tuple | ShareLock | f
> 7175 | feature | tuple | ShareLock | f
> 7183 | feature | tuple | ShareLock | f
> 7206 | feature | relation | ShareUpdateExclusiveLock | t
>
> LOCKS ON DATANODE2:
>
> [venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5434 -U postgres -d
> adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks,
> pg_class where relation=oid and relname not like 'pg_%' order by mode;"
> pid | relname | locktype | mode |
> granted
>
> ------+-----------------------+----------+--------------------------+---------
> 7185 | feature_id | relation | AccessShareLock | t
> 7179 | feature_value_id | relation | AccessShareLock | t
> 7179 | prd_id | relation | AccessShareLock | t
> 7184 | feature_id | relation | AccessShareLock | t
> 7180 | feature_id | relation | AccessShareLock | t
> 7179 | feature_id | relation | AccessShareLock | t
> 7186 | feature | tuple | ExclusiveLock | t
> 7184 | prd_feature_id | relation | RowExclusiveLock | t
> 7184 | product_feature | relation | RowExclusiveLock | t
> 7186 | feature_semid | relation | RowExclusiveLock | t
> 7186 | feature_id | relation | RowExclusiveLock | t
> 7186 | feature | relation | RowExclusiveLock | t
> 7185 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7185 | prd_feature_id | relation | RowExclusiveLock | t
> 7185 | product_feature | relation | RowExclusiveLock | t
> 7184 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7180 | cat_prd_feature_semid | relation | RowExclusiveLock | t
> 7180 | prd_feature_id | relation | RowExclusiveLock | t
> 7180 | product_feature | relation | RowExclusiveLock | t
> 7179 | product_feature | relation | RowExclusiveLock | t
> 7202 | feature_semid | relation | RowExclusiveLock | t
> 7202 | feature_id | relation | RowExclusiveLock | t
> 7179 | product | relation | RowShareLock | t
> 7184 | feature | relation | RowShareLock | t
> 7179 | feature_value | relation | RowShareLock | t
> 7185 | category | relation | RowShareLock | t
> 7179 | feature | relation | RowShareLock | t
> 7179 | category | relation | RowShareLock | t
> 7180 | feature | relation | RowShareLock | t
> 7180 | category | relation | RowShareLock | t
> 7185 | feature | relation | RowShareLock | t
> 7184 | category | relation | RowShareLock | t
> 7185 | feature | tuple | ShareLock | f
> 7180 | feature | tuple | ShareLock | f
> 7184 | feature | tuple | ShareLock | f
> 7202 | feature | relation | ShareUpdateExclusiveLock | t
> (36 rows)
>
>
>
> ________________________________________
>
> Venky Kandaswamy
>
> Principal Engineer, Adchemy Inc.
>
> 925-200-7124
> ------------------------------
> *From:* Koichi Suzuki [koi...@gm...]
> *Sent:* Monday, April 08, 2013 10:41 PM
> *To:* Amit Khandekar
> *Cc:* Venky Kandaswamy; pos...@li...
> *Subject:* Re: [Postgres-xc-developers] PGXC hangs when run with
> concurrent inserts
>
> Because insert is being done in parallel, I'm afraid there could be a
> possibility that we have internal lock conflicts, which should not happen.
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> 2013/4/9 Amit Khandekar <ami...@en...>
>
>>
>>
>>
>> On 9 April 2013 06:46, Venky Kandaswamy <ve...@ad...> wrote:
>>
>>> All,
>>> We have been running into a hang issue on our app that appears to be
>>> related to PGXC. Our app processes messages from RabbitMQ and
>>> inserts/updates tables. We run 5 concurrent threads. The incoming queues
>>> are replicated, one feeding Postgres 9.1 and the other feeding PGXC
>>> (current git master). PGXC is hanging on inserts after processing a few
>>> transactions. It does not appear to be related to the actual data itself.
>>> IT looks like all the sessions are waiting for something. There is no
>>> information on locks available from pg_locks.
>>>
>>
>> Since most of the operations are inserts, it does not look like it is
>> due to locks, unless something has acquired table locks. But just to rule
>> out that possibility, it would be better if you check pg_locks on the
>> datanodes, if you have checked it only on coordinator so far.
>>
>>
>>>
>>> An strace simply says recfrom(10.
>>>
>>> The are no errors in the logs from gtm, coordinator or datanodes.
>>>
>>> The tables have referential integrity and use a shared sequence to
>>> get the next id. Is it possible that something is going on with the logic
>>> to retrieve sequence numbers? The tables are all replicated.
>>>
>>> Unfortunately, we have not been able to reproduce a reliable test
>>> case.
>>>
>>> [postgres@gnode0 pgxc]$ /usr/local/pgsql/bin/psql -p 5433 -U
>>> postgres -d postgres -c 'select * from pg_catalog.pg_stat_activity;'
>>> datid | datname | pid | usesysid | usename | application_name
>>> | client_addr | client_hostname | client_port |
>>> backend_start | xact_start |
>>> query_start | state_change | waiting |
>>> state |
>>>
>>> query
>>>
>>>
>>> -------+--------------+-------+----------+----------+------------------+----------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+---------------------+--------------------
>>>
>>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> 12893 | postgres | 22330 | 10 | postgres | pgxc |
>>> 192.168.53.109 | | 47025 | 2013-03-31
>>> 21:42:16.724845-07 | | 2013-04-08
>>> 15:43:52.313325-07 | 2013-04-08 15:26:11.444754-07 | f |
>>> idle | COMMIT PREPARED 'T1
>>> 32273'
>>> 16393 | master | 4267 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 54961 | 2013-04-08
>>> 15:24:28.668023-07 | | 2013-04-08
>>> 15:33:17.586836-07 | 2013-04-08 15:33:17.587942-07 | f |
>>> idle | SELECT count(*) FRO
>>> M ONLY bicommon.account_datasource WHERE true
>>> 16395 | adchemy10013 | 4363 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55084 | 2013-04-08
>>> 15:28:48.822939-07 | | 2013-04-08
>>> 15:50:21.650727-07 | 2013-04-08 15:50:07.916753-07 | f |
>>> idle | SELECT prd_id, prd_
>>> semid, prd_name, prd_line, prd_model, prd_brand, prd_image_url,
>>> prd_dest_url, created_ts, updated_ts, source_msg_ts, modified_by FROM
>>> biods.product
>>> 16393 | master | 4486 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55246 | 2013-04-08
>>> 15:33:21.019388-07 | | 2013-04-08
>>> 15:43:51.321376-07 | 2013-04-08 15:43:51.322675-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16393 | master | 4781 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55515 | 2013-04-08
>>> 15:42:42.122785-07 | | 2013-04-08
>>> 17:02:21.023713-07 | 2013-04-08 17:02:20.804751-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16393 | master | 4787 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55521 | 2013-04-08
>>> 15:42:42.142662-07 | | 2013-04-08
>>> 16:17:19.26364-07 | 2013-04-08 16:17:19.126163-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16393 | master | 4792 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55526 | 2013-04-08
>>> 15:42:42.159009-07 | | 2013-04-08
>>> 15:45:11.915026-07 | 2013-04-08 15:45:11.886392-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16393 | master | 4799 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55533 | 2013-04-08
>>> 15:42:42.678387-07 | | 2013-04-08
>>> 17:02:21.195332-07 | 2013-04-08 17:02:20.805074-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16393 | master | 4804 | 16392 | xcadmin | pgxc |
>>> 192.168.53.109 | | 55538 | 2013-04-08
>>> 15:42:42.694802-07 | | 2013-04-08
>>> 15:45:11.904619-07 | 2013-04-08 15:45:11.888493-07 | f |
>>> idle | SET SESSION AUTHORI
>>> ZATION DEFAULT;RESET ALL;
>>> 16395 | adchemy10013 | 4977 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 55732 | 2013-04-08
>>> 15:47:34.901175-07 | 2013-04-08 15:48:08.345331-07 | 2013-04-08
>>> 15:48:08.528818-07 | 2013-04-08 15:48:08.410815-07 | f | idle in
>>> transaction | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 4979 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 55734 | 2013-04-08
>>> 15:47:35.042778-07 | 2013-04-08 15:48:16.384763-07 | 2013-04-08
>>> 15:48:16.506899-07 | 2013-04-08 15:48:16.388503-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 4985 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 55740 | 2013-04-08 15:47:
>>> 35.235945-07 | 2013-04-08 15:48:14.38895-07 | 2013-04-08
>>> 15:48:14.445351-07 | 2013-04-08 15:48:14.446752-07 | t |
>>> active | UPDATE biods.featur
>>> e SET feature_semid = $3, feature_name = $2, created_ts = $1, updated_ts
>>> = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id = $7)
>>> 16395 | adchemy10013 | 4986 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 55741 | 2013-04-08 15:47:
>>> 35.238843-07 | 2013-04-08 15:48:18.201043-07 | 2013-04-08
>>> 15:48:18.273204-07 | 2013-04-08 15:48:18.205647-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 4998 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 55753 | 2013-04-08
>>> 15:47:35.910309-07 | 2013-04-08 15:48:08.412038-07 | 2013-04-08
>>> 15:48:08.566945-07 | 2013-04-08 15:48:08.415026-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 6340 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57002 | 2013-04-08
>>> 16:31:44.414804-07 | 2013-04-08 16:31:50.293828-07 | 2013-04-08
>>> 16:31:50.433988-07 | 2013-04-08 16:31:50.297752-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 6341 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57003 | 2013-04-08
>>> 16:31:44.418356-07 | 2013-04-08 16:31:49.450704-07 | 2013-04-08
>>> 16:31:49.599946-07 | 2013-04-08 16:31:49.45562-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 6348 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57010 | 2013-04-08 16:31:
>>> 45.065767-07 | 2013-04-08 16:31:50.699979-07 | 2013-04-08
>>> 16:31:50.817425-07 | 2013-04-08 16:31:50.704669-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 6349 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57011 | 2013-04-08
>>> 16:31:45.06926-07 | 2013-04-08 16:31:51.528207-07 | 2013-04-08
>>> 16:31:51.582036-07 | 2013-04-08 16:31:51.532618-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 6350 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57012 | 2013-04-08 16:31:
>>> 45.072711-07 | 2013-04-08 16:31:50.085336-07 | 2013-04-08
>>> 16:31:50.223221-07 | 2013-04-08 16:31:50.088908-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7269 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57774 | 2013-04-08
>>> 16:57:15.563006-07 | 2013-04-08 16:57:21.849156-07 | 2013-04-08
>>> 16:57:21.978984-07 | 2013-04-08 16:57:21.853289-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7271 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57776 | 2013-04-08
>>> 16:57:15.63199-07 | 2013-04-08 16:57:16.575535-07 | 2013-04-08
>>> 16:57:17.00605-07 | 2013-04-08 16:57:17.007747-07 | t |
>>> active | INSERT INTO biods.f
>>> eature_value (feature_value_id, feature_value_semid, feature_value,
>>> feature_semid, feature_id, created_ts, updated_ts, source_msg_ts,
>>> modified_by) VALUES ($9, $5, $4, $3, $2, $1, $8, $7, $6)
>>> 16395 | adchemy10013 | 7283 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57788 | 2013-04-08
>>> 16:57:16.292702-07 | 2013-04-08 16:57:21.849125-07 | 2013-04-08
>>> 16:57:21.978824-07 | 2013-04-08 16:57:21.853251-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7284 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57789 | 2013-04-08
>>> 16:57:16.295879-07 | 2013-04-08 16:57:24.233166-07 | 2013-04-08
>>> 16:57:24.321938-07 | 2013-04-08 16:57:24.237514-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7285 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57790 | 2013-04-08
>>> 16:57:16.299271-07 | 2013-04-08 16:57:22.119868-07 | 2013-04-08
>>> 16:57:22.197213-07 | 2013-04-08 16:57:22.128357-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7465 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57954 | 2013-04-08
>>> 17:01:54.750113-07 | 2013-04-08 17:02:00.17336-07 | 2013-04-08
>>> 17:02:00.320469-07 | 2013-04-08 17:02:00.177758-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7466 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57955 | 2013-04-08
>>> 17:01:54.753559-07 | 2013-04-08 17:01:59.49003-07 | 2013-04-08
>>> 17:01:59.602925-07 | 2013-04-08 17:01:59.493732-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7467 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57956 | 2013-04-08
>>> 17:01:54.75699-07 | 2013-04-08 17:01:58.262083-07 | 2013-04-08
>>> 17:01:58.349452-07 | 2013-04-08 17:01:58.350822-07 | t |
>>> active | INSERT INTO biods.f
>>> eature_value (feature_value_id, feature_value_semid, feature_value,
>>> feature_semid, feature_id, created_ts, updated_ts, source_msg_ts,
>>> modified_by) VALUES ($9, $5, $4, $3, $2, $1, $8, $7, $6)
>>> 16395 | adchemy10013 | 7473 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57963 | 2013-04-08
>>> 17:01:55.49134-07 | 2013-04-08 17:02:00.313138-07 | 2013-04-08
>>> 17:02:00.420405-07 | 2013-04-08 17:02:00.318887-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 16395 | adchemy10013 | 7474 | 17361 | adchemy | pgxc |
>>> 192.168.53.109 | | 57964 | 2013-04-08
>>> 17:01:55.494777-07 | 2013-04-08 17:02:00.514142-07 | 2013-04-08
>>> 17:02:00.577239-07 | 2013-04-08 17:02:00.519572-07 | t |
>>> active | INSERT INTO biods.p
>>> roduct_feature (prd_feature_id, category_id, prd_id, feature_id,
>>> feature_value_id, category_semid, prd_semid, feature_semid,
>>> feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1,
>>> $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
>>> 12893 | postgres | 8517 | 10 | postgres | psql
>>> | | | -1 | 2013-04-08
>>> 17:35:28.217934-07 | | 2013-04-08
>>> 17:35:28.220366-07 | 2013-04-08 17:35:28.220369-07 | f |
>>> active | select * from pg_ca
>>> talog.pg_stat_activity;
>>> (30 rows)
>>>
>>>
>>> ________________________________________
>>>
>>> Venky Kandaswamy
>>>
>>> Principal Engineer, Adchemy Inc.
>>>
>>> 925-200-7124
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Precog is a next-generation analytics platform capable of advanced
>>> analytics on semi-structured data. The platform includes APIs for
>>> building
>>> apps and a phenomenal toolset for data science. Developers can use
>>> our toolset for easy data analysis & visualization. Get a free account!
>>> http://www2.precog.com/precogplatform/slashdotnewsletter
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Precog is a next-generation analytics platform capable of advanced
>> analytics on semi-structured data. The platform includes APIs for building
>> apps and a phenomenal toolset for data science. Developers can use
>> our toolset for easy data analysis & visualization. Get a free account!
>> http://www2.precog.com/precogplatform/slashdotnewsletter
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
--
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company
|
