Menu
▾
▴
Re: [Postgres-xc-general] Pgxc_ctl Primer draft
|
From: Dorian H. <dor...@gm...> - 2014-05-07 12:00:41
|
Currently there is no open-source/free alternative(infinisql looks ~dead, cubrid doesn't have multi-node-statements + has middleware) that provides global transactions (most commercial offerings are in-memory(voltdb,memsql,nuodb). Even nosql dbs that don't have transactions (mongo(the hipster)db, has mongod,replica-servers,control-servers,mongos,balancer| hbase/hypertable). Only couchbase (no transactions,it is consistent, while riak/cassandra are eventually-consistent). Only Translattice looks to be 1node. So having 2 components is actually pretty good. If it makes sense(don't know internals), for certain workloads, when only 1-node-transactions are required, maybe the gtm could be eleminated? On Wed, May 7, 2014 at 10:21 AM, 鈴木 幸市 <ko...@in...> wrote: > Please understand that GTM is not a performance bottleneck. It may need > dedicated network segment and a server, but GTM’s load average is quite > low. I understand this ends up with bad workload balance and this can be > a concern for some people. Without GTM, each node may have to do much > more calculation to and exchange much more data to get snapshot and > determine if a given row can be vacuumed/vacuum frozen. I don’t know how > effective this could be. > > Regards; > --- > Koichi Suzuki > > 2014/05/07 17:10、ZhangJulian <jul...@ou...> のメール： > > Of course it is attractive. The performance bottleneck, GTM, could be > removed. And the cluster will be comprised of only one type of unified > component. :) > > Thanks > Julian > > ------------------------------ > From: ko...@in... > To: jul...@ou... > CC: koi...@gm...; dor...@gm...; > pos...@li... > Subject: Re: [Postgres-xc-general] Pgxc_ctl Primer draft > Date: Wed, 7 May 2014 08:00:12 +0000 > > Right. Maybe I can find how to calculate this without GTM and GXID. > Anyway, I thing we should keep track of root XID and local XID. I’m now > designing how to do this. Hope we can share the outcome as well soon. > Algorithm could be complicated but cluster configuration may look > significantly simpler. > > How do you think providing global MVCC without GTM/GXID attractive? > > Regards; > --- > Koichi Suzuki > > 2014/05/07 16:51、ZhangJulian <jul...@ou...> のメール： > > Oh, yes. > The oldest GXID must be in the snapshot of the oldest alive GXID. So if we > can know the old alive GXID, we can derive the oldest GXID which is still > referred. > > ------------------------------ > From: ko...@in... > To: jul...@ou... > CC: koi...@gm...; dor...@gm...; > pos...@li... > Subject: Re: [Postgres-xc-general] Pgxc_ctl Primer draft > Date: Wed, 7 May 2014 04:00:25 +0000 > > Oldest alive GXID is not correct. We need referred oldest GXID, which is > the oldest GXID appears in all the snapshot being used. Please consider > that in the case of long, repeated-read transaction, lifetime of snapshot > can be very long. > > Regards; > --- > Koichi Suzuki > > 2014/05/07 12:25、ZhangJulian <jul...@ou...> のメール： > > I said 'time' as the clock value. > You had considered more than I had known. > > For the VACUUM, as my understanding, if some data which can be vacuumed, > but is not vacuumed in time, this is OK. So if we collect the oldest alive > GXID, even it is smaller than the current accurate value, it still can > guide to VACUUM. Am I right? > > Thanks > Julian > > ------------------------------ > From: ko...@in... > To: jul...@ou... > CC: koi...@gm...; dor...@gm...; > pos...@li... > Subject: Re: [Postgres-xc-general] Pgxc_ctl Primer draft > Date: Wed, 7 May 2014 02:40:43 +0000 > > What do you mean by “time-based policy”, does it based on (G)XID, or real > clock value? To my view, it depend upon what “time” we depend on. > > Yes, I’m now studying if we can use real “clock” value for this. In > this case, we may not need GTM if the clock is accurate enough among > servers involved. > > If you mean not to use global “snapshot” and if it is feasible, we may > not need GTM. If we associate each local transaction to its “root” > transaction, which is the transaction application generated directly, we > can maintaing the visibility by calculating the “snapshot” each time > needed, by collecting it from all the other nodes. > > We need to consider the “vacuum”. I’ve not found a good way to > determine if some “deleted” rows can be removed from the database and if > some “live” row’s xmim value can be frozen. > > Regards; > --- > Koichi Suzuki > > 2014/05/07 11:19、ZhangJulian <jul...@ou...> のメール： > > Is it possible to do the row visibility check based on a time based > policy? That is, > > 1. Each data node maintains a data structure: gtid - start time - end > time. Only the gtids modifying data on current data node are contained. > 2. Each data node maintains the oldest alive gtid, which may not be > updated synchronously. > 3. GTM is only responsible to generate a sequence of GTID, which is only > an integer value. > 4. The time in different data nodes may be not consistent, but I think in > some scenario, the application can bear the little difference. > > Is there any potential issues? > > Thanks > > > Date: Sun, 4 May 2014 19:36:20 +0900 > > From: koi...@gm... > > To: dor...@gm... > > CC: pos...@li... > > Subject: Re: [Postgres-xc-general] Pgxc_ctl Primer draft > > > > As discussed in the last year's XC-day, GTM proxy should be integrated > > as postmaster backend. Maybe GTM can be. Coordinator/Datanode > > can also be integrated into one. > > > > Apparently, this is the direction we should take. At first, there > > were no such good experience to start with. Before version 1.0, we > > determined that the datanode and the coordinator can share the same > > binary. It is true that we started with the idea to provide > > cluster-wide MVCC and now we found the next direction. > > > > With this integration and when start with only one node, we don't need > > GTM, which looks identical to standalone PG. When we add the server, > > at present we do need GTM. Only accumulating local transactions in > > the nodes cannot maintain cluster-wide database consistency. > > > > I'm still investigating an idea how to get rid of GTM. We need to do > > the following: > > > > 1) To provide cluster wide MVCC, > > 2) To provide good means to determine which row can be vacuumed. > > > > My current idea is: if we associate any local XID to the root > > transaction (the transaction which application created), we may be > > able to provide cluster wide MVCC by calculating cluster-wide snapshot > > when needed. I don't know how efficient it is and t don't have good > > idea how to determine if a given row can be vacuumed. > > > > This is the current situation. > > > > Hope to have much more input on this. > > > > Anyway, hope my draft helps people who is trying to use Postgres-XC. > > > > Best; > > --- > > Koichi Suzuki > > > > > > 2014-05-04 19:05 GMT+09:00 Dorian Hoxha <dor...@gm...>: > > > Probably even the gtm-proxy need to be merged with > datanode+coordinator from > > > what i read. > > > > > > If you make only local transactions (inside 1 datanode) + not using > global > > > sequences, will there be no traffic to the GTM for that transaction ? > > > > > > > > > On Sun, May 4, 2014 at 6:24 AM, Michael Paquier < > mic...@gm...> > > > wrote: > > >> > > >> On Sun, May 4, 2014 at 12:59 AM, Dorian Hoxha <dor...@gm... > > > > >> wrote: > > >> >> You just need commodity INTEL server runnign Linux. > > >> > Are INTEL cpu required ? If not INTEL can be removed ? (also running > > >> > typo) > > >> Not really... I agree to what you mean here. > > >> > > >> >> For datawarehouse > > >> >> > > >> >> applications, you may need separate patch which devides complexed > query > > >> >> into smaller > > >> >> > > >> >> chunks which run in datanodes in parallel. StormDB will provide > such > > >> >> patche. > > >> > > > >> > Wasn't stormdb bought by another company ? Is there an opensource > > >> > alternative ? Fix the "patche" typo ? > > >> > > > >> > A way to make it simpler is by merging coordinator and datanode > into 1 > > >> > and > > >> > making it possible for a 'node' to not hold data (be a coordinator > > >> > only), > > >> > like in elastic-search, but you probably already know that. > > >> +1. This would alleviate data transfer between cross-node joins where > > >> Coordinator and Datanodes are on separate servers. You could always > > >> have both nodes on the same server with the XC of now... But that's > > >> double number of nodes to monitor. > > >> > > >> > What exact things does the gtm-proxy do? For example, a single row > > >> > insert > > >> > wouldn't need the gtm (coordinator just inserts it to the right > > >> > data-node)(asumming no sequences, since for that the gtm is needed)? > > >> Grouping messages between Coordinator/Datanode and GTM to reduce > > >> package interferences and improve performance. > > >> > > >> > If multiple tables are sharded on the same key (example: user_id). > Will > > >> > all > > >> > the rows, from the same user in different tables be in the same > > >> > data-node ? > > >> Yep. Node choice algorithm is based using the data type of the key. > > >> -- > > >> Michael > > > > > > > > > > > ------------------------------------------------------------------------------ > > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE > > Instantly run your Selenium tests across 300+ browser/OS combos. Get > > unparalleled scalability from the best Selenium testing platform > available. > > Simple to use. Nothing to install. Get started now for free." > > http://p.sf.net/sfu/SauceLabs > > _______________________________________________ > > Postgres-xc-general mailing list > > Pos...@li... > > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce_______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > > > > |
