kai-devel Mailing List for kai
Kai is a distributed key-value datastore
Status: Beta
Brought to you by:
takemaru
Menu
▾
▴
kai-devel — For Kai developers
You can subscribe to this list here.
| 2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(7) |
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
|---|
|
From: Takeru I. <tak...@gm...> - 2008-08-09 10:13:33
|
Hi,
I conducted detailed experiments to reveals the performance of Kai.
I'm happy to let you know the results are pretty good.
* Experimental environments
The experiments were conducted in the following environments.
Kai nodes:
CPU Xeon 2.13 GHz x4
Memory 8GB
1000Base-T (RTT 90 us)
Debian Etch
OTP R12B-3
Clients:
Perl Cache::Memcached
Kai:
branches/takemaru_connection_pooling (r66)
Conditions:
Data size: 1 KB
# of operations: around hundreds of thousands
# of nodes: shown in each experiment
# of clients: shown in each experiment
* As a stand alone server
In this experiment, Kai ran as a stand alone server like memcached.
Conditions:
# of nodes: 1
# of clients: 12-14
Configurations (kai.config):
n: 1 (# of replication)
r: 1
w: 1
max_connections: 90
memcache_max_connections: 30
Results:
set:
rate: 7136 qps
latency:
median: 1.55 ms
99%: 2.76 ms
CPU: 370 %
get:
rate: 10193 qps
latency:
median: 1.21 ms
99%: 3.71 ms
CPU: 355 %
I was surprised that Kai showed over 10,000 qps for get.
The latency was also good.
The value of 99 percentile as well as median was less than 4 ms.
Kai is a kind of persistent memory storage, not a transient cache like
memcached.
However, it's performance is not much worse than memcached.
* As a cluster system
Next, I conducted larger scale experiments with the Kai cluster, which
consists of five nodes.
Conditions:
# of nodes: 5
# of clients: 30-40
Configurations (kai.config):
n: 2 (# of replication)
r: 1
w: 2
max_connections: 180
memcache_max_connections: 60
I presented the experimental results in the following paragraph.
The value "x (y/z)" means "total (set/get").
For example, rate of "12155 (2176/9979)" shows the total rate is
12155, that of set is 2176, and that of get is 9979.
Results:
rate: 12155 (2176/9979) qps
latency:
median: (2.06/2.01) ms
99%: (17.1/14.5) ms
CPU: 175 %
Kai executes quite complicated tasks; it calculates consistent
hashing, routes requests to the appropriate nodes, replicates data,
and manages the version.
However, the performance is still good.
The nodes have four cores and the limitation of CPU usage is 400 %,
but the experimental results were less than 200%.
Unfortunately, the current implementation has inefficient points, such
as internal communication.
I guess that the performance can be more than double when the issue
will be resolved.
--
Takeru INOUE <tak...@gm...>
|
|
From: Takeru I. <tak...@gm...> - 2008-07-23 00:06:55
|
Since Kai is based on Apache License 2.0, feel free to build it into Dynomite. Kai has some useful features like memcache API and tcp_server, which implements TCP process pooling. Regards, On Wed, Jul 23, 2008 at 8:58 AM, Takeru INOUE <tak...@gm...> wrote: > Thanks a lot. > We're implementing VectorClocks right now. > > I'm expecting Dynomite will be used in production soon. > > On Wed, Jul 23, 2008 at 2:05 AM, Cliff Moon <cl...@mo...> wrote: >> I'm currently planning on rolling out an early version for a low-latency >> read system in production at Powerset. I want to decouple partitioning from >> partition placement in the membership server before I do that. After that I >> plan on getting synchronization w/ merkle trees up and running. >> >> I noticed that Kai doesn't currently have a vector clock implementation. I >> have a working implementation in the Dynomite source here: >> http://github.com/cliffmoon/dynomite/tree/master/elibs/vector_clock.erl and >> the tests are here: >> http://github.com/cliffmoon/dynomite/tree/master/etest/vector_clock_test.erl. >> Feel free to integrate that in with Kai if you think it will be useful. >> >> Takeru INOUE wrote: >>> >>> Thank you for letting me know your project. >>> It's interesting for us, too. >>> >>> Currently, Kai has basic functions of Dynamo, like consistent hashing, >>> membership management, and data synchronization. >>> We conducted simple tests with 16 nodes, and got fine results. >>> Our future work includes Merkle tree, Chord and so on. >>> >>> I found that you are from Powerset Inc. >>> Will Dynomite used in Powerset search in the near future? >>> >>> >>> On Tue, Jul 22, 2008 at 2:00 PM, Cliff Moon <cl...@mo...> >>> wrote: >>> >>>> >>>> I'm currently working on another erlang based implementation of dynamo, >>>> called dynomite. I only just found out about kai, and it's interesting >>>> to see how far along you all are. Dynomite is hosted on github at: >>>> http://github.com/cliffmoon/dynomite/tree/master >>>> >>>> ------------------------------------------------------------------------- >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>>> challenge >>>> Build the coolest Linux based applications with Moblin SDK & win great >>>> prizes >>>> Grand prize is a trip for two to an Open Source event anywhere in the >>>> world >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>> _______________________________________________ >>>> Kai-devel mailing list >>>> Kai...@li... >>>> https://lists.sourceforge.net/lists/listinfo/kai-devel >>>> >>>> >>> >>> >>> >>> >> >> > > > > -- > Takeru INOUE <tak...@gm...> > -- Takeru INOUE <tak...@gm...> |
|
From: Takeru I. <tak...@gm...> - 2008-07-22 23:58:20
|
Thanks a lot. We're implementing VectorClocks right now. I'm expecting Dynomite will be used in production soon. On Wed, Jul 23, 2008 at 2:05 AM, Cliff Moon <cl...@mo...> wrote: > I'm currently planning on rolling out an early version for a low-latency > read system in production at Powerset. I want to decouple partitioning from > partition placement in the membership server before I do that. After that I > plan on getting synchronization w/ merkle trees up and running. > > I noticed that Kai doesn't currently have a vector clock implementation. I > have a working implementation in the Dynomite source here: > http://github.com/cliffmoon/dynomite/tree/master/elibs/vector_clock.erl and > the tests are here: > http://github.com/cliffmoon/dynomite/tree/master/etest/vector_clock_test.erl. > Feel free to integrate that in with Kai if you think it will be useful. > > Takeru INOUE wrote: >> >> Thank you for letting me know your project. >> It's interesting for us, too. >> >> Currently, Kai has basic functions of Dynamo, like consistent hashing, >> membership management, and data synchronization. >> We conducted simple tests with 16 nodes, and got fine results. >> Our future work includes Merkle tree, Chord and so on. >> >> I found that you are from Powerset Inc. >> Will Dynomite used in Powerset search in the near future? >> >> >> On Tue, Jul 22, 2008 at 2:00 PM, Cliff Moon <cl...@mo...> >> wrote: >> >>> >>> I'm currently working on another erlang based implementation of dynamo, >>> called dynomite. I only just found out about kai, and it's interesting >>> to see how far along you all are. Dynomite is hosted on github at: >>> http://github.com/cliffmoon/dynomite/tree/master >>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>> challenge >>> Build the coolest Linux based applications with Moblin SDK & win great >>> prizes >>> Grand prize is a trip for two to an Open Source event anywhere in the >>> world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Kai-devel mailing list >>> Kai...@li... >>> https://lists.sourceforge.net/lists/listinfo/kai-devel >>> >>> >> >> >> >> > > -- Takeru INOUE <tak...@gm...> |
|
From: Cliff M. <cl...@mo...> - 2008-07-22 17:05:38
|
I'm currently planning on rolling out an early version for a low-latency read system in production at Powerset. I want to decouple partitioning from partition placement in the membership server before I do that. After that I plan on getting synchronization w/ merkle trees up and running. I noticed that Kai doesn't currently have a vector clock implementation. I have a working implementation in the Dynomite source here: http://github.com/cliffmoon/dynomite/tree/master/elibs/vector_clock.erl and the tests are here: http://github.com/cliffmoon/dynomite/tree/master/etest/vector_clock_test.erl. Feel free to integrate that in with Kai if you think it will be useful. Takeru INOUE wrote: > Thank you for letting me know your project. > It's interesting for us, too. > > Currently, Kai has basic functions of Dynamo, like consistent hashing, > membership management, and data synchronization. > We conducted simple tests with 16 nodes, and got fine results. > Our future work includes Merkle tree, Chord and so on. > > I found that you are from Powerset Inc. > Will Dynomite used in Powerset search in the near future? > > > On Tue, Jul 22, 2008 at 2:00 PM, Cliff Moon <cl...@mo...> wrote: > >> I'm currently working on another erlang based implementation of dynamo, >> called dynomite. I only just found out about kai, and it's interesting >> to see how far along you all are. Dynomite is hosted on github at: >> http://github.com/cliffmoon/dynomite/tree/master >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Kai-devel mailing list >> Kai...@li... >> https://lists.sourceforge.net/lists/listinfo/kai-devel >> >> > > > > |
|
From: Takeru I. <tak...@gm...> - 2008-07-22 13:30:37
|
Thank you for letting me know your project. It's interesting for us, too. Currently, Kai has basic functions of Dynamo, like consistent hashing, membership management, and data synchronization. We conducted simple tests with 16 nodes, and got fine results. Our future work includes Merkle tree, Chord and so on. I found that you are from Powerset Inc. Will Dynomite used in Powerset search in the near future? On Tue, Jul 22, 2008 at 2:00 PM, Cliff Moon <cl...@mo...> wrote: > I'm currently working on another erlang based implementation of dynamo, > called dynomite. I only just found out about kai, and it's interesting > to see how far along you all are. Dynomite is hosted on github at: > http://github.com/cliffmoon/dynomite/tree/master > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Kai-devel mailing list > Kai...@li... > https://lists.sourceforge.net/lists/listinfo/kai-devel > -- Takeru INOUE <tak...@gm...> |
|
From: Cliff M. <cl...@mo...> - 2008-07-22 05:00:53
|
I'm currently working on another erlang based implementation of dynamo, called dynomite. I only just found out about kai, and it's interesting to see how far along you all are. Dynomite is hosted on github at: http://github.com/cliffmoon/dynomite/tree/master |
|
From: Takeru I. <tak...@gm...> - 2008-07-08 13:58:50
|
Hi Ian, > I came across this project when looking for merkle trees in erlang. Thank you for your interest. We're planning to implement Merkle tree in the near future. > Are you guys mainly developing in Japanese or is english acceptable? Yes, mainly in Japanese since all developers are Japanese. But, English is also acceptable, and we are trying to provide slides and wiki in English. > also do you have a PDF of your slides? all I found was on slideshare > which is hard to read offline. You can get PDF by cicking "Download file" at slideshare (I failed downloading it with firefox3, but safari is ok). Regards, > Regards > Ian > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Kai-devel mailing list > Kai...@li... > https://lists.sourceforge.net/lists/listinfo/kai-devel > -- Takeru INOUE <tak...@gm...> |
|
From: Ian H. <li...@ho...> - 2008-07-08 07:10:18
|
Hi. I came across this project when looking for merkle trees in erlang. Are you guys mainly developing in Japanese or is english acceptable? also do you have a PDF of your slides? all I found was on slideshare which is hard to read offline. Regards Ian |
