For Developers

Running Tests

Before performing self-diagnostic test, activate common_test in your Erlang system.

% make test

Modules

This section may be obsoleted.

• kai_config: kai_config manages configured parameters of the Kai instance. The values can be retrieved by using get/1.
  *kai_log: kai_log provides logging facilities. By default, messages are shown in standard output. Filename can be set by setting logfile parameter.
• kai_hash: kai_hash implements consistent hashing. Calling find_nodes/1 with a key, you can get nodes which have the corresponding value. To add nodes to Kai cluster or remove them from it, update/2 method is called.
• kai_store: kai_store manages data storage. This module provides get/1, put/1, and delete/1 functions. ets and dets can be chosen, see [[Configuration]] in detail.
• kai_version: kai_version updates version information of each data, and generates ordering of them.
• kai_connection: kai_connection implements connection pool and manages sockets. In order to connect other nodes, a socket in the pool is leased. Replacement algorithm used in this modules is LRU.
• kai_sync: kai_sync compares buckets with other nodes, and synchronize them if needed.
• kai_membership: kai_membership monitors membership of the cluster, and updates consistent hashing.
• kai_coordinator: kai_coordinator has a role of coordinator, which routes requests and resolves inconsistency of responses by using quorum algorithm.
• kai_tcp_server: kai_tcp_server implements basic behaviour of generic TCP servers. This modules is used in kai_api and kai_memcache.
• kai_rpc: kai_rpc provides internal RPC, which enables RPC calls to kai_hash, kai_store, and kai_membership.
• kai_memcache: kai_memcache provides memcache API.
• kai_sup:kai_sup is a supervisor module of Kai.
• kai: kai is an application module of Kai.

Coding Standards

Basically follow "Programming Erlang" and "Programming Rules and Conventions"

Indent using 4 spaces for each level

;; .emacs for erlang-mode
(require 'erlang-start)
(add-hook 'erlang-mode-hook
         '(lambda ()
            (progn
              (setq indent-tabs-mode nil)
              (setq tab-width 4)
              )))

Return values of functions are like: * {ok, [...]}, that is tagged * Value or undefined, when simple accessors like get and put * {error, Reason}, if something wrong occurs

Reading List for Developers

• Amazon's Dynamo
• Programming Erlang
• Programming Rules and Conventions
• Efficiency Guide

Roadmap

This is a roadmap of Kai's development.
Current status is in basic implementation.

• Basic implementation
  16 nodes
  2 GB/node, on memory
  1,000 qps for whole system
  < 300ms for requests of 99%
  fairly balancing loads
  kai_hash: consitent hashing with virtual nodes and buckets
  kai_store: memory storage (ets)
  kai_coordinator: quorum
  kai_sync: sync data one by another
  kai_membership: gossipe-based protocol
  ** kai_memcache: set, get, and delete
• Almost Dynamo
  64 nodes
  8 GB/node, in persistent storage
  10,000 qps for whole system
  < 300ms for requests of 99.9%
  fairly balancing loads
  kai_hash: read operations are called without gen_server:call
  kai_store: persistent storage without capacity limitation
  kai_sync: bulk and paralell transport
  kai_version: vector clocks
  kai_coordinator: requests from clients will be routed to coordinators
  kai_rpc: process pool
  kai_memcache: process pool, cas
• Dynamo
  256 nodes
  32 GB/node
  100,000 qps for whole system
  < 300ms for requests of 99.99%
  fairly balancing loads
  kai_hash: physical placement
  kai_store: delaying deletion
  kai_sync: merkle tree
  kai_membership: Chord or Kademlia
  kai_rpc: connection pool