[OpenCoin-devel] A description of my code

[mathew r. <mat...@gm...>]

A brief introduction to the workings of my sandbox, in generally the
order it was written in.
Note: there may be errors in this, but I try to describe things in a
way that build up to the logic of messages.

==================
==================
message.py

message.py is the beginning work of the protocol. It was envisioned as
the interface between the logic of the parts which use the system and
the data passing through the network. To do this, some parts were
needed. First, MessageType.

-----------------------
message.MessageType:
MessageType is used to hold all the applicable parts of a
conversation. Different MessageHandlers are registered to it. Also,
fuctions register callbacks with the MessageType to be informed of all
messages which are sent or received.

When it receives input (via the function addInput) it first gets a
MessageHandler object for the message it received. The MessageType
then handles the message. It finally calls all the callback functions
with the result of the handling (which is the Message itself).

MessageTypes have two special holders for their state:

globals is used to hold the registered MessageHandlers, information to
verify proper ordering of messages, and the status.
The status is used to ensure that messages are being sent and received
by the proper half of the conversation. For instance, a wallet should
never try to handle a DSDB_KEY_REQUEST from an issuer. Likewise, the
issuer should never try to handle the DSDB_KEY_PASS from the wallet.

persistant is used as a placeholder for the MessageHandlers to put the
information encoded in them. This allows some decoupling between the
logic of actual processes from the encoding of the messages.
persistant is cleared everytime the conversation passes through a Goodbye/Hello.
For example: DSDB_KEY_REQUEST -> DSDB_KEY_PASS ->
MINTING_KEY_REQUEST_KEY_ID. The only allowed suffix for DSDB_KEY_PASS
is [Goodbye]. The only allowed prefix for MINTING_KEY_REQUEST_KEYID is
[Hello]. Persistant  gets cleared

----------------------
message.MessageHandler:
MessageHandler is the base class for all messages. Each message which
can be sent over the network is a derived class of the MessageHandler.

Each MessageHandler has various variables to define it's workings.

---Interaction with the MessageType---
identifier is the message identifier (e.g. 'DSDB_KEY_REQUEST',
'MINTING_KEY_FETCH_DENOMINATION', and 'REDEEM_COINS_REQUEST'). Each
identifier is a different message which has a different meaning

prefixes is all the messages which are allowed to come directly before
this message, as a list. The identifier of the allowed messages is
used. The special prefix Hello (which is an instance of the
HelloGoodbye class) is used to show when the message can start from a
Hello state (it can start a brand new conversation)

suffixes is the similar to prefixes. Strings of allowed identifiers to
come after the message are stated.  The special suffix Goodbye (which
is an instance of the HelloGoodbye class) is used to show when the
message can end a conversation.

handlestatus, encodestatus, and mustalreadyhave give requirements to
the MessageType. handlestatus gives the required MessageStatus of the
MessageType when used to handle. encodestatus gives the required
MessageStatus when used to encode. mustalreadyhave defines if the
MessageType has to already have the required status already active to
use.

----Other variables----
grammer is the grammar for the message. This is planned to be used to
put the encoding information for the message.

func is a function to be called whenever the MessageHandler handles or encodes

messageLayer is a link to the MessageType that created the message

sentence is where the python representation of the different parts of
the message goes. This is what the grammar encodes from and decodes
to.


----Functions----
new() is used to create a new copy of the MessageHandler. This copy is
then filled with the information needed.

handle() checks the appropriate status. If we have the appropriate
status, we then handle the Message. Finally, if mustalreadyhave is
False, we set the required status.

encode() is similar to handle except that it called _encode instead of _handle

------------------
HelloGoodbye is a class used to denote the Hello and Goodbye states of
the protocol. It's two copies of it's one instance, Hello and Goodbye
are used as prefixes and suffixes.

------------------
MessageStatusBase and MessageStatusPrivilege define the statuses and
privileges used in defining the allowed ordering of messages.

MessageStatusPrivilege accepts a string status.

MessageStatusBase holds the allowed and current privileges.
__init__() accepts a basestatus, which is the allowed privileges, and
the status (active privileges).

elevate(status) tries to add the privilege to the MessageStatusBase.
If it cannot, a MessageError is raised

restrict(status) tries to remove the privilege from the active
privileges of the MessageStatusBase. If it is not an elevated status,
it raises a MessageStatusError.

reset() removes all currently elevated privileges

can(status) returns if status is a privilege which can be elevated

has(status) returns if status is currently elevated

----------------------
MessageStatuses is the base for all privileges and bases.
PrivilegeClient is the client privilege
PrivilegeServer is the server privilege
getBaseClient returns a MessageStatusBase with the PrivilegeClient as
the allowed privilege
getBaseServer returns a MessageStatusBase with the privilegeServer as
the allowed privilege
getBaseNode returns a MessageStatusBase with both PrivilegeClient and
privilegeServer as the allowed privileges

-------------------------
statusHandleClient sets up a message where the handle() function
requires PrivilegeClient, the encode() function requires
PrivilegeServer, and the privileges are required to be elevated prior
to the message being handled or encoded

statusHandleServer is similar to statusHandleClient except that
handle() requires PrivilegeServer and encode() requires
PrivilegeClient

statusHandleServerSet is similar to statusHandleServer. The difference
is that the ability to elevate a privilege is checked, and after the
message is handled or encoded, the privilege is raised.

---------------------------
BlankPresent(MessageHandler) is the message 'BLANK_PRESENT'.
It has the identifier 'BLANK_PRESENT'.
It is prefixed with [Hello]. It has allowed suffixes of
['BLANK_FAILURE', 'BLANK_REJECT', 'BLANK_ACCEPT'].
It has statusHandlerServerSet meaning that the ability to receive
PrivilegeServer for handle(), PrivilegeClient for encode() is checked,
and afterwords, the Privilege is elevated.

_handle() and _encode() move information from sentence to the
MessageLayer(MessageType)'s persistant area

--------------------------------
All other classes are similar to BlankPresent.

-------Notes-----------
Creating MessageTypes and MessageHandlers in this fashion is to allow
rapid evolution of the protocol. For instance, if we decide to require
the IS and DSDB to be at the same location, and a LOCK_COINS_ACCEPT
from the issuer to the wallet has to be followed by an
UNLOCK_COINS_REQUEST or a REDEEM_COINS_REQUEST, we would only have to
change three places: the Prefix for REDEEM_COINS_REQUEST and
UNLOCK_COINS_REQUEST to be 'LOCK_COINS_ACCEPT' and the suffixes of LCA
to be RCR and UCR.
We don't seem to have the rapid evolution I was expecting.

Now, in practice this has worked well for me. The variety of checks on
both the privileges of the MessageType and the strict prefixes and
suffixes for all messages have caught numerous errors in other areas
of code.

The function object of the MessageHandler allows me to key in whenever
a certain MessageHandler is used. I use this to notice that the next
part of my conversation is occuring.

The callback mechanism of the MessageTypes allows me to see every
message occuring over the MessageType. This helps me keep my code
cleaner and more specific.

A weakness of my current implementation (besides the lack of encoding)
is that it is possible to get input and do nothing with it, because no
callbacks are registered at the MessageType and the message itself
does not have a function to call.





======================
crypto.py:
Classes for base SigningAlgorithm, EncryptionAlgorithm,
BlindingAlgorithm, and HashingAlgorithms. Also, the KeyPair.
The base class definitions should not really be used, unless the RSA
counterparts use them.
RSAKeyPair, RSAEncryptionAlgorithm, RSABlindingAlgorithm, and
RSASigningAlgorithm are implementations the keypair and Algorithms
using RSA.
They are wrappers are pycrypto's Crypto.Util.PublicKey.RSA.

The actual implementation of performs all functions directly using the
RSAKeyPair's key (which is a pycrypto rsa key).

RSABlindingAlgorithm also can create the blinding_factor and stores it
for future use.

createRSAKeyPair(N) returns a created an RSA keypair of N bits.

SHA256HashingAlgorithm is a HashingAlgorithm using pycrypto's
implementation of SHA256.

Random is a smart RandomPool from pycrypto which ensures there is
sufficient randomness in the pool prior to giving out random numbers.
If there is not sufficient entropy, it employ's RandomPool's randomize
function which gets randomness from (by default) /dev/urandom.

_r is an instance of Random.

CryptoContainer is used in the containers for the issuer_cipher_suite
of CurrencyDescriptionDocument and the cipher of DSDBKey.

---------Notes---------
I do not like the interface of the different Algorithms. I think the
interface will be much cleaner if there are no update or reset
functions. I also think passing the HashingAlgorithm into the
SigningAlgorithm when we want to sign or verify makes sense.

RSAPublicKey has a __str__ function to print out the public key in
base64 to minimize mess while I was working. The actual working of
base64 should be handled from the encoding point

CryptoContainers has a __str__ function to only print out the
algorithm names for use in the Containers with CryptoContainers


=======================
containers.py:
containers.py attempts to give python classes to the containers
specified in protocol-containers.

Container is the base class for ContainerWithBase,

Signature is class holding a keyprint and a signature

Containers and signatures are used to create the
CurrencyDescriptionDocuments, MintKeys, and DSDBKeys.

------------------------------
ContainerWithBase is derived from Container and the base class for
ContainerWithSignature and ContainerWithAdSignature.

ContainerWithBase has a two functions:

_verifyASignature(self, signature_algorithm, hashing_algorithm,
signature, key, content_part) returns whether a signature is verified
on the content part.

_performSigning(self, key, signing_algorith, hashing_algorithm)
returns a Signature for the content part

-------------------------------
ContainerWithSignature is derived from ContainerWithBase.
It has one signature, signed by the issuer.

ContainerWithSignature has two functions:
verifySignature(...) verifies the signature against the key
setSignature(...) sets the variable self.signature to the signature

-------------------------------
ContainerWithAdSignatures is derived from ContainerWithBase
It has multiple additional signatures, not necessarily signed by a
specific party.

ContainerWithAdSignatures has two functions:
verifyAdSignatures(...) finds the same keyprint and verifies the signature
addAdSignature(...) adds a signature to the existing additional signatures

----------------------------------
CurrencyDescriptionDocument:
The Currency Description Document from an issuer.

content_part() returns a string of the content of the container

verify_self() verifies the CDD using verifySignature()

sign_self() sets the CDD signature


---------------------
MintKey:
the Mint Key for a denomination at a specific slice of time.

content_part() returns a string of the content of the container

verify_with_CDD(cdd): verifies the mint key with the CDD (ensuring the
mintkey is valid)

verify_time(time): returns if the MintKey is currently valid at a specific time.
To be valid, it returns if the MintKey can be used to mint and redeem.

-----------------------
DSDBKey:
the DSDB key used in obfuscating the blanks from one wallet to another

content_part() returns a string of the content of the container

verify_with_CDD(cdd): verifies the dsdb key with the CDD (checking if
the DSDBKey has been signed by the issuer master public key)

verify_time(time): verifies the dsdb key is valid to use with the DSDB


--------------------
--------------------

CurrencyBase:
The base class used for the three Currency classes, CurrencyBlank,
CurrencyObfuscatedBlank, and CurrencyCoin

All three Currency classes have the standard identifier, currency
identifier, denomination, key identifier and serial.

Function validate_with_CDD_and_MintingKey verifies the information of
the blank with the minting key and the minting key with the cdd
(and the curency's standard_identifier with the cdd's standard_version)

------------------------------
CurrencyBlank:
The blank generated for use with minting.

Functions:

generateSerial() generates a 128 bit random string for use as a serial

blind_blank() returns the value returned from the BlindingAlgorithm to
use with minting. blind_blank also stores the blinding factor used,
and the value returned from the blinding (although that is not useful)

unblind_signature() returns the unblinded value of the blinded signature.

newCoin() returns a new CurrencyCoin using the unblinded signature and
verifies the coin is valid if we pass in the CDD and MintingKey

-----------------------------------
CurrencyCoin:
The actual coin with signature.

validate_with_CDD_and_MintingKey(cdd, minting_key): Ensures the coin is valid.

check_similar_to_obfuscated_blank(blank): used when the wallet has
received both the obfuscated blank and the real blank and checks that
they are similar.

check_obfuscated_blank_serial(blank, dsdb_key): used to check the
serial of the coin is the same as the serial of the obfuscated blank.
This can be used in conjunction with check_similar_to_obfuscated_blank
to verify that the obfuscated blank is the same as the coin we
received.
**CAUTION** This check_obfuscated_blank_serial may not always be
usable, depending on EncryptionAlgorithms

newObfuscatedBlank(): Creates a new CurrencyObfuscatedBlank from a coin

------------------------------------
CurrencyObfuscatedBlank:
The blank sent to the other wallet when performing a transaction.

-------Notes--------
ContainerWithBase._verifyASignature(...content_part) has content_part
passed in to minimize recomputation of the content_part with multiple
signatures. In retrospect, this doesn't make any sense, since we can
match the keyprint of the signature

In the implementation right now, without encoding the information to
text, the CurrencyDescriptionDocument, DSDBKey, and MintKeys are
passed around with the RSAKeyPairs including the private keys. When
encoding is implemented, the KeyPair held by the
CurrencyDescriptionDocument, the DSDBKey, and the MintKey would not
have the private key. No changes to the implementation of the
containers should have to occur.

CurrencyDescriptionDocument should be subclassed from
ContainerWithAdSignature as well to allow third party signatures.

content_part() is an encoding of the information of the container. It
does not have to be the same representation as the one used in
transmitting, but it makes sense to have them be the same.

One of the design goals of containers.py is to put all wallet
cryptography in one place. I believe all (or almost all) of it is in
containers.py

==========================
entity.py:
Entity.py is different from everything else in the sandbox. It was
created since I needed a place for persistant information between
transactions. It also has helper functions for creating various
Containers.

For differentiating between the two states of a wallet, it can be the
wallet transferring the coins from or transferring the coins to, I
have given them different names.

A consumer wallet is the one purchasing goods with coins, so it gives
the coins to the other wallet, the merchant wallet.

A merchant wallet is the one receiving the coins from the other
wallet, checking if they are enough, verifying the DSDB, accepting the
coins, and getting new ones minted from the DSDB.

-----------------
UniverseContainer is how I find and get different entities.
I can add an is, dsdb, or a merchant wallet

They get registered using an address and the entity.

When we want to connect to one of the entities in the universe, we
call getNew*MessageType(address) and a new MessageType to the entity
is called.

--------------------
createCDD, createMK, createDSDBCertificate, and makeCoin are helper
functions for this stage of testing.

createCDD(..) returns a new CDD for the OpenCent

createMK(..) returns a new MintKey of a certain denomination, and time
restraints for a certian CDD

createDSDBCertificate(..) returns a new DSDBKey signed by the CDD

makeCoin(..) returns a new coin which has not been through the issuer.
This is used to have currency which is valid for trying out the
spending protocol.

-------------------------
linkMessageTypes(m1, m2) connects the output of one messageType to the
input of the other messageType and vice versa for testing.

--------------------------
WalletEntity is the entity of a wallet. It stores persistant state
during and between transactions.

cdds is a dictionary of cdds by the currency identifier which the
wallet knows and will accept.

coins is a list of coins we have. The implementation right now only
accepts coins we have the minting_key for. It makes sense to do a test
for coin.validate_with_CDD_and_MintingKey as well to test the validity
of the coin.

minting_keys_key_id is a dictionary of known minting_keys by their key_id

minting_keys_denomination is a dictionary of known minting keys by the
currency identifier and denomination

requests is a dictionary of requests by request_id with the blinding
factor and blanks. This is used to do FetchMintedRequests with the
issuer later (not implemented yet)

universe is a link back to the universe so we can connect to other entities

__init__(cdds, minting_keys, coins, requests, universe):
cdds is a dictionary of cdds by the currency identifier which the
wallet knows and will accept.

coins is a list of coins we have. The implementation right now only
accepts coins we have the minting_key for. It makes sense to do a test
for coin.validate_with_CDD_and_MintingKey as well to test the validity
of the coin.

minting_keys is a list of known minting_keys. We pass these in to fill
out minting_keys_denomination and minting_keys_key_id. We pass these
in because the wallet should have the minting_key for all coins it
has. It should also store every minting_key it has come across which
is still valid for redemption, within reason.

requests is a dictionary of requests by request_id with the blinding
factor and blanks. This is used to do FetchMintedRequests with the
issuer later (not implemented yet)

--
addMintingKeys(minting_keys) takes a list of mintingkeys, and adds
them to minting_keys_key_id and minting_keys_denomination

---
newMerchantWalletMessageType: returns a new
merchantwallet.MerchantWalletManager  (This is the merchant half of
the conversation. This is used by
universe.getMerchantWalletMessageType(address)).

connectTo*(messageType, *somethingelse*): connects the messageType
with the IS/DSDB/Wallet using _connectTo* and an address picked out of
*somethingelse* using linkMessageTypes.

_connectsTo*(address): returns self.universe.getNew*MessageType(address)

spendCoins(address, currency_identifier, denominations):
denominations is a list of denominations to spend.
spendCoins finds the coins meeting the list of denominations, connects
to the address, and starts a transaction with the merchant wallet.

---------------------------------
IssuerDSDBEntity:
Similar to WalletEntity. Implements both DSDB and Issuer for one currency.
The only real difference is attemptToMint() which mints all the
MintRequests and moves them to minted or mint_failures depending on if
they succeeded for failed.

---------Notes-----------
Many pieces of dsdb.py and issuer.py should be moved to the
IssuerDSDBEntity. There is a lot of code directly dealing with the
entity.

==========================
==========================
dsdb.py, consumerwallet.py, merchantwallet.py, issuer.py:

These four files contain the rest of the four implemented and
paritially tested conversations. They all have similar workings. They
also make up half the code.

dsdb.py has all the conversations with the DSDB from the POV of the DSDB.
issuer.py has all the conversations with the IS from the POV of the IS
consumerwallet.py has all the conversations with the merchant wallet
from the POV of the consumerwallet
merchantwallet.py has all the conversations with the consumerwallet,
IS, and DSDB from the POV of the merchantwallet.

To differentiate the two main ways of acting as a wallet, I have named
them consumer wallet and merchant wallet. If using the names from
protocol.txt, comsumer wallet is WalletAlice, and merchant wallet is
WalletBob.

-----------------------
---- Managers ----
-----------------------

Each of the modules defines a Manager.
dsdb.py: HandlerManager
issuer.py: HandlerManager
consumerwallet.py: ConsumerWalletManager
merchantwallet.py: MerchantWalletManager

Each of the manager's __init__ is passed in a messageType and an entity.
the ConsumerWalletManager is also passed in a set of coins to spend.
The messageType is a MessageType with the correct base privileges
(either server or client)

Each manager then sets up its first messageType.

First, it creates holders for the instances of messages with
callbacks. Those messages all come from other side of the
conversation. Note: Not all messages from the other side have callback
functions (See MerchantWallet BlankAndMintingKey), although that is
almost always the case

The manager then registers all the MessageHandlers with the
MessageType. This sets up everything allowed during the conversation.

ConsumerWalletManager and MerchantWalletManager both need to connect
to more than one entity. ConsumerWalletManager has connectToIS(cdd).
MerchantWalletManager has connectToIS(cdd) and connectToDSDB(dsdbKey).
They all setup a MessageType to the respective entity and setup the
MessageType for the conversation.

All the Managers have a resumeConversation or a startConversation
function. This function was setup as a callback from the
MessageHandlers which come from the other party in the conversation.
When the message is handled, the callback is called and goes into
resume/startConversation.

*Conversation has a switch statement to find the type of Message
caused the callback and then starts a ConversationHandler for that
Message family (see below).

ConsumerWalletManager has one special function, startConversation(),
which begins the transaction by sending a DSDB_KEY_REQUEST to the IS.
Only ConsumerWalletManager has this because every other Manager starts
out by getting some input from someone else.

----- ConversationHandler -------

Each Message family has a conversation handler. A message family is a
convenient grouping of messages to put together. For example, the DSDB
messages LOCK_COINS_* is in the message family LockCoins.
UNLOCK_COINS_* is in the message family UnlockCoins.

Having the message families grouped has worked well when we have
privilegeServer, since we receive a message from the message family,
and then we return messages from the message family. It does not work
as well in with privilegeClient because when we receive a message, we
send a message from a different message family.

However, grouping with message families together with the prefixes and
suffixes of the MessageHandlers does prove useful with both
privilegeClient and privilegeServer because the callbacks do force the
exact conversation between all parties to occur in a certain pattern.

ConversationHandlers for a message familty are implemented by classes
in the file.

An example which runs as a Server is dsdb.py:LockCoins.

when startConversation makes the LockCoins instance, it passes in the
manager and the firstMessage (which is promptly forgotten).

__init__ stores a link to the manager and adds self.handle() as a
callback for the manager.messageType. This works for the first message
and any future messages.

when LockCoins.handle is called with a message, it first checks that
the message is in the message family. If it is not, it raises a
MessageError.

LockCoins.handle then has a switch statement.
If we received a LockCoinsRequest, we move information from
messageType.persistant (the information from the message) to the
ConversationHandler, and call the lock function.

the lock function returns type, result. type is either 'ACCEPT' or
'REJECT'. It uses the information from the type and result to decide
what kind of message to return.

If type == 'ACCEPT', it moves information to messageType.persistant
for the next message, removes the callback for self.handle, and then
outputs the message.

This is an example message. There are minor differences in this
portion with most of the messages we receive.

---- Notes about common things ----
Most function calls from ConversationHandler.handle follow the type,
result format. This is to keep the logic cleaner rather than pushing
exceptions or logic up into the handle function. I use this
extensively to pass whether things succeed or fail up the chain. I
feel it's cleaner than trying to use exceptions. Python programmers
may disagree with me.

Almost all (if not every) member function does not need direct access
to the ConversationHandler. They should (almost) all have been passed
in every argument they need.

Many member functions should be part of the entity, not in the
ConversationHandler. A good example of this is the IS/DSDB
interactions. They are all acting on complex types which would change
if you used a sql database as a backend instead of python objects

---- Uncommon things ----

MerchantWalletHandler has success and failure functions. When I
started working with them, I figured this may be a way to return
information to the client. I haven't used them in this way yet

MerchantWalletHandler has a complex ConversationHandler
BlankAndMintingKeys. This Conversation Handler handles two families of
messages, the family with the conversation starting with a
'BLANK_PRESENT', and the family starting with
'MINTING_KEY_REQUEST_...'. It works in this fashion becuase it keeps
information local. When it gets the blanks, it goes to the issuer and
gets the minting keys it needs, both for the obfuscated blanks
presented and the blanks it creates for minting. It loops until it has
everything it needs