Recent changes to ReadComputeWrite

ReadComputeWrite modified by David Kyle

David Kyle — Tue, 28 Nov 2017 20:38:30 -0000

--- v3
+++ v4
@@ -103,21 +103,21 @@
 class NewThread : public RCWThread {
 private:
   int x;
-  Tracked<int> tx;
+  Tracked<int> x2;
   Tracked<std::string> s;
   Tracked<std::vector<int64_t>> vec;
 public:
   void setup(Transaction &tx) override {
      tx.build("x", x).init(42).add()
-     tx = 17;
-     tx.add_init("tx", tx);
+     x2 = 17;
+     tx.add_init("x2", x2);
      tx.build("s", s).init("Hello World").add();
      tx.build("vec", vec).init({2, 3, 5, 7, 11, 13, 17}).add();
   }

   void compute(const Transaction &tx) override {
-    cout << x << " " << tx << " " << s << " " << vec.size() << endl;
-    ++x; --tx; s+= "!"; vec.push_back(0);
+    cout << x << " " << x2 << " " << s << " " << vec.size() << endl;
+    ++x; --x2; s+= "!"; vec.push_back(0);
   }

   void cleanup(Transaction &tx) override {

ReadComputeWrite modified by David Kyle

David Kyle — Fri, 13 Oct 2017 17:05:39 -0000

--- v2
+++ v3
@@ -13,6 +13,8 @@
     rcw::Transaction tr(kb);

 This will be the knowledge base that the `Transaction` reads from and updates. Note that the `Transaction` object can safely outlive the `KnowledgeBase` object passed to it; the underlying knowledge base will be kept alive as long as the `Transaction` is alive.
+
+For an example of usage, see **`tests/rcw/test_rcw_transaction.cpp`**.

 # `add()`

@@ -67,8 +69,8 @@

 The available configuration options are:

-`ro()`: Mapping will `pull()` into given variable, but not `push()`
-`wo()`: `pull()` doesn't read from the knowledge base, and instead sets the variable to its default value. `push()` works normally.
+`readonly()`/`ro()`: Mapping will `pull()` into given variable, but not `push()`
+`writeonly()`/`wo()`: `pull()` doesn't read from the knowledge base, and instead sets the variable to its default value. `push()` works normally.
 `init()`: Immediately (when `add()` is called) update the knowledge base with the current value of the variable.
 `init(value)`: Immediately (when called) set the variable to `value`; then when `add()` is called, update the knowledge base as with `init()`
 `prefix()`: The variable must be a `Tracked<std::vector<...>>`. Instead of storing the vector's contents as a native MADARA array, store it as multiple madara entries, one per element. The variable name is instead used as a prefix. The size of the array is stored in `$PREFIX.size`, while the elements are stored in `$PREFIX.0`, `$PREFIX.1`, etc.
@@ -80,11 +82,13 @@
 # Using `Tracked`
 By default, `Transaction` tracks whether a value has changed, and thus needs to be written back to the knowledge base, by keeping a copy of the original value as read during `pull()`, then checking if it has changed in `push()`. The `Tracked` template wraps arbitrary variable types, and provides its own tracking of modified state, eliminating that copy, and allowing you to treat a value as modified even if it hasn't actually changed.

-The `Tracked` template overloads most operators, allowing it to be used as if it were the wrapped type in most cases. However, to simply get the value, you must use the `*` operator (or the `get()` method), and to call methods or access member variables of the wrapped type you must use the `->` operator. The latter, however, only supports `const` methods and reading member variables. To call non-`const` methods and change member variables, call the `get_mut()` method (which immediately marks the value as modified), and use the returned reference.
+The `Tracked` template overloads most operators, allowing it to be used as if it were the wrapped type in most cases. However, to simply get the value, you must use the `*` operator (or the `get()` method), and to call methods or access member variables of the wrapped type you must use the `->` operator. The latter, however, only supports `const` methods and reading member variables. To call non-`const` methods and change member variables, call the `get_mut()` (or its synonym `get_mutable()`) method, which immediately marks the value as modified, and use the returned reference.

 There are two specializations of `Tracked` provided; one for `Tracked<std::string>` and `Tracked<std::vector>`. The former simply provides most `std::string` methods itself, allowing usage through the `.` operator, and allowing more accurate modification tracking than `get_mut()`.

 The latter also forwards most `std::vector` methods, but also provides additional tracking which makes `pull()`, and space usage, much more efficient than than the generic implementation would provide, or that `std::vector<Tracked<...>>` provides. With `Tracked<std::vector<...>>`, there is only a single bit of overhead for each element to track modification status. Without this specialization, there would, for example, likely be an 8-byte overhead per element on 64-bit systems.
+
+For an example of usage, see **`tests/rcw/test_rcw_tracked.cpp`**.

 # Using `RCWThread`
 The `RCWThread` class provides an easy way to implement a thread with Read-Compute-Write semantics. It handles creating and managing a `Transaction` for you. Much like `BaseThread`, you have three (albeit differently named) virtual member functions to override:
@@ -124,6 +128,8 @@

 During `compute()`, you can call `add()`/`build()` (except with `_init`/`init()`) and `remove()`. If you add a new variable, it won't get updated (with `pull()`) until the next time `compute()` is called.

+For an example of usage, see **`tests/rcw/test_rcw_prodcon.cpp`**.
+
 # Adding Custom Types to a `Transaction`
 By default, you can add variables that are:

@@ -149,3 +155,5 @@
 * `void clear_dirty(Type &o)`: clear the modification status

 If you implement the above, `Transaction` will use your type as if it were `Tracked<...>`.
+
+For an example of usage, see **`tests/rcw/test_rcw_custom.cpp`**.

ReadComputeWrite modified by David Kyle

David Kyle — Tue, 10 Oct 2017 14:11:42 -0000

--- v1
+++ v2
@@ -34,7 +34,7 @@

 ## `push()` and `pull()`

-Unlike MADARA containers, variables added to a transaction do not automatically read and write an entry in the knowledge base. Instead, the `Transaction` provides two methods, `push()` and  `pull()`, to write and read, respectively, the values in the knowledge base. If you are using an `RCWThread`, do not call `push()` or `pull()` directly. The thread object will handle that for you.
+Unlike MADARA containers, variables added to a transaction do not automatically read and write an entry in the knowledge base. Instead, the `Transaction` provides two methods, `push()` and  `pull()`, to write and read, respectively, the values in the knowledge base. If you are using an `RCWThread`, you need not call `push()` or `pull()` directly. The thread object will handle that for you.

 For example, if using a `Transaction` directly:

@@ -73,14 +73,18 @@
 `init(value)`: Immediately (when called) set the variable to `value`; then when `add()` is called, update the knowledge base as with `init()`
 `prefix()`: The variable must be a `Tracked<std::vector<...>>`. Instead of storing the vector's contents as a native MADARA array, store it as multiple madara entries, one per element. The variable name is instead used as a prefix. The size of the array is stored in `$PREFIX.size`, while the elements are stored in `$PREFIX.0`, `$PREFIX.1`, etc.

+## `const Transaction` Objects
+
+The `const`-ness of a `Transaction` refers to access to its underlying knowledge base, not the object itself. Therefore, a `const Transaction` can have variables added or removed, but cannot `push()`, `pull()`, or use the `_init` forms of `add()`. This is especially useful for `RCWThread`, where the `compute()` method takes a `const Transaction &`, and thus prevents accidentally breaking the read-comput-write pattern in that method.
+
 # Using `Tracked`
 By default, `Transaction` tracks whether a value has changed, and thus needs to be written back to the knowledge base, by keeping a copy of the original value as read during `pull()`, then checking if it has changed in `push()`. The `Tracked` template wraps arbitrary variable types, and provides its own tracking of modified state, eliminating that copy, and allowing you to treat a value as modified even if it hasn't actually changed.

 The `Tracked` template overloads most operators, allowing it to be used as if it were the wrapped type in most cases. However, to simply get the value, you must use the `*` operator (or the `get()` method), and to call methods or access member variables of the wrapped type you must use the `->` operator. The latter, however, only supports `const` methods and reading member variables. To call non-`const` methods and change member variables, call the `get_mut()` method (which immediately marks the value as modified), and use the returned reference.

-There are two specializations of `Tracked` provided; one for `Tracked<std::string>` and `Tracked<std::vector>`. The former simply provides most of the `std::string` methods itself, allowing usage through the `.` operator, and allowing more accurate modification tracking than `get_mut()`.
+There are two specializations of `Tracked` provided; one for `Tracked<std::string>` and `Tracked<std::vector>`. The former simply provides most `std::string` methods itself, allowing usage through the `.` operator, and allowing more accurate modification tracking than `get_mut()`.

-The latter also forwards most of `std::vector` methods, but also provides additional tracking which makes `pull()`, and space usage, much more efficient than than the generic implementation would provide, or that `std::vector<Tracked<...>>` provides. With `Tracked<std::vector<...>>`, there is only a single bit of overhead for each element to track modification status. Without this specialization, there would generally be an 8-byte overhead per element on 64-bit systems.
+The latter also forwards most `std::vector` methods, but also provides additional tracking which makes `pull()`, and space usage, much more efficient than than the generic implementation would provide, or that `std::vector<Tracked<...>>` provides. With `Tracked<std::vector<...>>`, there is only a single bit of overhead for each element to track modification status. Without this specialization, there would, for example, likely be an 8-byte overhead per element on 64-bit systems.

 # Using `RCWThread`
 The `RCWThread` class provides an easy way to implement a thread with Read-Compute-Write semantics. It handles creating and managing a `Transaction` for you. Much like `BaseThread`, you have three (albeit differently named) virtual member functions to override:
@@ -89,6 +93,59 @@
 * `compute()`, which operates on your variables, and
 * `cleanup()`, optionally, to handle termination of the thread.

-Ensure you do not call `push()`, `pull()`, or an `add()` or `build()` which initializes a variable in the knowledge base. These break the RCW pattern, and can produce unexpected results.
+As a simple example of an `RCWThread`:

-It is, however, OK to call `add()`/`build()` and `remove()` otherwise. Note that if you add a new variable, it won't get updated until the next time `compute()` is called.
+~~~
+class NewThread : public RCWThread {
+private:
+  int x;
+  Tracked<int> tx;
+  Tracked<std::string> s;
+  Tracked<std::vector<int64_t>> vec;
+public:
+  void setup(Transaction &tx) override {
+     tx.build("x", x).init(42).add()
+     tx = 17;
+     tx.add_init("tx", tx);
+     tx.build("s", s).init("Hello World").add();
+     tx.build("vec", vec).init({2, 3, 5, 7, 11, 13, 17}).add();
+  }
+  
+  void compute(const Transaction &tx) override {
+    cout << x << " " << tx << " " << s << " " << vec.size() << endl;
+    ++x; --tx; s+= "!"; vec.push_back(0);
+  }
+  
+  void cleanup(Transaction &tx) override {
+      // Not needed for this example
+  }
+};
+~~~
+
+During `compute()`, you can call `add()`/`build()` (except with `_init`/`init()`) and `remove()`. If you add a new variable, it won't get updated (with `pull()`) until the next time `compute()` is called.
+
+# Adding Custom Types to a `Transaction`
+By default, you can add variables that are:
+
+* any primitive type
+* `std::string`
+* `std::vector<int64_t>` or `std::vector<double>`
+* `madara::knowledge::KnowledgeRecord`
+
+If you add using the `prefix()` build option, you can add any std::vector<T>, where T is itself supported in a non-`prefix()` mapping.
+
+You can extend support to new types by implementing a set of functions in the same namespace as the type itself (or in `madara::knowledge::rcw`, but the former is preferred).
+
+For the most basic support, implement these functions, where `Type` is the name of your type, and `Basic` is one of the above default supported types:
+
+* `Basic get_value(const Type &o)`: given some object of your `Type`, return one of the default supported types.
+* `set_value(Type &o, const Basic &i)`: given the value, set the object of your `Type` accordingly
+
+With this basic support, your can add your type directly (in which case a copy will be kept with every `pull()` for later comparison), or within `Tracked<...>`.
+
+If your type tracks its own modification status,  you can also implement the following:
+
+* `bool is_dirty(const Type &o)`: return `true` if `o` has been modified; `false` if not
+* `void clear_dirty(Type &o)`: clear the modification status
+
+If you implement the above, `Transaction` will use your type as if it were `Tracked<...>`.

ReadComputeWrite modified by David Kyle

David Kyle — Mon, 09 Oct 2017 21:40:33 -0000

One way to simplify reasoning about parallel tasks is to confine them to a Read-Compute-Write cycle; i.e., all information in the shared knowledge base is read at once, then computed upon (without accessing any shared state), then written back. This allows the compute phase to avoid any locking, the read and write phases can hold the knowledge base lock for as short of time as possible.

To support this pattern, MADARA offers a Transaction class. This class handles reading from the knowledge base into variables of your choosing, and writing them back, if they have changed. MADARA also provides a Tracked template, which when applied to a type, provides a type which tracks is "dirty" status, i.e., whether it has been modified.

Finally, MADARA also provides an RCWThread which extends BaseThread and allows a thread to easily fit the Read-Compute-Write pattern.

Using `Transaction`

If you are using an RCWThread, it will create a Transaction for you; you should not create one yourself. Otherwise, to create a Transaction, simply construct it with a KnowledgeBase:

using namespace madara::knowledge;
KnowledgeBase kb;
rcw::Transaction tr(kb);

This will be the knowledge base that the Transaction reads from and updates. Note that the Transaction object can safely outlive the KnowledgeBase object passed to it; the underlying knowledge base will be kept alive as long as the Transaction is alive.

`add()`

Next, you should initialize the Transaction by adding variables to it:

int x;
tr.add("my.x", x);
double y;
tr.add("my.y", y);
std::string s;
tr.add("my.s", s);

The string given is the name of the MADARA variable. It can be any legal MADARA key, and need not have any relationship with the variable given. If it does not exist, it will be created immediately with the default 0 value. If it already exists, the value will not be touched. Any current value of the second parameter is ignored. The second parameter will have its address taken and stored. Ensure the given object lives long enough. If an object added to a Transaction is destructed, it is undefined behavior to do anything with that Transaction object except destruct it.

If you wish the initialize the corresponding entry in the MADARA knowledge base, use the add_init() method instead:

int z = 42;
tr.add_init("my.z", z);

`push()` and `pull()`

Unlike MADARA containers, variables added to a transaction do not automatically read and write an entry in the knowledge base. Instead, the Transaction provides two methods, push() and pull(), to write and read, respectively, the values in the knowledge base. If you are using an RCWThread, do not call push() or pull() directly. The thread object will handle that for you.

For example, if using a Transaction directly:

x = 9;
std::cout << x << " " << y << " " << z << " \"" << s << "\"" << std::endl;
/* Output: 9 0 42 "0" */
tr.pull();
std::cout << x << " " << y << " " << z << " \"" << s << "\"" << std::endl;
/* Output: 0 0 42 "0" */
x = 13;
std::cout << kb.get("my.x").to_integer() << std::endl; /* Output: 0 */
tr.push();
std::cout << kb.get("my.x").to_integer() << std::endl; /* Output: 13 */

`remove()`

After adding a variable to a Transaction, you can remove it with the remove() method. You can pass either the string name of MADARA entry, or a pointer to the variable you originally added.

tr.remove(&x);
tr.remove("my.y");

`build()`

The build() method provides a means for greater control of how added variables are handled by a Transaction. The add() and various add_*() methods are simply shorthand for the more fully feature build() method.

When you call build() on a transaction (with the same arguments as add()), it returns a Transaction::Builder object, which provides several methods for customizing options. Each of these methods returns the builder object itself, so you can chain these methods. To actually add the variable to the Transaction, call the add() method on the builder object. For example, to add a write-only variable that will be initialized with a given value immediately:

int b;
tr.build("my.b", b).wo().init(7).add();

The available configuration options are:

ro(): Mapping will pull() into given variable, but not push()
wo(): pull() doesn't read from the knowledge base, and instead sets the variable to its default value. push() works normally.
init(): Immediately (when add() is called) update the knowledge base with the current value of the variable.
init(value): Immediately (when called) set the variable to value; then when add() is called, update the knowledge base as with init()
prefix(): The variable must be a Tracked<std::vector<...>>. Instead of storing the vector's contents as a native MADARA array, store it as multiple madara entries, one per element. The variable name is instead used as a prefix. The size of the array is stored in $PREFIX.size, while the elements are stored in $PREFIX.0, $PREFIX.1, etc.

Using `Tracked`

By default, Transaction tracks whether a value has changed, and thus needs to be written back to the knowledge base, by keeping a copy of the original value as read during pull(), then checking if it has changed in push(). The Tracked template wraps arbitrary variable types, and provides its own tracking of modified state, eliminating that copy, and allowing you to treat a value as modified even if it hasn't actually changed.

The Tracked template overloads most operators, allowing it to be used as if it were the wrapped type in most cases. However, to simply get the value, you must use the * operator (or the get() method), and to call methods or access member variables of the wrapped type you must use the -> operator. The latter, however, only supports const methods and reading member variables. To call non-const methods and change member variables, call the get_mut() method (which immediately marks the value as modified), and use the returned reference.

There are two specializations of Tracked provided; one for Tracked<std::string> and Tracked<std::vector>. The former simply provides most of the std::string methods itself, allowing usage through the . operator, and allowing more accurate modification tracking than get_mut().

The latter also forwards most of std::vector methods, but also provides additional tracking which makes pull(), and space usage, much more efficient than than the generic implementation would provide, or that std::vector<Tracked<...>> provides. With Tracked<std::vector<...>>, there is only a single bit of overhead for each element to track modification status. Without this specialization, there would generally be an 8-byte overhead per element on 64-bit systems.

Using `RCWThread`

The RCWThread class provides an easy way to implement a thread with Read-Compute-Write semantics. It handles creating and managing a Transaction for you. Much like BaseThread, you have three (albeit differently named) virtual member functions to override:

setup(), in which you should add() your variables (typically member variables of your thread class inheriting from RCWThread) to the Transaction passed into it
compute(), which operates on your variables, and
cleanup(), optionally, to handle termination of the thread.

Ensure you do not call push(), pull(), or an add() or build() which initializes a variable in the knowledge base. These break the RCW pattern, and can produce unexpected results.

It is, however, OK to call add()/build() and remove() otherwise. Note that if you add a new variable, it won't get updated until the next time compute() is called.

Recent changes to ReadComputeWrite

ReadComputeWrite modified by David Kyle

ReadComputeWrite modified by David Kyle

ReadComputeWrite modified by David Kyle

ReadComputeWrite modified by David Kyle

Using Transaction

add()

push() and pull()

remove()

build()

Using Tracked

Using RCWThread

Using `Transaction`

`add()`

`push()` and `pull()`

`remove()`

`build()`

Using `Tracked`

Using `RCWThread`