How Actor Model Can Be Used Inside Single Process C++ Applications

Over the last few years the Actor model has become rather popular. But it seems that Erlang and Akka (widely used and well known implementations of the actor model) are primarily used in development of distributed applications. This can make you think that the actor model is oriented only towards distributed application development.

It's hard to say that this impression is completely wrong. The development of some distributed application is significantly easier when the actor model is used. But this is just a consequence of the main feature of that model: usage of asynchronous interaction between isolated actors. Due to asynchronous message passing, sometimes, there is hardly any difference between in-process and inter-process communications.

The usage of the actor model has sense even in development of single process applications. Even traditional desktop GUI applications can benefit if the actor model and async message passing are used inside them. Especially, when modern CPUs are getting more and more cores and modern OSes can effectively manage thousands of threads.

We argue that usage of the actor model in complex C++ applications has even more sense. It is because C++ is not a safe language and multithreading programming in C++ is hard. So, any approach and any tool which simplifies the development of multithreaded program in C++ can save a lot of time and reduce the number of defects.

A C++ framework with implementation of the actor model provides at least two important things for application developers:

1. Worker threads management. Actions like: starting, stopping and load balancing can and should be done via framework. Developers shouldn't care about them.
2. Ready-to-use facilities for asynchronous message passing must be provided via framework. The developer shouldn't implement various types of message queues himself. This includes not only inter-thread message transfer but also the control of message instance lifetime. Message lifetime control is very important for one-to-many message distribution.

There are several actor frameworks for C++, available under open-source and/or commercial licenses:

• CAF (C++ Actor Framework);
• Just::Thread Pro;
• SObjectizer;
• QP (note that QP is targeted for embedded area).

Suppose you have chosen one of them. Which cases can you use it in?

The first use case seems obvious: any problem where a lot of independent or semi-independent activities must be handled concurrently. For example: a web-service with RESTful API where each REST-request can be handled by a separate actor. Or an application which performs some complex simulation where actors play various roles inside the simulation process.

This use case has the following distinctive features:

• there could be many actors. Tens of thousands, hundreds of thousands, millions, tens of millions or even more. Actors can be created and destroyed at very high rates. That’s why an actor must be as lightweight as possible and the cost of actor creation and deletion must be minimal;
• there is no need for precise control of working context for most of the actors. All actors can work on the same working thread. Or all actors can work on a single pool of working threads with some appropriate scheduling policies (like work requesting or work stealing);
• message processing is almost always a very quick operation (sometimes the whole message processing is just a repacking of one message into another message).

The second use case is not so obvious. It is the case where an application requires precise control of its working threads. For example, if an application performs long CPU-intensive calculations and needs to have dedicated threads for such operations. Or if an application does some manipulations with a hardware device attached to the computer and requires separate threads for low-level operations via I/O ports. Or if an application does blocking calls to 3rd-party libraries or external services and requires dedicated threads for that.

This use case is rarely described as appropriate for the actor model. But we have found that the actor model can be successfully used in such applications and the usage of it significantly simplifies the development and maintenance.

This use case has different distinctive features:

• there are not so many actors in the application as in the first use case: typically hundreds or less, may be thousands, tens of thousands (rarely). Often all actors are created in the very beginning and live during the work of the application;
• some messages can be handled within long time intervals (seconds or even minutes);
• the developer must control the working context of its agents;
• significant amount of actors have rather complex behaviour.

Historically, the main area of application of SObjectizer framework was the second use case. This is why there is such thing as dispatcher which allows to control the working context of actors (agents in SObjectizer's terminology).

For example, SObjectizer was used in SMS/USSD aggregation platform which supported dozens SMPP/UCP/etc connections to SMS/USSD-centers on one side and different types of connections from content-providers on the other side plus did intensive work with RDBMS. The work was distributed among hundreds groups of agents which were bound to different instances of various dispatchers. As a result, each group had its own working contexts which were used for different operations (like network reads/writes, DB-manipulations, business-logic related actions and so on). All these working threads were controlled by SObjectizer and developers didn’t need to manage threads manually.

There weren't any different thread-specific problems like deadlocks or data races because all communications between agents were performed via asynchronous messages. Likewise, there wasn’t synchronization code inside agents. SObjectizer guarantees that the agent is always used in thread safe conditions.

We came to conclusion that these features of an actor framework significantly simplified the application development and maintenance. That’s why it’s a good idea to consider the usage of actor framework to make the development of multithreaded applications in C++ easier. It can save a lot of your time.