A Strange Difference in SObjectizer 5.5.15 and C++Actor Framework 0.14.4 Performance

The strange difference was noticed while comparing SObjectizer 5.5.15 and C++ Actor Framework (CAF) 0.14.4 performance. We have created CAF-version of thread_pool_disp benchmark from SO-5.5. The benchmark shows the performance of event scheduling on pool of worker threads and has the following description. A bunch of agents (actors) are created. Every agent sends several messages to itself at the start and then continues sending such messages until some limit is reached. After that each agent sends a special completion signal to the main agent-controller and the agent-controller finishes the benchmark when all completion signals are received.

Surprisingly to us, during benchmarking with different parameters we have found a very interesting effect: working time for CAF version grew much faster than for SO-5 version of the benchmark.

Series of benchmarks were done under 64-bit Windows 8.1 and Linux Mint 17 and the following results were obtained:

Benchmarks were performed on Intel Core i5-3450S CPU @ 2.80 GHz with 8GiB RAM with
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.23506 for x64 on Windows and GCC 4.9.3 (Ubuntu 4.9.3-8ubuntu2~14.04) on Linux.

It is a very unexpected result for us. Probably, it is our misunderstanding of CAF. In particular, how to tune CAF dispatcher for such workload profile. At the same time, it is possible that there are some flaws in CAF's dispatcher implementation and these flaws lead to performance degradation of the benchmark. We simply can’t judge because we are not CAF's experts.

However, this case shows that if you are selecting an actor framework it is better to create your own application-domain specific benchmarks. Otherwise, you can get into a difficult situation where your framework shows great numbers in synthetic benchmarks whereas it has very different performance under the actual workload.