#2 Subscribing and sending abstract messages

[Boris Sivko]

Good day!

In the base functionality of SO5 I'd like to see the next features.

Firstly, a possibility to subscribe of abstract types of messages. For instance, we have

class A: public so_5::rt::message_t { ... } //< Abtract
class B: public A { ... }
class C: public A { ... }

And, the subscribes like:

//! All messages will be got here.
void
evt_all(
    const so_5::rt::event_data_t< so_5::rt::message_t > & );

//! All B-C-messages will be got here.
void
    evt_a(
    const so_5::rt::event_data_t< A > & );

// ...

so_subscribe( m_self_mbox )
        .event( &a_hello_t::evt_all );
so_subscribe( m_self_mbox )
        .event( &a_hello_t::evt_a );

Secondly, a possibility to send the messages an abstract way. Here the example:

std::unique_ptr< so_5::rt::message_t > m( new msg_hello );

so_environment().single_timer(
    std::move( m ),
    m_self_mbox,
    2*1000 );

At the 5.1 version of SO5 both of ways doesn't work. The hello_delay example which is changed according to descriptions above, is attached to this forward ticket.

This example works as:

Mon Oct 21 17:02:08 2013
a_hello_t::so_evt_start()
so_5::rt::message_t

and an infinite loop.

My current conclusion is: SO 5.1 works with messages as strict types, either sending or delivering. We can't send an abstract message, and we can't receive an abstract message. On top of everything else, I think this kind of behaviour is unexpected for the end-user.

In my opinion, both of showed functions allow create agents and subsystems of widespread solutions, and also, make it possible to implement the high abstract logic for applications.

[Yauheni Akhotnikau]

This is because SO5 uses typeid(MSG) to identify messages subscribers. Each mbox has a map from typeid(MSG) to agents-subscribers. And inside deliver_message method there is simple one-step search for subscribers by typeid(MSG). So if you change type indication of your message from concrete class to its base class you change typeid and mbox can't find subscribers why made subscription with different typeid.

I don't think it is a good idea to implement complex polymorphic analyses of real message type, all its bases and so on during message delivery. It will just take too much time and slow SO5 speed down.

And I don't see many areas where such behaviour will be critical. If your application-specific agents need to interact with polymorphic data structures then why not to use polymorphic pointer inside non-polimorphic message, like this:

class SomeBase { ... };
class AppSpecificData : public SomeBase { ... };
class AnotherAppSpecificData : public SomeBase { ... };

class msg_app_data : public so_5::rt::message {
  public :
    std::unique_ptr< SomeBase > m_data;
    msg_app_data( SomeBase * d ) : m_data( d ) {}
};

mbox->deliver_message( new msg_app_data( new AnotherAppSpecificData(...) ) );

Or if you need to intercept (or just receive) all messages of some type inside your application for monitoring or debuging purposes, it should be done by different tools (for example via some sort of message interception API).

Something with polymorphic messages could be implemented if we create some sort of type-bounded mboxes, e.g. mboxes with limited message types. For example:

auto my_mbox = so_env->create_msg_channel< SomeBase >();
// mbox will have type like type_limited_mbox_t<SomeBase>.
...
void a_one::evt_all( const event_data_t< SomeBase > & e ) { ... }
void a_two::evt_specific_data( const event_data_t< AppSpecificData > & e ) { ... }
...
void a_one::so_define_agent() {
  so_subscribe(my_mbox).event( &a_one::evt_all );
  ...
}
void a_two::so_define_agent() {
  so_subscribe(my_mbox).event( &a_two::evt_specific_data );
  ...
}
...
// a_one and a_two will receive that message.
my_mbox->deliver_message( new AppSpecificData(...) );
...
// only a_one will receive that message.
my_mbox->deliver_message( new AnotherAppSpecificData(...) );

But I don't think it will be useful in practice.

[Boris Sivko]

The first example looks like some approach and is valuable, but I suppose, we can't push through the message to the another Process (with MBAPI for example), and we have to code more by hand (what could be error-prone and tedious).

My first practice example is the typical actions for groups of messages. Let's consider the same hierarchy ( so_5::rt::message_t <- A, A<-B, A<-C ). Assume that at every A-group-message receiving we have to do some typical action ( like monitoring update, logging ). In this case a piece of code for the SO 5.1 can looks like:

void
agent::evt_b( const so_5::rt::event_data_t< B > & msg )
{
    do_typical_action( msg );

    do_special_action_B();
}

void
agent::evt_c( const so_5::rt::event_data_t< C > & msg )
{
    do_typical_action( msg );

    do_special_action_C();
}

We have to write do_typical_action() at every event. It would be more handy in such way:

void
agent::evt_a( const so_5::rt::event_data_t< A > & msg )
{
    // typical_action
}

void
agent::evt_b( const so_5::rt::event_data_t< B > & msg )
{
    // special_action_B
}

void
agent::evt_c( const so_5::rt::event_data_t< C > & msg )
{
    // special_action_C
}

The second practice example is about "template/pattern-agents". I don't know how to name it more correctly. In the case where the agents do some valuable work and they do not know anything about concrete type.

Let's consider some typical task. A lot of agents in a system send a message to refresh some database to the agent X. However, if this event occur very often, it will overload the system. A solution: the X-event must occur not more often than in some period of time.
For this case we can create an agent Y, which receives the refresh-request-message from all subsystems and pass this message to the agent X. Thus, Y can control the sequence of messages in time, and pass (for example) not more than one message in period T (by collecting of a queue, throw away unnecessary messages, make a priority queue and so on). Besides that, we decrease the complexity of the agent X and alleviate our work.

The point is that we can use this agent not only for the DB refreshing. We can implement this kind of agent once, and use it as a pattern in a lot of projects. For example, to defend users from mass e-mail alarming. Although, some part of these problems can be successfully solved with cpp-templates.

And, from this point of view, we find the third solution for our first ABC-example. It can looks like:

void
Y::evt_a( const so_5::rt::event_data_t< A > & msg )
{
    // typical_action

    // pass msg to the agent X
}

void
X::evt_b( const so_5::rt::event_data_t< B > & msg )
{
    // special_action_B
}

void
X::evt_c( const so_5::rt::event_data_t< C > & msg )
{
    // special_action_C
}

In my mind, the last case looks very pretty (;

[Yauheni Akhotnikau]

My first practice example is the typical actions for groups of messages. Let's consider the same hierarchy ( so_5::rt::message_t <- A, A<-B, A<-C ). Assume that at every A-group-message receiving we have to do some typical action ( like monitoring update, logging ).

First of all I don't see some practical example here. Tasks like monitoring or logging don't imply message type hierarhies.

Second, there is another factor. In SO4 you can subscribe several events to one message in one state, but all those events must have different priorities. This is because SO cannot guarante order of event invocation in case of equal priorities. It means that if evt_a and evt_b are subscribed to MSG then on the first MSG occurence evt_a could be called before evt_b, but on second MSG occurence order could be the opposite: evt_b, then evt_a.

SO5 has no event priorities. Because of that you can't subscribe more than one event to one message in any state.

Your example rely on particular order of event invocation. It is not the case for SO.

The point is that we can use this agent not only for the DB refreshing. We can implement this kind of agent once, and use it as a pattern in a lot of projects. For example, to defend users from mass e-mail alarming. Although, some part of these problems can be successfully solved with cpp-templates.

And why can't you do that with current SO5 implementation? Is there any problem with templated agents in SO5?

[Boris Sivko]

First of all I don't see some practical example here. Tasks like monitoring or logging don't imply message type hierarhies.

For me it means that we have different experience and I could not explain my thoughts.

SO5 has no event priorities. Because of that you can't subscribe more than one event to one message in any state.

I never see such kind of problems. If I subscribe some magazine two times at the post office, I will receive two times at every month, and postal workers have no problems with order of delivery. Conclusion: if there are no order, the order doesn't matter.

And why can't you do that with current SO5 implementation? Is there any problem with templated agents in SO5?

Currently I didn't try to impelement any kind of template agents. Not so soon and SO could be changed at any time. But I hope and suppose the template agents will and must work.

Still, I tried to implement polymorph and fail.

[Yauheni Akhotnikau]

SO5 has no event priorities. Because of that you can't subscribe more than one event to one message in any state.
I never see such kind of problems. If I subscribe some magazine two times at the post office, I will receive two times at every month, and postal workers have no problems with order of delivery. Conclusion: if there are no order, the order doesn't matter.

Just for clarification.

Lets suppose we have an agent with two different message handlers for one message for one state:

void a_my::so_define_agent()
{
  so_subscribe( mbox ).event( &a_my::evt_first );
  so_subscribe( mbox ).event( &a_my::evt_second );
}
void a_my::evt_first( const event_data_t<MSG> & d )
{
  ...// some stuff.
  do_something_useful(); // This is a virtual call.
  ...// some stuff.
}
void a_my::evt_second( const event_data_t<MSG> & d )
{
  ...// some stuff.
  do_another_thing(); // This is a virtual call.
  ...// some stuff.
}

And now imagine that in some future someone reimplement a_my::do_something_useful and make call to so_change_state in it. Now the behaviour of agent is depending on order of invocation. If evt_first is called before, then agent changes its state and evt_second is never called (because agent switched to another state). If evt_second is called before, then evt_first is called next. And both scenaries could apply to the same agent at its life time!

[Boris Sivko]

The order of invocation - is a problem?

I want to notice, that the order of invocation is already a problem in the SO.

My example a little bit more complicated, but I tried to simplify it as possible.

There will be three agents: A, B, C.
To make example more clear, A and B works on one thread, C is external. In all example there is one mbox.

A and B receive message M in their events (EAM and EBM).

In the end of EAM A generates message E to agent B, and in the end of EBM B generates message F to agent A.
Event of message E changes the state of B, and after this, agent B will not receive messages M. F - the same behaviour ( changes the state of A, and afterwards, A can't receive messages M).

C sends M two times: at a 0 time point, and after 100 ms.
EAM and EBM works 70 ms.
All another actions do not waste any time.

So, it's time to run.

C sends the first M. SO will put messages to queues of the agents A and B. And only one of our events (EAM and EBM) will start in 0 point of time.

Scenario 1:

1. 0.000 EAM.
2. 0.070 EAM sends E and finish.
3. 0.070 EBM starts.
4. 0.100 C sends the second M.
5. 0.140 EBM sends F and finish.
6. 0.140 B processes E. As a result, B will never get the second message M.
7. 0.140 EAM.

We have: two EAM, only one EBM.

Scenario 2:

1. 0.000 EBM.
2. 0.070 EBM sends F and finish.
3. 0.070 EAM starts.
4. 0.100 C sends the second M.
5. 0.140 EAM sends E and finish.
6. 0.140 A processes F. As a result, A will never get the second message M.
7. 0.140 EBM.

We have: two EBM, only one EAM.

Conclusion: behaviour and order of invocation depends on the SO.

[Yauheni Akhotnikau]

For me it means that we have different experience and I could not explain my thoughts.

Just use the real problem. What the exact problem you have, the sample of solution in SO5 and the sample of solution with polymorphic messages?

[Boris Sivko]

The sample of solution is just a sample to show a current behaviour of SO and my exploring.

And the last problem is a long story...

In a nutshell, it's a attempt to solve the problem of receiving mbapi-messages from one endpoint to two or more agents. If polymorph is possible, we could just send the message to some internal mbox, and other agents can get particular messages which are interested in.

Some time ago I had a solution at SO4, when there were two agents which were inherited from a base agent-class. The base class subscribed all necessary messages and used "postmans". The concrete agents implement an interface of message processing. If some message comes, base class did all stuff of general actions and called vitual function with concrete actions. However, this kind of implementation is not possible at the current SO5. I mean, it has no easy solution and doesn't matches like 1 to 1.

Right now for my particular task, Nicolay implement a special agent (splitter), which receives a binary message from mbapi and send this to all endpoints in his list. Other agents of this story have their own endpoints, and control subscribing. If they want to send message in return, they send it to the splitter endpoint and afterwards, it send to the real destination.
This solution has some drawbacks: only one point to send smth in return, two more endpoints in the environment, one more agent to do all of this stuff.

Polymorph was a just one of the ways to simply solve the problem. But in common, in my vision, this feature has a great potential. And, by the way, it increases our flexibility of expressing our wishes at SObjectizer writing.

[Yauheni Akhotnikau]

In a nutshell, it's a attempt to solve the problem of receiving mbapi-messages from one endpoint to two or more agents. If polymorph is possible, we could just send the message to some internal mbox, and other agents can get particular messages which are interested in.

I think you are trying to mix different problems. MBAPI should be seen as an additional framework on top of SO. And it is not necessary that MBAPI-problem would be solved at SO-level. I prefer to solve it at MBAPI-level.

Yet again: the problem with splitting MBAPI messages flow to different agents doesn't seem related to polymorphism. The root of that problem is an assumption that MBAPI endpoint (+stagepoints) would be used as plain and straight message flow, without any flow forking/merging and message interception. May be this is MBAPI4 design flaw and we shoult attack it in further MBAPI development.

MBAPI3 on top of SO4 has great flexibility. User could use different message analyzers, interceptors and postmans and it gave many opportunities for developers. But price was (and is in current SO4/MBAPI3 applications) a very, very poor perfomance. Something around 60K MBAPI3 messages/sec. But before that the perfomance was yet more awful -- from 4K to 6K MBAPI3 msg/sec. And that was a really BIG PROBLEM. After some optimization and tweeking perfomance of MBAPI3 was improved but:
a) it is steel not sufficient;
b) MBAPI3 message dispatching scheme has big flaws in design. For example, you could make chain of message processors (agent A, then agent B, then agent C). But if agent B is gone by some reason whole chain will not be broken and will work, but without B. In many cases it is not appropriate.

So in MBAPI4 we address some of MBAPI3 drawbacks: poor perfomance and poor control on message processing chains. But because MAPI4 is at very early stage of its life cycle there are steel problems to solve. Forking/merging message flow inside stagepoints/endpoints is one of them. And there are few approaches to solve it. Will it be a message polymorphism or direct fork/merge point description? I don't know yet. But it seems that fork/merge description is more straight, declarative and more efficient solution.

[Boris Sivko]

Foremost, I want to notice, that my example (mbapi-problem) it's not as an example, which must be solved with a polymorph. It's just an example, where the SO flexibility could solve some class of problems.

I think it's clear that we can solve any problem with Turing Machine, but question is: the cost.

From my first point of view, the task is: how to create an agent, which will be able to receive groups of messages (of different types) in one event (input-point).

Now I'll try to show it through other example.

In C++, we can process groups of parameters in one function:

class A
{
    public:
        void
        do_smth();
};

class B : public A
{
    // ...
};

void
func( A & a )
{
    a.do_smth();
}

int main()
{
    B b;
    A a;

    func( a );
    func( b );

    return 0;
}

and it would be useful to do stmh similar in an event processing.

In fact, that's all that I wanted to say.

[Boris Sivko]

Unfortunately, we have a crunch-time now and I will continue discussion in a week or two.

[Boris Sivko]

I'm back. And I'm back with some good news.

First of all, I want to introduce a decision for some part of discussed problems. It's an agent "a_sluice", and an example of which is attached to this message.

It's a template-agent (templates works (; ), has compiled and tested at the last version of 5.1 (out of the branch).

"a_sluice" solves the showed problem:

Let's consider some typical task. A lot of agents ... Although, some part of these problems can be successfully solved with cpp-templates.

By the way, "a_sluice" is already used in a project (and at three places), and soon will be in an action application.

By experience, "a_sluice" helps to move out its specific logic from the application and makes behaviour in the system more clear.

[Boris Sivko]

BTW, is it correct (according to the SO ideology) to send msg of "so_5::rt::message_t" type and receive msg of "so_5::rt::message_t" type? As it happens at the first post:

Mon Oct 21 17:02:08 2013
a_hello_t::so_evt_start()
so_5::rt::message_t

[Yauheni Akhotnikau]

BTW, is it correct (according to the SO ideology) to send msg of "so_5::rt::message_t" type and receive msg of "so_5::rt::message_t" type?

From technical point of view there is nothing wrong with such approach. But generally speaking is has no sense.

The first reason is the presence of message_t as a base class for all messages. It is here not to say to user "you should make object hierarhies of your message classes (as for your exception classes)". The need in message_t is because somewhere should be a store for service information (like reference counters and so on). In SO4 any class may be used as message class. But the price for that is an additional object message_wrapper created by SO4 for storing service information. It means two allocation for message instead of just one allocation in SO5.

Because of that message_t should be regarded not as root of complex class hierarhy, but as a small mixin for adding store space for service information in your message objects.

The second reason is the usage of message type as message identifier. In SO4 there were different thing: message name (part of message indentification) and message type. It was like weak type system (it allows some useful tricks like handling messages of different types but at the price of damaging something in case of error). As result there were poor perfomance and possibility of misinterpretation of message object type. So, in SO5 there is only one thing: message type, which is used as message indentifier and as part of more stronger type system.

And another reason. SObjectizer is a simple tool for delivering message of concrete type A to recipients of message objects of type A. If some user wants to handle of messages similar way to handling exceptions (with deep exception hierarhies) then there is, probably, a problem for such user.