#833 Events do not work (API_EventTimeout) if the client is multi threaded.

[Giacomo S]

The attached command line application is a client reading one attribute for each of the ones specified in the command line.
For each command line argument, a thread is created and subscribe_event is called within the thread itself. It is easily demonstrated that as soon as the number of reading threads is greater than a single one, the events stop working, as you can see in the pictures that you can find in the same archive.
The application is the simplest Qt CLI application possible to reproduce the issue.
I think this bug has been hanging around for a long time but been masked by other issues causing the same symptoms. I marked it high priority cause it prevents tango events from working in a multi threaded environment.

Package versions under test:
tango-9.2.2
omni-4.2.1
zeromq-4.0.7

Building instructions:
qmake (qt5)
make

[Bourtembourg Reynald]

Hi Giacomo,

Thanks for the bug report.
I managed to reproduce the bug with your code and with tango 9.2.5.
I also found a small work-around.
I just added the following line at the end of the loop which creates all the threads:

    sleep(1);

On my test machine, this simple work-around seems to work.
I suspect there is a concurrency issue during the Tango event initialization phase or subscription phase...

We'll try to find a solution...
Cheers,
Reynald

[Giacomo S]

Thank You Reynald.
Hope you are doing fine and let's stay in touch then!
let me know when something is ready to be tested!

P.S. write me to giacomo.strangolino@elettra.eu for work, here for
everything else!

Bye!

Giacomo.

Il giorno lun 30 gen 2017 alle ore 13:47 Bourtembourg Reynald bourtemb@users.sf.net ha scritto:

Hi Giacomo,

Thanks for the bug report.
I managed to reproduce the bug with your code and with tango 9.2.5.
I also found a small work-around.
I just added the following line at the end of the loop which creates all
the threads:

sleep(1);

On my test machine, this simple work-around seems to work.
I suspect there is a concurrency issue during the Tango event
initialization phase or subscription phase...

We'll try to find a solution...
Cheers,
Reynald

---

• [bugs:#833] https://sourceforge.net/p/tango-cs/bugs/833/ Events do not
  work (API_EventTimeout) if the client is multi threaded.*

Status: open
Labels: kernel events multithread
Created: Mon Jan 30, 2017 11:00 AM UTC by Giacomo S
Last Updated: Mon Jan 30, 2017 11:00 AM UTC
Owner: nobody
Attachments:

• tangoclithreadtest.tar.gz
  https://sourceforge.net/p/tango-cs/bugs/833/attachment/tangoclithreadtest.tar.gz
  (322.5 kB; application/gzip)

The attached command line application is a client reading one attribute
for each of the ones specified in the command line.
For each command line argument, a thread is created and subscribe_event is
called within the thread itself. It is easily demonstrated that as soon as
the number of reading threads is greater than a single one, the events stop
working, as you can see in the pictures that you can find in the same
archive.
The application is the simplest Qt CLI application possible to reproduce
the issue.
I think this bug has been hanging around for a long time but been masked
by other issues causing the same symptoms. I marked it high priority cause
it prevents tango events from working in a multi threaded environment.

Package versions under test:
tango-9.2.2
omni-4.2.1
zeromq-4.0.7

Building instructions:
qmake (qt5)
make

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/tango-cs/bugs/833/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Giacomo S]

Hello everyone.
The attachment provided contains a simple pthread client that works the same way as the Qt example: one thread for each command line argument (device/attribute).
Qt is not involved this time, but the behavior is the same as it was in the Qt example, as you can see from the attached PNG pic (which you can find also in the tar/gz archive).
There are 4 threads involved, the main, the two created by the client, and a fourth where push_event is called.
pthread experts can dig into this simple code, but its correctness seems to be confirmed by the behaviour at exit time, which is as expected. The four threads are evidentiated by different colors. Note that subscribe and unsubscribe take place in the very same thread. As in the qt example, the push_event is called from another thread.

Giacomo.

[Bourtembourg Reynald]

Hi Giacomo,

Thank you very much for sending the pthread client.
There is an easy way to make the events work with this client.
Simply add

omni_thread::ensure_self se;

in ReadingPThread class.

This does not work with the Qt example. So the 2 clients are not fully equivalent.
I think this is because ReadingPThread constructor is invoked from a new thread in the pthread example whereas it is invoked from the main thread in the Qt example.
Adding

omni_thread::ensure_self se;

at the beginning of ReadingThread::run() method seems to solve the problem on the Qt example.

All these problems are coming from the fact that Tango is checking whether we are in the main thread or in another thread in various locations of the code but these checks are done using omni thread IDs. In the examples you gave, the threads which were created were not omnithreads, so they had no omnithread IDs...
Using omni_thread::ensure_self will create a dummy omni_thread for your current thread which will keep the same ID as long as the ensure_self object is alive.
I'll try to see whether there is a way to avoid to add the above line of code and I will investigate a bit deeper to ensure there is no side effect with the work around I suggested above...

Cheers,
Reynald

[Giacomo S]

Hello Reynald!
Great!!
Thanks a lot!
Keep me up to date!
Giacomo

Il giorno mer 1 feb 2017 alle ore 11:35 Bourtembourg Reynald bourtemb@users.sf.net ha scritto:

Hi Giacomo,

Thank you very much for sending the pthread client.
There is an easy way to make the events work with this client.
Simply add

omni_thread::ensure_self se;

in ReadingPThread class.

This does not work with the Qt example. So the 2 clients are not fully
equivalent.
I think this is because ReadingPThread constructor is invoked from a new
thread in the pthread example whereas it is invoked from the main thread in
the Qt example.
Adding

omni_thread::ensure_self se;

at the beginning of ReadingThread::run() method seems to solve the problem
on the Qt example.

All these problems are coming from the fact that Tango is checking whether
we are in the main thread or in another thread in various locations of the
code but these checks are done using omni thread IDs. In the examples you
gave, the threads which were created were not omnithreads, so they had no
omnithread IDs...
Using omni_thread::ensure_self will create a dummy omni_thread for your
current thread which will keep the same ID as long as the ensure_self
object is alive.
I'll try to see whether there is a way to avoid to add the above line of
code and I will investigate a bit deeper to ensure there is no side effect
with the work around I suggested above...

Cheers,
Reynald

---

• [bugs:#833] https://sourceforge.net/p/tango-cs/bugs/833/ Events do not
  work (API_EventTimeout) if the client is multi threaded.*

Status: open
Labels: kernel events multithread
Created: Mon Jan 30, 2017 11:00 AM UTC by Giacomo S

Last Updated: Tue Jan 31, 2017 04:38 PM UTC
Owner: nobody
Attachments:

• tangoclithreadtest.tar.gz
  https://sourceforge.net/p/tango-cs/bugs/833/attachment/tangoclithreadtest.tar.gz
  (322.5 kB; application/gzip)

The attached command line application is a client reading one attribute
for each of the ones specified in the command line.
For each command line argument, a thread is created and subscribe_event is
called within the thread itself. It is easily demonstrated that as soon as
the number of reading threads is greater than a single one, the events stop
working, as you can see in the pictures that you can find in the same
archive.
The application is the simplest Qt CLI application possible to reproduce
the issue.
I think this bug has been hanging around for a long time but been masked
by other issues causing the same symptoms. I marked it high priority cause
it prevents tango events from working in a multi threaded environment.

Package versions under test:
tango-9.2.2
omni-4.2.1
zeromq-4.0.7

Building instructions:
qmake (qt5)
make

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/tango-cs/bugs/833/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Giacomo S]

One observation.
In the pthread client example, I added a

omni_thread::ensure_self se;

as a class member, to ensure that its life lasts as long as the object exists.

Terminating the application produces in this case this output

terminate called after throwing an instance of 'omni_thread_invalid'
Aborted (core dumped)

2 0x00007fa393d714ed in __gnu_cxx::__verbose_terminate_handler () at /build/gcc/src/gcc/libstdc++-v3/libsupc++/vterminate.cc:95

3 0x00007fa393d6f2a6 in __cxxabiv1::__terminate (handler=<optimized out="">) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47</optimized>

4 0x00007fa393d6e239 in __cxa_call_terminate (ue_header=ue_header@entry=0xb14e50) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_call.cc:54

5 0x00007fa393d6ebed in __cxxabiv1::__gxx_personality_v0 (version=<optimized out="">, actions=<optimized out="">, exception_class=5138137972254386944, ue_header=<optimized out="">, context=0x7ffcbe6204a0)</optimized></optimized></optimized>

at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_personality.cc:676

6 0x00007fa3937d4f43 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0xb14e50, context=context@entry=0x7ffcbe6204a0) at /build/gcc/src/gcc/libgcc/unwind.inc:62

7 0x00007fa3937d52ab in _Unwind_RaiseException (exc=exc@entry=0xb14e50) at /build/gcc/src/gcc/libgcc/unwind.inc:131

8 0x00007fa393d6f4fb in __cxxabiv1::__cxa_throw (obj=0xb14e70, tinfo=0x7fa3948dfcd0 <typeinfo for="" omni_thread_invalid="">, dest=0x0) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc:82</typeinfo>

9 0x00007fa3946de9f1 in omni_thread::release_dummy() () from /usr/local/omniorb-4.2.1/lib/libomnithread.so.4

10 0x0000000000402d78 in omni_thread::ensure_self::~ensure_self (this=0x7fa38c000980, __in_chrg=<optimized out="">) at /usr/local/omniorb-4.2.1/include/omnithread.h:644</optimized>

11 0x0000000000402c15 in ReadingPThread::~ReadingPThread (this=0x7fa38c0008c0, __in_chrg=<optimized out="">) at readingpthread.cpp:124</optimized>

12 0x0000000000402c5c in ReadingPThread::~ReadingPThread (this=0x7fa38c0008c0, __in_chrg=<optimized out="">) at readingpthread.cpp:127</optimized>

13 0x000000000040319a in clean () at main.cpp:23

I guess it's normal in this case, but it would be perfect if we can avoid this behavior.

Thanks again.
Giacomo

[Bourtembourg Reynald]

Hello,

You're right.
If you put :

omni_thread::ensure_self se;

at the beginning of ReadingPThread::subscribe() method, it should work fine without the crash at exit. (Don't forget to remove the ensure_self ReadingPThread class member in this case).

Hoping this helps,
Reynald

[Giacomo S]

Hello Reynald.
Yes, it helps, thank you very much!

Before adding this to the qtango framework (looks like a dirty hack to me),
I would like to understand what it actually does,
and why for example putting it as a class member behaves differently
respect to putting it in the subscribe method.
In both cases omni_thread::ensure_self se is created lives and is destroyed
in the same thread (correct me if I am wrong)

Thank You for providing this quick workaround and for any information you
will kindly give me.

Best regards, Giacomo.

Il giorno gio 2 feb 2017 alle ore 10:34 Bourtembourg Reynald bourtemb@users.sf.net ha scritto:

Hello,

You're right.
If you put :

omni_thread::ensure_self se;

at the beginning of ReadingPThread::subscribe() method, it should work
fine without the crash at exit. (Don't forget to remove the ensure_self
ReadingPThread class member in this case.

Hoping this helps,
Reynald

---

• [bugs:#833] https://sourceforge.net/p/tango-cs/bugs/833/ Events do not
  work (API_EventTimeout) if the client is multi threaded.*

Status: open
Labels: kernel events multithread
Created: Mon Jan 30, 2017 11:00 AM UTC by Giacomo S

Last Updated: Wed Feb 01, 2017 01:19 PM UTC
Owner: nobody
Attachments:

• tangoclithreadtest.tar.gz
  https://sourceforge.net/p/tango-cs/bugs/833/attachment/tangoclithreadtest.tar.gz
  (322.5 kB; application/gzip)

The attached command line application is a client reading one attribute
for each of the ones specified in the command line.
For each command line argument, a thread is created and subscribe_event is
called within the thread itself. It is easily demonstrated that as soon as
the number of reading threads is greater than a single one, the events stop
working, as you can see in the pictures that you can find in the same
archive.
The application is the simplest Qt CLI application possible to reproduce
the issue.
I think this bug has been hanging around for a long time but been masked
by other issues causing the same symptoms. I marked it high priority cause
it prevents tango events from working in a multi threaded environment.

Package versions under test:
tango-9.2.2
omni-4.2.1
zeromq-4.0.7

Building instructions:
qmake (qt5)
make

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/tango-cs/bugs/833/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Bourtembourg Reynald]

As I wrote before, using omni_thread::ensure_self will create a dummy omni_thread for your
current thread which will keep the same ID as long as the ensure_self object is alive.
The omnithread documentation states the following about release_dummy() function called from the destructor of ensure_self:

This function MUST be called before the thread exits. Throws omni_thread_invalid if the calling thread does not have a dummy omni_thread.

This explains why when you put the ensure_self object as a member of ReadingPThread class, you get the omni_thread_invalid exception which is thrown, because ~ReadingPThread destructor is called from the main class in your case, not from the created thread.
If you delete ReadingPThreads objects at the end of create_reader() function it won't crash, even if you put ensure_self as class member of ReadingPThread.

Cheers,
Reynald

[Giacomo S]

Thank You Reynald.
I've just noticed that the deletion takes place in the main thread. I was
convinced it did not, but now the reason you addressed is clear to me.
One more thing. In Claudio's omni thread example, the events were received
in different threads, according to where they had done the subscription.
In the pthread/Qt examples, another thread involves push_event. Is this
(apparently weird) behavior in accord with Tango relying on omnithread and
not dealing with pthreads?

Thanks, Giacomo.

Il giorno gio 2 feb 2017 alle ore 11:32 Bourtembourg Reynald bourtemb@users.sf.net ha scritto:

As I wrote before, using omni_thread::ensure_self will create a dummy
omni_thread for your

current thread which will keep the same ID as long as the ensure_self
object is alive.

The omnithread documentation states the following about release_dummy()
function called from the destructor of ensure_self:

This function MUST be called before the thread exits. Throws
omni_thread_invalid if the calling thread does not have a dummy omni_thread.

This explains why when you put the ensure_self object as a member of
ReadingPThread class, you get the omni_thread_invalid exception which is
thrown, because ~ReadingPThread destructor is called from the main class in
your case, not from the created thread.
If you delete ReadingPTreads objects at the end of create_reader()
function it won't crash, even if you put ensure_self as class member of
ReadingPThread.

Cheers,
Reynald

---

• [bugs:#833] https://sourceforge.net/p/tango-cs/bugs/833/ Events do not
  work (API_EventTimeout) if the client is multi threaded.*

Status: open
Labels: kernel events multithread
Created: Mon Jan 30, 2017 11:00 AM UTC by Giacomo S

Last Updated: Thu Feb 02, 2017 09:34 AM UTC
Owner: nobody
Attachments:

• tangoclithreadtest.tar.gz
  https://sourceforge.net/p/tango-cs/bugs/833/attachment/tangoclithreadtest.tar.gz
  (322.5 kB; application/gzip)

The attached command line application is a client reading one attribute
for each of the ones specified in the command line.
For each command line argument, a thread is created and subscribe_event is
called within the thread itself. It is easily demonstrated that as soon as
the number of reading threads is greater than a single one, the events stop
working, as you can see in the pictures that you can find in the same
archive.
The application is the simplest Qt CLI application possible to reproduce
the issue.
I think this bug has been hanging around for a long time but been masked
by other issues causing the same symptoms. I marked it high priority cause
it prevents tango events from working in a multi threaded environment.

Package versions under test:
tango-9.2.2
omni-4.2.1
zeromq-4.0.7

Building instructions:
qmake (qt5)
make

---

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/tango-cs/bugs/833/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Bourtembourg Reynald]

I think you didn't send the code for the omnithread example... but according to the output you've sent, it seems like the only difference is for the first received events which correspond to the synchronous calls ocurring during the subscription.
I think the answer to your question might be found in the following section of the code with some comments illustrating why it was done like that:
https://github.com/tango-controls/cppTango/blob/tango-9-lts/cppapi/client/event.cpp#L1214
(cppapi/client/event.cpp lines 1214-1239)
Here are the comments:

// Following code is for the case of event subscription in one event callback
// In such a case, we have to do  the subscription in a thread  otherwise we have a deadlock due to the
// event consumer thread which can not at the same time execute the call back and register the new event

I guess Claudio's omni thread example is going through this part of the code, which is creating a new thread for the event subscription because Tango thinks the subscription was done from a callback (which is probably wrong in this case). And since some deadlocks have been observed in the past in this particular case, tango is doing the subscription in another thread.
This is just my guess...
Is this really a problem?