Menu ▾ ▴

#689 Problems while getting events on a remote machine

closed-fixed
nobody
None
Miscellaneous
9
2014-12-17
2014-10-03
IMS ERAS
No

Hi,
Whithin one of the ERAS projects we are trying to make use of the events in PyTango.

We are doing something rather simple as having a tango server generating events then in a python script we do something like:

device = PyTango.DeviceProxy(args.tango_server)
try:
event_id = device.subscribe_event(
'writablecmd', PyTango.EventType.CHANGE_EVENT, event_callback,
filters=[], stateless=True
)

def event_callback(event):
"""Callback function used to receive Tango events."""
if event.errors:
print(event.errors)
return
try:
msg = event.attr_value.value
............

When we run this python script on the same machine were the tango server is running then everythin is perfectly fine and run as expected.

If we instead run the script on a remote machine (were we set accordingly the TANGO_HOST) then, when we generate an sigle event like this:

dev.writablecmd = 'MOVE Cube -5 -5 -5'
we get those kind of errors:

received MOVE Cube -5 -5 -5
(DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)
received MOVE Cube -5 -5 -5
(DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)
received MOVE Cube -5 -5 -5

So what is happening is that the event triggered only once is then periodically received every 10 sec (which seems corresponding to the keep alive thread).

If we run the same script at the same time on the local machine then everything is fine.
Changing from stateless: True <-> False does not change much this behaviour.

Below some more tracing obtained via printing the whole event.

Thanks a lot for your help

Regards

Franco

EventData[
attr_name = 'tango://10.202.104.51:10000/c3/testtango/testtango/writablecmd'
attr_value = None
device = PyTestTango(c3/testtango/testtango)
err = True
errors = (DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)
event = 'change'
reception_date = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 839816)]

EventData[
attr_name = 'tango://10.202.104.51:10000/c3/testtango/testtango/writablecmd'
attr_value = DeviceAttribute(data_format = PyTango._PyTango.AttrDataFormat.SCALAR, dim_x = 1, dim_y = 0, has_failed = False, is_empty = False, name = 'writablecmd', nb_read = 1, nb_written = 1, quality = PyTango._PyTango.AttrQuality.ATTR_VALID, r_dimension = AttributeDimension(dim_x = 1, dim_y = 0), time = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 889623), type = PyTango._PyTango.CmdArgType.DevString, value = 'MOVE Cube 0 0 0', w_dim_x = 1, w_dim_y = 0, w_dimension = AttributeDimension(dim_x = 1, dim_y = 0), w_value = 'MOVE Cube 0 0 0')
device = PyTestTango(c3/testtango/testtango)
err = False
errors = ()
event = 'change'
reception_date = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 847228)]

Related

Bugs: #689

Discussion

  • Andy Gotz

    Andy Gotz - 2014-10-04

    It sounds like you are not getting events across the network. What version of PyTango are you running? With which patches? How many network interfaces are there on your hosts?

     

  • IMS ERAS

    IMS ERAS - 2014-10-04

    We do are getting the events over the network, even too many! as a said, a single event on the tango server side, trigger an infinite loop of events with 10 sec periodicity.

    We are running on both machines:

    • Ubuntu 14.04 were we replace ZeroMQ 4 (the default on Ubuntu 14.04) with ZeroMQ 3 via dpkg (othervise events do not work at all, not event on the same machine)

    • PyTango.version
      '8.1.1'
      on python3 and installed via plain: "apt-get install python3-pytango"

    • we only apply the attached patch (that should have nothing to do with this behaviour)

    In our host we have only one interface active at the time. We tried using only wifi on both machines or using only wired connection to a router. Behaviour is exactly the same.

    I should add that all the rest is running fine, in the sense that if I run jive I can see correctly the database on the remote machine and I can launch indifferently tango servers in the local (TANGO_HOST) or remote machine without problems

    The problem occur in a completely simmetric way, in the sense that if I swap the local (TANGO_HOST) and the remote machine the behaviour is exactly the same.

     
    patch_pytango3.py

    • Andy Gotz

      Andy Gotz - 2014-10-04

      On 04/10/2014 08:24, IMS ERAS wrote:

      We do are getting the events over the network, even too many! as a said, a single event on the tango server side, trigger an infinite loop of events with 10 sec periodicity.

      Hi Franco,
      I see. It sounds like a bug to me. If you send only one event (did you
      send it using a push event call?) then you should not receive anymore
      until you push a new event. The 10 second periodicity comes from the
      keep_alive thread. It should just check that the server is still alive
      and throw an exception/error if not. If the server is alive it should
      remain silent. It should not send an event. Strangely I heard of a
      similar problem yesterday from someone else running the same version of
      Tango. I wonder if this is not a bug. I let the experts comment on this.
      If I have time I will try to reproduce your problem. It must be
      reproducible if I install the same versions as you.

      Regards

      Andy

       

  • Emmanuel Taurel

    Emmanuel Taurel - 2014-10-08

    Franco,

    I had a look at your log files. It seems to me that something is wrong in event names.
    The device server process send the heartbeat event "tango://v-eras-0:10000/dserver/xxxx" while the client is waiting for "tango://10.202.104.50:10000/dserver/xxxx"

    What are the values of your TANGO_HOST env. variables on both the client and server sides?

    Good luck

    Emmanuel

     

  • IMS ERAS

    IMS ERAS - 2014-10-08

    Dear Emmanuel,
    I have set on the local machine:
    TANGO_HOST v-eras-0:10000 (were v-eras-0 is the name of the local machine)
    on the remote machine I have set:
    TANGO_HOST 10.202.104.50:10000
    were that address correspond to the v-eras-0 machine IP address.(I have not DNS available)

    With this setting the rest of tango functionalities, like the exploration of the v-eras-0 database from the remote machine via jive is working fine.

    This evening I will try to set on both machines the use of explicit IP address (that may imagine is your implicit suggestion)

    Regards

    Franco

    From: "Emmanuel Taurel" taurel@users.sf.net
    To: "[tango-cs:bugs] " 689@bugs.tango-cs.p.re.sf.net
    Cc:
    Date: Wed, 08 Oct 2014 09:15:31 +0000
    Subject: [tango-cs:bugs] #689 Problems while getting events on a remote machine

    Franco,
    I had a look at your log files. It seems to me that something is wrong in event names.

    The device server process send the heartbeat event "tango://v-eras-0:10000/dserver/xxxx" while the client is waiting for "tango://10.202.104.50:10000/dserver/xxxx"
    What are the values of your TANGO_HOST env. variables on both the client and server sides?
    Good luck
    Emmanuel

    [bugs:#689] Problems while getting events on a remote machine
    Status: open

    Created: Fri Oct 03, 2014 09:36 PM UTC by IMS ERAS

    Last Updated: Mon Oct 06, 2014 12:56 PM UTC

    Owner: nobody
    Hi,

    Whithin one of the ERAS projects we are trying to make use of the events in PyTango.
    We are doing something rather simple as having a tango server generating events then in a python script we do something like:
    device = PyTango.DeviceProxy(args.tango_server)

    try:

     event_id = device.subscribe_event(

 'writablecmd', PyTango.EventType.CHANGE_EVENT, event_callback,

   filters=[], stateless=True

    )
    def event_callback(event):

    """Callback function used to receive Tango events."""

if event.errors:

    print(event.errors)

    return

try:

    msg = event.attr_value.value

............

    When we run this python script on the same machine were the tango server is running then everythin is perfectly fine and run as expected.
    If we instead run the script on a remote machine (were we set accordingly the TANGO_HOST) then, when we generate an sigle event like this:

    dev.writablecmd = 'MOVE Cube -5 -5 -5'

    we get those kind of errors:

    received MOVE Cube -5 -5 -5

    (DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)

    received MOVE Cube -5 -5 -5

    (DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)

    received MOVE Cube -5 -5 -5
    So what is happening is that the event triggered only once is then periodically received every 10 sec (which seems corresponding to the keep alive thread).
    If we run the same script at the same time on the local machine then everything is fine.

    Changing from stateless: True <-> False does not change much this behaviour.
    Below some more tracing obtained via printing the whole event.
    Thanks a lot for your help
    Regards
    Franco

    EventData

     attr_name = 'tango://10.202.104.51:10000/c3/testtango/testtango/writablecmd'

attr_value = None

    device = PyTestTango(c3/testtango/testtango)

       err = True

    errors = (DevError(desc = 'Event channel is not responding anymore, maybe the server or event system is down', origin = 'EventConsumer::KeepAliveThread()', reason = 'API_EventTimeout', severity = PyTango._PyTango.ErrSeverity.ERR),)

     event = 'change'

    reception_date = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 839816)
    EventData

     attr_name = 'tango://10.202.104.51:10000/c3/testtango/testtango/writablecmd'

attr_value = DeviceAttribute(data_format = PyTango._PyTango.AttrDataFormat.SCALAR, dim_x = 1, dim_y = 0, has_failed = False, is_empty = False, name = 'writablecmd', nb_read = 1, nb_written = 1, quality = PyTango._PyTango.AttrQuality.ATTR_VALID, r_dimension = AttributeDimension(dim_x = 1, dim_y = 0), time = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 889623), type = PyTango._PyTango.CmdArgType.DevString, value = 'MOVE Cube 0 0 0', w_dim_x = 1, w_dim_y = 0, w_dimension = AttributeDimension(dim_x = 1, dim_y = 0), w_value = 'MOVE Cube 0 0 0')

    device = PyTestTango(c3/testtango/testtango)

       err = False

    errors = ()

     event = 'change'

    reception_date = TimeVal(tv_nsec = 0, tv_sec = 1412364160, tv_usec = 847228)

    Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/tango-cs/bugs/689/
    To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

    Franco Carbognani
    Science and Technology Coordinator
    Italian Mars Society
    E-mail: franco.carbognani@marssociety.it
    Tel: +39 050 936038

     

    Related

    Bugs: #689

    alternate

  • IMS ERAS

    IMS ERAS - 2014-10-08

    Hi everybody,
    I have set my two machines so that
    on the local machine:
    TANGO_HOST 192.168.1.100:10000 (address of the local machine)
    on the remote machine:
    TANGO_HOST 192.168.1.100:10000 (same IP address)

    And now everything is running fine, I can se the events received correctly in both machines and our Blender simulations move sincronously the test cube around.

    So we can proceed now further in our tests

    Thanks a lot for your support, this ticket can be closed

     

  • Andy Gotz

    Andy Gotz - 2014-10-08

    Dear Franco,

    great news that problem is solved by specifying TANGO_HOST with the same syntax on client and server.

    For your information the reason for this behavior is the fully qualified name (i.e. including the TANGO_HOST) is used as an identifier for the zmq events. If the TANGO_HOST's are specified differently the events will be filtered out by zmq and not be sent to the client due to the different names.

    We are discussing how to change this behaviour for example by converting all TANGO_HOST names to ip addresses. This is a larger change because the event name is used in different places so we cannot promise anything soon. In the meantime the solution is to ensure TANGO_HOST has the same all over. You could use /etc/hosts to resolve ip names if you prefer them to ip addresses and don't have DNS.

    Thanks for the bug report it will help other people to avoid this problem in the future.

    Andy

     

  • Tiago Coutinho

    Tiago Coutinho - 2014-12-17

    According to reporter the ticket can be closed.
    I move the Category to miscellaneous since it is not PyTango specific.

     

  • Tiago Coutinho

    Tiago Coutinho - 2014-12-17
    • status: open --> closed-fixed
    • Category: PyTango --> Miscellaneous
     

Log in to post a comment.

Monday.com Logo