Re: [Pythomnic3k-questions] send an email when error is logged

[<dm...@ta...>]

> But since both smsc and php app are mission critical I
> need to be notified when any of them goes down. Basically when
> connection to smpp fails, bind fails, enquire ping, submitSm/deliverSm
> commands fail and etc., also on any timeouts to php application. If I
> remember correctly - all those are logged as ERR messages. So the
> easiest solution (from my point of view) would be to attach my custom
> module/function to a logger.

Yes, I see the problem. Philosophically, log errors that you are trying
to intercept are too low level and application agnostic. You will be
getting noise and have trouble filtering it out.

In my opinion the right way to do it with Pythomnic would be this:

1. Start a separate health_monitor cage.
2. It starts periodically probing all the cages it discovers
    around itself, ex. once in 30 seconds. Each probe comes as
    an RPC call to a particular cage, see 
health_monitor_event.py::probe()
3. That probe method is empty by default, therefore from 
health_monitor's
    point of view, any cage that is accessible via RPC is doing well.
    Therefore you get for free monitoring of the cage itself.
4. Then in cage's probe() you implement some application-specific 
testing,
    which in your case would be sending some SMPP packet (or, better
    yet a complete SMS to some test number) and seeing whether it goes
    through.
5. If probe() throws, health_monitor reports that the cage is down
    by invoking *its own* health_monitor_event.py::cage_down() in which
    you should put anything, like regular e-mail sending.
6. How do you "probe" the liveness of the PHP client, I can't tell as
    I have no idea how it speaks to the cage. But I'm sure you got the 
idea
    and can figure something out.
7. You can even use that memory buffered queue of errors for detecting
    failures. This way it will be in probe() that you read the error 
queue
    and you wouldn't need local scheduled calls.

So, health_monitor is useful for just that kind of monitoring that you 
need,
at least give it a thought.

If you have any questions, don't hesitate to ask.

Sincerely,
Dmitry Dvoinikov

On 2017-02-13 08:53, Egidijus | Canarius wrote:
> Hello,
> 
> thank you for such comprehensive and quick response.
> 
> I think I'll go with suggestion number five.
> I'm gonna describe my use case and why I need such functionality -
> maybe I'm looking at a problem from wrong direction.
> 
> I need to "connect" sms center server (accessed trough smpp) and a php
> application (accessed trough http). Both of them are quite unstable. I
> need to route messages (smpp and http requests) from one to another.
> Pythomnic works perfectly for this use case. I've already have a
> working cage. But since both smsc and php app are mission critical (I
> hope stability issues will be fixed at some point in the future) - I
> need to be notified when any of them goes down. Basically when
> connection to smpp fails, bind fails, enquire ping, submitSm/deliverSm
> commands fail and etc., also on any timeouts to php application. If I
> remember correctly - all those are logged as ERR messages. So the
> easiest solution (from my point of view) would be to attach my custom
> module/function to a logger.
> 
> Your suggestion about queue for logs got me thinking - maybe it could
> be added to Pythomnic3k itself? Like an in memory (or in state) buffer
> of logs that can be accessed from schedule (or any) cage. So instead
> logging directly to file pythomnic3k would log to this buffer and
> buffer would emit an event, call attached custom functions, notify
> health monitor or broadcast a rpc message - and then log to a file.
> This would allow to easily attach custom logic to critical events of
> the system.
> I see one downside to it - if for some reason pythomnic/buffer object
> crashes some logs would be gone. But I think this could happen with
> current implementation also - if pythomnic3k crashes while log file is
> not yet synced (correct me if I'm wrong).
> 
> Anyway, I'll let you know how it goes when I implement
> InterlockedQueue suggestion. Also I'm gonna send notification emails
> not using pmnc.transaction, so no infinite loop of errors.
>