Re: [pysnmp-dev] Regarding use of Dispatcher with Multiprocessing

[Ilya E. <il...@gl...>]

Hi Ani,

There’s similar app snmpfwd [1], however it does yet not support SNMP notifications at the moment.

The most straightforward way to do a proxy with pysnmp is to run both sides — Notification Receiver & Notification Originator together within a single main loop. Since pysnmp natively supports asynchronous communication, while one request is waiting for I/O, the other could be served concurrently. But two strictly simultaneous requests would queue up at shared SNMP engine waiting for its services. Here’s an example script [2] that implements SNMP Commands forwarding in the above design. It could be modified to support Notifications. With pysnmp API all the low-level processing and ACKs could be done automatically so you could deal with PDUs and targets.

If you wish to use threads, you'd have to run many independent SNMP engines (and main loops) local to each thread. Take a look at this example [3] to get an idea. In case of notifications I imagine you would have the receiving parts sitting each in its own main loop waiting for incoming queries and pushing received requests to a queue. The sending parts would  run their main loops briefly, just to push out request and optionally wait for response (informs). So most of the time the sending parts would spend waiting for queue to pull request from.

From my perspective, the threading approach may give better load distribution (but unlikely better throughput considering GIL which does not let you run threads truly simultaneously), however it may require more work from your side and larger memory footprint due to SNMP engine duplication.

The third option could be to run multiple instances of your purely asynchronous proxy (receivers) to improve concurrency.

-ilya

[1] http://sf.net/projects/snmpfwd/ <http://sf.net/projects/snmpfwd/>
[2] http://pysnmp.sourceforge.net/examples/current/v3arch/proxy/cmd/v3-to-v2c-conversion.html <http://pysnmp.sourceforge.net/examples/current/v3arch/proxy/cmd/v3-to-v2c-conversion.html>
[3] http://pysnmp.sourceforge.net/examples/current/v3arch/oneliner/manager/cmdgen/get-threaded-multiple-transports-and-protocols.html <http://pysnmp.sourceforge.net/examples/current/v3arch/oneliner/manager/cmdgen/get-threaded-multiple-transports-and-protocols.html>


> On 27 Feb 2015, at 08:01, Ani A <ani...@gm...> wrote:
> 
> Hi Ilya,
> 
> I am trying to create something like a _proxy_, the goal is to be able to intercept traps from the local 
> SNMP Agent, and then peeking into the varbinds, decide which one of the remote trap-receivers to send to.
> 
> Traps are the first goal, then, for Informs, I need to be able to get the ack, and send it back to the main agent as well.
> 
> Since I am using Multiprocessing module [1], do I still have to worry about MT safe? 
> 
> (the networking part of this, is that the receivers are all on different networks
>  [ different Linux namespaces ], so I have to get the SNMP payload and then build a separate UDP packet and 
> send, unless there is an easier way/API that I can use to just change the IP and send)
> 
> Hope I explained the problem [that I am trying to solve], clearly. Any help/tips/guidance very much welcome.
> 
> 
> [1] https://docs.python.org/2/library/multiprocessing.html <https://docs.python.org/2/library/multiprocessing.html>
> 
> --
> Ani
> 
> 
> On Fri, Feb 27, 2015 at 1:09 AM, Ilya Etingof <il...@gl... <mailto:il...@gl...>> wrote:
> Hi Ani,
> 
> At what point you are going to employ threading? Just for SNMP receiving part (what is not easy since a single instance of SNMP engine is not MT-safe) or for some sort of custom processing logic beyond SNMP engine?
> 
> -ilya
> 
> > On 24 Feb 2015, at 09:44, Ani A <ani...@gm... <mailto:ani...@gm...>> wrote:
> >
> > Hello,
> >
> > I was running through the examples in pysnmp (for Trap receiver with V3 case)
> > and I have a few doubts (maybe trivial! but somehow I am not getting it)
> >
> > I want to use pysnmp, with Python's Multiprocessing module, so in
> > each Process's callback function, do I run the following ?
> >
> >    try:
> >        snmpEngine.transportDispatcher.runDispatcher()
> >    except:
> >        snmpEngine.transportDispatcher.closeDispatcher()
> >        raise
> >
> > i.e, what should my main-loop wait on? , since I have another Queue as well
> > which I need to check.
> >
> > Thanks.
> > —
> > Ani
> 
> 
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the 
> conversation now. http://goparallel.sourceforge.net/_______________________________________________
> pysnmp-dev mailing list
> pys...@li...
> https://lists.sourceforge.net/lists/listinfo/pysnmp-dev