Re: [pysnmp-users] cmdgen and SNMPv3

[Ilya E. <il...@gl...>]

Hi Vincent,

> So, I had a look at PySNMP 4.3 but it seems that CommandGenerator is no
> longer using AsynCommandGenerator. I can provide my own SnmpEngine but
> SnmpEngine is still not thread-safe.

The 4.3 version has many changes. In particular, SNMP high-level API has been unified across different transports and moved into pysnmp.hlapi.* .
I’d advise not to use legacy interface (e.g. pysnmp.entity.rfc3413.oneliner) for multithreading stuff - it has many concurrency issues.
Use pysnmp.hlapi.asyncore or pysnmp.hlapi (synchronous) instead.

> A quick summary: I am trying to use multiple threads with several
> managers per thread (but a given manager is created and lives in a
> single thread). I am also trying to avoid the costly creation of a
> CommandGenerator (which parses MIB).

The only feasible design to me is to have one SnmpEngine per thread. SnmpEngine is expensive to create and populate, so try to keep your SnmpEngine’s in a thread pool for as long as you can rather than re-create them frequently.

http://pysnmp.sourceforge.net/examples/hlapi/asyncore/sync/manager/cmdgen/advanced-topics.html#query-agents-from-multiple-threads

[Asyn]CommandGenerator is gone - it was reimplemented as a function in the new API.

> What does not work:
> 
> 1. Having CommandGenerator as a singleton and using a single
>    thread. This fails when spawning commands from the same command
>    generator with different USM. This should not and using the
>    structure of usmUserTable (which is for a single agent) to store
>    stuff for a manager talking to multiple agents is a bad idea. The
>    cache should not just be a usmUserTable. It should also be indexed
>    by other security parameters or by the target agent (but we don't
>    know it yet when creating this structure).

It is not quite clear to me what exactly fails here.

> 2. Having CommandGenerator as a singleton and using multiple threads
>    with SNMPv1/v2 managers. This fails because SnmpEngine is not
>    thread-safe.

That’s right.

> 3. Having SnmpEngine as a singleton and using multiple threads with
>    SNMPv1/v2 managers, each manager having its own CommandGenerator
>    using the global SnmpEngine. SnmpEngine is not thread-safe and this
>    doesn't work either.

Fair enough.

> 4. Gaving CommandGenerator as a thread-local storage singleton. This
>   fails again with SNMPv3 when the same threads has several managers
>   (talking to several agents).

Important thing is to have SnmpEngine local to thread.

> What works:
> 
> 1. Having one CommandGenerator for each manager. Works fine. Slow as
>    hell when you need to spawn them fast.

Right.

> 2. Having CommandGenerator as a thread-local storage singleton with
>    SNMPv1/v2 managers. Works fine. However, spawning multiple threads
>    is slow as hell.

Right.

> Since many stuff doesn't seem to be thread safe and that SnmpEngine is
> only parsing MIB stuff to get constant data, this part should be
> extracted to do it once. The other idea would be to stuff the context of

Well, not quite. MIB structures can be modified at real-time (configuration and statistics, not to mention remote configuration features).

> a new SnmpEngine with the mibViewController of a previous SnmpEngine.

Yeah, but what do you think of the idea when you keep a bunch of SnmpEngine persistent and per-thread local passing queries to them?

The other approach is asyncio:

http://pysnmp.sourceforge.net/examples/hlapi/asyncio/manager/cmdgen/advanced-topics.html#concurrent-queries

What do you think?

-ilya