[Net-snmp-patches] [ net-snmp-Patches-1627049 ] netsnmp api extension to deal with large file handles

[SourceForge.net <no...@so...>]

Patches item #1627049, was opened at 2007-01-03 07:19
Message generated for change (Comment added) made by rwestphal
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=312694&aid=1627049&group_id=12694

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: Accepted
Priority: 3
Private: No
Submitted By: joyceanmachine (joyceanmachine)
Assigned to: Dave Shield (dts12)
Summary: netsnmp api extension to deal with large file handles

Initial Comment:
Hi all,

Currently the netsnmp library crashes when you try to use file desriptors larger than FD_SETSIZE. The snmp_read and snmp_select_info methods rely on filedescriptors being communicated through fd_sets,
which can contain only filedescriptors up to FD_SETSIZE (which is 1024 on most linux systems).

The attached patch contains an extension of the netsnmp api to deal with this problem. It allows the user to interact with netsnmp using arrays with arbitrarily large filehandles. 

Is there already someone working on this? If not, what would be the procedure to get this fix into the official distribution? 

Enabling netsnmp to deal with arbitrarily large filedescriptors is achieved (while maintinging backward compatibility) by introducing an interface in the style of the poll() command.

The patch introduces four new functions:
* snmp_read_extd
* snmp_sess_read_extd
* snmp_poll_info
* _sess_selpol_info
and modifies the interface of the function
* _sess_read

These methods all stick as closely as possible to the interface of the poll() command. They been tested with a couple of small examples on aix and linux.

Who would be the person to talk to about this stuff? 

Feedback and questions are most welcome.
Best regards,
Thijs

----------------------------------------------------------------------

Comment By: Renato Westphal (rwestphal)
Date: 2012-10-02 05:59

Message:
Hi all,

"I reserve the right to consider the pollfdarr solution a more elegant
solution... ;-)"

Not only more elegant but far more efficient. Using the net-snmp current
API I just can't take advantage of the Linux epoll performance improvements
over the traditional select/poll interfaces.

"The reason we submitted this patch is that it allows the user to choose
whichever system calls it wants to use inside his/her event loop. The
struct pollfdarr is just a more convenient means to transport the file
descriptors than using fd_set if you have large filedescriptors. Also, it
can be used in any event handling framework. It doesn't force you to use
poll(). It only uses to the pollfdarr to transport the filedescriptors."

I think that's the main point here. The fd_set struct returned by
snmp_select_info() is very inconvenient if you are not using select()...

It would be great if the net-snmp team could review this patch once again,
I think it adds a lot to the project.

Renato.

----------------------------------------------------------------------

Comment By: joyceanmachine (joyceanmachine)
Date: 2010-08-27 10:18

Message:
To Bart: Thanks for the quick response.

At the first glance, that looks like it should do the trick as well. That's
great!
Will look at it more carefully over the coming weeks (I will probably have
to adapt our code to deal with the new interface), so I'll get back to you
if it doesn't satisfy our needs... ;-)

I reserve the right to consider the pollfdarr solution a more elegant
solution... ;-)

Have a nice weekend!
Thijs

----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2010-08-27 08:23

Message:
To Thijs: recent Net-SNMP versions already have support for more than 1024
file descriptors. Please have a look at the data structures and functions
declared in include/net-snmp/library/large_fd_set.h, and also at the
functions in include/net-snmp/session_api.h that accept a  large fd set as
argument. Do these satisfy your needs ?

----------------------------------------------------------------------

Comment By: joyceanmachine (joyceanmachine)
Date: 2010-08-27 07:58

Message:
Hello, haven't looked at this thing in ages. I think I classified it in the
back of my brain as 'done' after the last post from dts12.

To Magnus: Yes, ideally this is what we want (including pony ;-) ).
Actually, nice guess: we do have an ACE-style IO handling class. :-)
However, as you noted yourself below, creating callbacks would require a
(major) rewrite (of the interface). Instead, we just calculate the deltas
ourselves.

To Bart: Yes, this is what would be useful for us. :-) That's why I
submitted this patch.

Ok, maybe a bit more explanation is justified:
Our primary objective was not something that's faster than fd_set. Our
primary objective was to create a convenient way for netsnmp to communicate
filedescriptors > 1024 to the user, without completely changing the
interface. 
The reason we submitted this patch is that it allows the user to choose
whichever system calls it wants to use inside his/her event loop. The
struct pollfdarr is just a more convenient means to transport the
filedescriptors than using fd_set if you have large filedescriptors. Also,
it can be used in any event handling framework. It doesn't force you to use
poll(). It only uses to the pollfdarr to transport the filedescriptors.

Hope this clarifies a bit.

So what happens now?

Cheers,
Thijs



----------------------------------------------------------------------

Comment By: Magnus Fromreide (magfr)
Date: 2009-04-18 10:39

Message:
To Bart: Not for me, but possibly for the original poster.

If I were to get what I want (and a pony!) then i really would like to
get a callbacks when the library wants to
 * Register a socket for reading (today: when it get added to the fdset)
 * Deregister a socket for reading (today: when it get removed from
   the fdset)
 * A timeout is created
 * A timeout is removed since it is uninteresting

And yes - the reason this interests me is that I look at general event
handling frameworks that have their own abstractions - quite effective
ones but they do not match neither the poll nor the select model.

Other examples of an alternate event handlers are
 * imagine that we want to enable the messages to be handled by
   GetMessage in Windows.
 * imagine that we want the messages to be handled by an ACE IO
   handling class.


----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2009-04-18 10:21

Message:
To Magnus: would it be a solution if an snmp_poll_info() function would be
available that a.o. fills in a struct pollfd array ?

----------------------------------------------------------------------

Comment By: Magnus Fromreide (magfr)
Date: 2009-04-18 10:08

Message:
To Bart: No, snmp_select_info is made to work with select. While it is
possible to dig the file descriptors out of the returned fd_set and do
whatever processing you want with them it still pretends to know that
you wish to run select and that is what I think the original poster
had issues with as well since he wanted to use poll (there are
different cost calculations for poll and select).


----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2009-04-18 09:51

Message:
To Magnus: I agree that a user should be able to query the Net-SNMP file
descriptor and timeouts such that the user can build its own event loop. If
I'm not mistaken this is already possible via the snmp_select_info()
function ?


----------------------------------------------------------------------

Comment By: Magnus Fromreide (magfr)
Date: 2009-04-18 09:44

Message:
While I agree with Bart that it is horrible with all the event loops
that are hidden in netsnmp I think that the real problem is that the
library misbehaves and thinks that it should own the event loop - this
is generally false for libraries.

Thus I think that what really is needed is to change the library
interface to provide callbacks that the library user get file
descriptors and timeouts with and then it is up to the user if they
wish to use select, epoll or some windowsy method to figure out when
there is traffic on the descriptors or some timeout and then call back
to the lib and tell it that it must do something now.

Of course I fear that this is a rather large change put that is one of
my long (sadly very long) term visions for Net-SNMP.


----------------------------------------------------------------------

Comment By: Thomas Anders (tanders)
Date: 2009-04-18 06:43

Message:
Bart, would you be able/willing to propose an alternate patch, ideally
within the 5.5.preX period?

----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2009-04-17 12:35

Message:
Note: if you would like to add API functions for Linux that work faster
than select() for large numbers of socket descriptors, you need the
epoll_create() / epoll_ctl() / epoll_wait() system calls in their
implementation, not the poll() system call. See e.g.
http://lwn.net/Articles/16026/ or <sys/epoll.h>.

----------------------------------------------------------------------

Comment By:  (bvassche)
Date: 2009-04-17 10:48

Message:
This patch has been written for the wrong reason. At least on Windows and
on Linux it is possible to pass more than 1024 file descriptors / socket
descriptors to the select() function -- I'm not sure about other Unix
systems. You can do this as follows on Linux:
* Make sure that /proc/sys/fs/file-max is large enough (the default value
on my Linux PC is 344507).
* Allocate (nfds + 8 * sizeof(int) - 1) / 8 bytes of memory instead of
using fd_set.
* When passing fd_set's between functions, always pass a pointer to an
fd_set, never the fd_set itself.
* See also http://opengroup.org/onlinepubs/007908775/xsh/select.html for
the SUSv2 specification of select().
* See also the source file fs/select.c in the Linux kernel tree for the
select() implementation.
And on Windows if you want to pass more than 64 sockets at a time to
select(), proceed as follows:
* Allocate sizeof(u_int) + nfds * sizeof(SOCKET) bytes of memory and use
that memory instead of an fd_set.
* When passing fd_set's between functions, always pass a pointer to an
fd_set, never the fd_set itself.
* See also http://msdn.microsoft.com/en-us/library/ms740141(VS.85).aspx and
http://msdn.microsoft.com/en-us/library/ms737873(VS.85).aspx.


----------------------------------------------------------------------

Comment By: Dave Shield (dts12)
Date: 2007-03-23 01:59

Message:
Logged In: YES 
user_id=88893
Originator: NO

Thanks for the patch!  It has been applied to the current
development code, and will appear in the next major release
of the Net-SNMP package.


----------------------------------------------------------------------

Comment By: Dave Shield (dts12)
Date: 2007-03-23 01:59

Message:
Logged In: YES 
user_id=88893
Originator: NO

Note that I haven't attempted to address the issues you raised
(which is why this tracker entry has been left open).  But
with the code in place, there's more chance of people actually
commenting on these!

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2007-01-04 03:39

Message:
Logged In: NO 

Ah yes, to get the patch working, you need to link against openssl >=
0.9.8.d

----------------------------------------------------------------------

Comment By: joyceanmachine (joyceanmachine)
Date: 2007-01-03 07:37

Message:
Logged In: YES 
user_id=1682471
Originator: YES

Open issues:
- The patch adds the line #include <sys/poll.h> to
include/net-snmp/library/snmp_api.h This is unelegant at best, since this
file does not contain any other includes (and probably shouldn't). Don't
know how to solve this properly (ok, I admit it, I didn't spend a lot of
time trying to figure it out...)

- It adds poll.h to configure and configure.in. Don't know if this is
correct and / or sufficient.

- Not sure if we want the POLLRDNORM and POLLRDBAND stuff in snmp_api.c

- FIXME in snmp_api.c which leads to quadratic behaviour in the number of
sessions. To fix this we need a hash table.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=312694&aid=1627049&group_id=12694