Re: [webmin-l] LDAP Users and Groups module performance

[Jamie C. <jca...@we...>]

On 11/Mar/2007 17:25 Murray Trainer wrote ..
> On Fri, 2007-03-09 at 09:43 -0800, Jamie Cameron wrote:
> > On 8/Mar/2007 15:37 Murray Trainer wrote ..
> > > On Thu, 2007-03-08 at 13:00 -0800, Jamie Cameron wrote:
> > > > On 7/Mar/2007 23:16 Murray Trainer wrote ..
> > > > > On Mon, 2007-02-26 at 15:19 +0900, Murray Trainer wrote:
> > > > > > On Sun, 2007-01-21 at 22:27 -0800, Jamie Cameron wrote:
> > > > > > > On 21/Jan/2007 16:11 Murray Trainer wrote ..
> > > > > > > > On Fri, 2007-01-19 at 00:23 -0800, Jamie Cameron wrote:
> > > > > > > > > On 18/Jan/2007 22:46 Murray Trainer wrote ..
> > > > > > > > > > Hi Jamie
> > > > > > > > > > 
> > > > > > > > > > We now have over 1000 users we manage with the LDAP Users
> > > module.
> > > > > > > > I
> > > > > > > > > > have noticed lately that it is becoming very slow bringing
> > > up
> > > > > the
> > > > > > > > > > initial screen.  I can imagine things only getting worse
> > > as the
> > > > > > > > > > directory gets larger.  I haven't checked the queries
> My
> > > guess
> > > > > is that
> > > > > > > > > > the module tries to search the LDAP directory for the
> whole
> > > list
> > > > > of
> > > > > > > > > > users, then after realising there are too many, it then
> says
> > > > > there
> > > > > > > > are
> > > > > > > > > > too many to display and fails.  Even if I set the maximum
> > > number
> > > > > of
> > > > > > > > > > users or groups to display to 0 or 1 it still takes a
> fair
> > > few
> > > > > seconds
> > > > > > > > > > to display the opening screen.  
> > > > > > > > > > 
> > > > > > > > > > The recommed method is to process each LDAP user entry
> one
> > > at
> > > > > a time
> > > > > > > > as
> > > > > > > > > > they are returned. The module would stop when it reached
> > > the
> > > > > limit
> > > > > > > > of
> > > > > > > > > > users for that page which could be a configuration value.
> > > The
> > > > > module
> > > > > > > > > > appears to be using the NET::LDAP library.  I had a look
> > > at the
> > > > > > > > > > documentation below and they recommend processing each
> entry
> > > > > as it
> > > > > > > > is
> > > > > > > > > > returned:  
> > > > > > > > > > 
> > > > > > > > > > http://search.cpan.org/~gbarr/perl-ldap-0.33/lib/Net/LDAP/FAQ.pod#USING_THE_CALLBACK_SUBROUTINE_APPROACH
> > > > > > > > > > 
> > > > > > > > > > If we want 100 users to be displayed on the page we print
> > > the
> > > > > first
> > > > > > > > 100
> > > > > > > > > > entries then quit.  If we want users 100-200 we would
> ignore
> > > > > the first
> > > > > > > > > > hundred entries and print the second hundred, and so
> on.
> > > > > > > > > > 
> > > > > > > > > > Hope the above idea has merit and its possible for you
> to
> > > implement
> > > > > > > > at
> > > > > > > > > > some stage.
> > > > > > > > > 
> > > > > > > > > Actually, that is what the module already does. The main
> page
> > > does
> > > > > an
> > > > > > > > > LDAP search for matching users, and then checks the number
> > > of results
> > > > > > > > available.
> > > > > > > > > If this is higher than the maximum set on the Module Config
> > > page,
> > > > > it
> > > > > > > > just
> > > > > > > > > displays the search form without fetching the full user
> list.
> > > > > > > > > 
> > > > > > > > > Of course, for this to be useful I assume that the LDAP
> server
> > > > > doesn't
> > > > > > > > > fetch users from it's database until actually asked to..
> > > > > > > > > 
> > > > > > > > >  - Jamie
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > If it does things in the sequential manner then I am not
> sure
> > > why
> > > > > it is
> > > > > > > > so slow for me.  There is nothing wrong with the performance
> > > of the
> > > > > ldap
> > > > > > > > server.  I can do an ldapsearch from a client machine and
> dump
> > > the
> > > > > whole
> > > > > > > > list of users into a text file in less than a second.  Going
> > > into
> > > > > the
> > > > > > > > first ldap users and groups screen with the list of users
> takes
> > > >
> > > > > 10
> > > > > > > > secs even with the amount of users to display set to 0. I
> am
> > > not
> > > > > sure
> > > > > > > > where the bottleneck in performance is?
> > > > > > > 
> > > > > > > I took another look at the code for this module, and it seems
> that
> > > > > even
> > > > > > > though Webmin just calls the Net::LDAP module's search function,
> > > this
> > > > > > > takes a long time to respond on large LDAP databases even though
> > > no
> > > > > > > actual results are requested!
> > > > > > > 
> > > > > > > However, I managed to find another way to reduce the search
> size,
> > > by
> > > > > > > specifying a maximum result size to the LDAP server. This will
> > > go into
> > > > > > > the next (1.330) Webmin release.
> > > > > > > 
> > > > > > >  - Jamie
> > > > > > 
> > > > > > Hi Jamie,
> > > > > > 
> > > > > > I noticed that you had added the fix above to the 1.328 version.
> > > I
> > > > > > downloaded it and tested it.  It displays the initial list of
> > > > > > users/groups immediately if it exceeds the limit for the page.
> Thanks
> > > > > > for your work on that - it will improve things a lot in our setup.
> > > > > 
> > > > > Hi Jamie,
> > > > > 
> > > > > I have installed 1.330 and the initial list of users displays quickly
> > > as
> > > > > for the earlier test version but I noticed that when clicking on
> create
> > > > > a new ldap user or opening an existing one, the screen takes a
> fair
> > > > > while to display (15-20 secs).  Not sure why that should be so
> slow?
> > > -
> > > > > in the first case it only needs to look up the next available uid.
> > > > > Bringing up an existing user might be a bit slower but the ldap
> search
> > > > > should be pretty quick.
> > > > 
> > > > I'll have to look into this .. do you have a large number of groups
> too?
> > > > 
> > > >  - Jamie
> > > 
> > > We have about 1500 users.  We don't have a large number of groups but
> a
> > > few groups have lots of members ie. 1000+.
> > 
> > Ok, I found another bug in the code that can cause a slowdown - it tries
> > to scan all users in LDAP to get a free UID, and to find uses shells.
> The
> > former is un-needed for editing existing users, and the latter can be
> disabled
> > on the Module Config page.
> > 
> > Let me know if you'd like a beta version of the fixed module to try out..
> > 
> >  - Jamie
> > 
> 
> Thanks, a beta version would be great.

The 1.333 development version of Webmin from http://www.webmin.com/devel.html
includes this fix. Just make sure that on the Module Config page, you have
'System users' unchecked for the 'Build list of shells from' setting.

 - Jamie