cpu-users

[Cpu-users] New packages

[Blake M. <bma...@pu...>]

Hey Everyone,
I have repalced the 1.4.1 packages on the website with new ones. There was a
three line patch needed to fix compilation for RedHat 7.3. If you are running
RedHat 7.3 and use cpu, please grab the new packages.

-Blake

[Cpu-users] Slow user creation

[Terrence M. <tm...@ph...>]

I am noticing that when adding new users the command take a considerable 
amount of time to come back. Presumably this is as the result of 
searching for unused gid or uid.

 From my slapd log.

Sep 24 13:21:01 llama slapd[2434]: conn=1630 op=1062 SRCH attr=gidNumber
Sep 24 13:21:01 llama slapd[2434]: conn=1630 op=1062 SEARCH RESULT 
tag=101 err=0 nentries=1 text=
Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SRCH 
base="ou=Group,dc..." scope=2 filter="(gidNumber=5537)"
Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SRCH attr=gidNumber
Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SEARCH RESULT 
tag=101 err=0 nentries=1 text=
Sep 24 13:21:01 llama slapd[4906]: conn=1630 op=1064 SRCH 
base="ou=Group,dc..." scope=2 filter="(gidNumber=5538)"

It can take 30 seconds to add a users and moments to delete one. Is 
there anyway to speed this up? Either via some additional indexing in 
OpenLDAP or changes to the config file for cpu?

I do not necessarily need to re-use user ids as my number of users is 
only in the hundreds range. As a note I start at 5000.

Terrence

Re: [Cpu-users] Slow user creation

[Blake M. <bma...@pu...>]

In cpu.conf, change RANDOM = "false" to RANDOM = "true". You will no longer
get a linear search for IDs (e.g. if the next unused ID is X you will most
likely not get to use X) but it will drop that time significantly. If you
notice in the log below, it is doing a linear search hitting 5537, 5538,
5539.... until an unused ID is found. Change RANDOM to true will hit an
unused ID quickly as a seeded rand() is used to generate an integer value
between MAX_UIDNUMBER and MIN_UIDNUMBER.

You could also change MIN_UIDNUMBER/MIN_GIDNUMBER to a number higher than the
largest number in your directory, but this probably is not a good way to fix
things.

In the future, CPU will probably need to cache the last number used for a
linear search, and start from that position the next time a useradd/groupadd
takes place.

Let me know if that fixes things.

-Blake

Whatchu talkin' 'bout, Willis?
> I am noticing that when adding new users the command take a considerable 
> amount of time to come back. Presumably this is as the result of 
> searching for unused gid or uid.
> 
> From my slapd log.
> 
> Sep 24 13:21:01 llama slapd[2434]: conn=1630 op=1062 SRCH attr=gidNumber
> Sep 24 13:21:01 llama slapd[2434]: conn=1630 op=1062 SEARCH RESULT 
> tag=101 err=0 nentries=1 text=
> Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SRCH 
> base="ou=Group,dc..." scope=2 filter="(gidNumber=5537)"
> Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SRCH attr=gidNumber
> Sep 24 13:21:01 llama slapd[2443]: conn=1630 op=1063 SEARCH RESULT 
> tag=101 err=0 nentries=1 text=
> Sep 24 13:21:01 llama slapd[4906]: conn=1630 op=1064 SRCH 
> base="ou=Group,dc..." scope=2 filter="(gidNumber=5538)"
> 
> It can take 30 seconds to add a users and moments to delete one. Is 
> there anyway to speed this up? Either via some additional indexing in 
> OpenLDAP or changes to the config file for cpu?
> 
> I do not necessarily need to re-use user ids as my number of users is 
> only in the hundreds range. As a note I start at 5000.
> 
> Terrence
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Cpu-users mailing list
> Cpu...@li...
> https://lists.sourceforge.net/lists/listinfo/cpu-users

-- 
Blake Matheny           "... one of the main causes of the fall of the
bma...@pu...      Roman Empire was that, lacking zero, they had
http://www.mkfifo.net    no way to indicate successful termination of
http://ovmj.org/GNUnet/  their C programs." --Robert Firth

Re: [Cpu-users] Slow user creation

[Eric A. <and...@ce...>]

Blake Matheny wrote:

>In cpu.conf, change RANDOM = "false" to RANDOM = "true". You will no longer
>get a linear search for IDs (e.g. if the next unused ID is X you will most
>likely not get to use X) but it will drop that time significantly. If you
>notice in the log below, it is doing a linear search hitting 5537, 5538,
>5539.... until an unused ID is found. Change RANDOM to true will hit an
>unused ID quickly as a seeded rand() is used to generate an integer value
>between MAX_UIDNUMBER and MIN_UIDNUMBER.
>
>You could also change MIN_UIDNUMBER/MIN_GIDNUMBER to a number higher than the
>largest number in your directory, but this probably is not a good way to fix
>things.
>
>In the future, CPU will probably need to cache the last number used for a
>linear search, and start from that position the next time a useradd/groupadd
>takes place.
>  
>
Why not make the  search algorithm smarter?  A binary search might be 
faster, or even a different kind of search.  I haven't looked at the 
code, but shouldn't it be fast anyhow if the MIN_UIDNUMBER is set to the 
current highest UID?  Unless you are adding 10000 users, shouldn't it 
stay fast?

Also, if it is doing an ldap search for each uid it has to check, why 
not do the search for, say, 10 uids at a time, and then just parse 
through those 10?

Anyway, just ideas.. Ignore if you like.. :)

Eric


-- 
------------------------------------------------------------------
Eric Anderson	   Systems Administrator      Centaur Technology
All generalizations are false, including this one.
------------------------------------------------------------------

Re: [Cpu-users] Slow user creation

[Blake M. <bma...@pu...>]

Well, there are a couple of issues here to consider. If there is a large
userbase, grabbing all of the IDs could be terribly slow. If there isn't a
large user base, the problem is that the order in which users (or groups) are
returned is not by UID/GID, but rather by time of last modification (usually).
So as we are grabbing UIDs/GIDs, we would need to insert them into a binary
tree (or something similar), then search the tree. We could do this also with
a bitvector (which would likely be faster and smaller) but both take up extra
memory and time. I'll have to think about it.

Of course, setting MIN_UID to the MAX_UID in use would still allow for fast
linear searches as Eric suggested, but it's not ideal.

Ideas welcome.

-Blake

Whatchu talkin' 'bout, Willis?
> 
> That should but I would like both, linear and relatively quick.
> 
> CPU seems to do a lot of small searchs, could it not get the list of 
> UIDs in one go and then figure out the UID to use from there internally? 
> A cpu -w cat comes back in a little over 2 seconds on my system and only 
> makes one query to the directory rather than searching one entry at a 
> time to determine its relatively usefulness.
> 
> For example if you run the following you quickly get a sorted list of 
> all the existing UID's
> 
> cpu -w cat| awk -F : {'print $3'} | sort | uniq
> 
> Then just grab the last one, increment by one and you are done. I can do 
> this by hand of course but having CPU do it for me is nicer. :)
> 
> That would also avoid the need to cache the UID.
> 
> Terrence
> 

-- 
Blake Matheny           "... one of the main causes of the fall of the
bma...@pu...      Roman Empire was that, lacking zero, they had
http://www.mkfifo.net    no way to indicate successful termination of
http://ovmj.org/GNUnet/  their C programs." --Robert Firth

Re: [Cpu-users] Slow user creation

[Lukas K. <lu...@kn...>]

Am Mi, 2003-09-24 um 22.27 schrieb Terrence Martin:
> I am noticing that when adding new users the command take a considerable 
> amount of time to come back. Presumably this is as the result of 
> searching for unused gid or uid.

A qoute from cpu.conf(5):
-- snip --
       RANDOM = true or false
              If RANDOM is true, then a random number will  be  generated  and
              searched  for  (this number, if unused in the directory, will be
              the users uid or a groups gid). If a user or group with that  ID
              exists,  the process will continue for ID_MAX_PASSES. If true, a
              linear scan will be done starting at MIN_UIDNUMBER  (or  GIDNUM-
              BER) and will not stop until an unused ID is found or the number
              of scans is equal to ID_MAX_PASSES
-- snap --

If I interpret this correctly, it would solve your problem (speed),
because there don't have to be much ldap-questions for finding a free
uid.


> Terrence

-- 
bye
  Lukas