Menu

#77 Properly calculate the number of users in a directory/mysql

closed-duplicate
nobody
None
5
2009-03-24
2006-12-20
No

This patch fixes the way vpopmail determines when to create a new hash directory when using users-big-dir, and also allows the DB to be reset so that mysql will "backfill" any holes in existing directories. This patch only implements the vcount_dir_users function for the MySQL backend, but the concept should be adaptable to all backends - this is just the one I use and am thus familiar with.

Background: When using the big-dir hash directory structure, vpopmail normally creates a new hash directory for every 100 users on the system. The problem is that it does it for every 100 total users on the system (if (total % 100 = 0), create new dir), not 100 users in the current directory. Thus, it is possible to have directories that get heavily overloaded, and other directories that have very few users before a new directory is created. Take the following example:

1) 100 users are created, which go into directory 0
0 -> 100, total 100
2) 99 users are created, which go into directory 1
0 -> 100, 1 -> 99, total 199
3) 98 users are now deleted from directory 0
0 -> 2, 1 -> 99, total 101
4) 99 users are now created; the first 98 go in directory 1, and the last 1 goes into directory 2
0 -> 2, 1 -> 197, 2 -> 1, total 200
5) 1 user is now deleted from directory 2
0 -> 2, 1 -> 197, 2 -> 0, total 199
6) 1 user is now created, and goes into directory 3!
0 -> 2, 1 -> 197, 2-> 0, 3-> 1, total 200

While backfilling is possible by resetting the current directory, you can see that the advancement of the pointer is not based on the number of users in the current directory. This patch counts the users in the directory and only advances the pointer if the number of current users is more than MAX_USERS_PER_DIR, which defaults to 100.

With this patch, the above scenario becomes:

1) 100 users are created, which go into directory 0
0 -> 100, total 100
2) 99 users are created, which go into directory 1
0 -> 100, 1 -> 99, total 199
3) 98 users are now deleted from directory 0
0 -> 2, 1 -> 99, total 101
4) 99 users are now created; the first goes into directory 1, the rest go into directory 2
0 -> 2, 1 -> 100, 2 -> 98, total 200
5) 1 user is now deleted from directory 2
0 -> 2, 1 -> 100, 2 -> 97, total 199
6) 1 user is now created, and goes into directory 2
0 -> 2, 1 -> 100, 2 -> 98, total 200

If the directory pointer had been reset to 0 before step 4, the results would have been:

4) 99 users are now created; the first 98 go into directory 0, the last into directory 1
0 -> 100, 1 -> 100, total 200

If the user deleted in step 5 was from directory 0, the results of step 6 would be
0 -> 99, 1 -> 100, 2 -> 1, total 200

If the user was deleted from directory 1, the new user would have simply replaced them.

As you can see, this results in a much more even distribution of the users, which I believe was the original intent of vpopmail. Ideally, the pointer would be reset periodically to promote the backfilling of the directory structure, but it's really only needed in situations where there are large numbers of user deletions that throw off the count down the line.

This has been tested extensively with the MySQL backend in an environment where ~600 users were both added and deleted on an annual basis, but I'm sure it needs to be extended to the other backends before it could be incorporated into the main tree.

Discussion

  • Joshua Megerman

    Joshua Megerman - 2006-12-20

    Patch File

     
  • Matt Brookings

    Matt Brookings - 2009-03-24

    Fixed by backfill patch

     
  • Matt Brookings

    Matt Brookings - 2009-03-24
    • status: open --> closed-duplicate
     

Log in to post a comment.