Poor performance using Paged Search

[Anuj]

Hi Guys,

We are migrating to UnboundID LDAP SK as we were facing some performance issues with sun java / jndi based LDAP operations.

As part o fthis migrating, I have created a method to search groups from AD and with paged control search it is taking more then a minuet to fetch 11000 groups.

However with regular JNDI search it was taking lest then 30 seconds. IT seems the paged searches are taking lot of time, any ideas how to increase the performance, I am using following code to do the search:

try {
final LDAPConnection conn = new LDAPConnection(serverAddress, serverPort,
bindDN, bindPW);
final SearchRequest searchRequest =
new SearchRequest(baseDN, scope, filter);
searchRequest.setControls(new SimplePagedResultsControl(pageSize, true));
int iterationNumber = 1;
while (true)
{
final SearchResult searchResult = conn.search(searchRequest);
System.out.println("Iteration " + (iterationNumber++) + " returned " +
searchResult.getEntryCount() + " entries");
final SimplePagedResultsControl responseControl =
(SimplePagedResultsControl) searchResult.getResponseControl(
SimplePagedResultsControl.PAGED_RESULTS_OID);
final ASN1OctetString cookie = responseControl.getCookie();
if ((cookie == null) || (cookie.getValueLength() == 0))
{
System.out.println("There are no more entries to return.");
break;
}
searchRequest.setControls(
new SimplePagedResultsControl(pageSize, cookie, true));
}
conn.close();
} catch (LDAPSearchException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (LDAPException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

[Neil Wilson]

I wrote some simple test code to compare the difference in performing a subtree search to retrieve all entries in a server using the simple paged results control using the UnboundID LDAP SDK for Java and JNDI. I tested with a directory containing one million entries using a filter of “(objectClass=*)” to retrieve all entries from the server using pages of 100 entries. In my testing, I saw a performance difference of less than 10% when using the UnboundID LDAP SDK and JNDI, which is nothing like the >100% difference you’re reporting. Are you sure that the code you have using JNDI and the code you have using the UnboundID LDAP SDK do the same thing?

The code I used to test the UnboundID LDAP SDK was:

import com.unboundid.ldap.sdk.LDAPConnection;
import com.unboundid.ldap.sdk.SearchRequest;
import com.unboundid.ldap.sdk.SearchResult;
import com.unboundid.ldap.sdk.SearchScope;
import com.unboundid.ldap.sdk.controls.SimplePagedResultsControl;

/**
 * This class provides an example of performing a search that uses JNDI to
 * perform a search with the simple paged results control to retrieve all
 * entries matching a given set of search criteria.
 */
final class TestUnboundIDLDAPSDKPagedSearch
{
  /**
   * Uses the provided information to establish an authenticated LDAP connection
   * and perform a subtree search using the simple paged results control to
   * iterate through all entries matching a set of search criteria.
   *
   * @param  serverAddress  The directory server address.
   * @param  serverPort     The directory server port.
   * @param  bindDN         The DN of the user as whom to authenticate.
   * @param  bindPW         The password for the user as whom to authenticate.
   * @param  baseDN         The base DN to use for the search.
   * @param  filter         The filter to use for the search.
   * @param  pageSize       The maximum number of entries to retrieve per page.
   *
   * @throws  Exception  If a problem is encountered during processing.
   */
  static void doSearch(final String serverAddress, final int serverPort,
                       final String bindDN, final String bindPW,
                       final String baseDN, final String filter,
                       final int pageSize)
         throws Exception
  {

    try (LDAPConnection conn =
              new LDAPConnection(serverAddress, serverPort, bindDN, bindPW))
    {
      final SearchRequest searchRequest =
           new SearchRequest(baseDN, SearchScope.SUB, filter);
      searchRequest.setControls(new SimplePagedResultsControl(pageSize, true));

      int numPages = 0;
      int totalEntries = 0;
      final long startTime = System.currentTimeMillis();

      while (true)
      {
        final SearchResult searchResult = conn.search(searchRequest);

        numPages++;
        final int iterationEntries = searchResult.getEntryCount();
        totalEntries += iterationEntries;

        final SimplePagedResultsControl responseControl =
             SimplePagedResultsControl.get(searchResult);
        if (responseControl.moreResultsToReturn())
        {
          searchRequest.setControls(new SimplePagedResultsControl(pageSize,
               responseControl.getCookie(), true));
        }
        else
        {
          break;
        }
      }

      final long elapsedTimeMillis = System.currentTimeMillis() - startTime;
      System.out.println("Retrieved " + totalEntries + " entries across " +
           numPages + " pages in " + elapsedTimeMillis +
           "ms with the UnboundID LDAP SDK for Java.");
    }
  }
}

And the code I used to test JNDI was:

import java.util.Hashtable;
import javax.naming.Context;
import javax.naming.NamingEnumeration;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;
import javax.naming.ldap.Control;
import javax.naming.ldap.InitialLdapContext;
import javax.naming.ldap.LdapContext;
import javax.naming.ldap.PagedResultsControl;
import javax.naming.ldap.PagedResultsResponseControl;

/**
 * This class provides an example of performing a search that uses the UnboundID
 * LDAP SDK for Java to perform a search with the simple paged results control
 * to retrieve all entries matching a given set of search criteria.
 */
final class TestJNDIPagedSearch
{
  /**
   * Uses the provided information to establish an authenticated LDAP connection
   * and perform a subtree search using the simple paged results control to
   * iterate through all entries matching a set of search criteria.
   *
   * @param  serverAddress  The directory server address.
   * @param  serverPort     The directory server port.
   * @param  bindDN         The DN of the user as whom to authenticate.
   * @param  bindPW         The password for the user as whom to authenticate.
   * @param  baseDN         The base DN to use for the search.
   * @param  filter         The filter to use for the search.
   * @param  pageSize       The maximum number of entries to retrieve per page.
   *
   * @throws  Exception  If a problem is encountered during processing.
   */
  static void doSearch(final String serverAddress, final int serverPort,
                       final String bindDN, final String bindPW,
                       final String baseDN, final String filter,
                       final int pageSize)
         throws Exception
  {
    final Hashtable<String,String> jndiProperties = new Hashtable<>(10);
    jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY,
         "com.sun.jndi.ldap.LdapCtxFactory");
    jndiProperties.put(Context.PROVIDER_URL,
         "ldap://" + serverAddress + ':' + serverPort + '/');
    jndiProperties.put(Context.SECURITY_PRINCIPAL, bindDN);
    jndiProperties.put(Context.SECURITY_CREDENTIALS, bindPW);

    final LdapContext ctx = new InitialLdapContext(jndiProperties, null);
    try
    {
      final PagedResultsControl initialPagedResultsControl =
           new PagedResultsControl(pageSize, null, true);
      ctx.setRequestControls(new Control[] { initialPagedResultsControl });

      final SearchControls searchControls = new SearchControls(
           SearchControls.SUBTREE_SCOPE, 0, 0, null, false, false);

      int numPages = 0;
      int totalEntries = 0;
      final long startTime = System.currentTimeMillis();

      while (true)
      {
        final NamingEnumeration<SearchResult> results =
             ctx.search(baseDN, filter, searchControls);
        while (results.hasMore())
        {
          results.next();
          totalEntries++;
        }

        numPages++;

        final Control[] responseControls = ctx.getResponseControls();
        if (responseControls != null)
        {
          byte[] cookie = null;
          for (final Control responseControl : responseControls)
          {
            if (responseControl instanceof PagedResultsResponseControl)
            {
              final PagedResultsResponseControl pagedResultsResponseControl =
                   (PagedResultsResponseControl) responseControl;
              cookie = pagedResultsResponseControl.getCookie();
              break;
            }
          }

          if ((cookie == null) || (cookie.length == 0))
          {
            break;
          }
          else
          {
            final PagedResultsControl nextPagedResultsControl =
                 new PagedResultsControl(pageSize, cookie, true);
            ctx.setRequestControls(new Control[] { nextPagedResultsControl });
          }
        }
      }

      final long elapsedTimeMillis = System.currentTimeMillis() - startTime;
      System.out.println("Retrieved " + totalEntries + " entries across " +
           numPages + " pages in " + elapsedTimeMillis + "ms with JNDI.");
    }
    finally
    {
      ctx.close();
    }
  }
}