LDAP: How to return more than 1000 results (java)

Question

I am using the LDAP SDK from this site: https://www.unboundid.com/products/ldap-sdk/ . I would like to make a search operation which returns a lot of entries.

According to the FAQ's site, ( https://www.unboundid.com/products/ldap-sdk/docs/ldapsdk-faq.php#search ) I have to use a SearchResultListener implementation.

So here is what I did:

 public class UpdateThread extends Thread implements SearchResultListener {
 ...
 // create request
 final SearchRequest request = new SearchRequest(this, instance.getBaseDN(),SearchScope.SUB, filter);
 // Setting size limit of results.
 request.setSizeLimit(2000);

 ...

 // Get every result one by one.
 @Override
public void searchEntryReturned(SearchResultEntry arg0) {
    System.out.println("entry "+arg0.getDN());

}

The problem is that "searchEntryReturned" returns a maximum of 1000 results. Even if I set the size limit to "2000".

Answer 1

The LDAP client is setting a "client-requested" size limit of 2000. This client-requested limit cannot override the limits set in the configuration of the server. No matter what the client-requested size limit is, the server's size limit overrides it. Contact your directory server administrator and ask that the size limit be increased.

Answer 2

It is pretty simple to implement a paged LDAP query using standard java, by using the adding a PagedResultsControl to the LdapContext, without using a third party API as per Neil's answer above.

Hashtable<String, Object> env = new Hashtable<String, Object>(11);
env
    .put(Context.INITIAL_CONTEXT_FACTORY,
        "com.sun.jndi.ldap.LdapCtxFactory");

/* Specify host and port to use for directory service */
env.put(Context.PROVIDER_URL,
    "ldap://localhost:389/ou=People,o=JNDITutorial");

try {
  LdapContext ctx = new InitialLdapContext(env, null);

  // Activate paged results
  int pageSize = 5;
  byte[] cookie = null;
  ctx.setRequestControls(new Control[] { new PagedResultsControl(pageSize,
      Control.NONCRITICAL) });
  int total;

  do {
    /* perform the search */
    NamingEnumeration results = ctx.search("", "(objectclass=*)",
        new SearchControls());

    /* for each entry print out name + all attrs and values */
    while (results != null && results.hasMore()) {
      SearchResult entry = (SearchResult) results.next();
      System.out.println(entry.getName());
    }

    // Examine the paged results control response
    Control[] controls = ctx.getResponseControls();
    if (controls != null) {
      for (int i = 0; i < controls.length; i++) {
        if (controls[i] instanceof PagedResultsResponseControl) {
          PagedResultsResponseControl prrc = (PagedResultsResponseControl) controls[i];
          total = prrc.getResultSize();
          if (total != 0) {
            System.out.println("***************** END-OF-PAGE "
                + "(total : " + total + ") *****************\n");
          } else {
            System.out.println("***************** END-OF-PAGE "
                + "(total: unknown) ***************\n");
          }
          cookie = prrc.getCookie();
        }
      }
    } else {
      System.out.println("No controls were sent from the server");
    }
    // Re-activate paged results
    ctx.setRequestControls(new Control[] { new PagedResultsControl(
        pageSize, cookie, Control.CRITICAL) });

  } while (cookie != null);

  ctx.close();

Example copied from here.

Answer 3

I solved like @PeterK , but with some modifications

    public List<MyUser> listUsers() {
    LOG.info("listUsers() inicio");
    List<MyUser> users = new ArrayList<MyUser>();

    Hashtable env = new Hashtable();
    env.put(Context.INITIAL_CONTEXT_FACTORY, INITIAL_CTX);
    env.put(Context.PROVIDER_URL, 'ldap://192.168.10.10:389');
    env.put(Context.SECURITY_AUTHENTICATION, CONNECTION_TYPE);
    env.put(Context.SECURITY_PRINCIPAL, USER_ADMIN_PASSWORD);
    env.put(Context.SECURITY_CREDENTIALS, USER_ADMIN);

    try {
        LdapContext ctx = new InitialLdapContext(env, null);

        // Activate paged results
        int pageSize = 1000;
        byte[] cookie = null;
        ctx.setRequestControls(new Control[] { new PagedResultsControl(pageSize, Control.NONCRITICAL) });
        int total;

        do {
            /* perform the search */
            SearchControls sc = new SearchControls();
            sc.setSearchScope(SearchControls.SUBTREE_SCOPE);
            String filtro = "(&(sAMAccountName=*)&(objectClass=user))";
            NamingEnumeration results = ctx.search(getBaseDn(ctx), filtro, sc);

            /* for each entry */
            while (results.hasMoreElements()) {
                SearchResult result = (SearchResult) results.nextElement();
                Attributes attributes = result.getAttributes();
                //convert to MyUser class
                MyUser user = toUser(attributes);
                users.add(user);
            }

            // Examine the paged results control response
            Control[] controls = ctx.getResponseControls();
            if (controls != null) {
                for (int i = 0; i < controls.length; i++) {
                    if (controls[i] instanceof PagedResultsResponseControl) {
                        PagedResultsResponseControl prrc = (PagedResultsResponseControl) controls[i];
                        total = prrc.getResultSize();
                        if (total != 0) {
                            System.out.println("***************** END-OF-PAGE " + "(total : " + total + ") *****************\n");
                        } else {
                            System.out.println("***************** END-OF-PAGE " + "(total: unknown) ***************\n");
                        }
                        cookie = prrc.getCookie();
                    }
                }
            } else {
                System.out.println("No controls were sent from the server");
            }
            // Re-activate paged results
            ctx.setRequestControls(new Control[] { new PagedResultsControl(pageSize, cookie, Control.CRITICAL) });

        } while (cookie != null);

        ctx.close();

    } catch (NamingException e) {
        System.err.println("PagedSearch failed.");
        e.printStackTrace();
    } catch (IOException ie) {
        System.err.println("PagedSearch failed.");
        ie.printStackTrace();
    } catch (Exception ie) {
        System.err.println("PagedSearch failed.");
        ie.printStackTrace();
    }

    LOG.info("listUsers() size = " + (users.size()));
    LOG.info("listUsers() fim");

    return users;
}


private MyUser toUser(Attributes attributes) throws NamingException {
    if (attributes != null) {
        String fullName = attributes.get("distinguishedName") != null ? attributes.get("distinguishedName").get().toString() : null;
        String mail = attributes.get("mail") != null ? attributes.get("mail").get().toString() : null;
        String userName = attributes.get("cn") != null ? attributes.get("cn").get().toString() : null;
        String userPrincipalName = attributes.get("userPrincipalName") != null ? attributes.get("userPrincipalName").get().toString() : null;

        if (userPrincipalName != null) {
            String[] user = userPrincipalName.split("@");
            if (user != null && user.length > 0) {
                userName = user[0];
            }
        }

        MyUser user = new MyUser();
        user.setFullName(fullName);
        user.setEmail(mail);
        user.setName(userName);
        user.setUserPrincipalName(userPrincipalName);
        user.setRoles(getRolesUser(attributes));

        return user;
    }

    return null;
}

Answer 4

Although it's almost certainly the case that the server is enforcing the size limit of 1000 entries, there are potentially ways to get around that by issuing the request in multiple parts.

If the server supports the use of the simple paged results control (as defined in RFC 2696 and supported in the LDAP SDK as per https://docs.ldap.com/ldap-sdk/docs/javadoc/com/unboundid/ldap/sdk/controls/SimplePagedResultsControl.html), then you can use it to iterate through the results in "pages" containing a specified number of entries.

Alternately, the virtual list view (VLV) request control (https://www.unboundid.com/products/ldap-sdk/docs/javadoc/index.html?com/unboundid/ldap/sdk/controls/VirtualListViewRequestControl.html) could be used, but I would probably only recommend that if the server doesn't support the simple paged results control because the VLV request control also requires that the results be sorted, and that likely either requires special configuration in the server or some pretty expensive processing in order to be able to service the request.