ldap-sdk-discuss — Discussions related to the LDAP SDK for Java

Re: [ldap-sdk-discuss] How can I make Entry.diff consider case?

[Lucas R. <lu...@me...>]

On Mar 17, 2015, at 11:17 PM, Neil A. Wilson <nei...@un...> wrote:

> On 03/17/2015 10:26 AM, Lucas Rockwell wrote:
>> Hello,
>> 
>> I am using the UnboundID LDAP SDK to edit a directory, and the issue I have is that Entry.diff seems to compare attribute values based on the attribute syntax if a schema is provided, and the "default matching rule" if no schema is provided (which is what I assume happens because, up until now, I have never provided a schema and I never see the SDK query cn=schema on the server). The use case for exact case matching is simple: a user may wish to change their name from 'LUCAS' to 'Lucas'.
> 
> You are correct that the LDAP SDK makes use of schema information if
> it's available in order to determine which matching rule should be used
> for a given operation.  There are methods for providing a schema for
> client-side objects like Attribute and Entry.  If you're communicating
> with a directory server, then you can use the
> LDAPConnectionOptions.setUseSchema method to cause the SDK to retrieve
> the schema from the server and try to use it for client-side processing.

Ah, I will take a look at that option.

>> So, my question is, how can I either set the default matching rule to be case sensitive, or how to I properly pass the schema to the entries to compare? The first method would be preferred as I would hate to have to maintain a special schema just to change all of the attributes to caseExact*Match.
> 
> It would be very problematic to provide a method to change the default
> matching rule on an SDK-wide basis.  Use of matching rules is pervasive
> in the LDAP SDK, and if there's any chance of multiple threads
> interacting with the LDAP SDK, then there is too much risk of breaking
> things or getting unexpected behavior.  There would likely also be a
> performance impact associated with doing this in order to avoid
> potential thread safety issues.

That makes total sense.

> The problem with the example that you're using below is that each
> Attribute object has its own MatchingRule instance, and you're not
> explicitly specifying it (or providing a Schema to allow the SDK to
> select the matching rule).  In the particular constructor that you're
> using, the Schema argument isn't really used for much at creation time,
> but it is used in other ways for various method calls (e.g., it makes it
> possible to reference an attribute with multiple names or its numeric
> OID).  But if you're providing Attribute objects to an entry, then each
> Attribute will use the matching rule that it already has when performing
> matching operations.
> 
> You could get around this by explicitly specifying the matching rule
> when you create the Attribute.  For example, instead of using:
> 
>     Attribute.new('givenName', 'LUCAS')
> 
> you could use:
> 
>     Attribute.new('givenName',
>          CaseExactStringMatchingRule.getInstance(), 'LUCAS')

This, of course, works perfectly!

> Or if you have a Schema that has givenName defined with the desired
> matching rule, you could use:
> 
>     Attribute.new('givenName', schema, 'LUCAS)
> 
> 
> I can consider adding an option for diff to always do byte-for-byte
> matching of values so that it triggers a modification if there is any
> difference at all, even if the values are still logically equivalent.

That would be cool, but I can make use of the CaseExactStringMatchingRule as I already wrap my addAttribute and setAttribute methods in helper methods and it will be trivial to add the new code.

Thank you very much!

-lucas

>> I have tired a simple test of modifying the schema for name and givenName and supplying that to the Entry constructor (see code below), and it still does not work. I feel like this is possible, but I am somehow missing some subtlety.
>> 
>> Many thanks.
>> 
>> -lucas
>> 
>> Example code using a modified Schema (using JRuby):
>> 
>> java_import 'com.unboundid.ldap.sdk.Entry'
>> java_import 'com.unboundid.ldap.sdk.Attribute'
>> java_import 'com.unboundid.ldap.sdk.schema.Schema'
>> 
>> def schema
>>    s = Entry.new(
>>        "dn: cn=schema",
>>        "objectclass: top",
>>        "objectclass: ldapSubentry",
>>        "objectclass: subschema",
>>        "attributeTypes: ( 2.5.4.41 NAME 'name' EQUALITY caseExactMatch
>>  SUBSTR caseExactSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768}
>>  X-ORIGIN 'RFC 4519' )",
>>        "attributeTypes: ( 2.5.4.42 NAME 'givenName' EQUALITY caseExactMatch SUBSTR caseExactSubstringsMatch SUP name SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 4519' )"
>>        )
>>    return Schema.new(s)
>> end
>> 
>> entry1 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'LUCAS'))
>> entry2 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'Lucas'))
>> 
>> mods = Entry::diff(entry1, entry2, true, true)
>> 
>> mods.each {|mod|
>>    puts mod.toString()
>> }
>> 
>> The above outputs nothing. But if I change 'LUCAS' to 'Luke', then it outputs:
>> 
>> LDAPModification(type=add, attr=givenName, values={'Luke'})
>> LDAPModification(type=delete, attr=givenName, values={'LUCAS'})
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the 
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>> 
> 
> 
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the 
> conversation now. http://goparallel.sourceforge.net/_______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss

Re: [ldap-sdk-discuss] How can I make Entry.diff consider case?

[Neil A. W. <nei...@un...>]

On 03/17/2015 10:26 AM, Lucas Rockwell wrote:
> Hello,
> 
> I am using the UnboundID LDAP SDK to edit a directory, and the issue I have is that Entry.diff seems to compare attribute values based on the attribute syntax if a schema is provided, and the "default matching rule" if no schema is provided (which is what I assume happens because, up until now, I have never provided a schema and I never see the SDK query cn=schema on the server). The use case for exact case matching is simple: a user may wish to change their name from 'LUCAS' to 'Lucas'.

You are correct that the LDAP SDK makes use of schema information if
it's available in order to determine which matching rule should be used
for a given operation.  There are methods for providing a schema for
client-side objects like Attribute and Entry.  If you're communicating
with a directory server, then you can use the
LDAPConnectionOptions.setUseSchema method to cause the SDK to retrieve
the schema from the server and try to use it for client-side processing.


> So, my question is, how can I either set the default matching rule to be case sensitive, or how to I properly pass the schema to the entries to compare? The first method would be preferred as I would hate to have to maintain a special schema just to change all of the attributes to caseExact*Match.

It would be very problematic to provide a method to change the default
matching rule on an SDK-wide basis.  Use of matching rules is pervasive
in the LDAP SDK, and if there's any chance of multiple threads
interacting with the LDAP SDK, then there is too much risk of breaking
things or getting unexpected behavior.  There would likely also be a
performance impact associated with doing this in order to avoid
potential thread safety issues.

The problem with the example that you're using below is that each
Attribute object has its own MatchingRule instance, and you're not
explicitly specifying it (or providing a Schema to allow the SDK to
select the matching rule).  In the particular constructor that you're
using, the Schema argument isn't really used for much at creation time,
but it is used in other ways for various method calls (e.g., it makes it
possible to reference an attribute with multiple names or its numeric
OID).  But if you're providing Attribute objects to an entry, then each
Attribute will use the matching rule that it already has when performing
matching operations.

You could get around this by explicitly specifying the matching rule
when you create the Attribute.  For example, instead of using:

     Attribute.new('givenName', 'LUCAS')

you could use:

     Attribute.new('givenName',
          CaseExactStringMatchingRule.getInstance(), 'LUCAS')

Or if you have a Schema that has givenName defined with the desired
matching rule, you could use:

     Attribute.new('givenName', schema, 'LUCAS)


I can consider adding an option for diff to always do byte-for-byte
matching of values so that it triggers a modification if there is any
difference at all, even if the values are still logically equivalent.


> 
> I have tired a simple test of modifying the schema for name and givenName and supplying that to the Entry constructor (see code below), and it still does not work. I feel like this is possible, but I am somehow missing some subtlety.
> 
> Many thanks.
> 
> -lucas
> 
> Example code using a modified Schema (using JRuby):
> 
> java_import 'com.unboundid.ldap.sdk.Entry'
> java_import 'com.unboundid.ldap.sdk.Attribute'
> java_import 'com.unboundid.ldap.sdk.schema.Schema'
> 
> def schema
>     s = Entry.new(
>         "dn: cn=schema",
>         "objectclass: top",
>         "objectclass: ldapSubentry",
>         "objectclass: subschema",
>         "attributeTypes: ( 2.5.4.41 NAME 'name' EQUALITY caseExactMatch
>   SUBSTR caseExactSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768}
>   X-ORIGIN 'RFC 4519' )",
>         "attributeTypes: ( 2.5.4.42 NAME 'givenName' EQUALITY caseExactMatch SUBSTR caseExactSubstringsMatch SUP name SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 4519' )"
>         )
>     return Schema.new(s)
> end
> 
> entry1 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'LUCAS'))
> entry2 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'Lucas'))
> 
> mods = Entry::diff(entry1, entry2, true, true)
> 
> mods.each {|mod|
>     puts mod.toString()
> }
> 
> The above outputs nothing. But if I change 'LUCAS' to 'Luke', then it outputs:
> 
> LDAPModification(type=add, attr=givenName, values={'Luke'})
> LDAPModification(type=delete, attr=givenName, values={'LUCAS'})
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the 
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
> 

[ldap-sdk-discuss] How can I make Entry.diff consider case?

[Lucas R. <lu...@me...>]

Hello,

I am using the UnboundID LDAP SDK to edit a directory, and the issue I have is that Entry.diff seems to compare attribute values based on the attribute syntax if a schema is provided, and the "default matching rule" if no schema is provided (which is what I assume happens because, up until now, I have never provided a schema and I never see the SDK query cn=schema on the server). The use case for exact case matching is simple: a user may wish to change their name from 'LUCAS' to 'Lucas'.

So, my question is, how can I either set the default matching rule to be case sensitive, or how to I properly pass the schema to the entries to compare? The first method would be preferred as I would hate to have to maintain a special schema just to change all of the attributes to caseExact*Match.

I have tired a simple test of modifying the schema for name and givenName and supplying that to the Entry constructor (see code below), and it still does not work. I feel like this is possible, but I am somehow missing some subtlety.

Many thanks.

-lucas

Example code using a modified Schema (using JRuby):

java_import 'com.unboundid.ldap.sdk.Entry'
java_import 'com.unboundid.ldap.sdk.Attribute'
java_import 'com.unboundid.ldap.sdk.schema.Schema'

def schema
    s = Entry.new(
        "dn: cn=schema",
        "objectclass: top",
        "objectclass: ldapSubentry",
        "objectclass: subschema",
        "attributeTypes: ( 2.5.4.41 NAME 'name' EQUALITY caseExactMatch
  SUBSTR caseExactSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768}
  X-ORIGIN 'RFC 4519' )",
        "attributeTypes: ( 2.5.4.42 NAME 'givenName' EQUALITY caseExactMatch SUBSTR caseExactSubstringsMatch SUP name SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 4519' )"
        )
    return Schema.new(s)
end

entry1 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'LUCAS'))
entry2 = Entry.new('uid=someentry', schema, Attribute.new('givenName', 'Lucas'))

mods = Entry::diff(entry1, entry2, true, true)

mods.each {|mod|
    puts mod.toString()
}

The above outputs nothing. But if I change 'LUCAS' to 'Luke', then it outputs:

LDAPModification(type=add, attr=givenName, values={'Luke'})
LDAPModification(type=delete, attr=givenName, values={'LUCAS'})

Re: [ldap-sdk-discuss] In Memory LDAP Operations

[Handy,Brad <bh...@ve...>]

I found the answer to my question of how to modify the request and what
happens afterward.  Thanks for the insight.


Regards,

Brad Handy





On 2/11/15, 10:03 AM, "Handy,Brad" <bh...@ve...> wrote:

>I¹ve seen the interfaces involved with your suggestions prior to sending
>this out.  However, I found the intercepted operations use ReadOnly
>versions of the request so I wasn¹t sure how this would work.  I took
>another look at the ReadOnly*Request interfaces and I see the duplicate
>methods now.
>
>Given your suggestions below, would I duplicate the read-only request and
>then set it back on the intercepted request instance to modify the
>request?  For example, duplicate a ReadOnlyAddRequest, make the
>modifications, and then set it back on the InMemoryInterceptedAddRequest
>instance.
>
>
>Regards,
>
>Brad Handy
>
>
>
>
>
>On 2/10/15, 3:21 PM, "Neil A. Wilson" <nei...@un...> wrote:
>
>>On 02/10/2015 01:03 PM, Handy,Brad wrote:
>>> I have some legacy code which uses an actual LDAP server for its ³unit²
>>>tests.  I would like to migrate the tests to use the InMemory server.
>>>However, the legacy logic depends on some functionality I have yet to
>>>reproduce.
>>>
>>> I need to find a way to:
>>> * hash the userPassword attribute when a modify/add operation is taking
>>>place.
>>> * find a way to maintain password history on the record by generating a
>>>new attribute/value pair when the password is modified.
>>>
>>> Our Production systems use OpenLDAP with some custom plugin code to
>>>accomplish all of this, but I have not been able to find something in
>>>the SDK which would allow this.
>>>
>>> Is this even possible with the InMemory server?
>>
>>The in-memory directory server does not provide any direct support for
>>any kind of password policy functionality like password history, nor
>>does it offer support for storing passwords in any format other than
>>clear-text.  The in-memory directory server tries to be a pretty
>>standards-compliant server and right now most password policy
>>implementations are at least partially proprietary.
>>
>>However, if you're willing to write some code you could probably
>>accomplish this using the LDAP SDK's InMemoryOperationInterceptor API.
>>This API allows you to intercept and potentially alter or short-circuit
>>requests before the server processes them, and to intercept and
>>potentially alter the results of operations after the server processes
>>them.
>>
>>In particular, you would probably need to do the following:
>>
>>- Implement the processAddRequest method and encode any password
>>contained in the entry being added.
>>
>>- Implement the processSimpleBindRequest method and encode the
>>clear-text password contained in the bind request so that the server can
>>do a simple byte-for-byte comparison against the stored value.  This
>>might require retrieving the user entry from the server in order to get
>>the current encoded password if the encoding mechanism depends on
>>something like a salt that varies from one entry to another.
>>
>>- If you need to support any SASL-based password mechanisms, then you
>>would need to implement processSASLBindRequest accordingly as well.
>>
>>- Implement the processModifyRequest method to check if any password in
>>the request is in the history.  If it is, then reject the operation.  If
>>not, then encode the password contained in the modify request and add a
>>new modification to put the previous password in the history, and if
>>appropriate to remove the oldest password from the history.
>>
>>- If you need to support the password modify extended operation, then
>>you'll need to implement the ProcessExtendedRequest method to encode the
>>current password (if present) in the same way required for simple binds,
>>and the new password (if present) in the same way required for add and
>>modify.  This will also need to do the same history checking as for the
>>modify operation.
>>
>>
>>Use the InMemoryDirectoryServerConfig.addInMemoryOperationInterceptor
>>method to install the interceptor for use in the server.
>>
>>This would probably not require much code to implement, especially if
>>you don't need to handle SASL authentication or the password modify
>>extended operation.
>>
>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Brad Handy
>>
>>
>
>
>--------------------------------------------------------------------------
>----
>Dive into the World of Parallel Programming. The Go Parallel Website,
>sponsored by Intel and developed in partnership with Slashdot Media, is
>your
>hub for all things parallel software development, from weekly thought
>leadership blogs to news, videos, case studies, tutorials and more. Take a
>look and join the conversation now. http://goparallel.sourceforge.net/
>_______________________________________________
>ldap-sdk-discuss mailing list
>lda...@li...
>https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss

Re: [ldap-sdk-discuss] In Memory LDAP Operations

[Handy,Brad <bh...@ve...>]

I¹ve seen the interfaces involved with your suggestions prior to sending
this out.  However, I found the intercepted operations use ReadOnly
versions of the request so I wasn¹t sure how this would work.  I took
another look at the ReadOnly*Request interfaces and I see the duplicate
methods now.

Given your suggestions below, would I duplicate the read-only request and
then set it back on the intercepted request instance to modify the
request?  For example, duplicate a ReadOnlyAddRequest, make the
modifications, and then set it back on the InMemoryInterceptedAddRequest
instance.


Regards,

Brad Handy





On 2/10/15, 3:21 PM, "Neil A. Wilson" <nei...@un...> wrote:

>On 02/10/2015 01:03 PM, Handy,Brad wrote:
>> I have some legacy code which uses an actual LDAP server for its ³unit²
>>tests.  I would like to migrate the tests to use the InMemory server.
>>However, the legacy logic depends on some functionality I have yet to
>>reproduce.
>>
>> I need to find a way to:
>> * hash the userPassword attribute when a modify/add operation is taking
>>place.
>> * find a way to maintain password history on the record by generating a
>>new attribute/value pair when the password is modified.
>>
>> Our Production systems use OpenLDAP with some custom plugin code to
>>accomplish all of this, but I have not been able to find something in
>>the SDK which would allow this.
>>
>> Is this even possible with the InMemory server?
>
>The in-memory directory server does not provide any direct support for
>any kind of password policy functionality like password history, nor
>does it offer support for storing passwords in any format other than
>clear-text.  The in-memory directory server tries to be a pretty
>standards-compliant server and right now most password policy
>implementations are at least partially proprietary.
>
>However, if you're willing to write some code you could probably
>accomplish this using the LDAP SDK's InMemoryOperationInterceptor API.
>This API allows you to intercept and potentially alter or short-circuit
>requests before the server processes them, and to intercept and
>potentially alter the results of operations after the server processes
>them.
>
>In particular, you would probably need to do the following:
>
>- Implement the processAddRequest method and encode any password
>contained in the entry being added.
>
>- Implement the processSimpleBindRequest method and encode the
>clear-text password contained in the bind request so that the server can
>do a simple byte-for-byte comparison against the stored value.  This
>might require retrieving the user entry from the server in order to get
>the current encoded password if the encoding mechanism depends on
>something like a salt that varies from one entry to another.
>
>- If you need to support any SASL-based password mechanisms, then you
>would need to implement processSASLBindRequest accordingly as well.
>
>- Implement the processModifyRequest method to check if any password in
>the request is in the history.  If it is, then reject the operation.  If
>not, then encode the password contained in the modify request and add a
>new modification to put the previous password in the history, and if
>appropriate to remove the oldest password from the history.
>
>- If you need to support the password modify extended operation, then
>you'll need to implement the ProcessExtendedRequest method to encode the
>current password (if present) in the same way required for simple binds,
>and the new password (if present) in the same way required for add and
>modify.  This will also need to do the same history checking as for the
>modify operation.
>
>
>Use the InMemoryDirectoryServerConfig.addInMemoryOperationInterceptor
>method to install the interceptor for use in the server.
>
>This would probably not require much code to implement, especially if
>you don't need to handle SASL authentication or the password modify
>extended operation.
>
>
>>
>>
>>
>> Regards,
>>
>> Brad Handy
>
>

Re: [ldap-sdk-discuss] In Memory LDAP Operations

[Neil A. W. <nei...@un...>]

On 02/10/2015 01:03 PM, Handy,Brad wrote:
> I have some legacy code which uses an actual LDAP server for its “unit” tests.  I would like to migrate the tests to use the InMemory server.  However, the legacy logic depends on some functionality I have yet to reproduce.
>
> I need to find a way to:
> * hash the userPassword attribute when a modify/add operation is taking place.
> * find a way to maintain password history on the record by generating a new attribute/value pair when the password is modified.
>
> Our Production systems use OpenLDAP with some custom plugin code to accomplish all of this, but I have not been able to find something in the SDK which would allow this.
>
> Is this even possible with the InMemory server?

The in-memory directory server does not provide any direct support for 
any kind of password policy functionality like password history, nor 
does it offer support for storing passwords in any format other than 
clear-text.  The in-memory directory server tries to be a pretty 
standards-compliant server and right now most password policy 
implementations are at least partially proprietary.

However, if you're willing to write some code you could probably 
accomplish this using the LDAP SDK's InMemoryOperationInterceptor API. 
This API allows you to intercept and potentially alter or short-circuit 
requests before the server processes them, and to intercept and 
potentially alter the results of operations after the server processes them.

In particular, you would probably need to do the following:

- Implement the processAddRequest method and encode any password 
contained in the entry being added.

- Implement the processSimpleBindRequest method and encode the 
clear-text password contained in the bind request so that the server can 
do a simple byte-for-byte comparison against the stored value.  This 
might require retrieving the user entry from the server in order to get 
the current encoded password if the encoding mechanism depends on 
something like a salt that varies from one entry to another.

- If you need to support any SASL-based password mechanisms, then you 
would need to implement processSASLBindRequest accordingly as well.

- Implement the processModifyRequest method to check if any password in 
the request is in the history.  If it is, then reject the operation.  If 
not, then encode the password contained in the modify request and add a 
new modification to put the previous password in the history, and if 
appropriate to remove the oldest password from the history.

- If you need to support the password modify extended operation, then 
you'll need to implement the ProcessExtendedRequest method to encode the 
current password (if present) in the same way required for simple binds, 
and the new password (if present) in the same way required for add and 
modify.  This will also need to do the same history checking as for the 
modify operation.


Use the InMemoryDirectoryServerConfig.addInMemoryOperationInterceptor 
method to install the interceptor for use in the server.

This would probably not require much code to implement, especially if 
you don't need to handle SASL authentication or the password modify 
extended operation.


>
>
>
> Regards,
>
> Brad Handy

[ldap-sdk-discuss] In Memory LDAP Operations

[Handy,Brad <bh...@ve...>]

I have some legacy code which uses an actual LDAP server for its “unit” tests.  I would like to migrate the tests to use the InMemory server.  However, the legacy logic depends on some functionality I have yet to reproduce.

I need to find a way to:
* hash the userPassword attribute when a modify/add operation is taking place.
* find a way to maintain password history on the record by generating a new attribute/value pair when the password is modified.

Our Production systems use OpenLDAP with some custom plugin code to accomplish all of this, but I have not been able to find something in the SDK which would allow this.

Is this even possible with the InMemory server?



Regards,

Brad Handy

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[Neil A. W. <nei...@un...>]

On 02/09/2015 06:01 PM, pankaj.tolani wrote:
> On Mon, Feb 9, 2015 at 11:12 PM, Neil A. Wilson <nei...@un...>
> wrote:
>
>> On 02/09/2015 04:48 PM, pankaj.tolani wrote:
>>
>>> With ConcurrentSkipListMap, the performance halved.
>>>
>>
>> Performance halved over not using any synchronization at all, or over the
>> fully-synchronized implementation?
>>
>> For TreeMap -> ConcurrentSkipListMap
>
>>
>>   If the modify operations are going to be on disjoint entries in the
>>> entryMap, and sequential within a thread, isnt TreeMap still OK?
>>>
>>
>> No.  It is not safe for multiple threads to interact with a non-concurrent
>> map without synchronization if there is any chance that the content of the
>> map may change (by adding new entries, or by removing or replacing existing
>> entries).  This will create the possibility for a race condition that could
>> result in a deadlock or a ConcurrentModificationException.  It may be
>> rare that you actually encounter it, but it's a definite possibility and
>> should not be ignored.
>>
> I do not add/remove entries. Wonder why replacing values of existing
> entries would be unsafe unless my execution paths need using the iterator.

I am not intimately familiar with the TreeMap implementation, but the 
general rule is that unless a map is explicitly stated to be threadsafe, 
you should assume that it is not safe to alter it in any way while other 
threads might access it.

Further, I should also reiterate that there are other parts of the 
in-memory directory server code that depend on the thread safety 
umbrella that the aggressive synchronization provides.  Indexes are 
definitely one such component, but there may be others.

If you need the highest possible performance and scalability, and can 
accomplish that by returning canned or generated responses instead of 
storing data, then the custom request handler route is the way to go.



>
>>
>> Also, I just realized that the ConcurrentSkipListMap suggestion isn't safe
>> if you're using indexes because the "synchronized (entryMap)" block ensures
>> that accesses to indexes (as well as other features like change logging)
>> are safe.
>>
> I stopped using indexes, as I got a big jump by using a BASE scope search
> instead of SUB scope search.
>
>>
>> If you expect to have a lot more reads than writes, then you could
>> probably convert the existing synchronization to be based on a read-write
>> lock, where read operations like bind, compare, and search acquire the read
>> lock (which will allow them to operate concurrently) and other operations
>> like add, delete, modify, and modify DN have to get the write lock (which
>> would ensure that operation is the only one in progress).
>>
> Thats a good suggestion. But we have nearly same number of reads/writes.
>
>>
>> Beyond that, trying to turn the in-memory directory server into a high
>> performance, highly-concurrent directory server that is safe under all
>> conditions would be a substantial undertaking.  It wasn't designed with
>> those goals in mind.  However, if you only have a very specific subset of
>> operations to support, and you are more concerned about testing the
>> performance of your application, then you may be able to create a custom
>> LDAPListenerRequestHandler implementation that satisfies your application
>> but doesn't actually need to store any data.  For example, the
>> CannedResponseRequestHandler will return a predetermined response to any
>> request and that will be very fast.  I'm not saying that the
>> CannedResponseRequestHandler is suitable for what you need, but you may be
>> able to create something similar that does suit your needs without any
>> concerns about safety, and also without any concerns about the
>> maintainability of your code (because if you make changes to the LDAP SDK
>> code itself, then you'll have to re-apply those changes if you want to
>> switch to a newer version of the LDAP SDK).
>>
>> Yes. thats the plan. Simulated response.
>
> Again, many thanks for your help.
>
>>
>>
>>
>>>
>>> On Mon, Feb 9, 2015 at 10:35 PM, Neil A. Wilson <
>>> nei...@un...>
>>> wrote:
>>>
>>>   On 02/09/2015 04:21 PM, pankaj.tolani wrote:
>>>>
>>>>   Many thanks Neil, for your detailed response.
>>>>>
>>>>> So, our idea is to internal factory acceptance performance test an
>>>>> application.
>>>>>
>>>>> Now, we mainly have just search and modify operations. No adds and
>>>>> deletes.
>>>>> Also multiple threads are going to operate on disjoint set of entries.
>>>>> With
>>>>> this in mind, I was hoping we could remove the synchronized(entryMap) {}
>>>>> on
>>>>> all the operations. Does this sound OK?
>>>>>
>>>>>
>>>> This would not be safe if you're performing modify operations, because
>>>> modify operations change the contents of the map.
>>>>
>>>> You might be able to get away with doing that if you changed entryMap
>>>> from
>>>> a TreeMap to a ConcurrentSkipListMap.  I haven't looked at the code
>>>> closely
>>>> enough to know if that will eliminate the worst possibilities of
>>>> eliminating synchronization (in particular, deadlocks and concurrent
>>>> modification exceptions), but I know that it won't guarantee correct
>>>> behavior in all circumstances (e.g., if multiple threads try to alter the
>>>> same entry at the same time).
>>>>
>>>>
>>>>
>>>>
>>>>   Also, I have been able to convert my SUB scope query to a BASE query.
>>>>>
>>>>> With this, I am getting a decent performance for 10000 subscribers, 2-3
>>>>> msec for one search and one update operation.
>>>>>
>>>>> On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <
>>>>> nei...@un...
>>>>>
>>>>>>
>>>>>>   wrote:
>>>>>
>>>>>    On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>>>>>
>>>>>>
>>>>>>    BTW with search/update, I meant 1 search + 1 update. Just to be
>>>>>> clear.
>>>>>>
>>>>>>>
>>>>>>> And I want to avoid the linear loop in
>>>>>>> InMemoryRequestHandler.processSearchRequest which reads
>>>>>>>
>>>>>>>
>>>>>>>   First, I should point out that the in-memory directory server isn't
>>>>>> meant
>>>>>> to be fast, nor should it ever under any circumstances be used in any
>>>>>> circumstance that requires a production-quality directory server. The
>>>>>> in-memory directory server is meant to be a stable, correct,
>>>>>> standards-compliant directory server that you can easily use for
>>>>>> development and testing purposes.  There are many essential features
>>>>>> for
>>>>>> production deployments that there are no plans to provide, like access
>>>>>> control, replication/high availability, and data durability.
>>>>>>
>>>>>> I'm not really sure what value there is in conducting
>>>>>> load/performance/scalability tests against the server because it is not
>>>>>> intended to be fast or scalable.  It makes use of very heavy locking as
>>>>>> an
>>>>>> easy way to ensure correct behavior by ensuring that the server may
>>>>>> only
>>>>>> ever attempt to process one operation at a time.  There are much better
>>>>>> ways to provide concurrency, but the approach it currently uses is both
>>>>>> very simple to implement and a very easy way to guarantee that there is
>>>>>> no
>>>>>> concern about conflicts between operations.
>>>>>>
>>>>>> However, depending on the type of search that you are performing, it
>>>>>> might
>>>>>> be possible to define one or more indexes that would dramatically
>>>>>> improve
>>>>>> performance and eliminate the need to iterate across all of the
>>>>>> entries.
>>>>>> Since you haven't indicated what kinds of searches you are performing,
>>>>>> it's
>>>>>> not possible to determine whether indexes can help. However, if your
>>>>>> searches are simple filters like "(uid=user.1234)", then you could
>>>>>> define
>>>>>> an index for the uid attribute and get much better performance.
>>>>>>
>>>>>> At present, the in-memory directory server only provides support for
>>>>>> equality indexes.  This can help improve the performance of searches
>>>>>> with
>>>>>> filters that look like "(indexedAttribute=value)".  Equality indexes
>>>>>> can
>>>>>> also be used for AND filters that contain at least one equality
>>>>>> component
>>>>>> that targets the indexed attribute, like "(&(indexedAttribute=value)(
>>>>>> someOtherAttribute=someOtherValue))", and for OR filters in which all
>>>>>> of
>>>>>> the components are equality components targeting the indexed attribute.
>>>>>>
>>>>>> To configure the in-memory directory server with one or more equality
>>>>>> indexes, use the InMemoryDirectoryServerConfig.
>>>>>> setEqualityIndexAttributes
>>>>>> method and pass in the names of the attributes you want to index.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>    for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>>>>>
>>>>>>>               {
>>>>>>>                 final DN dn = me.getKey();
>>>>>>>                 final Entry entry = me.getValue();
>>>>>>>                 try
>>>>>>>                 {
>>>>>>>                   if (dn.matchesBaseAndScope(baseDN, scope) &&
>>>>>>>                        filter.matchesEntry(entry, schema))
>>>>>>>                   {
>>>>>>>                     processSearchEntry(entry, includeSubEntries,
>>>>>>> includeChangeLog,
>>>>>>>                          hasManageDsaIT, fullEntryList, referenceList);
>>>>>>>                   }
>>>>>>>                 }
>>>>>>>                 catch (final Exception e)
>>>>>>>                 {
>>>>>>>                   Debug.debugException(e);
>>>>>>>                 }
>>>>>>>               }
>>>>>>>
>>>>>>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <
>>>>>>> pan...@gm...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>     Thanks Neil. Just before you replied, a colleague helped me with
>>>>>>> it.
>>>>>>>
>>>>>>>
>>>>>>>> I am doing a search/update load test with 4000 vs 40000 entires
>>>>>>>> against
>>>>>>>> InMemoryServer. Doing a load of search/update on 4000 entries takes
>>>>>>>> 40
>>>>>>>> odd
>>>>>>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>>>>>>> profiler output, it seems most of the time is spent in
>>>>>>>> InMemoryRequestHandler.processSearchRequest - calls to
>>>>>>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>>>>>>> filter
>>>>>>>> indexes might help. But I dont know what they mean and do they apply
>>>>>>>> to
>>>>>>>> InMemoryDirectoryServer.
>>>>>>>>
>>>>>>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>>>>>>             if (candidateDNs == null)
>>>>>>>>             {
>>>>>>>> // my execution lands up here
>>>>>>>>             }
>>>>>>>>             else {
>>>>>>>> // how do I come here
>>>>>>>>             }
>>>>>>>>
>>>>>>>> My seach is Scope SUB with a search filter which is an OR of two
>>>>>>>> object
>>>>>>>> classses.
>>>>>>>>
>>>>>>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>>>>>>> nei...@un...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>     On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>>     Hello,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I have checked out the ldap-sdk-code from svn://
>>>>>>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>>>>>>
>>>>>>>>>> I get lots of errors, examples being
>>>>>>>>>>
>>>>>>>>>> ASN1Boolean.java - import static com.unboundid.asn1.
>>>>>>>>>> ASN1Messages.*;
>>>>>>>>>>
>>>>>>>>>> InMemoryDirectoryServer.java - import static
>>>>>>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>>>>>>
>>>>>>>>>> What steps am I missing?
>>>>>>>>>>
>>>>>>>>>> regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    How are you trying to compile it?  It sounds as if you might
>>>>>>>>>> just be
>>>>>>>>>>
>>>>>>>>> doing a javac of everything in the src directory, and that's not
>>>>>>>>> good
>>>>>>>>> enough because the build process needs to do other work first (e.g.,
>>>>>>>>> generate source code containing build properties like product name,
>>>>>>>>> version, build date, etc., and also to generate source code from the
>>>>>>>>> messages properties files to guarantee that the code can't try to
>>>>>>>>> use
>>>>>>>>> any
>>>>>>>>> messages that aren't defined).
>>>>>>>>>
>>>>>>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>>>>>>> shell script on UNIX-based systems, or the build-se.bat batch file
>>>>>>>>> on
>>>>>>>>> Windows.  Alternately, if you've got Apache ant installed, then you
>>>>>>>>> can
>>>>>>>>> run
>>>>>>>>> "ant -buildfile build-se.xml".
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Neil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ------------------
>>>>>>>>> Dive into the World of Parallel Programming. The Go Parallel
>>>>>>>>> Website,
>>>>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>>>>> is
>>>>>>>>> your
>>>>>>>>> hub for all things parallel software development, from weekly
>>>>>>>>> thought
>>>>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>>>>> Take a
>>>>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>>>>> net/
>>>>>>>>> _______________________________________________
>>>>>>>>> ldap-sdk-discuss mailing list
>>>>>>>>> lda...@li...
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   --
>>>>>>>> पंकज एम तोलानी
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ------------------
>>>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>>> is
>>>>>>> your
>>>>>>> hub for all things parallel software development, from weekly thought
>>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>>> Take a
>>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>>> net/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ldap-sdk-discuss mailing list
>>>>>>> lda...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------------
>>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>>>> your
>>>>>> hub for all things parallel software development, from weekly thought
>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>> Take
>>>>>> a
>>>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>>> _______________________________________________
>>>>>> ldap-sdk-discuss mailing list
>>>>>> lda...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------
>>>>> ------------------
>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>>> your
>>>>> hub for all things parallel software development, from weekly thought
>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>> Take a
>>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ldap-sdk-discuss mailing list
>>>>> lda...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>> your
>>>> hub for all things parallel software development, from weekly thought
>>>> leadership blogs to news, videos, case studies, tutorials and more. Take
>>>> a
>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> ldap-sdk-discuss mailing list
>>>> lda...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>
>>>
>>>
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

On Mon, Feb 9, 2015 at 11:12 PM, Neil A. Wilson <nei...@un...>
wrote:

> On 02/09/2015 04:48 PM, pankaj.tolani wrote:
>
>> With ConcurrentSkipListMap, the performance halved.
>>
>
> Performance halved over not using any synchronization at all, or over the
> fully-synchronized implementation?
>
> For TreeMap -> ConcurrentSkipListMap

>
>  If the modify operations are going to be on disjoint entries in the
>> entryMap, and sequential within a thread, isnt TreeMap still OK?
>>
>
> No.  It is not safe for multiple threads to interact with a non-concurrent
> map without synchronization if there is any chance that the content of the
> map may change (by adding new entries, or by removing or replacing existing
> entries).  This will create the possibility for a race condition that could
> result in a deadlock or a ConcurrentModificationException.  It may be
> rare that you actually encounter it, but it's a definite possibility and
> should not be ignored.
>
I do not add/remove entries. Wonder why replacing values of existing
entries would be unsafe unless my execution paths need using the iterator.

>
> Also, I just realized that the ConcurrentSkipListMap suggestion isn't safe
> if you're using indexes because the "synchronized (entryMap)" block ensures
> that accesses to indexes (as well as other features like change logging)
> are safe.
>
I stopped using indexes, as I got a big jump by using a BASE scope search
instead of SUB scope search.

>
> If you expect to have a lot more reads than writes, then you could
> probably convert the existing synchronization to be based on a read-write
> lock, where read operations like bind, compare, and search acquire the read
> lock (which will allow them to operate concurrently) and other operations
> like add, delete, modify, and modify DN have to get the write lock (which
> would ensure that operation is the only one in progress).
>
Thats a good suggestion. But we have nearly same number of reads/writes.

>
> Beyond that, trying to turn the in-memory directory server into a high
> performance, highly-concurrent directory server that is safe under all
> conditions would be a substantial undertaking.  It wasn't designed with
> those goals in mind.  However, if you only have a very specific subset of
> operations to support, and you are more concerned about testing the
> performance of your application, then you may be able to create a custom
> LDAPListenerRequestHandler implementation that satisfies your application
> but doesn't actually need to store any data.  For example, the
> CannedResponseRequestHandler will return a predetermined response to any
> request and that will be very fast.  I'm not saying that the
> CannedResponseRequestHandler is suitable for what you need, but you may be
> able to create something similar that does suit your needs without any
> concerns about safety, and also without any concerns about the
> maintainability of your code (because if you make changes to the LDAP SDK
> code itself, then you'll have to re-apply those changes if you want to
> switch to a newer version of the LDAP SDK).
>
> Yes. thats the plan. Simulated response.

Again, many thanks for your help.

>
>
>
>>
>> On Mon, Feb 9, 2015 at 10:35 PM, Neil A. Wilson <
>> nei...@un...>
>> wrote:
>>
>>  On 02/09/2015 04:21 PM, pankaj.tolani wrote:
>>>
>>>  Many thanks Neil, for your detailed response.
>>>>
>>>> So, our idea is to internal factory acceptance performance test an
>>>> application.
>>>>
>>>> Now, we mainly have just search and modify operations. No adds and
>>>> deletes.
>>>> Also multiple threads are going to operate on disjoint set of entries.
>>>> With
>>>> this in mind, I was hoping we could remove the synchronized(entryMap) {}
>>>> on
>>>> all the operations. Does this sound OK?
>>>>
>>>>
>>> This would not be safe if you're performing modify operations, because
>>> modify operations change the contents of the map.
>>>
>>> You might be able to get away with doing that if you changed entryMap
>>> from
>>> a TreeMap to a ConcurrentSkipListMap.  I haven't looked at the code
>>> closely
>>> enough to know if that will eliminate the worst possibilities of
>>> eliminating synchronization (in particular, deadlocks and concurrent
>>> modification exceptions), but I know that it won't guarantee correct
>>> behavior in all circumstances (e.g., if multiple threads try to alter the
>>> same entry at the same time).
>>>
>>>
>>>
>>>
>>>  Also, I have been able to convert my SUB scope query to a BASE query.
>>>>
>>>> With this, I am getting a decent performance for 10000 subscribers, 2-3
>>>> msec for one search and one update operation.
>>>>
>>>> On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <
>>>> nei...@un...
>>>>
>>>>>
>>>>>  wrote:
>>>>
>>>>   On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>>>>
>>>>>
>>>>>   BTW with search/update, I meant 1 search + 1 update. Just to be
>>>>> clear.
>>>>>
>>>>>>
>>>>>> And I want to avoid the linear loop in
>>>>>> InMemoryRequestHandler.processSearchRequest which reads
>>>>>>
>>>>>>
>>>>>>  First, I should point out that the in-memory directory server isn't
>>>>> meant
>>>>> to be fast, nor should it ever under any circumstances be used in any
>>>>> circumstance that requires a production-quality directory server. The
>>>>> in-memory directory server is meant to be a stable, correct,
>>>>> standards-compliant directory server that you can easily use for
>>>>> development and testing purposes.  There are many essential features
>>>>> for
>>>>> production deployments that there are no plans to provide, like access
>>>>> control, replication/high availability, and data durability.
>>>>>
>>>>> I'm not really sure what value there is in conducting
>>>>> load/performance/scalability tests against the server because it is not
>>>>> intended to be fast or scalable.  It makes use of very heavy locking as
>>>>> an
>>>>> easy way to ensure correct behavior by ensuring that the server may
>>>>> only
>>>>> ever attempt to process one operation at a time.  There are much better
>>>>> ways to provide concurrency, but the approach it currently uses is both
>>>>> very simple to implement and a very easy way to guarantee that there is
>>>>> no
>>>>> concern about conflicts between operations.
>>>>>
>>>>> However, depending on the type of search that you are performing, it
>>>>> might
>>>>> be possible to define one or more indexes that would dramatically
>>>>> improve
>>>>> performance and eliminate the need to iterate across all of the
>>>>> entries.
>>>>> Since you haven't indicated what kinds of searches you are performing,
>>>>> it's
>>>>> not possible to determine whether indexes can help. However, if your
>>>>> searches are simple filters like "(uid=user.1234)", then you could
>>>>> define
>>>>> an index for the uid attribute and get much better performance.
>>>>>
>>>>> At present, the in-memory directory server only provides support for
>>>>> equality indexes.  This can help improve the performance of searches
>>>>> with
>>>>> filters that look like "(indexedAttribute=value)".  Equality indexes
>>>>> can
>>>>> also be used for AND filters that contain at least one equality
>>>>> component
>>>>> that targets the indexed attribute, like "(&(indexedAttribute=value)(
>>>>> someOtherAttribute=someOtherValue))", and for OR filters in which all
>>>>> of
>>>>> the components are equality components targeting the indexed attribute.
>>>>>
>>>>> To configure the in-memory directory server with one or more equality
>>>>> indexes, use the InMemoryDirectoryServerConfig.
>>>>> setEqualityIndexAttributes
>>>>> method and pass in the names of the attributes you want to index.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>   for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>>>>
>>>>>>              {
>>>>>>                final DN dn = me.getKey();
>>>>>>                final Entry entry = me.getValue();
>>>>>>                try
>>>>>>                {
>>>>>>                  if (dn.matchesBaseAndScope(baseDN, scope) &&
>>>>>>                       filter.matchesEntry(entry, schema))
>>>>>>                  {
>>>>>>                    processSearchEntry(entry, includeSubEntries,
>>>>>> includeChangeLog,
>>>>>>                         hasManageDsaIT, fullEntryList, referenceList);
>>>>>>                  }
>>>>>>                }
>>>>>>                catch (final Exception e)
>>>>>>                {
>>>>>>                  Debug.debugException(e);
>>>>>>                }
>>>>>>              }
>>>>>>
>>>>>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <
>>>>>> pan...@gm...>
>>>>>> wrote:
>>>>>>
>>>>>>    Thanks Neil. Just before you replied, a colleague helped me with
>>>>>> it.
>>>>>>
>>>>>>
>>>>>>> I am doing a search/update load test with 4000 vs 40000 entires
>>>>>>> against
>>>>>>> InMemoryServer. Doing a load of search/update on 4000 entries takes
>>>>>>> 40
>>>>>>> odd
>>>>>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>>>>>> profiler output, it seems most of the time is spent in
>>>>>>> InMemoryRequestHandler.processSearchRequest - calls to
>>>>>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>>>>>> filter
>>>>>>> indexes might help. But I dont know what they mean and do they apply
>>>>>>> to
>>>>>>> InMemoryDirectoryServer.
>>>>>>>
>>>>>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>>>>>            if (candidateDNs == null)
>>>>>>>            {
>>>>>>> // my execution lands up here
>>>>>>>            }
>>>>>>>            else {
>>>>>>> // how do I come here
>>>>>>>            }
>>>>>>>
>>>>>>> My seach is Scope SUB with a search filter which is an OR of two
>>>>>>> object
>>>>>>> classses.
>>>>>>>
>>>>>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>>>>>> nei...@un...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>    On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>>>>
>>>>>>>
>>>>>>>>    Hello,
>>>>>>>>
>>>>>>>>
>>>>>>>>> I have checked out the ldap-sdk-code from svn://
>>>>>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>>>>>
>>>>>>>>> I get lots of errors, examples being
>>>>>>>>>
>>>>>>>>> ASN1Boolean.java - import static com.unboundid.asn1.
>>>>>>>>> ASN1Messages.*;
>>>>>>>>>
>>>>>>>>> InMemoryDirectoryServer.java - import static
>>>>>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>>>>>
>>>>>>>>> What steps am I missing?
>>>>>>>>>
>>>>>>>>> regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   How are you trying to compile it?  It sounds as if you might
>>>>>>>>> just be
>>>>>>>>>
>>>>>>>> doing a javac of everything in the src directory, and that's not
>>>>>>>> good
>>>>>>>> enough because the build process needs to do other work first (e.g.,
>>>>>>>> generate source code containing build properties like product name,
>>>>>>>> version, build date, etc., and also to generate source code from the
>>>>>>>> messages properties files to guarantee that the code can't try to
>>>>>>>> use
>>>>>>>> any
>>>>>>>> messages that aren't defined).
>>>>>>>>
>>>>>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>>>>>> shell script on UNIX-based systems, or the build-se.bat batch file
>>>>>>>> on
>>>>>>>> Windows.  Alternately, if you've got Apache ant installed, then you
>>>>>>>> can
>>>>>>>> run
>>>>>>>> "ant -buildfile build-se.xml".
>>>>>>>>
>>>>>>>>
>>>>>>>> Neil
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> ------------------
>>>>>>>> Dive into the World of Parallel Programming. The Go Parallel
>>>>>>>> Website,
>>>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>>>> is
>>>>>>>> your
>>>>>>>> hub for all things parallel software development, from weekly
>>>>>>>> thought
>>>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>>>> Take a
>>>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>>>> net/
>>>>>>>> _______________________________________________
>>>>>>>> ldap-sdk-discuss mailing list
>>>>>>>> lda...@li...
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  --
>>>>>>> पंकज एम तोलानी
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------------
>>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>> is
>>>>>> your
>>>>>> hub for all things parallel software development, from weekly thought
>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>> Take a
>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>> net/
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ldap-sdk-discuss mailing list
>>>>>> lda...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ------------------------------------------------------------
>>>>> ------------------
>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>>> your
>>>>> hub for all things parallel software development, from weekly thought
>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>> Take
>>>>> a
>>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>> _______________________________________________
>>>>> ldap-sdk-discuss mailing list
>>>>> lda...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>> your
>>>> hub for all things parallel software development, from weekly thought
>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>> Take a
>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ldap-sdk-discuss mailing list
>>>> lda...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take
>>> a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>>
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>
>


-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[Neil A. W. <nei...@un...>]

On 02/09/2015 04:48 PM, pankaj.tolani wrote:
> With ConcurrentSkipListMap, the performance halved.

Performance halved over not using any synchronization at all, or over 
the fully-synchronized implementation?


> If the modify operations are going to be on disjoint entries in the
> entryMap, and sequential within a thread, isnt TreeMap still OK?

No.  It is not safe for multiple threads to interact with a 
non-concurrent map without synchronization if there is any chance that 
the content of the map may change (by adding new entries, or by removing 
or replacing existing entries).  This will create the possibility for a 
race condition that could result in a deadlock or a 
ConcurrentModificationException.  It may be rare that you actually 
encounter it, but it's a definite possibility and should not be ignored.

Also, I just realized that the ConcurrentSkipListMap suggestion isn't 
safe if you're using indexes because the "synchronized (entryMap)" block 
ensures that accesses to indexes (as well as other features like change 
logging) are safe.

If you expect to have a lot more reads than writes, then you could 
probably convert the existing synchronization to be based on a 
read-write lock, where read operations like bind, compare, and search 
acquire the read lock (which will allow them to operate concurrently) 
and other operations like add, delete, modify, and modify DN have to get 
the write lock (which would ensure that operation is the only one in 
progress).

Beyond that, trying to turn the in-memory directory server into a high 
performance, highly-concurrent directory server that is safe under all 
conditions would be a substantial undertaking.  It wasn't designed with 
those goals in mind.  However, if you only have a very specific subset 
of operations to support, and you are more concerned about testing the 
performance of your application, then you may be able to create a custom 
LDAPListenerRequestHandler implementation that satisfies your 
application but doesn't actually need to store any data.  For example, 
the CannedResponseRequestHandler will return a predetermined response to 
any request and that will be very fast.  I'm not saying that the 
CannedResponseRequestHandler is suitable for what you need, but you may 
be able to create something similar that does suit your needs without 
any concerns about safety, and also without any concerns about the 
maintainability of your code (because if you make changes to the LDAP 
SDK code itself, then you'll have to re-apply those changes if you want 
to switch to a newer version of the LDAP SDK).


>
>
> On Mon, Feb 9, 2015 at 10:35 PM, Neil A. Wilson <nei...@un...>
> wrote:
>
>> On 02/09/2015 04:21 PM, pankaj.tolani wrote:
>>
>>> Many thanks Neil, for your detailed response.
>>>
>>> So, our idea is to internal factory acceptance performance test an
>>> application.
>>>
>>> Now, we mainly have just search and modify operations. No adds and
>>> deletes.
>>> Also multiple threads are going to operate on disjoint set of entries.
>>> With
>>> this in mind, I was hoping we could remove the synchronized(entryMap) {}
>>> on
>>> all the operations. Does this sound OK?
>>>
>>
>> This would not be safe if you're performing modify operations, because
>> modify operations change the contents of the map.
>>
>> You might be able to get away with doing that if you changed entryMap from
>> a TreeMap to a ConcurrentSkipListMap.  I haven't looked at the code closely
>> enough to know if that will eliminate the worst possibilities of
>> eliminating synchronization (in particular, deadlocks and concurrent
>> modification exceptions), but I know that it won't guarantee correct
>> behavior in all circumstances (e.g., if multiple threads try to alter the
>> same entry at the same time).
>>
>>
>>
>>
>>> Also, I have been able to convert my SUB scope query to a BASE query.
>>>
>>> With this, I am getting a decent performance for 10000 subscribers, 2-3
>>> msec for one search and one update operation.
>>>
>>> On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <nei...@un...
>>>>
>>> wrote:
>>>
>>>   On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>>>>
>>>>   BTW with search/update, I meant 1 search + 1 update. Just to be clear.
>>>>>
>>>>> And I want to avoid the linear loop in
>>>>> InMemoryRequestHandler.processSearchRequest which reads
>>>>>
>>>>>
>>>> First, I should point out that the in-memory directory server isn't meant
>>>> to be fast, nor should it ever under any circumstances be used in any
>>>> circumstance that requires a production-quality directory server. The
>>>> in-memory directory server is meant to be a stable, correct,
>>>> standards-compliant directory server that you can easily use for
>>>> development and testing purposes.  There are many essential features for
>>>> production deployments that there are no plans to provide, like access
>>>> control, replication/high availability, and data durability.
>>>>
>>>> I'm not really sure what value there is in conducting
>>>> load/performance/scalability tests against the server because it is not
>>>> intended to be fast or scalable.  It makes use of very heavy locking as
>>>> an
>>>> easy way to ensure correct behavior by ensuring that the server may only
>>>> ever attempt to process one operation at a time.  There are much better
>>>> ways to provide concurrency, but the approach it currently uses is both
>>>> very simple to implement and a very easy way to guarantee that there is
>>>> no
>>>> concern about conflicts between operations.
>>>>
>>>> However, depending on the type of search that you are performing, it
>>>> might
>>>> be possible to define one or more indexes that would dramatically improve
>>>> performance and eliminate the need to iterate across all of the entries.
>>>> Since you haven't indicated what kinds of searches you are performing,
>>>> it's
>>>> not possible to determine whether indexes can help. However, if your
>>>> searches are simple filters like "(uid=user.1234)", then you could define
>>>> an index for the uid attribute and get much better performance.
>>>>
>>>> At present, the in-memory directory server only provides support for
>>>> equality indexes.  This can help improve the performance of searches with
>>>> filters that look like "(indexedAttribute=value)".  Equality indexes can
>>>> also be used for AND filters that contain at least one equality component
>>>> that targets the indexed attribute, like "(&(indexedAttribute=value)(
>>>> someOtherAttribute=someOtherValue))", and for OR filters in which all of
>>>> the components are equality components targeting the indexed attribute.
>>>>
>>>> To configure the in-memory directory server with one or more equality
>>>> indexes, use the InMemoryDirectoryServerConfig.
>>>> setEqualityIndexAttributes
>>>> method and pass in the names of the attributes you want to index.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>   for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>>>>              {
>>>>>                final DN dn = me.getKey();
>>>>>                final Entry entry = me.getValue();
>>>>>                try
>>>>>                {
>>>>>                  if (dn.matchesBaseAndScope(baseDN, scope) &&
>>>>>                       filter.matchesEntry(entry, schema))
>>>>>                  {
>>>>>                    processSearchEntry(entry, includeSubEntries,
>>>>> includeChangeLog,
>>>>>                         hasManageDsaIT, fullEntryList, referenceList);
>>>>>                  }
>>>>>                }
>>>>>                catch (final Exception e)
>>>>>                {
>>>>>                  Debug.debugException(e);
>>>>>                }
>>>>>              }
>>>>>
>>>>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
>>>>> wrote:
>>>>>
>>>>>    Thanks Neil. Just before you replied, a colleague helped me with it.
>>>>>
>>>>>>
>>>>>> I am doing a search/update load test with 4000 vs 40000 entires against
>>>>>> InMemoryServer. Doing a load of search/update on 4000 entries takes 40
>>>>>> odd
>>>>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>>>>> profiler output, it seems most of the time is spent in
>>>>>> InMemoryRequestHandler.processSearchRequest - calls to
>>>>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>>>>> filter
>>>>>> indexes might help. But I dont know what they mean and do they apply to
>>>>>> InMemoryDirectoryServer.
>>>>>>
>>>>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>>>>            if (candidateDNs == null)
>>>>>>            {
>>>>>> // my execution lands up here
>>>>>>            }
>>>>>>            else {
>>>>>> // how do I come here
>>>>>>            }
>>>>>>
>>>>>> My seach is Scope SUB with a search filter which is an OR of two object
>>>>>> classses.
>>>>>>
>>>>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>>>>> nei...@un...>
>>>>>> wrote:
>>>>>>
>>>>>>    On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>>>
>>>>>>>
>>>>>>>    Hello,
>>>>>>>
>>>>>>>>
>>>>>>>> I have checked out the ldap-sdk-code from svn://
>>>>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>>>>
>>>>>>>> I get lots of errors, examples being
>>>>>>>>
>>>>>>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>>>>>>
>>>>>>>> InMemoryDirectoryServer.java - import static
>>>>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>>>>
>>>>>>>> What steps am I missing?
>>>>>>>>
>>>>>>>> regards,
>>>>>>>>
>>>>>>>>
>>>>>>>>   How are you trying to compile it?  It sounds as if you might just be
>>>>>>> doing a javac of everything in the src directory, and that's not good
>>>>>>> enough because the build process needs to do other work first (e.g.,
>>>>>>> generate source code containing build properties like product name,
>>>>>>> version, build date, etc., and also to generate source code from the
>>>>>>> messages properties files to guarantee that the code can't try to use
>>>>>>> any
>>>>>>> messages that aren't defined).
>>>>>>>
>>>>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>>>>> shell script on UNIX-based systems, or the build-se.bat batch file on
>>>>>>> Windows.  Alternately, if you've got Apache ant installed, then you
>>>>>>> can
>>>>>>> run
>>>>>>> "ant -buildfile build-se.xml".
>>>>>>>
>>>>>>>
>>>>>>> Neil
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ------------------
>>>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>>> is
>>>>>>> your
>>>>>>> hub for all things parallel software development, from weekly thought
>>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>>> Take a
>>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>>> net/
>>>>>>> _______________________________________________
>>>>>>> ldap-sdk-discuss mailing list
>>>>>>> lda...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> पंकज एम तोलानी
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------
>>>>> ------------------
>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>>> your
>>>>> hub for all things parallel software development, from weekly thought
>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>> Take a
>>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ldap-sdk-discuss mailing list
>>>>> lda...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>> your
>>>> hub for all things parallel software development, from weekly thought
>>>> leadership blogs to news, videos, case studies, tutorials and more. Take
>>>> a
>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> ldap-sdk-discuss mailing list
>>>> lda...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>
>>>
>>>
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

With ConcurrentSkipListMap, the performance halved.

If the modify operations are going to be on disjoint entries in the
entryMap, and sequential within a thread, isnt TreeMap still OK?


On Mon, Feb 9, 2015 at 10:35 PM, Neil A. Wilson <nei...@un...>
wrote:

> On 02/09/2015 04:21 PM, pankaj.tolani wrote:
>
>> Many thanks Neil, for your detailed response.
>>
>> So, our idea is to internal factory acceptance performance test an
>> application.
>>
>> Now, we mainly have just search and modify operations. No adds and
>> deletes.
>> Also multiple threads are going to operate on disjoint set of entries.
>> With
>> this in mind, I was hoping we could remove the synchronized(entryMap) {}
>> on
>> all the operations. Does this sound OK?
>>
>
> This would not be safe if you're performing modify operations, because
> modify operations change the contents of the map.
>
> You might be able to get away with doing that if you changed entryMap from
> a TreeMap to a ConcurrentSkipListMap.  I haven't looked at the code closely
> enough to know if that will eliminate the worst possibilities of
> eliminating synchronization (in particular, deadlocks and concurrent
> modification exceptions), but I know that it won't guarantee correct
> behavior in all circumstances (e.g., if multiple threads try to alter the
> same entry at the same time).
>
>
>
>
>> Also, I have been able to convert my SUB scope query to a BASE query.
>>
>> With this, I am getting a decent performance for 10000 subscribers, 2-3
>> msec for one search and one update operation.
>>
>> On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <nei...@un...
>> >
>> wrote:
>>
>>  On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>>>
>>>  BTW with search/update, I meant 1 search + 1 update. Just to be clear.
>>>>
>>>> And I want to avoid the linear loop in
>>>> InMemoryRequestHandler.processSearchRequest which reads
>>>>
>>>>
>>> First, I should point out that the in-memory directory server isn't meant
>>> to be fast, nor should it ever under any circumstances be used in any
>>> circumstance that requires a production-quality directory server. The
>>> in-memory directory server is meant to be a stable, correct,
>>> standards-compliant directory server that you can easily use for
>>> development and testing purposes.  There are many essential features for
>>> production deployments that there are no plans to provide, like access
>>> control, replication/high availability, and data durability.
>>>
>>> I'm not really sure what value there is in conducting
>>> load/performance/scalability tests against the server because it is not
>>> intended to be fast or scalable.  It makes use of very heavy locking as
>>> an
>>> easy way to ensure correct behavior by ensuring that the server may only
>>> ever attempt to process one operation at a time.  There are much better
>>> ways to provide concurrency, but the approach it currently uses is both
>>> very simple to implement and a very easy way to guarantee that there is
>>> no
>>> concern about conflicts between operations.
>>>
>>> However, depending on the type of search that you are performing, it
>>> might
>>> be possible to define one or more indexes that would dramatically improve
>>> performance and eliminate the need to iterate across all of the entries.
>>> Since you haven't indicated what kinds of searches you are performing,
>>> it's
>>> not possible to determine whether indexes can help. However, if your
>>> searches are simple filters like "(uid=user.1234)", then you could define
>>> an index for the uid attribute and get much better performance.
>>>
>>> At present, the in-memory directory server only provides support for
>>> equality indexes.  This can help improve the performance of searches with
>>> filters that look like "(indexedAttribute=value)".  Equality indexes can
>>> also be used for AND filters that contain at least one equality component
>>> that targets the indexed attribute, like "(&(indexedAttribute=value)(
>>> someOtherAttribute=someOtherValue))", and for OR filters in which all of
>>> the components are equality components targeting the indexed attribute.
>>>
>>> To configure the in-memory directory server with one or more equality
>>> indexes, use the InMemoryDirectoryServerConfig.
>>> setEqualityIndexAttributes
>>> method and pass in the names of the attributes you want to index.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>  for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>>>             {
>>>>               final DN dn = me.getKey();
>>>>               final Entry entry = me.getValue();
>>>>               try
>>>>               {
>>>>                 if (dn.matchesBaseAndScope(baseDN, scope) &&
>>>>                      filter.matchesEntry(entry, schema))
>>>>                 {
>>>>                   processSearchEntry(entry, includeSubEntries,
>>>> includeChangeLog,
>>>>                        hasManageDsaIT, fullEntryList, referenceList);
>>>>                 }
>>>>               }
>>>>               catch (final Exception e)
>>>>               {
>>>>                 Debug.debugException(e);
>>>>               }
>>>>             }
>>>>
>>>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
>>>> wrote:
>>>>
>>>>   Thanks Neil. Just before you replied, a colleague helped me with it.
>>>>
>>>>>
>>>>> I am doing a search/update load test with 4000 vs 40000 entires against
>>>>> InMemoryServer. Doing a load of search/update on 4000 entries takes 40
>>>>> odd
>>>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>>>> profiler output, it seems most of the time is spent in
>>>>> InMemoryRequestHandler.processSearchRequest - calls to
>>>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>>>> filter
>>>>> indexes might help. But I dont know what they mean and do they apply to
>>>>> InMemoryDirectoryServer.
>>>>>
>>>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>>>           if (candidateDNs == null)
>>>>>           {
>>>>> // my execution lands up here
>>>>>           }
>>>>>           else {
>>>>> // how do I come here
>>>>>           }
>>>>>
>>>>> My seach is Scope SUB with a search filter which is an OR of two object
>>>>> classses.
>>>>>
>>>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>>>> nei...@un...>
>>>>> wrote:
>>>>>
>>>>>   On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>>
>>>>>>
>>>>>>   Hello,
>>>>>>
>>>>>>>
>>>>>>> I have checked out the ldap-sdk-code from svn://
>>>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>>>
>>>>>>> I get lots of errors, examples being
>>>>>>>
>>>>>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>>>>>
>>>>>>> InMemoryDirectoryServer.java - import static
>>>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>>>
>>>>>>> What steps am I missing?
>>>>>>>
>>>>>>> regards,
>>>>>>>
>>>>>>>
>>>>>>>  How are you trying to compile it?  It sounds as if you might just be
>>>>>> doing a javac of everything in the src directory, and that's not good
>>>>>> enough because the build process needs to do other work first (e.g.,
>>>>>> generate source code containing build properties like product name,
>>>>>> version, build date, etc., and also to generate source code from the
>>>>>> messages properties files to guarantee that the code can't try to use
>>>>>> any
>>>>>> messages that aren't defined).
>>>>>>
>>>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>>>> shell script on UNIX-based systems, or the build-se.bat batch file on
>>>>>> Windows.  Alternately, if you've got Apache ant installed, then you
>>>>>> can
>>>>>> run
>>>>>> "ant -buildfile build-se.xml".
>>>>>>
>>>>>>
>>>>>> Neil
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------------
>>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>>> sponsored by Intel and developed in partnership with Slashdot Media,
>>>>>> is
>>>>>> your
>>>>>> hub for all things parallel software development, from weekly thought
>>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>>> Take a
>>>>>> look and join the conversation now. http://goparallel.sourceforge.
>>>>>> net/
>>>>>> _______________________________________________
>>>>>> ldap-sdk-discuss mailing list
>>>>>> lda...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> पंकज एम तोलानी
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>> your
>>>> hub for all things parallel software development, from weekly thought
>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>> Take a
>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ldap-sdk-discuss mailing list
>>>> lda...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take
>>> a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>>
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>
>


-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[Neil A. W. <nei...@un...>]

On 02/09/2015 04:21 PM, pankaj.tolani wrote:
> Many thanks Neil, for your detailed response.
>
> So, our idea is to internal factory acceptance performance test an
> application.
>
> Now, we mainly have just search and modify operations. No adds and deletes.
> Also multiple threads are going to operate on disjoint set of entries. With
> this in mind, I was hoping we could remove the synchronized(entryMap) {} on
> all the operations. Does this sound OK?

This would not be safe if you're performing modify operations, because 
modify operations change the contents of the map.

You might be able to get away with doing that if you changed entryMap 
from a TreeMap to a ConcurrentSkipListMap.  I haven't looked at the code 
closely enough to know if that will eliminate the worst possibilities of 
eliminating synchronization (in particular, deadlocks and concurrent 
modification exceptions), but I know that it won't guarantee correct 
behavior in all circumstances (e.g., if multiple threads try to alter 
the same entry at the same time).


>
> Also, I have been able to convert my SUB scope query to a BASE query.
>
> With this, I am getting a decent performance for 10000 subscribers, 2-3
> msec for one search and one update operation.
>
> On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <nei...@un...>
> wrote:
>
>> On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>>
>>> BTW with search/update, I meant 1 search + 1 update. Just to be clear.
>>>
>>> And I want to avoid the linear loop in
>>> InMemoryRequestHandler.processSearchRequest which reads
>>>
>>
>> First, I should point out that the in-memory directory server isn't meant
>> to be fast, nor should it ever under any circumstances be used in any
>> circumstance that requires a production-quality directory server. The
>> in-memory directory server is meant to be a stable, correct,
>> standards-compliant directory server that you can easily use for
>> development and testing purposes.  There are many essential features for
>> production deployments that there are no plans to provide, like access
>> control, replication/high availability, and data durability.
>>
>> I'm not really sure what value there is in conducting
>> load/performance/scalability tests against the server because it is not
>> intended to be fast or scalable.  It makes use of very heavy locking as an
>> easy way to ensure correct behavior by ensuring that the server may only
>> ever attempt to process one operation at a time.  There are much better
>> ways to provide concurrency, but the approach it currently uses is both
>> very simple to implement and a very easy way to guarantee that there is no
>> concern about conflicts between operations.
>>
>> However, depending on the type of search that you are performing, it might
>> be possible to define one or more indexes that would dramatically improve
>> performance and eliminate the need to iterate across all of the entries.
>> Since you haven't indicated what kinds of searches you are performing, it's
>> not possible to determine whether indexes can help. However, if your
>> searches are simple filters like "(uid=user.1234)", then you could define
>> an index for the uid attribute and get much better performance.
>>
>> At present, the in-memory directory server only provides support for
>> equality indexes.  This can help improve the performance of searches with
>> filters that look like "(indexedAttribute=value)".  Equality indexes can
>> also be used for AND filters that contain at least one equality component
>> that targets the indexed attribute, like "(&(indexedAttribute=value)(
>> someOtherAttribute=someOtherValue))", and for OR filters in which all of
>> the components are equality components targeting the indexed attribute.
>>
>> To configure the in-memory directory server with one or more equality
>> indexes, use the InMemoryDirectoryServerConfig.setEqualityIndexAttributes
>> method and pass in the names of the attributes you want to index.
>>
>>
>>
>>
>>
>>
>>
>>
>>> for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>>             {
>>>               final DN dn = me.getKey();
>>>               final Entry entry = me.getValue();
>>>               try
>>>               {
>>>                 if (dn.matchesBaseAndScope(baseDN, scope) &&
>>>                      filter.matchesEntry(entry, schema))
>>>                 {
>>>                   processSearchEntry(entry, includeSubEntries,
>>> includeChangeLog,
>>>                        hasManageDsaIT, fullEntryList, referenceList);
>>>                 }
>>>               }
>>>               catch (final Exception e)
>>>               {
>>>                 Debug.debugException(e);
>>>               }
>>>             }
>>>
>>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
>>> wrote:
>>>
>>>   Thanks Neil. Just before you replied, a colleague helped me with it.
>>>>
>>>> I am doing a search/update load test with 4000 vs 40000 entires against
>>>> InMemoryServer. Doing a load of search/update on 4000 entries takes 40
>>>> odd
>>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>>> profiler output, it seems most of the time is spent in
>>>> InMemoryRequestHandler.processSearchRequest - calls to
>>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>>> filter
>>>> indexes might help. But I dont know what they mean and do they apply to
>>>> InMemoryDirectoryServer.
>>>>
>>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>>           if (candidateDNs == null)
>>>>           {
>>>> // my execution lands up here
>>>>           }
>>>>           else {
>>>> // how do I come here
>>>>           }
>>>>
>>>> My seach is Scope SUB with a search filter which is an OR of two object
>>>> classses.
>>>>
>>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>>> nei...@un...>
>>>> wrote:
>>>>
>>>>   On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>>
>>>>>   Hello,
>>>>>>
>>>>>> I have checked out the ldap-sdk-code from svn://
>>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>>
>>>>>> I get lots of errors, examples being
>>>>>>
>>>>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>>>>
>>>>>> InMemoryDirectoryServer.java - import static
>>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>>
>>>>>> What steps am I missing?
>>>>>>
>>>>>> regards,
>>>>>>
>>>>>>
>>>>> How are you trying to compile it?  It sounds as if you might just be
>>>>> doing a javac of everything in the src directory, and that's not good
>>>>> enough because the build process needs to do other work first (e.g.,
>>>>> generate source code containing build properties like product name,
>>>>> version, build date, etc., and also to generate source code from the
>>>>> messages properties files to guarantee that the code can't try to use
>>>>> any
>>>>> messages that aren't defined).
>>>>>
>>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>>> shell script on UNIX-based systems, or the build-se.bat batch file on
>>>>> Windows.  Alternately, if you've got Apache ant installed, then you can
>>>>> run
>>>>> "ant -buildfile build-se.xml".
>>>>>
>>>>>
>>>>> Neil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------
>>>>> ------------------
>>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>>> your
>>>>> hub for all things parallel software development, from weekly thought
>>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>>> Take a
>>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>>> _______________________________________________
>>>>> ldap-sdk-discuss mailing list
>>>>> lda...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> पंकज एम तोलानी
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>
>>>
>>>
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

Many thanks Neil, for your detailed response.

So, our idea is to internal factory acceptance performance test an
application.

Now, we mainly have just search and modify operations. No adds and deletes.
Also multiple threads are going to operate on disjoint set of entries. With
this in mind, I was hoping we could remove the synchronized(entryMap) {} on
all the operations. Does this sound OK?

Also, I have been able to convert my SUB scope query to a BASE query.

With this, I am getting a decent performance for 10000 subscribers, 2-3
msec for one search and one update operation.

On Mon, Feb 9, 2015 at 6:08 PM, Neil A. Wilson <nei...@un...>
wrote:

> On 02/09/2015 11:47 AM, pankaj.tolani wrote:
>
>> BTW with search/update, I meant 1 search + 1 update. Just to be clear.
>>
>> And I want to avoid the linear loop in
>> InMemoryRequestHandler.processSearchRequest which reads
>>
>
> First, I should point out that the in-memory directory server isn't meant
> to be fast, nor should it ever under any circumstances be used in any
> circumstance that requires a production-quality directory server. The
> in-memory directory server is meant to be a stable, correct,
> standards-compliant directory server that you can easily use for
> development and testing purposes.  There are many essential features for
> production deployments that there are no plans to provide, like access
> control, replication/high availability, and data durability.
>
> I'm not really sure what value there is in conducting
> load/performance/scalability tests against the server because it is not
> intended to be fast or scalable.  It makes use of very heavy locking as an
> easy way to ensure correct behavior by ensuring that the server may only
> ever attempt to process one operation at a time.  There are much better
> ways to provide concurrency, but the approach it currently uses is both
> very simple to implement and a very easy way to guarantee that there is no
> concern about conflicts between operations.
>
> However, depending on the type of search that you are performing, it might
> be possible to define one or more indexes that would dramatically improve
> performance and eliminate the need to iterate across all of the entries.
> Since you haven't indicated what kinds of searches you are performing, it's
> not possible to determine whether indexes can help. However, if your
> searches are simple filters like "(uid=user.1234)", then you could define
> an index for the uid attribute and get much better performance.
>
> At present, the in-memory directory server only provides support for
> equality indexes.  This can help improve the performance of searches with
> filters that look like "(indexedAttribute=value)".  Equality indexes can
> also be used for AND filters that contain at least one equality component
> that targets the indexed attribute, like "(&(indexedAttribute=value)(
> someOtherAttribute=someOtherValue))", and for OR filters in which all of
> the components are equality components targeting the indexed attribute.
>
> To configure the in-memory directory server with one or more equality
> indexes, use the InMemoryDirectoryServerConfig.setEqualityIndexAttributes
> method and pass in the names of the attributes you want to index.
>
>
>
>
>
>
>
>
>> for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>>            {
>>              final DN dn = me.getKey();
>>              final Entry entry = me.getValue();
>>              try
>>              {
>>                if (dn.matchesBaseAndScope(baseDN, scope) &&
>>                     filter.matchesEntry(entry, schema))
>>                {
>>                  processSearchEntry(entry, includeSubEntries,
>> includeChangeLog,
>>                       hasManageDsaIT, fullEntryList, referenceList);
>>                }
>>              }
>>              catch (final Exception e)
>>              {
>>                Debug.debugException(e);
>>              }
>>            }
>>
>> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
>> wrote:
>>
>>  Thanks Neil. Just before you replied, a colleague helped me with it.
>>>
>>> I am doing a search/update load test with 4000 vs 40000 entires against
>>> InMemoryServer. Doing a load of search/update on 4000 entries takes 40
>>> odd
>>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>>> profiler output, it seems most of the time is spent in
>>> InMemoryRequestHandler.processSearchRequest - calls to
>>> DN.matchesBaseAndScope. Looking at the code around this, seems like
>>> filter
>>> indexes might help. But I dont know what they mean and do they apply to
>>> InMemoryDirectoryServer.
>>>
>>> final Set<DN> candidateDNs = indexSearch(filter);
>>>          if (candidateDNs == null)
>>>          {
>>> // my execution lands up here
>>>          }
>>>          else {
>>> // how do I come here
>>>          }
>>>
>>> My seach is Scope SUB with a search filter which is an OR of two object
>>> classses.
>>>
>>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <
>>> nei...@un...>
>>> wrote:
>>>
>>>  On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>>
>>>>  Hello,
>>>>>
>>>>> I have checked out the ldap-sdk-code from svn://
>>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>>
>>>>> I get lots of errors, examples being
>>>>>
>>>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>>>
>>>>> InMemoryDirectoryServer.java - import static
>>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>>
>>>>> What steps am I missing?
>>>>>
>>>>> regards,
>>>>>
>>>>>
>>>> How are you trying to compile it?  It sounds as if you might just be
>>>> doing a javac of everything in the src directory, and that's not good
>>>> enough because the build process needs to do other work first (e.g.,
>>>> generate source code containing build properties like product name,
>>>> version, build date, etc., and also to generate source code from the
>>>> messages properties files to guarantee that the code can't try to use
>>>> any
>>>> messages that aren't defined).
>>>>
>>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>>> shell script on UNIX-based systems, or the build-se.bat batch file on
>>>> Windows.  Alternately, if you've got Apache ant installed, then you can
>>>> run
>>>> "ant -buildfile build-se.xml".
>>>>
>>>>
>>>> Neil
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>>> your
>>>> hub for all things parallel software development, from weekly thought
>>>> leadership blogs to news, videos, case studies, tutorials and more.
>>>> Take a
>>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>>> _______________________________________________
>>>> ldap-sdk-discuss mailing list
>>>> lda...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>>
>>>>
>>>>
>>>
>>> --
>>> पंकज एम तोलानी
>>>
>>>
>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>>
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>
>


-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[Neil A. W. <nei...@un...>]

On 02/09/2015 11:47 AM, pankaj.tolani wrote:
> BTW with search/update, I meant 1 search + 1 update. Just to be clear.
>
> And I want to avoid the linear loop in
> InMemoryRequestHandler.processSearchRequest which reads

First, I should point out that the in-memory directory server isn't 
meant to be fast, nor should it ever under any circumstances be used in 
any circumstance that requires a production-quality directory server. 
The in-memory directory server is meant to be a stable, correct, 
standards-compliant directory server that you can easily use for 
development and testing purposes.  There are many essential features for 
production deployments that there are no plans to provide, like access 
control, replication/high availability, and data durability.

I'm not really sure what value there is in conducting 
load/performance/scalability tests against the server because it is not 
intended to be fast or scalable.  It makes use of very heavy locking as 
an easy way to ensure correct behavior by ensuring that the server may 
only ever attempt to process one operation at a time.  There are much 
better ways to provide concurrency, but the approach it currently uses 
is both very simple to implement and a very easy way to guarantee that 
there is no concern about conflicts between operations.

However, depending on the type of search that you are performing, it 
might be possible to define one or more indexes that would dramatically 
improve performance and eliminate the need to iterate across all of the 
entries.  Since you haven't indicated what kinds of searches you are 
performing, it's not possible to determine whether indexes can help. 
However, if your searches are simple filters like "(uid=user.1234)", 
then you could define an index for the uid attribute and get much better 
performance.

At present, the in-memory directory server only provides support for 
equality indexes.  This can help improve the performance of searches 
with filters that look like "(indexedAttribute=value)".  Equality 
indexes can also be used for AND filters that contain at least one 
equality component that targets the indexed attribute, like 
"(&(indexedAttribute=value)(someOtherAttribute=someOtherValue))", and 
for OR filters in which all of the components are equality components 
targeting the indexed attribute.

To configure the in-memory directory server with one or more equality 
indexes, use the 
InMemoryDirectoryServerConfig.setEqualityIndexAttributes method and pass 
in the names of the attributes you want to index.






>
> for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
>            {
>              final DN dn = me.getKey();
>              final Entry entry = me.getValue();
>              try
>              {
>                if (dn.matchesBaseAndScope(baseDN, scope) &&
>                     filter.matchesEntry(entry, schema))
>                {
>                  processSearchEntry(entry, includeSubEntries,
> includeChangeLog,
>                       hasManageDsaIT, fullEntryList, referenceList);
>                }
>              }
>              catch (final Exception e)
>              {
>                Debug.debugException(e);
>              }
>            }
>
> On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
> wrote:
>
>> Thanks Neil. Just before you replied, a colleague helped me with it.
>>
>> I am doing a search/update load test with 4000 vs 40000 entires against
>> InMemoryServer. Doing a load of search/update on 4000 entries takes 40 odd
>> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
>> profiler output, it seems most of the time is spent in
>> InMemoryRequestHandler.processSearchRequest - calls to
>> DN.matchesBaseAndScope. Looking at the code around this, seems like filter
>> indexes might help. But I dont know what they mean and do they apply to
>> InMemoryDirectoryServer.
>>
>> final Set<DN> candidateDNs = indexSearch(filter);
>>          if (candidateDNs == null)
>>          {
>> // my execution lands up here
>>          }
>>          else {
>> // how do I come here
>>          }
>>
>> My seach is Scope SUB with a search filter which is an OR of two object
>> classses.
>>
>> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <nei...@un...>
>> wrote:
>>
>>> On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>>
>>>> Hello,
>>>>
>>>> I have checked out the ldap-sdk-code from svn://
>>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>>
>>>> I get lots of errors, examples being
>>>>
>>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>>
>>>> InMemoryDirectoryServer.java - import static
>>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>>
>>>> What steps am I missing?
>>>>
>>>> regards,
>>>>
>>>
>>> How are you trying to compile it?  It sounds as if you might just be
>>> doing a javac of everything in the src directory, and that's not good
>>> enough because the build process needs to do other work first (e.g.,
>>> generate source code containing build properties like product name,
>>> version, build date, etc., and also to generate source code from the
>>> messages properties files to guarantee that the code can't try to use any
>>> messages that aren't defined).
>>>
>>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>>> shell script on UNIX-based systems, or the build-se.bat batch file on
>>> Windows.  Alternately, if you've got Apache ant installed, then you can run
>>> "ant -buildfile build-se.xml".
>>>
>>>
>>> Neil
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> ldap-sdk-discuss mailing list
>>> lda...@li...
>>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>>
>>>
>>
>>
>> --
>> पंकज एम तोलानी
>>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

BTW with search/update, I meant 1 search + 1 update. Just to be clear.

And I want to avoid the linear loop in
InMemoryRequestHandler.processSearchRequest which reads

for (final Map.Entry<DN,ReadOnlyEntry> me : entryMap.entrySet())
          {
            final DN dn = me.getKey();
            final Entry entry = me.getValue();
            try
            {
              if (dn.matchesBaseAndScope(baseDN, scope) &&
                   filter.matchesEntry(entry, schema))
              {
                processSearchEntry(entry, includeSubEntries,
includeChangeLog,
                     hasManageDsaIT, fullEntryList, referenceList);
              }
            }
            catch (final Exception e)
            {
              Debug.debugException(e);
            }
          }

On Mon, Feb 9, 2015 at 5:43 PM, pankaj.tolani <pan...@gm...>
wrote:

> Thanks Neil. Just before you replied, a colleague helped me with it.
>
> I am doing a search/update load test with 4000 vs 40000 entires against
> InMemoryServer. Doing a load of search/update on 4000 entries takes 40 odd
> seconds, whereas 350 odd seconds with 40000 entries. Looking at the
> profiler output, it seems most of the time is spent in
> InMemoryRequestHandler.processSearchRequest - calls to
> DN.matchesBaseAndScope. Looking at the code around this, seems like filter
> indexes might help. But I dont know what they mean and do they apply to
> InMemoryDirectoryServer.
>
> final Set<DN> candidateDNs = indexSearch(filter);
>         if (candidateDNs == null)
>         {
> // my execution lands up here
>         }
>         else {
> // how do I come here
>         }
>
> My seach is Scope SUB with a search filter which is an OR of two object
> classses.
>
> On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <nei...@un...>
> wrote:
>
>> On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>>
>>> Hello,
>>>
>>> I have checked out the ldap-sdk-code from svn://
>>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>>
>>> I get lots of errors, examples being
>>>
>>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>>
>>> InMemoryDirectoryServer.java - import static
>>> com.unboundid.ldap.listener.ListenerMessages.*;
>>>
>>> What steps am I missing?
>>>
>>> regards,
>>>
>>
>> How are you trying to compile it?  It sounds as if you might just be
>> doing a javac of everything in the src directory, and that's not good
>> enough because the build process needs to do other work first (e.g.,
>> generate source code containing build properties like product name,
>> version, build date, etc., and also to generate source code from the
>> messages properties files to guarantee that the code can't try to use any
>> messages that aren't defined).
>>
>> The easiest way to build the LDAP SDK is to just run the build-se.sh
>> shell script on UNIX-based systems, or the build-se.bat batch file on
>> Windows.  Alternately, if you've got Apache ant installed, then you can run
>> "ant -buildfile build-se.xml".
>>
>>
>> Neil
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>>
>
>
> --
> पंकज एम तोलानी
>



-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

Thanks Neil. Just before you replied, a colleague helped me with it.

I am doing a search/update load test with 4000 vs 40000 entires against
InMemoryServer. Doing a load of search/update on 4000 entries takes 40 odd
seconds, whereas 350 odd seconds with 40000 entries. Looking at the
profiler output, it seems most of the time is spent in
InMemoryRequestHandler.processSearchRequest - calls to
DN.matchesBaseAndScope. Looking at the code around this, seems like filter
indexes might help. But I dont know what they mean and do they apply to
InMemoryDirectoryServer.

final Set<DN> candidateDNs = indexSearch(filter);
        if (candidateDNs == null)
        {
// my execution lands up here
        }
        else {
// how do I come here
        }

My seach is Scope SUB with a search filter which is an OR of two object
classses.

On Mon, Feb 9, 2015 at 4:59 PM, Neil A. Wilson <nei...@un...>
wrote:

> On 02/09/2015 04:36 AM, pankaj.tolani wrote:
>
>> Hello,
>>
>> I have checked out the ldap-sdk-code from svn://
>> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>>
>> I get lots of errors, examples being
>>
>> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>>
>> InMemoryDirectoryServer.java - import static
>> com.unboundid.ldap.listener.ListenerMessages.*;
>>
>> What steps am I missing?
>>
>> regards,
>>
>
> How are you trying to compile it?  It sounds as if you might just be doing
> a javac of everything in the src directory, and that's not good enough
> because the build process needs to do other work first (e.g., generate
> source code containing build properties like product name, version, build
> date, etc., and also to generate source code from the messages properties
> files to guarantee that the code can't try to use any messages that aren't
> defined).
>
> The easiest way to build the LDAP SDK is to just run the build-se.sh shell
> script on UNIX-based systems, or the build-se.bat batch file on Windows.
> Alternately, if you've got Apache ant installed, then you can run "ant
> -buildfile build-se.xml".
>
>
> Neil
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>
>


-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] compiling ldap-sdk-code

[Neil A. W. <nei...@un...>]

On 02/09/2015 04:36 AM, pankaj.tolani wrote:
> Hello,
>
> I have checked out the ldap-sdk-code from svn://
> svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.
>
> I get lots of errors, examples being
>
> ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;
>
> InMemoryDirectoryServer.java - import static
> com.unboundid.ldap.listener.ListenerMessages.*;
>
> What steps am I missing?
>
> regards,

How are you trying to compile it?  It sounds as if you might just be 
doing a javac of everything in the src directory, and that's not good 
enough because the build process needs to do other work first (e.g., 
generate source code containing build properties like product name, 
version, build date, etc., and also to generate source code from the 
messages properties files to guarantee that the code can't try to use 
any messages that aren't defined).

The easiest way to build the LDAP SDK is to just run the build-se.sh 
shell script on UNIX-based systems, or the build-se.bat batch file on 
Windows.  Alternately, if you've got Apache ant installed, then you can 
run "ant -buildfile build-se.xml".


Neil

[ldap-sdk-discuss] compiling ldap-sdk-code

[pankaj.tolani <pan...@gm...>]

Hello,

I have checked out the ldap-sdk-code from svn://
svn.code.sf.net/p/ldap-sdk/code/trunk, but it does not compile.

I get lots of errors, examples being

ASN1Boolean.java - import static com.unboundid.asn1.ASN1Messages.*;

InMemoryDirectoryServer.java - import static
com.unboundid.ldap.listener.ListenerMessages.*;

What steps am I missing?

regards,
-- 
पंकज एम तोलानी

Re: [ldap-sdk-discuss] UnboundId 2.3.8 SimplePagedResultsControl throws ASN.1 stream reader exception if page size is above certain value

[Phoebe S. <ps...@vm...>]

Hi Neil,

Thanks for the prompt reply.

Somehow I cannot reproduce the problem now.  :(   I ran the test several times but now it always worked fine.  I bumped the pages size to 100 and 1000, both works perfectly.    I will keep an eye on it and provide the full stack if I see it again.

Thanks!

phoebe 

________________________________________
From: Neil A. Wilson <nei...@un...>
Sent: Monday, December 29, 2014 6:29 PM
To: Phoebe Shao; lda...@un...
Cc: lda...@li...
Subject: Re: UnboundId 2.3.8 SimplePagedResultsControl throws ASN.1 stream reader exception if page size is above certain value

The exception message suggests that the server is sending a malformed
response to the client.  LDAP messages are encoded in ASN.1, and each
ASN.1 element specifies up front how big the element's value will be.
It is also very common for the value of one ASN.1 element to be
comprised of one or more other ASN.1 elements.

It would be useful to have the full stack trace to see exactly where
this is happening, but it looks like the problem here is that the length
of a nested element did not match up with the length of the element that
contained it.  While it could be a problem with the LDAP SDK, no one
else has reported this problem, and this code is very heavily used, so
it suggests that the problem may be with the response that the server is
sending.

What directory server are you using?  Could you provide the complete
stack trace (including class names and line numbers) for the exception?


On 12/29/2014 02:52 PM, Phoebe Shao wrote:
> Hi,
>
> With the following code (copied from the https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.ldap.com_ldap-2Dsdk_docs_javadoc_index.html-29-3A&d=AwIF-g&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=z940VUKgiVziZS78LuciTQ&m=wPrUk-rZarPiJMn7i10YkMPUUR4EddfO9vlN6-jWPZA&s=E0yrZ1McLjOxxOzp2oYHBU8xqyAQn561ztD48_BSEh0&e=
>
>      public void searchPagedTest(String base, String[] attrsToReturn,
>              LdapSearchFilter filter, Object... filterArgs) throws LDAPException {
>          int numSearches = 0;
>          int totalEntriesReturned = 0;
>
>          ASN1OctetString resumeCookie = null;
>
>          SearchRequest searchRequest = new SearchRequest(base, SearchScope.SUB,
>                  Filter.createEqualityFilter("objectClass", "person"));
>
>          while (true) {
>              searchRequest.setControls(new SimplePagedResultsControl(100,
>                      resumeCookie));
>              SearchResult searchResult = ldapConn.search(searchRequest);
>              numSearches++;
>              totalEntriesReturned += searchResult.getEntryCount();
>              for (SearchResultEntry e : searchResult.getSearchEntries()) {
>                  // Do something with each entry...
>              }
>
>              System.out.println(numSearches + " " + totalEntriesReturned);
>
>              LDAPTestUtils.assertHasControl(searchResult,
>                      SimplePagedResultsControl.PAGED_RESULTS_OID);
>              SimplePagedResultsControl responseControl = SimplePagedResultsControl
>                      .get(searchResult);
>              if (responseControl.moreResultsToReturn()) {
>                  resumeCookie = responseControl.getCookie();
>              } else {
>                  break;
>              }
>          }
>      }
>
> As you can see from the code, it prints numSearches and totalEntriesReturned each page is fetched.
>
> After it prints:
> numSearches=55, totalEntriesReturned=5500
>
> It throws this error:
> Caused by: LDAPException(resultCode=84 (decoding error), errorMessage='Unable to read or decode an LDAP attribute:  The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).com.unboundid.asn1.ASN1Exception: The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).')
>
> If I  changed the page size from 100 to 10, then it works fine and fetched all29637?? entries from the AD.
>
> Looks like a bug.  Or is there any any parameter I can tweak to get around this?
>
> I don's have a support license.  It is not clear to me if this is the correct channel to report bugs.  If it is not, please instruct.
>
> I was using 2.3.6, then upgraded to 2.3.8, encounter the same problem on both.
>
> Thanks,
>
> phoebe

Re: [ldap-sdk-discuss] UnboundId 2.3.8 SimplePagedResultsControl throws ASN.1 stream reader exception if page size is above certain value

[Neil A. W. <nei...@un...>]

The exception message suggests that the server is sending a malformed 
response to the client.  LDAP messages are encoded in ASN.1, and each 
ASN.1 element specifies up front how big the element's value will be. 
It is also very common for the value of one ASN.1 element to be 
comprised of one or more other ASN.1 elements.

It would be useful to have the full stack trace to see exactly where 
this is happening, but it looks like the problem here is that the length 
of a nested element did not match up with the length of the element that 
contained it.  While it could be a problem with the LDAP SDK, no one 
else has reported this problem, and this code is very heavily used, so 
it suggests that the problem may be with the response that the server is 
sending.

What directory server are you using?  Could you provide the complete 
stack trace (including class names and line numbers) for the exception?


On 12/29/2014 02:52 PM, Phoebe Shao wrote:
> Hi,
>
> With the following code (copied from the https://docs.ldap.com/ldap-sdk/docs/javadoc/index.html):
>
>      public void searchPagedTest(String base, String[] attrsToReturn,
>              LdapSearchFilter filter, Object... filterArgs) throws LDAPException {
>          int numSearches = 0;
>          int totalEntriesReturned = 0;
>
>          ASN1OctetString resumeCookie = null;
>
>          SearchRequest searchRequest = new SearchRequest(base, SearchScope.SUB,
>                  Filter.createEqualityFilter("objectClass", "person"));
>
>          while (true) {
>              searchRequest.setControls(new SimplePagedResultsControl(100,
>                      resumeCookie));
>              SearchResult searchResult = ldapConn.search(searchRequest);
>              numSearches++;
>              totalEntriesReturned += searchResult.getEntryCount();
>              for (SearchResultEntry e : searchResult.getSearchEntries()) {
>                  // Do something with each entry...
>              }
>
>              System.out.println(numSearches + " " + totalEntriesReturned);
>
>              LDAPTestUtils.assertHasControl(searchResult,
>                      SimplePagedResultsControl.PAGED_RESULTS_OID);
>              SimplePagedResultsControl responseControl = SimplePagedResultsControl
>                      .get(searchResult);
>              if (responseControl.moreResultsToReturn()) {
>                  resumeCookie = responseControl.getCookie();
>              } else {
>                  break;
>              }
>          }
>      }
>
> As you can see from the code, it prints numSearches and totalEntriesReturned each page is fetched.
>
> After it prints:
> numSearches=55, totalEntriesReturned=5500
>
> It throws this error:
> Caused by: LDAPException(resultCode=84 (decoding error), errorMessage='Unable to read or decode an LDAP attribute:  The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).com.unboundid.asn1.ASN1Exception: The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).')
>
> If I  changed the page size from 100 to 10, then it works fine and fetched all29637?? entries from the AD.
>
> Looks like a bug.  Or is there any any parameter I can tweak to get around this?
>
> I don's have a support license.  It is not clear to me if this is the correct channel to report bugs.  If it is not, please instruct.
>
> I was using 2.3.6, then upgraded to 2.3.8, encounter the same problem on both.
>
> Thanks,
>
> phoebe

[ldap-sdk-discuss] UnboundId 2.3.8 SimplePagedResultsControl throws ASN.1 stream reader exception if page size is above certain value

[Phoebe S. <ps...@vm...>]

Hi,


With the following code (copied from the https://docs.ldap.com/ldap-sdk/docs/javadoc/index.html):


    public void searchPagedTest(String base, String[] attrsToReturn,

            LdapSearchFilter filter, Object... filterArgs) throws LDAPException {

        int numSearches = 0;

        int totalEntriesReturned = 0;


        ASN1OctetString resumeCookie = null;


        SearchRequest searchRequest = new SearchRequest(base, SearchScope.SUB,

                Filter.createEqualityFilter("objectClass", "person"));


        while (true) {

            searchRequest.setControls(new SimplePagedResultsControl(100,

                    resumeCookie));

            SearchResult searchResult = ldapConn.search(searchRequest);

            numSearches++;

            totalEntriesReturned += searchResult.getEntryCount();

            for (SearchResultEntry e : searchResult.getSearchEntries()) {

                // Do something with each entry...

            }


            System.out.println(numSearches + " " + totalEntriesReturned);


            LDAPTestUtils.assertHasControl(searchResult,

                    SimplePagedResultsControl.PAGED_RESULTS_OID);

            SimplePagedResultsControl responseControl = SimplePagedResultsControl

                    .get(searchResult);

            if (responseControl.moreResultsToReturn()) {

                resumeCookie = responseControl.getCookie();

            } else {

                break;

            }

        }

    }



As you can see from the code, it prints numSearches and totalEntriesReturned each page is fetched.


After it prints:

numSearches=55, totalEntriesReturned=5500


It throws this error:

Caused by: LDAPException(resultCode=84 (decoding error), errorMessage='Unable to read or decode an LDAP attribute:  The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).com.unboundid.asn1.ASN1Exception: The ASN.1 stream reader has already read beyond the end of this set (expected set of length 4 to end at 8,711,433 bytes into the stream, but 8,711,543 bytes have already been read from the stream).')


If I  changed the page size from 100 to 10, then it works fine and fetched all29637?? entries from the AD.


Looks like a bug.  Or is there any any parameter I can tweak to get around this?


I don's have a support license.  It is not clear to me if this is the correct channel to report bugs.  If it is not, please instruct.


I was using 2.3.6, then upgraded to 2.3.8, encounter the same problem on both.


Thanks,


phoebe

Re: [ldap-sdk-discuss] GSSAPIBindRequestProperties and suppressedSystemProperties

[Alon Bar-L. <alo...@gm...>]

Thank you!

On Wed, Jun 25, 2014 at 11:49 PM, Neil A. Wilson
<nei...@un...> wrote:
> On 06/25/2014 06:05 AM, Alon Bar-Lev wrote:
>>
>> Hello,
>>
>> Thank you for adding that!
>>
>> Very usable for shared applications use case.
>>
>> I believe that:
>>
>>          System.clearProperty(PROPERTY_KDC_ADDRESS);
>>          System.clearProperty(PROPERTY_REALM);
>>
>> Should follow the same logic, and be excluded if within
>> suppressedSystemProperties.
>
>
> You are correct.  This is now done.
>
>
>>
>> Thanks!
>> Alon
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Open source business process management suite built on Java and Eclipse
>> Turn processes into business applications with Bonita BPM Community
>> Edition
>> Quickly connect people, data, and systems into organized workflows
>> Winner of BOSSIE, CODIE, OW2 and Gartner awards
>> http://p.sf.net/sfu/Bonitasoft
>>
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>
>
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] LDAPConnection::getStartTLSRequest - please make it public

[Alon Bar-L. <alo...@gm...>]

Thank you!


On Wed, Jun 25, 2014 at 11:51 PM, Neil A. Wilson
<nei...@un...> wrote:
>
> On 06/25/2014 08:09 AM, Alon Bar-Lev wrote:
>>
>> Hello,
>>
>> This function is required in order to enable programs to implement Referral
>> Connector with full functionality.
>>
>> The missing bit is to be able to call getStartTLSRequest.
>>
>> It is the only function that has no access modifier, maybe it should have
>> been public.
>
>
> It's not the only method without an access modifier -- there are several other non-public methods in the LDAPConnection class, and they are generally intended to only be called within the LDAP SDK itself and not by third-party code.
>
> However, the getStartTLSRequest method is now public.
>
>
>
>>
>> Thanks,
>> Alon
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Open source business process management suite built on Java and Eclipse
>> Turn processes into business applications with Bonita BPM Community Edition
>> Quickly connect people, data, and systems into organized workflows
>> Winner of BOSSIE, CODIE, OW2 and Gartner awards
>> http://p.sf.net/sfu/Bonitasoft
>>
>>
>>
>> _______________________________________________
>> ldap-sdk-discuss mailing list
>> lda...@li...
>> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>>
>
>
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] LDAPConnection::getStartTLSRequest - please make it public

[Neil A. W. <nei...@un...>]

On 06/25/2014 08:09 AM, Alon Bar-Lev wrote:
> Hello,
>
> This function is required in order to enable programs to implement Referral
> Connector with full functionality.
>
> The missing bit is to be able to call getStartTLSRequest.
>
> It is the only function that has no access modifier, maybe it should have
> been public.

It's not the only method without an access modifier -- there are several 
other non-public methods in the LDAPConnection class, and they are 
generally intended to only be called within the LDAP SDK itself and not 
by third-party code.

However, the getStartTLSRequest method is now public.



>
> Thanks,
> Alon
>
>
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>

Re: [ldap-sdk-discuss] GSSAPIBindRequestProperties and suppressedSystemProperties

[Neil A. W. <nei...@un...>]

On 06/25/2014 06:05 AM, Alon Bar-Lev wrote:
> Hello,
>
> Thank you for adding that!
>
> Very usable for shared applications use case.
>
> I believe that:
>
>          System.clearProperty(PROPERTY_KDC_ADDRESS);
>          System.clearProperty(PROPERTY_REALM);
>
> Should follow the same logic, and be excluded if within
> suppressedSystemProperties.

You are correct.  This is now done.


>
> Thanks!
> Alon
>
>
>
> ------------------------------------------------------------------------------
> Open source business process management suite built on Java and Eclipse
> Turn processes into business applications with Bonita BPM Community Edition
> Quickly connect people, data, and systems into organized workflows
> Winner of BOSSIE, CODIE, OW2 and Gartner awards
> http://p.sf.net/sfu/Bonitasoft
>
>
>
> _______________________________________________
> ldap-sdk-discuss mailing list
> lda...@li...
> https://lists.sourceforge.net/lists/listinfo/ldap-sdk-discuss
>