Are there any limitations in the number of used attributes?
I use a lot of attributes, and i have the problem that some attribute values are lost from a certain number of used attributes. Is this hash table size limit the cause of the problem (BOOST_LOG_HASH_TABLE_SIZE_LOG)?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There is a limit on the number of attribute names you can use throughout the application, it's (2^32 - 1). Note that whenever an attribute name is created, the resource is taken and never released until the application terminates.
There is no special limit on the number of attributes that can be in a set (global, thread- or source-specific), nor there is any limit for attribute values. The BOOST_LOG_HASH_TABLE_SIZE_LOG constant defines the internal hash table size, it does not limit the number of elements that can be stored in the set. However, if the number of elements exceed the hash table size performance will start degrading rather quickly.
If you're using same-named attributes in different sets you will see that only one of them is used to obtain attribute values for log records. See here for more details.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I use at the moment 19 attributes, most of them are source-specific. There are only 4 global attributes, all others are source-specific (application context-specific). Each attribute gets an ID in attribute_set, and my conclusion is that this ID is this limit that you mentioned (it is an 32bit integer).
I can iterate over all attributes in attribute_set, all are available. But the problem is attribute_set::find() method. One of the internal implementation methods looks like this:
buckets::static_size is 16 (because BOOST_LOG_HASH_TABLE_SIZE_LOG is 4, so array size is 2^4). Attribute "Message" gets id=2, and attribute "UserID" gets id=18. For both id's i get m_Buckets[2] element, so that the first attribute "Message" can't be found (element is found, but because id does not match empty value is returned for attribute "Message"). As result i get a log message without message text.
I have compiled the Boost.Log code with BOOST_LOG_HASH_TABLE_SIZE_LOG=5 and it works as expected (i get also message text). It works also if i use less then 16 attributes. But if i use more, i get this problem.
Did I do something wrong to provoke this behavior? Or is this a bug (or unwanted limit)? All my attributes have unique names.
Thanks in advance!
Last edit: ZagorTeNej 2014-06-19
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
get_bucket() returns a bucket, which is basically a section within a list of elements; all elements in this section have keys that match this bucket.
The bucket is not the result of the find() operation. You can see it in its code. After the bucket is found, the function iterates through its elements and looks for an element with the key to be found. So in your case the bucket should have both "UserID" and "Message" attributes, and the lookup loop should find either of them. That is unless I'm missing something. If it doesn't work that way then it's definitely a bug.
It'd be easier for me if I could reproduce it. Could you provide a minimal code sample that shows the problem?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Are there any limitations in the number of used attributes?
I use a lot of attributes, and i have the problem that some attribute values are lost from a certain number of used attributes. Is this hash table size limit the cause of the problem (BOOST_LOG_HASH_TABLE_SIZE_LOG)?
There is a limit on the number of attribute names you can use throughout the application, it's (2^32 - 1). Note that whenever an attribute name is created, the resource is taken and never released until the application terminates.
There is no special limit on the number of attributes that can be in a set (global, thread- or source-specific), nor there is any limit for attribute values. The BOOST_LOG_HASH_TABLE_SIZE_LOG constant defines the internal hash table size, it does not limit the number of elements that can be stored in the set. However, if the number of elements exceed the hash table size performance will start degrading rather quickly.
If you're using same-named attributes in different sets you will see that only one of them is used to obtain attribute values for log records. See here for more details.
I use at the moment 19 attributes, most of them are source-specific. There are only 4 global attributes, all others are source-specific (application context-specific). Each attribute gets an ID in attribute_set, and my conclusion is that this ID is this limit that you mentioned (it is an 32bit integer).
I can iterate over all attributes in attribute_set, all are available. But the problem is attribute_set::find() method. One of the internal implementation methods looks like this:
bucket& get_bucket(id_type id)
{
return m_Buckets[id & (buckets::static_size - 1)];
}
buckets::static_size is 16 (because BOOST_LOG_HASH_TABLE_SIZE_LOG is 4, so array size is 2^4). Attribute "Message" gets id=2, and attribute "UserID" gets id=18. For both id's i get m_Buckets[2] element, so that the first attribute "Message" can't be found (element is found, but because id does not match empty value is returned for attribute "Message"). As result i get a log message without message text.
I have compiled the Boost.Log code with BOOST_LOG_HASH_TABLE_SIZE_LOG=5 and it works as expected (i get also message text). It works also if i use less then 16 attributes. But if i use more, i get this problem.
Did I do something wrong to provoke this behavior? Or is this a bug (or unwanted limit)? All my attributes have unique names.
Thanks in advance!
Last edit: ZagorTeNej 2014-06-19
get_bucket() returns a bucket, which is basically a section within a list of elements; all elements in this section have keys that match this bucket.
The bucket is not the result of the find() operation. You can see it in its code. After the bucket is found, the function iterates through its elements and looks for an element with the key to be found. So in your case the bucket should have both "UserID" and "Message" attributes, and the lookup loop should find either of them. That is unless I'm missing something. If it doesn't work that way then it's definitely a bug.
It'd be easier for me if I could reproduce it. Could you provide a minimal code sample that shows the problem?
This simple application should help to reproduce it.
Thanks for the helpfulness!
P.S.
I use Linux and boost_1_55_0
Last edit: ZagorTeNej 2014-06-20
Fixed. Thank you for the report.
Last edit: Andrey Semashev 2014-06-21
Will this fix also be ported into boost version 1.56 (or even in 1.55)?
Great library, very powerful. We use it in a very large enterprise application, and are very happy with it :-)
1.56 is not yet released. I've merged the fix to master, so it will be released. No plans for 1.55.