I would like to know if there are recommendation to use stxxl::vector inside a multi-threaded environment. In fact, I several thread, each thread manage its own structure containing a stxxl::vector.
I don't know if it can cause a problem, but in this configuration i get the following error:
> msvcp80d.dll!std::_Debug_message(const wchar_t * message=0x00551038, const wchar_t * file=0x005509d0, unsigned int line=293) Line 24 C++
tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator==(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651)) Line 293 + 0x17 bytes C++
tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator!=(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651)) Line 305 + 0xc bytes C++
tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BID<2097152> * begin=0x0036f740, stxxl::BID<2097152> * end=0x0036f750) Line 586 + 0x34 bytes C++
tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BIDArray<2097152> & bids={...}) Line 544 C++
tdb.exe!stxxl::block_manager::new_blocks_int<stxxl::BID<2097152>,stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(const unsigned int nblocks=1, stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > out={storage=0x0036dee8 offset=39845888 }) Line 1198 + 0x1a bytes C++
tdb.exe!stxxl::block_manager::new_blocks<stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidbegin={storage=0x0036dee8 offset=39845888 }, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidend={storage=0xfdfdfdfd offset=-76843842582893653 }) Line 1251 C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::reserve(unsigned __int64 n=1) Line 595 + 0xb9 bytes C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::resize(unsigned __int64 n=1) Line 636 C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::push_back(const test::tickdb::QuoteInfo & obj={...}) Line 657 C++
The error message from vs2005 is: _DEBUG_ERROR("map/set iterators incompatible") in the xtree file from vc8 line 293
Thank you
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The problem is that the Stxxl Block Management (BM) layer, stxxl::vectors are calling, is not thread-safe. The thread safety for the BM layer has been introduced in one of the developement branches and will come in the next release. However, we have just merged these changes in the trunk version which you can check out using subversion:
Thank you for this information. You realize a REALLY good work.
I've a question around the container and the block management system. In fact, i've to sort gigas of value. It works great. However I would like to know if it possible to use a "part" of the available memory. In fact, it seems that the buffer files configured inside the .stxxl are always used. I would like to know if I can tell the BM to use a part of the memory available on my computer ? Or what to do you think on linux to create the buffer file on a tmpfs space ?
Thank you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
if you do not know what to do with the unused main memory ;-) it is better to invest it into the caches of stxxl data structures and/or memory buffers used by sorting/scanning. If your problem fits into this memory, almost no disk I/O will be needed by stxxl.
Roman
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi !
I would like to know if there are recommendation to use stxxl::vector inside a multi-threaded environment. In fact, I several thread, each thread manage its own structure containing a stxxl::vector.
I don't know if it can cause a problem, but in this configuration i get the following error:
> msvcp80d.dll!std::_Debug_message(const wchar_t * message=0x00551038, const wchar_t * file=0x005509d0, unsigned int line=293) Line 24 C++
tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator==(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651)) Line 293 + 0x17 bytes C++
tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator!=(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651)) Line 305 + 0xc bytes C++
tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BID<2097152> * begin=0x0036f740, stxxl::BID<2097152> * end=0x0036f750) Line 586 + 0x34 bytes C++
tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BIDArray<2097152> & bids={...}) Line 544 C++
tdb.exe!stxxl::block_manager::new_blocks_int<stxxl::BID<2097152>,stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(const unsigned int nblocks=1, stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > out={storage=0x0036dee8 offset=39845888 }) Line 1198 + 0x1a bytes C++
tdb.exe!stxxl::block_manager::new_blocks<stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidbegin={storage=0x0036dee8 offset=39845888 }, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidend={storage=0xfdfdfdfd offset=-76843842582893653 }) Line 1251 C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::reserve(unsigned __int64 n=1) Line 595 + 0xb9 bytes C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::resize(unsigned __int64 n=1) Line 636 C++
tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::push_back(const test::tickdb::QuoteInfo & obj={...}) Line 657 C++
The error message from vs2005 is: _DEBUG_ERROR("map/set iterators incompatible") in the xtree file from vc8 line 293
Thank you
Hi,
The problem is that the Stxxl Block Management (BM) layer, stxxl::vectors are calling, is not thread-safe. The thread safety for the BM layer has been introduced in one of the developement branches and will come in the next release. However, we have just merged these changes in the trunk version which you can check out using subversion:
svn https://stxxl.svn.sourceforge.net/svnroot/stxxl/trunk stxxl_trunk
I hope the trunk version will work for you.
Roman
Thank you for this information. You realize a REALLY good work.
I've a question around the container and the block management system. In fact, i've to sort gigas of value. It works great. However I would like to know if it possible to use a "part" of the available memory. In fact, it seems that the buffer files configured inside the .stxxl are always used. I would like to know if I can tell the BM to use a part of the memory available on my computer ? Or what to do you think on linux to create the buffer file on a tmpfs space ?
Thank you.
Hi,
if you do not know what to do with the unused main memory ;-) it is better to invest it into the caches of stxxl data structures and/or memory buffers used by sorting/scanning. If your problem fits into this memory, almost no disk I/O will be needed by stxxl.
Roman
Fixed in the 1.2.0 release.
Andreas