stxxl::vector in multithreaded environment

2007-09-13
2013-04-25
  • Nobody/Anonymous

    Hi !

    I would like to know if there are recommendation to use stxxl::vector inside a multi-threaded environment. In fact, I several thread, each thread manage its own structure containing a stxxl::vector.

    I don't know if it can cause a problem, but in this configuration i get the following error:

    >    msvcp80d.dll!std::_Debug_message(const wchar_t * message=0x00551038, const wchar_t * file=0x005509d0, unsigned int line=293)  Line 24    C++
         tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator==(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651))  Line 293 + 0x17 bytes    C++
         tdb.exe!std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator::operator!=(const std::_Tree<std::_Tmap_traits<__int64,__int64,std::less<__int64>,std::allocator<std::pair<__int64 const ,__int64> >,0> >::const_iterator & _Right=(-3617008641903833651,-3617008641903833651))  Line 305 + 0xc bytes    C++
         tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BID<2097152> * begin=0x0036f740, stxxl::BID<2097152> * end=0x0036f750)  Line 586 + 0x34 bytes    C++
         tdb.exe!stxxl::DiskAllocator::new_blocks<2097152>(stxxl::BIDArray<2097152> & bids={...})  Line 544    C++
         tdb.exe!stxxl::block_manager::new_blocks_int<stxxl::BID<2097152>,stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(const unsigned int nblocks=1, stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > out={storage=0x0036dee8 offset=39845888 })  Line 1198 + 0x1a bytes    C++
         tdb.exe!stxxl::block_manager::new_blocks<stxxl::offset_allocator<stxxl::RC>,std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(stxxl::offset_allocator<stxxl::RC> functor={...}, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidbegin={storage=0x0036dee8 offset=39845888 }, std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > bidend={storage=0xfdfdfdfd offset=-76843842582893653 })  Line 1251    C++
         tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::reserve(unsigned __int64 n=1)  Line 595 + 0xb9 bytes    C++
         tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::resize(unsigned __int64 n=1)  Line 636    C++
         tdb.exe!stxxl::vector<test::tickdb::QuoteInfo,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::push_back(const test::tickdb::QuoteInfo & obj={...})  Line 657    C++

    The error message from vs2005 is: _DEBUG_ERROR("map/set iterators incompatible") in the xtree file from vc8 line 293

    Thank you

     
    • Nobody/Anonymous

      Hi,

      The problem is that the Stxxl Block Management (BM) layer, stxxl::vectors are calling, is not thread-safe. The thread safety for the BM layer has been introduced in one of the developement branches and will come in the next release. However, we have just merged these changes in the trunk version which you can check out using subversion:

      svn https://stxxl.svn.sourceforge.net/svnroot/stxxl/trunk stxxl_trunk

      I hope the trunk version will work for you.

      Roman

       
    • Nobody/Anonymous

      Thank you for this information. You realize a REALLY good work.

      I've a question around the container and the block management system. In fact, i've to sort gigas of value. It works great. However I would like to know if it possible to use a "part" of the available memory. In fact, it seems that the buffer files configured inside the .stxxl are always used. I would like to know if I can tell the BM to use a part of the memory available on my computer ? Or what to do you think on linux to create the buffer file on a tmpfs space ?

      Thank you.

       
      • Nobody/Anonymous

        Hi,

        if you do not know what to do with the unused main memory ;-) it is better to invest it into the caches of stxxl data structures and/or memory buffers used by sorting/scanning. If your problem fits into this memory, almost no disk I/O will be needed by stxxl.

        Roman

         
    • Andreas Beckmann

      Fixed in the 1.2.0 release.

      Andreas

       

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks