Trouble setting up three disks

Roger Dahl
2008-07-31
2013-04-25
  • Roger Dahl
    Roger Dahl
    2008-07-31

    Hello,

    I'm having trouble with a crash in Stxxl. I ran out of room using a configuration with these two lines:

    disk=d:\stxxl.tmp,700000,wincall
    disk=f:\stxxl.tmp,460000,wincall

    So I attempted to add a little bit of space on a third disk by adding a third line:

    disk=c:\stxxl.tmp,40000,wincall

    However, when I add this line, I get the following error message:

    [STXXL-MSG] STXXL v1.2.0 + Boost 103500
    [STXXL-MSG] 3 disks are allocated, total space: 1,200,000 MB
    [STXXL-VERBOSE1] typed::block operator new[]: Meta info size: 0
    [STXXL-VERBOSE1] stxxl::aligned_alloc<4,096>(), size = 67,108,864, meta info size = 0
    [STXXL-VERBOSE1] stxxl::aligned_alloc<4,096>() address 0x1,6d1,000 lost 4,032 bytes
    [STXXL-VERBOSE1] stxxl::aligned_alloc<4,096>(), allocated at 1,6d0,040 returning 1,6d1,000
    ...
    [STXXL-VERBOSE2] Asking for a block with size: 2,097,152
    ... (260,000 copies of the line above)
    [STXXL-ERRMSG] External memory block allocation error: 183,251,238,912 bytes requested, 41,943,040,000 bytes free. Trying to extend the external memory space...

    And then Stxxl crashes in file::get_disk_number().

    The C: disk has room for the file I tried to set up but the crash happens with any number for the C: disk.

    When the crash occurs, the call stack is:

    >    mksieve.exe!stxxl::file::get_disk_number()  Line 181 + 0x5 bytes    C++
         mksieve.exe!stxxl::block_manager::delete_block<2097152>(const stxxl::BID<2097152> & bid={...})  Line 1364 + 0xd bytes    C++
         mksieve.exe!stxxl::block_manager::delete_blocks<std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > >(const std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > & bidbegin={storage=0x0000000000000000 offset=0 }, const std::_Vector_iterator<stxxl::BID<2097152>,std::allocator<stxxl::BID<2097152> > > & bidend={storage=0x00000000fdfdfdfd offset=0 })  Line 1378 + 0x1a bytes    C++
         mksieve.exe!stxxl::vector<collatz_record,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>::~vector<collatz_record,4,stxxl::lru_pager<8>,2097152,stxxl::RC,unsigned __int64>()  Line 1110 + 0x68 bytes    C++

    The problem appears to be that get_disk_number() is called on an uninitialized class.

    Thank you,

    Roger

     
    • Hi Roger,

      this looks strange. Unfortunately, I could not reproduce your problem on my machine. Could you please send me a shortened version of your program (here or using email) that produces the error? Does the error also occur if you use (1000 times) smaller inputs and smaller disk files?

      disk=d:\stxxl.tmp,700,wincall
      disk=f:\stxxl.tmp,460,wincall
      disk=c:\stxxl.tmp,40,wincall

      Best regards,
      Roman

       
    • Roger Dahl
      Roger Dahl
      2008-08-30

      Hi Roman,

      Sorry for the long delay in following up on this...

      I could not reproduce the error by reducing the problem to 1/1000 of the size. So this crashes:

      [STXXL-MSG] 3 disks are allocated, total space: 1,196,000 MB
      [STXXL-ERRMSG] External memory block allocation error: 183253336064 bytes requested, 37748736000 bytes free. Trying to extend the external memory space...

      But this does not crash:

      [STXXL-MSG] STXXL v1.2.0 + Boost 103500
      [STXXL-MSG] 3 disks are allocated, total space: 1,196 MB
      [STXXL-ERRMSG] External memory block allocation error: 180355072 bytes requested, 37748736 bytes free. Trying to extend the external memory space...

      All you need to do to reproduce the error is to set up the .stxxl file as described above, for a total of 1.2TB of space and then run something like the following:

      struct test_record {
        char buf[16];
      };

      stxxl::vector<test_record> test;
      test.reserve(1LL << 35); // 512GB

      Thanks,

      Roger