Omit header section in sorting a binary file

  • MikkoV

    Can I tell STXXL file or vector to omit a number of bytes, i.e., a header section, in the beginning of a file to be sorted?
    I tried to use stxxl::file::discard, but that doesn't seem to do anything in this case:

    stxxl::syscall_file f( filename, stxxl::file::DIRECT | stxxl::file::RDWR );
    // mark the header as unused space to be skipped in sorting
    f.discard( 0, number_of_skip_bytes ); // has no effect?
    const uint32_t block_size = sizeof(mystruct) * 4096;
    typedef stxxl::vector<mystruct, 1, stxxl::lru_pager<8>, block_size> vector_type;
    vector_type v(&f);
    stxxl::sort( v.begin(), v.end(), my_cmp(), memory_to_use );

    What am I missing?

  • STXXL vectors may only bind to files that contain data blocks only, no headers or anything else is supported.

    You could use stxxl::stream::sort(), write your own input loading routines, feed the data into the stream sorter and write the sorted output back to disk with you own method.

    stxxl::file.discard() has a completely different purpose (notifying the underlying storage system that some part of the file is not needed any more and the blocks could be deallocated if this is possible by the file system or T>RIM commands could be issued for an SSD)