fileo non-zero write pattern...possible?

Leon Kull
2012-07-01
2013-05-28
  • Leon Kull
    Leon Kull
    2012-07-01

    Is it possible to configure sysbench to write non-zero/random data when it prepares files ?

    TIA

     
  • Alexey Kopytov
    Alexey Kopytov
    2012-07-01

    Yes, when validation (-validate) is enabled on prepare, sysbench writes random data to test files.

     
  • Leon Kull
    Leon Kull
    2012-07-01

    I meant "fileio non-zero write pattern…possible?"

     
  • Leon Kull
    Leon Kull
    2012-07-01

    Thanks, kaamos!

     
  • Leon Kull
    Leon Kull
    2012-07-03

    I am using -validate on prepare:

    # sysbench --test=fileio --max-time=60 --max-requests=1000000  --validate --file-num=256 --file-io-mode=sync --file-fsync-freq=0  --file-total-size=1G --num-threads=16 --file-test-mode=rndrd prepare
    sysbench 0.4.12:  multi-threaded system evaluation benchmark
    256 files, 4096Kb each, 1024Mb total
    Creating files for the test...
    

    but getting FATAL errors when the test runs:

    # sysbench --test=fileio --max-time=60 --max-requests=1000000  --validate --file-num=256 --file-io-mode=sync --file-fsync-freq=0  --file-total-size=1G --num-threads=16 --file-test-mode=rndrd run
    sysbench 0.4.12:  multi-threaded system evaluation benchmark
    Running the test with following options:
    Number of threads: 16
    Additional request validation enabled.
    Extra file open flags: 0
    256 files, 4Mb each
    1Gb total file size
    Block size 16Kb
    Number of random requests for random IO: 1000000
    Read/Write ratio for combined random IO test: 1.50
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Using checksums validation.
    Doing random read test
    Threads started!
    FATAL: Checksum mismatch in block: 
    (last message repeated 2 times)
    FATAL:     Calculated value: 0x717bdc67    Stored value: 0x7528e3a8
    FATAL: Validation failed on file 21, block offset 0x140000, exiting...
    FATAL: Checksum mismatch in block: 
    FATAL:     Calculated value: 0xdb02b4dc    Stored value: 0x8477f623
    FATAL: Validation failed on file 186, block offset 0x2b4000, exiting...
    FATAL: Checksum mismatch in block: 
    (last message repeated 1 times)
    FATAL:     Calculated value: 0xbd70bf5f    Stored value: 0x26e606eb
    (last message repeated 1 times)
    FATAL:     Calculated value: 0x90ce0151    Stored value: 0x84f3355a
    FATAL: Checksum mismatch in block: 
    FATAL:     Calculated value: 0xc94abd2c    Stored value: 0xad11221b
    FATAL: Validation failed on file 47, block offset 0x104000, exiting...
    FATAL:     Calculated value: 0xe526dae3    Stored value: 0x7528e3a8
    FATAL: Validation failed on file 90, block offset 0x1c0000, exiting...
    FATAL: Offset mismatch in block:
    FATAL:    Actual offset: 98304    Stored offset: 2867200
    FATAL:     Calculated value: 0x3880740e    Stored value: 0xa78588bb
    FATAL: Checksum mismatch in block: 
    FATAL:     Calculated value: 0x324615f1    Stored value: 0x7528e3a8
    FATAL: Validation failed on file 26, block offset 0x18000, exiting...
    FATAL: Offset mismatch in block:
    FATAL:    Actual offset: 180224    Stored offset: 2834432
    FATAL: Validation failed on file 13, block offset 0x2c8000, exiting...
    FATAL: Validation failed on file 46, block offset 0x308000, exiting...
    FATAL:     Calculated value: 0x432aaaa4    Stored value: 0xc66b7367
    (last message repeated 1 times)
    FATAL: Validation failed on file 0, block offset 0x204000, exiting...
    FATAL: Validation failed on file 119, block offset 0x18c000, exiting...
    FATAL:     Calculated value: 0x6c10c3    Stored value: 0x99e72bf7
    FATAL: Validation failed on file 249, block offset 0x1d0000, exiting...
    FATAL: Offset mismatch in block:
    FATAL: Validation failed on file 221, block offset 0x2c000, exiting...
    FATAL: Checksum mismatch in block: 
    FATAL:     Calculated value: 0x2b3563ff    Stored value: 0x4da9c197
    FATAL: Validation failed on file 169, block offset 0x27c000, exiting...
    FATAL: Offset mismatch in block:
    FATAL:    Actual offset: 0    Stored offset: 1966080
    FATAL: Validation failed on file 0, block offset 0x0, exiting...
    FATAL:    Actual offset: 3014656    Stored offset: 3768320
    FATAL: Validation failed on file 28, block offset 0x2e0000, exiting...
    FATAL: Validation failed on file 232, block offset 0x2ac000, exiting...
    Done.
    Operations performed:  999985 Read, 0 Write, 0 Other = 999985 Total
    Read 15.259Gb  Written 0b  Total transferred 15.259Gb  (667.1Mb/sec)
    42694.46 Requests/sec executed
    Test execution summary:
        total time:                          23.4219s
        total number of events:              1000000
        total time taken by event execution: 3.0461
        per-request statistics:
             min:                                  0.00ms
             avg:                                  0.00ms
             max:                                  0.28ms
             approx.  95 percentile:               0.00ms
    Threads fairness:
        events (avg/stddev):           62500.0000/242057.59
        execution time (avg/stddev):   0.1904/0.74
    
     
  • Alexey Kopytov
    Alexey Kopytov
    2012-07-03

    -validate can only be used with single-threaded tests. sysbench does nothing to protect file blocks against concurrent access. That's an intentional behavior.

     
  • Leon Kull
    Leon Kull
    2012-07-03

    Thanks.

    It means -validate confllicts with  -num-threads > 1 option.
    Request for improvement: to display a warning in these cases.

     
  • Leon Kull
    Leon Kull
    2012-07-03

    Another question - is it correct that the files prepared for
    -validate -file-test-mode=rndwr
    cannot be tested in mode
    -validate -file-test-mode=rndrd
    ?

    In other words - each test mode needs a separate file set ?

    TIA

     
  • Alexey Kopytov
    Alexey Kopytov
    2012-07-03

    It means -validate confllicts with -num-threads > 1 option. Request for improvement: to display a warning in these cases.

    Right, I have reported https://bugs.launchpad.net/sysbench/+bug/1020648

     
  • Alexey Kopytov
    Alexey Kopytov
    2012-07-03

    Another question - is it correct that the files prepared for -validate -file-test-mode=rndwr cannot be tested in mode -validate -file-test-mode=rndrd ?

    Hm, I don't see a reason off the top of my head. How does it fail?

     
  • Leon Kull
    Leon Kull
    2012-07-05

    What does  "FATAL: Too large position discovered in request!" mean ?

    # sysbench -test=fileio -file-num=10240 -file-io-mode=sync -file-fsync-freq=0  -num-threads=1 -file-total-size=1T -max-requests=0 -file-test-mode=rndrd run
    WARNING: Both max-requests and max-time are 0, running endless test
    sysbench 0.4.12:  multi-threaded system evaluation benchmark

    Running the test with following options:
    Number of threads: 1

    Extra file open flags: 0
    10240 files, 102.4Mb each
    1024Gb total file size
    Block size 16Kb
    Number of random requests for random IO: 0
    Read/Write ratio for combined random IO test: 1.50
    Calling fsync() at the end of test, Enabled.
    Using synchronous I/O mode
    Doing random read test
    Threads started!
    FATAL: Too large position discovered in request!
    Done.

    Operations performed:  159 Read, 0 Write, 0 Other = 159 Total
    Read 2.4844Mb  Written 0b  Total transferred 2.4844Mb  (138.56Kb/sec)
        8.66 Requests/sec executed

    Test execution summary:
        total time:                          18.3607s
        total number of events:              159
        total time taken by event execution: 18.3352
        per-request statistics:
             min:                                  3.06ms
             avg:                                115.32ms
             max:                                536.28ms
             approx.  95 percentile:             280.21ms

    Threads fairness:
        events (avg/stddev):           159.0000/0.00
        execution time (avg/stddev):   18.3352/0.00

     
  • Leon Kull
    Leon Kull
    2012-07-05

    WIth -debug=on :

    DEBUG: Executing request, operation: 1, file_id: 0, pos: 107364352, size: 16384
    FATAL: Too large position discovered in request!

     
  • Alexey Kopytov
    Alexey Kopytov
    2012-07-05

    What does "FATAL: Too large position discovered in request!" mean ?

    That probably means a bug. I remember fixing something like this. Can you try sysbench from https://code.launchpad.net/~sysbench-developers/sysbench/0.5 and see if it has the same problem?

     

  • Anonymous
    2012-11-06

    @napobo3, "Too large position discovered in request" is some kind of bug related to rounding filesize. In my case I had "file-total-size=30G" and "file-num=100" when I hit this problem. Workaround is to make sure that total size is a multiplier of file count, eg "-file-total-size=$((100*300))M -file-num=100"

    This thread is a bit old, but I reproduced it today on the only version I found on EPEL (sysbench-0.4.12-5.el6.x86_64). Hope the workaround helps anybody googling for the error.