[Bigdata-developers] RW Scaling tests.

[Bryan T. <br...@sy...>]

Martyn,

I've gotten up to 1.7B loaded so far on the CI Perf machine into the RW Store.  Overall, I think that we are ready to bring the RWStore into the trunk.  I'm just looking for a spare moment to do that and then we can close out that branch.   

However, I would like to do a bit more scale testing for the RW store.  Given that we have so much better performance with the RWStore on SAS (due to the ability of SAS to merge writes based on the disk geometry), we should run the scaling tests on machines with SAS disks.  Luckily, the 8 node cluster has SAS disks (300GB 10K 6G 2.5 SAS DP HDD).  The cluster nodes listed below have four 300Gb drives setup with hardware RAID0.  LVM is also layered over this, giving effectively one ~1.2Tb partition.  This suggests that we could run scaling tests on the RW Store on nodes 206-209 as follows:

xx.xx.xx.206      uniprot   (2.5B)
xx.xx.xx.207      LUBM 8000 (1B)
xx.xx.xx.208      BTC 2009  (1.14B)
xx.xx.xx.209      BTC 2010  (3.2B)

Those data sets are currently on the CI Perf machine in /data.  We should also generate a larger LUBM data set, e.g., LUBM 100000. The scaling tests themselves are easy to set up (just checkout bigdata from SVN and use the templates in the bigdata-perf module) and can be run disconnected using nohup.  Please also nohup vmstat (vmstat -n 60) and iostat (iostat -x -N -d -k 60 -t) so we can get a good sense of how the IO system performs over time.  You have to do that in separate directories since each nohup process will write on 'nohup.out' in the local directory.

I'd like to get these runs done over the next week or so since we will need to have the cluster clear when we get into testing the quads query branch.

Thanks,
Bryan