Re: [Bigdata-developers] performance question

[Jeremy J C. <jj...@sy...>]

On Sep 10, 2013, at 1:08 PM, Bryan Thompson <br...@sy...> wrote:

> I would look at the performance of each query individually and see if any
> of them is an obvious outlier with a bad query plan.

the simplistic test environment prints a '.' after each query, and this come at a fairly steady rate … so I don't think it is query related. 

Thanks for the various suggestions, I am working through them, and a couple of my own or my colleagues ….
I will update here when I have a solution.


The "parallelizing the load" is the difference between the following two shell commands: nosetests is a python test harness, which runs all the tests in each file passed to it.


# run the client tests (11 queries) 6 times over, for 66 queries, one after the other
time nosetests python/syapse/apps/search/tests/syql_test.py python/syapse/apps/search/tests/syql_test.py python/syapse/apps/search/tests/syql_test.py python/syapse/apps/search/tests/syql_test.py python/syapse/apps/search/tests/syql_test.py python/syapse/apps/search/tests/syql_test.py


and


# run the client tests (11 queries) from 6 different parallel sub-shells, making 66 concurrent queries
for c in 1 2 3 4 5 6
do
   time nosetests python/syapse/apps/search/tests/syql_test.py &
done


Even with my OS limiting my h/w to one core, the parallel query has comparable performance compared with our previous solution which does not parallelize well.