RE: [Grinder-use] Statistics idea
Distributed load testing framework - Java, Jython, or Clojure scripts.
Brought to you by:
philipa
From: <seb...@ph...> - 2005-03-01 09:31:21
|
Hi Philipp, This is how I would do it. Definition: Let's say ni is the population and xi the response time for a test i. The Variance V is by definition equal to [sum(ni.xi^2)/sum(ni)] / mean^2 In our case ni is always equal to one because we get one response time per test. Thus : V=[sum(xi^2) / sum(ni)] - mean^2 I do some renaming for ease of writting. V=[Sa / Sb)] / M^2 If we know that at the time t1 we had V1, M1, Sa1 and Sb1 At the time t2 we can easily recalculate M2 which is equal to (M1 * sum(previous number of samples) + sum(new values)) / (sum(previous number of samples) + number of new samples) V2 can be recalculated from V1 as well with the new values we have V1 + M1^2 = [Sa1 / Sb1] We know Sb1 so we can calculate Sa1 If we add the sum(new samples values ^2) to Sa1 and then divide by (Sb1 + number of new samples) and finally substract the new mean M2 we then have the new Variance V2. The standard deviation is then easy to calculate. In my model the worker process would have to transmit the sum of the response times between time t1 and t2 and also the number of new samples. I am not sure this method could be implemented in your grinder distribution model. What is important to my opinion is that we get figures that are understandable by common people. I don't know how it is for other Grinder users but I communicate the test results to the project owners and explain them the figures. The common useful figures that everybody understand and that are easy to calculate at the end of a load-test are : - the mean (which we have already) - the min/max (which can be easily calculated) - the median (= the response time value where 50% of the tests where above and 50% under this value ; this would be more difficult to calculate in real time because it requires a sort all the results) - the standard deviation (which is possible to calculate in the console as demonstrated above). All together these figures give a good overview of the reaction of the target system. Any comment from your side ? Best regards, Sebastien "Philip Aston" <pa...@be...> 25/02/2005 17:15 To: Sebastien Lemeunier/PH/Novartis@PH, <gri...@li...> cc: Subject: RE: [Grinder-use] Statistics idea Category: Comments on the proposal in: http://sourceforge.net/mailarchive/message.php?msg_id=10692856 http://sourceforge.net/mailarchive/message.php?msg_id=10738329 ? Re: your second point: you are right that a modern browser is threaded and The Grinder is not. This is s deficiency in the model implemented by The Grinder. - Phil -----Original Message----- From: gri...@li... on behalf of seb...@ph... Sent: Thu 24/02/2005 15:20 To: gri...@li... Cc: Subject: [Grinder-use] Statistics idea Dear all, I think it would be useful to add three other columns in the standard Grinder statistics : - the standard deviation time - the minimum response time - the maximum response time. Actually it is difficult to say if a response time is valid for all virtual users. For instance it could have taken 10 minutes for one users to execute a test and 2 seconds for all other users. Displaying the standard deviation time would help to figure out this kind of problem and avoid mis-interpreting the results. Also, I have question regarding the interpretation of the results. Let's imagine I have a one page download which is decomposed in 10 individual tests. Can we say that that it take the sum of all average response times to display the page in the browser or does it depend on the number of threads the browser is using internally ? Thanks, Sebastien |