Re: [OpenSTA-devel] opensta results analysis and ideas

[Bernie V. <Ber...@iP...>]

Corey wrote:
>does anyone have other ideas? or is anyone working with OpenSTA in a unique
>way that users or developers might benefit from? suggestions welcome.. post 
>to
>the list.

Hi Corey,

My 2 cents worth... I've often thought about developing a reporting package 
for OpenSTA for resale. As it is more likely I'll get blood from a stone
then money from the average OpenSTA user, I'll share my ideas here.

I see room for improvement for both graphs and tabular reports. A useful 
analysis technique is to examine trends in response time and resource 
utilization as a function of load. For many tests, especially those where 
users are ramped up in batches, load increases with time. Two intuitive 
means to quantify load are the number of VUs and the count of transactions 
per second (I know, I know, its the arrival rate that is most important, but 
counting completed transactions is the next best thing). All the reports 
that show load on the X axis currently show ALL loads. The system under test 
is in a steady state condition for only a small fraction of the loads 
observed. For example, lets say I introduce users in batches of 90 (up to 
360) with a ramp up period of 10 minutes and time between batches of 20 
minutes. In this example, I am interested in reporting for 90, 180, 270, and 
360 users.

Tabular reports: I find it helpful to view a tabular report that, for 
example, would show min, max, avg, std dev, count, and percentiles 
(specified at report time) grouped by user (90, 180, 270, and 360 in this 
example) by timer. You can see an example (minus the percentiles) on page 20 
of http://www.iperformax.com/downloads/SamplePerformaxPerformanceReport.pdf

Graphs::Perf mon data: An example is worth a thousand words. See page 18 of 
the aforementioned .pdf file for an example where one can overlay multiple 
perf mon data points vs users. Its also helpful to see this same graph where 
#VUs is replaced by the count of (specified) timers that completed during 
the same time period.

Graphs::Response time data: An example is worth a thousand words. See page 
19 of the aforementioned .pdf file for an example where one can overlay 
multiple perf mon data points vs users. Its also helpful to see this same 
graph where #VUs is replaced by the count of (specified) timers that 
completed during the same time period.

At report run time, one would have to specify some combination of 'units for 
the x-axis' (ie users, wall clock time range, transaction rate),  'user 
populations' (i.e. 90, 180, 270, and 360 ), transactions to include in 
transaction count (a list of timer names), and time intervals (i.e. 10-30 
minutes, 40-60 minutes, 70-90 minutes, and 100-120 minutes).

-Bernie Velivis

www.iPerformax.com