Re: [OpenSTA-users] Severe load ramp-up limitation encountered

[Daniel S. <da...@Op...>]

Dan Downing wrote:
> Hi opensta gurus (you know who you are):

Bernie: do you ever feel left out that your name isn't Dan?  ;)

Sorry to hear about your issues Dan, I've heavily clipped your
mail and asked a few, hopefully relevant, questions below:

> After several years of using opensta successfully on dozens of
> customer load tests, in the last 2 weeks I have attempted two
> projects with aggressive load ramp-ups where opensta flattened
> my hefty load server and reported *unrealistically high*
> response times even at low load -- and thus failed miserably (to
> great embarrassment with the customer

Has this 'hefty load server' been used successfully on similar gigs
in the past?
If so, has ANYTHING changed on it since then?
Can you go back and recreate the successful load tests on it now?

> we eventually accomplished this using LR

Out of interest: how much were the LR licenses that allowed you to 
complete this task?

> The Script:? Could not be simpler -- a single page with nothing
> more than the customer's logo.?

So you're saying the content can't really be at fault and we can
create any simple HTML Web page with a graphic in it to simulate the
same problem ...

> PRIMARY GET URI
> "http://syn1.sellpoint.net/QA/smloadtest.html HTTP/1.0" ON 1?? HEADER
> DEFAULT_HEADERS ,WITH {"{the usual stuff here"}

Except: look at the content of this page ...
It's not well formed HTML at all just a bunch of javascript.

Are you doing any LOAD RESPONSE_INFO's with this content?
Do you have any WAITs left in your script or have they all been 
edited out?  SYNCHRONIZE used?

> The Test:? Two load servers (dual 1Ghz P4s, 1 GB memory).?  On
> each, test had 20 Task Groups with this one script, in v-user "burst"
> groups of 50, 100, 150,...1000, doing a single iteration.? Each group
> to be launch 3 minutes apart (allowing the web server to settle from
> the previous burst).?? Ramp rate of 300 users per minute (batch
> settings:? 1/5/1 (interval btw batches/vusers per batch/batch ramp time).

As your script is so short I would have considered doing looping in
the script rather than having the Test Executer doing the iterating -
it theoretically should save you some HP.  Your Task setup sounds a
little overcomplicated but that in itself should not cause you a
serious resource problem unless you are hitting a bug - besides from
another email you suggest that a much simpler task setup produce
similar awful results.

> The result:? (1)
> Test appeared to start up and run smoothly, until Group 8 (400-user burst)
> launched--at which point Error Log reports "Failed processing for TOF
> record for script line 59" (the PRIMARY GET); and what's worse, the Timer
> shows times at 50 users of about 2 seconds, rapidly climbing to 3.5 seconds at
> 100, to 10 seconds at 200, 20 seconds at 200, 30 seconds at 350...and so forth
> all the way to 60 seconds.

The "Failed processing TOF" error is usually accompanied by another,
more meaningful error - I'd be very interested if there is one and
if, what it is ...

> These timer results were proven "false" when
> (a) customer (and us) could hit the page manually and get <5 seconds
> response at something under 1000 v-users, and (b) subsequent test with LR
> showed these times to range from .7 seconds at just under 150 users, 5 seconds
> at 1000 users, and up to 9 seconds at 2000 users.
>
> Furthermore, perfmon showed that out dual-cpu server
> shoots up to 100% utilization when Task Group8 (400 users) starts--and stays
> there; and memory pages/sec. shoots up from an average of 8 to around 50, which
> spikes to 400 (I have a screen shot that shows this if want to see it).

As soon as the CPU and/or memory starts thrashing on a load generating
machine just throw the results away - they are worthless!
:shrug: Sorry - this will always be the case, and I'm sure it is with
LR and others as well.

> I believe understanding of and/or resolution of this apparent 'load-driving
> bottleneck' to be of pivotal interest to all serious opensta users. 

Absolutely!  Let's get to the bottom of what happened and then you can
pay me to fix it ;)
Cheers
/dan
-- 
Daniel Sutcliffe <Da...@Op...>
  OpenSTA part-time caretaker - http://OpenSTA.org/