curl-loader-devel

[ANN] - curl-loader virtual scripting in SVN

[Robert I. <cor...@gm...>]

Gentlemen,

Michael Moser has checked-in a new system for HTTP/FTP response body analyses,
which have WebLoad and JMeter tools.

Normally, body analyses in testing tools is done using JavaScript or
any other script,
which occupies a total thread, is blocking and is bad for performance.

Since curl-loader is about performance (written in C, using C-written
stacks of libcurl and openssl)
 Michael is developing a new scripting language, where the script will be:

- compiled to an object code,
- possessing its own VM, not occupying a whole thread;
- enjoying several entrance and exit points.

Y can see the major development of the "Virtual Script -VScript", yet
without integration to the
curl-loader configurationand practical usage and without support for
64-bit linux here:

http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/script/

or even better by importing the svn repository and looking at curl-loader/script
directories: http://sourceforge.net/svn/?group_id=179599

Your comments would be appreciated, but, please, keep Michael's  e-mail
in the list.

curl-loader has been configured to receive donations via sourceforge, which
will be used to accelerate development and integration of VScript:
http://sourceforge.net/project/project_donations.php?group_id=179599


Have a nice CURL-ing.
:)

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
An open-source web testing and traffic generation.

Re: [ANN] - curl-loader virtual scripting in SVN

[Aleksandar L. <al-...@no...>]

Hi,

On Fre 30.11.2007 10:41, Robert Iakobashvili wrote:
>Gentlemen,
>
>Michael Moser has checked-in a new system for HTTP/FTP response body
>analyses, which have WebLoad and JMeter tools.
>
>Normally, body analyses in testing tools is done using JavaScript or
>any other script, which occupies a total thread, is blocking and is bad
>for performance.
>
>Since curl-loader is about performance (written in C, using C-written
>stacks of libcurl and openssl) Michael is developing a new scripting
>language, where the script will be:

yust for my curiosity, have you looked into neko, lua or some other
'smalll and embadabble' language and found some nogos?!

Cheers

Aleks

Re: [ANN] - curl-loader virtual scripting in SVN

[Robert I. <cor...@gm...>]

Hi Aleks,

On Wed, Apr 30, 2008 at 12:56 AM, Aleksandar Lazic
<al-...@no...> wrote:
> Hi,
>
> On Fre 30.11.2007 10:41, Robert Iakobashvili wrote:
> >Gentlemen,
> >
> >Michael Moser has checked-in a new system for HTTP/FTP response body
> >analyses, which have WebLoad and JMeter tools.
> >
> >Normally, body analyses in testing tools is done using JavaScript or
> >any other script, which occupies a total thread, is blocking and is bad
> >for performance.
> >
> >Since curl-loader is about performance (written in C, using C-written
> >stacks of libcurl and openssl) Michael is developing a new scripting
> >language, where the script will be:
>
> yust for my curiosity, have you looked into neko, lua or some other
> 'smalll and embadabble' language and found some nogos?!

Sure we did.

Here is a new "pending revolution", which does Michael
Moser. I would let him to comment at his time-slot.

Michael, please ...

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Ghotit - Assistive technology that understands you
......................................................................

Re: [ANN] - curl-loader virtual scripting in SVN

[Michael M. <mos...@gm...>]

Hi,

I have looked into LUA, now lets try to reconstruct the reasoning (any
corrections/suggestions are welcome).

Our problem here is that we need several thousand simulated clients in one
process;

This poses the following question
1) do we have one virtual machine for each simulated  client, i.e. a
scripting language thread per client.
2) doe we have one virtual machine for all simulated clients.

We will come back to that question later.

Now lets look at the basic problem that we are facing with integrating any
language environment with server that is processing requests asynchronously
- xmethod calls (ie. calls to external methods written in C) usually these
calls are blocking.

[ the script would need at least one xmethod call that would perform the
next http request that would 1) initiate the http request, 2) suspend
execution of the vm for the current thread 3) be able to resume current
thread when needed]

now this problem can be solved by longjumps (or any other coprocedure
mechanism) at the expense of  some memory to hold the current stack when we
suspend current xmethod. Now

basically this kills the option  of using one vm for all simulated clients.

Ok, lets assume we are using longjump, this rules our sharing one vm for all
simulated clients,
How? lua is singlethreaded and with longjump we just suspended the current
thread that is running the vm, [also any pending lock will block the whole
VM if we branch out of it.]

Ok, now we have one VM per simulated client. Great, with LUA they have a
garbage collected heap, and holding such a heap per each simulated client is
a very large overhead.
(even if we could share one vm for all simulated clients then this garbage
collection schema is a real killer for predictable and reasonable
performance)


Our requirements is therefore
- vm supports xmethod calls that are truly asynchronous,  without  need for
longjumps.
- VM with very low footprint
- memory management for objects created by the script that is reference
counted;
- also when a VM exits it has to take its memory with it.


2008/4/30 Aleksandar Lazic <al-...@no...>:

> Hi,
>
> On Fre 30.11.2007 10:41, Robert Iakobashvili wrote:
>
> > Gentlemen,
> >
> > Michael Moser has checked-in a new system for HTTP/FTP response body
> > analyses, which have WebLoad and JMeter tools.
> >
> > Normally, body analyses in testing tools is done using JavaScript or
> > any other script, which occupies a total thread, is blocking and is bad
> > for performance.
> >
> > Since curl-loader is about performance (written in C, using C-written
> > stacks of libcurl and openssl) Michael is developing a new scripting
> > language, where the script will be:
> >
>
> yust for my curiosity, have you looked into neko, lua or some other
> 'smalll and embadabble' language and found some nogos?!
>
> Cheers
>
> Aleks
>

Re: [ANN] - curl-loader virtual scripting in SVN

[Michael M. <mos...@gm...>]

additional clarification regarding garbage collection: Lua uses mark and
sweep garbage collection, so when the heap fills up then everybody has to
wait until garbage is reclaimed; so nobody knows when and for how long these
wait cycles will happen.

2008/4/30 Michael Moser <mos...@gm...>:

> Hi,
>
> I have looked into LUA, now lets try to reconstruct the reasoning (any
> corrections/suggestions are welcome).
>
> Our problem here is that we need several thousand simulated clients in one
> process;
>
> This poses the following question
> 1) do we have one virtual machine for each simulated  client, i.e. a
> scripting language thread per client.
> 2) doe we have one virtual machine for all simulated clients.
>
> We will come back to that question later.
>
> Now lets look at the basic problem that we are facing with integrating any
> language environment with server that is processing requests asynchronously
> - xmethod calls (ie. calls to external methods written in C) usually these
> calls are blocking.
>
> [ the script would need at least one xmethod call that would perform the
> next http request that would 1) initiate the http request, 2) suspend
> execution of the vm for the current thread 3) be able to resume current
> thread when needed]
>
> now this problem can be solved by longjumps (or any other coprocedure
> mechanism) at the expense of  some memory to hold the current stack when we
> suspend current xmethod. Now
>
> basically this kills the option  of using one vm for all simulated
> clients.
>
> Ok, lets assume we are using longjump, this rules our sharing one vm for
> all simulated clients,
> How? lua is singlethreaded and with longjump we just suspended the current
> thread that is running the vm, [also any pending lock will block the whole
> VM if we branch out of it.]
>
> Ok, now we have one VM per simulated client. Great, with LUA they have a
> garbage collected heap, and holding such a heap per each simulated client is
> a very large overhead.
> (even if we could share one vm for all simulated clients then this garbage
> collection schema is a real killer for predictable and reasonable
> performance)
>
>
> Our requirements is therefore
> - vm supports xmethod calls that are truly asynchronous,  without  need
> for longjumps.
> - VM with very low footprint
> - memory management for objects created by the script that is reference
> counted;
> - also when a VM exits it has to take its memory with it.
>
>
> 2008/4/30 Aleksandar Lazic <al-...@no...>:
>
> Hi,
> >
> > On Fre 30.11.2007 10:41, Robert Iakobashvili wrote:
> >
> > > Gentlemen,
> > >
> > > Michael Moser has checked-in a new system for HTTP/FTP response body
> > > analyses, which have WebLoad and JMeter tools.
> > >
> > > Normally, body analyses in testing tools is done using JavaScript or
> > > any other script, which occupies a total thread, is blocking and is
> > > bad
> > > for performance.
> > >
> > > Since curl-loader is about performance (written in C, using C-written
> > > stacks of libcurl and openssl) Michael is developing a new scripting
> > > language, where the script will be:
> > >
> >
> > yust for my curiosity, have you looked into neko, lua or some other
> > 'smalll and embadabble' language and found some nogos?!
> >
> > Cheers
> >
> > Aleks
> >
>
>

Re: [ANN] - curl-loader virtual scripting in SVN

[Aleksandar L. <al-...@no...>]

Hi Michael,

On Mit 30.04.2008 13:59, Michael Moser wrote:

>additional clarification regarding garbage collection: Lua uses mark
>and sweep garbage collection, so when the heap fills up then everybody
>has to wait until garbage is reclaimed; so nobody knows when and for
>how long these wait cycles will happen.

thank you for your detailed description ;-)

I have take a look about a lot of the small languages but I haven't this
requirements.

Yust if you have time, what do you think about:

http://www.iolanguage.com/
http://nekovm.org/

Both looks to me very nice but I'am not sure if the fulfill the
requirements.

Neko use http://www.hpl.hp.com/personal/Hans_Boehm/gc/ for GC.

IOLanguage have developed there own 

http://github.com/stevedekorte/io/tree/master/libs/garbagecollector

afaIs

Cheers

Aleks

Addendum Re: [ANN] - curl-loader virtual scripting in SVN

[Aleksandar L. <al-...@no...>]

On Don 01.05.2008 08:32, Aleksandar Lazic wrote:

[snipp]

>IOLanguage have developed there own 
>
>http://github.com/stevedekorte/io/tree/master/libs/garbagecollector
>
>afaIs

http://www.iolanguage.com/scm/git/checkout/Io/docs/IoReference.html#Collector

---
Description

A singleton containing methods related to Io's garbage collector. Io
currently uses a incremental, non-moving, generational collector based
on the tri-color (black/gray/white) algorithm with a write-barrier.

Every N number of object allocs, the collector will walk some of the
objects marked as gray, marking their connected white objects as gray
and turning themselves black. Every M allocs, it will pause for a sweep
where it makes sure all grays are marked black and io_frees all whites.

If the sweepsPerGeneration is set to zero, it will immediately mark all
blacks as white again and mark the root objects as gray. Otherwise, it
will wait until the sweepsPerGeneration count is reached to do this. By
adjusting the allocsPerSweep and sweepsPerGeneration appropriately, the
collector can be tuned efficiently for various usage cases. Generally,
the more objects in your heap, the larger you'll want this number.
---

Don't look useable for curl-loader,imho.

Cheers

Aleks

Re: [ANN] - curl-loader virtual scripting in SVN

[Michael M. <mos...@gm...>]

Thanks a lot
I will look at these languages + vm's as well;
Michael

P.S.
somehow i still think that there might be a market for an vm + language
suited for asynchronous server design;  if time permits I will also continue
the project of doing that scripting language 'the right way (tm)' ;-)


2008/5/1 Aleksandar Lazic <al-...@no...>:

> Hi Michael,
>
> On Mit 30.04.2008 13:59, Michael Moser wrote:
>
>  additional clarification regarding garbage collection: Lua uses mark
> > and sweep garbage collection, so when the heap fills up then everybody
> > has to wait until garbage is reclaimed; so nobody knows when and for
> > how long these wait cycles will happen.
> >
>
> thank you for your detailed description ;-)
>
> I have take a look about a lot of the small languages but I haven't this
> requirements.
>
> Yust if you have time, what do you think about:
>
> http://www.iolanguage.com/
> http://nekovm.org/
>
> Both looks to me very nice but I'am not sure if the fulfill the
> requirements.
>
> Neko use http://www.hpl.hp.com/personal/Hans_Boehm/gc/ for GC.
>
> IOLanguage have developed there own
> http://github.com/stevedekorte/io/tree/master/libs/garbagecollector
>
> afaIs
>
> Cheers
>
> Aleks
>

Re: [ANN] - curl-loader virtual scripting in SVN

[Aleksandar L. <al-...@no...>]

On Don 01.05.2008 14:23, Michael Moser wrote:
>Thanks a lot

Your welcome ;-)

>P.S.
>somehow i still think that there might be a market for an vm + language
>suited for asynchronous server design; if time permits I will also
>continue the project of doing that scripting language 'the right way
>(tm)' ;-)

Full ack ;-)

Cheers

Aleks

Simulation scenario

[Niko W. S. S. <nik...@ya...>]

How to make in a configuration file a scenario like
this:
The 90% request is for the file with size less than
3KB and 90% of total load is from the file whose size
bigger than 100KB

I want to make the heavy tailed workload in a
configuration file (in curl-loader of course)


Best, 
Niko Wilfritz Sianipar
Teknik Informatika IT Telkom Bandung



      ________________________________________________________ 
Bergabunglah dengan orang-orang yang berwawasan, di di bidang Anda! Kunjungi Yahoo! Answers saat ini juga di http://id.answers.yahoo.com/

Re: Simulation scenario

[Robert I. <cor...@gm...>]

Hi Niko

On Thu, May 1, 2008 at 3:46 PM, Niko Wilfritz Sianipar Sianipar
<nik...@ya...> wrote:
> How to make in a configuration file a scenario like
>  this:
>  The 90% request is for the file with size less than
>  3KB and 90% of total load is from the file whose size
>  bigger than 100KB

Y mean 10% and 90%
>
>  I want to make the heavy tailed workload in a
>  configuration file (in curl-loader of course)


Y can find in FAQs http://curl-loader.sourceforge.net/doc/faq.html
the following tags:

FETCH_PROBABILITY - allows to fetch a url not as a must, but with a
certain run-time probability. The allowed values are in the range from
1 to 100 percents.

FETCH_PROBABILITY_ONCE when set to 1 configures each client to make
the decision regarding whether to fetch a URL marked by a
FETCH_PROBABILITY or not, to be done only once, namely, at the first
cycle.

And an example of doing that is in your curl-loader distribution, or
here with the same filename:

http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/conf-examples/fetch-probability.conf?view=markup

or
http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/conf-examples/fetch-probability-once.conf?view=markup

Take care,

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Ghotit - Assistive technology that understands you
......................................................................

Balasan: Re: Simulation scenario

[Niko W. S. S. <nik...@ya...>]

Clearly I want to make heavy tailed workload scenario.
The tag you mention is I think it's talking about
probability not the quantity of a load. Cause I
believe that heavy tailed workload has describe the
workload that web server nowadays faces.


--- Robert Iakobashvili <cor...@gm...> wrote:

> Hi Niko
> 
> On Thu, May 1, 2008 at 3:46 PM, Niko Wilfritz
> Sianipar Sianipar
> <nik...@ya...> wrote:
> > How to make in a configuration file a scenario
> like
> >  this:
> >  The 90% request is for the file with size less
> than
> >  3KB and 90% of total load is from the file whose
> size
> >  bigger than 100KB
> 
> Y mean 10% and 90%
> >
> >  I want to make the heavy tailed workload in a
> >  configuration file (in curl-loader of course)
> 
> 
> Y can find in FAQs
> http://curl-loader.sourceforge.net/doc/faq.html
> the following tags:
> 
> FETCH_PROBABILITY - allows to fetch a url not as a
> must, but with a
> certain run-time probability. The allowed values are
> in the range from
> 1 to 100 percents.
> 
> FETCH_PROBABILITY_ONCE when set to 1 configures each
> client to make
> the decision regarding whether to fetch a URL marked
> by a
> FETCH_PROBABILITY or not, to be done only once,
> namely, at the first
> cycle.
> 
> And an example of doing that is in your curl-loader
> distribution, or
> here with the same filename:
> 
>
http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/conf-examples/fetch-probability.conf?view=markup
> 
> or
>
http://curl-loader.svn.sourceforge.net/viewvc/curl-loader/trunk/curl-loader/conf-examples/fetch-probability-once.conf?view=markup
> 
> Take care,
> 
> -- 
> Truly,
> Robert Iakobashvili, Ph.D.
>
......................................................................
> www.ghotit.com
> Ghotit - Assistive technology that understands you
>
......................................................................
> 
>
-------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008
> JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still
> time to save $100. 
> Use priority code J8TL2D2. 
>
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> curl-loader-devel mailing list
> cur...@li...
>
https://lists.sourceforge.net/lists/listinfo/curl-loader-devel
> 


Best, 
Niko Wilfritz Sianipar
Teknik Informatika IT Telkom Bandung



      ________________________________________________________ 
Bergabunglah dengan orang-orang yang berwawasan, di di bidang Anda! Kunjungi Yahoo! Answers saat ini juga di http://id.answers.yahoo.com/

Re: Balasan: Re: Simulation scenario

[Robert I. <cor...@gm...>]

On Fri, May 2, 2008 at 12:01 PM, Niko Wilfritz Sianipar Sianipar
<nik...@ya...> wrote:
> Clearly I want to make heavy tailed workload scenario.
> The tag you mention is I think it's talking about
> probability not the quantity of a load.

What you mean is unclear? What is the quantification
of the quantity of load?

Y can easily place several URLs with the files of sizes, that
you wish and there are lot's of examples in conf-examples directory.

What it the scenario, and what is the question?

-- 
Truly,
Robert Iakobashvili, Ph.D.
......................................................................
www.ghotit.com
Ghotit - Assistive technology that understands you
......................................................................