tclwebtest-general

[tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

Hi.

At the moment I'm experimenting with such a script:

[gradha@ws5:1] [~/tclwebtest]$ cat selftest/cus.txt 
#do_request "http://192.168.1.50:8080/tclwebtest/show-group?id=5&gid=1"
do_request "$SERVER/tclwebtest/show-group?id=5&gid=1"
assert text "tclwebtest"

I saw the TESTHOME variable, but looks like that is set by tclwebtest
internally, I thought initially it could be an environment variable.
I can overcome this limitation wrapping the script around another
temporary script which sets the variables, but maybe there is an
easier way?

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)

Re: [tclwebtest] Passing variable parameters to test scripts

[Tilmann S. <ti...@ti...>]

* Grzegorz Adam Hankiewicz <gr...@ef...> [20030121 14:54]:
> On Mon, Jan 20, 2003 at 06:44:00PM +0000, Tilmann Singer wrote:
> > > I saw the TESTHOME variable, but looks like that is set by tclwebtest
> > > internally, I thought initially it could be an environment variable.
> > > I can overcome this limitation wrapping the script around another
> > > temporary script which sets the variables, but maybe there is an
> > > easier way?
> > 
> > How do these two things relate? And what do you want to achieve?
> 
> Basically, we are going to have scripts which run locally or remotely
> depending on a switch, and the easiest way to do it would be to
> set a variable with the base url of the test. I don't want this
> hardcoded into the scripts themselves, because then we would have
> two versions of the same script with one different line.
> 
> > Look at the openacs tests for an example. I think it would break
> > when you have your tests in several subdirectories, but I'm
> > not sure.
> 
> Well, all scripts are going to be stored in a database, so I'm
> going to run them iteratively, putting them out in some /tmp file
> and running tclwebtest over them. Since I'll be doing this, I could
> as well modify them to include the fist line "set SERVER_URL xxx".

What I meant would be interesting from the openacs scripts was the
fact that they are sourcing a common settings file, where you would
set your SERVER_URL variable, so that it's only stored in one place.

Still you could as well use an environment variable, why not? They
should be available in the env array within the test.

By the way, did you try to source tclwebtest into aolserver (not very
efficient, I know) like in the example I sent you and run the tests
you retrieve from the db with the run_test command? It might not be
necessary to put them in a /tmp/ file and run them from there.

Looking at run_test it occurs to me that Peter Marklund contributed a
nice -config_file switch, to solve the settings problem more
elegantly. Maybe this can be used in your project? It just passes a
filename to run_test where the file is read, but maybe it should be
changed so that the file is read before the invocation of run_test and
the contents should be passed instead, then run_test could be run more
flexibly. What do you think?



> PD: Weren't you meant to answer through the ML?

Yes, sorry. I'll send this reply over the list now.


til

Re: [tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

On Wed, Jan 22, 2003 at 06:23:44AM +0000, Tilmann Singer wrote:
> Looking at run_test it occurs to me that Peter Marklund contributed
> a nice -config_file switch, to solve the settings problem more
> elegantly. Maybe this can be used in your project?

Yes, this is what I would need, got it from CVS.

> By the way, did you try to source tclwebtest into aolserver (not
> very efficient, I know) like in the example I sent you and run
> the tests you retrieve from the db with the run_test command? It
> might not be necessary to put them in a /tmp/ file and run them
> from there.

Well, the example you sent still created temporary files. I'm
attaching a patch which adds a new switch to run_test modifying
the behaviour of log, debug and the other switches. I've run this
successfully over a serie of files.

The difference is that now all input/output is treated as a string,
more easy to handle from a superior script.

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)

Re: [tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

On Wed, Jan 22, 2003 at 06:32:57PM +0100, Grzegorz Adam Hankiewicz wrote:
> [...] I'm attaching a patch [...]

Running generate_docs I found about a wrong variable, so here's
a fixed patch, it also improves a little bit the documentation of
the procedures.

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)

Re: [tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

On Wed, Jan 22, 2003 at 07:13:35PM +0100, Grzegorz Adam Hankiewicz wrote:
> Running generate_docs I found about a wrong variable, so here's
> a fixed patch, it also improves a little bit the documentation of
> the procedures.

Ok, another thing which was missing in this patch was that when
something failed while using memory strings, tclwebtest wouldn't log
the lines which caused the error, since there's no file code_lines
can open.

Applying this patch over fixes that (you will note it also includes
the change of ::tclwebtest::log to public), and now five lines of
code are shown delimited with special markers. Example:

   ----- START: in_memory string at [27/Jan/2003:17:52:22] -----
   
   --- do_request for http://www.efaber.net/
   http status: >>200<<
   
   Assertion "[regexp -nocase $search_expr $::tclwebtest::body]" failed. 
   in script body line 13
           ... SOURCE CODE ...
   assert text "proporciona desarrollos de servicios web personalizados"
   
   #asdasd
   # verificación de código html
   assert full "<p><li><b>Opttimizar"
           ^^^^^ ERROR ^^^^^
   -----  FAILED: in_memory string (took 1s)               ----- 

The code to extract those lines is very rudimentary. I miss python's
in memory file object abstraction! At the moment it just takes the
code contained in the memory string and starts butchering it until
only five lines of text are left. I could write it much better if
[string first] allowed a starting character as parameter, because
I wouldn't need to chop so many times the string, but looks like
it doesn't.

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)

Re: [tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

> Applying this patch over fixes that [...]

Substitute that patch with the following one (the name should help
you find which one), I found a few corner cases where that algorithm
wouldn't find out the correct lines from the memory string. This
newer version also doesn't use chop so much, so maybe it's more
faster if TCL is faster retrieving indexes than chopping strings.

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)

Re: [tclwebtest] Passing variable parameters to test scripts

[Tilmann S. <ti...@ti...>]

Is your cambios-tclwebtest-2003-01-27b patch the latest one regarding
the issue of avoiding the usage of temporary files?

The approach I took with the temporary files looks like a huge hack to
me and it would be greatly appreciated if you'd remove that. I
remember that it seemed necessary in order to be able to retrieve the
_comment_ lines immediately before the command that failed, which
looks like an unnecessary requirement to me now. We should consider
introducing a better way to include msg-if-next-command-fails in the
tests.

One way would be to include a parameter for each of the commands that
can cause a test failure. I think that is the standard testing
framework approach.

Another idea would be to add a command like msg that takes one string
as argument, which will be saved internally and only output if a
following command produces a failure. To reset the fail_msg one would
simply call it with no arguments or an empty string. A test could thus
look like this:

msg { Logging in }

do_request $some_url
link follow "login"
field fill $password
form submit

msg


Or it could optionally wrap the code in a block so that the message is
only valid for that block:

msg { Logging in } {
    do_request $some_url
    link follow "login"
    field fill $password
    form submit
}


What is your opinion on the best syntax? We might also introduce a
common test declaration command along with it which could come at the
top of each file and which would make use of the api doc generator,
like this:

twt_test login {

    This test logs in to the website using the standard password.

    @author til

}

do_request ...


There are propably better names for these and maybe more that could be
learnt from other testing frameworks, but why I actually wrote it is
to say that the current requirement to be able to read the tcl
comments from the test source code is not really important, so don't
jump through big loops to preserve it when modifying tclwebtest to use
strings instead of temporary files, and please commit your patch with
these modifications, if not already done.



cheers, Til

Re: [tclwebtest] Passing variable parameters to test scripts

[Grzegorz A. H. <gr...@ef...>]

On Sat, Feb 15, 2003 at 11:11:13AM +0000, Tilmann Singer wrote:
> [...] but why I actually wrote it is to say that the current
> requirement to be able to read the tcl comments from the test
> source code is not really important, so don't jump through big
> loops to preserve it when modifying tclwebtest to use strings
> instead of temporary files, and please commit your patch with
> these modifications, if not already done.

Well, my patches with memory strings have problems when something
goes wrong (infinite loop/recursion). And most importantly, the
output log is buffered, while the current version logs as the test
unit proceeds, which can be important for slow/busy. So I guess
it's better to keep both ways.

-- 
 Grzegorz Adam Hankiewicz, gr...@ef.... Tel: +34-94-472 35 89.
 eFaber SL, Maria Diaz de Haro, 68, 2         http://www.efaber.net/
 48920 Portugalete, Bizkaia (SPAIN)