Re: [Tcllib-devel] Writing tests - how?

[Andreas K. <and...@ac...>]

> Andreas Kupries skrev:
> >> Well, what about a small helper package? One could make the above
> >>
> >> package require tcllib::testutilities
> >> tcllib::testutilities::init
> >>
> >> or something like that. The separate [tcllib::testutilities::init] call
> >> would be needed to retrieve the [info script] data, although now that I
> >> think about it, one could (if really devious) embed such a call into
> >> the [package ifneeded] script as well. An explicit call does however
> >> have the advantage that one can, if necessary, customise the
> >> initialisation (no, I don't have a clear use-case for that right now;
> >> maybe too tired).

> Having slept on the matter, I now have "configuration for nonstandard
> test environment" as use-case. The initialisation command put in the
> .test file would do a default init (e.g. assuming tcllib CVS
> organisation of files, using current file as reference), but extra
> commands given before sourcing the test file could specify overrides
> for the defaults.


> For example, one might do:
>
> package require tcllib::testutilities
> tcllib::testutilities::declaredir modules/math \
>    /Library/Tcl/tcllib-1.6/math
> source foo.test

Ok, this is different from what I thought it was per your first email.
There it looked to me as if the

	package require tcllib::testutilities

statement was inside of foo.test, replacing the big source/file/info script
construction.
Now it has moved outside, to whatever is invoking the test script.

Is my understanding of both older and current/here setup right?


> to specify an override that "whenever the test file wants to [use]
> something in modules/math, it should use the corresponding file in
> /Library/Tcl/tcllib-1.6/math". There's no harm in doing [package
> require tcllib::testutilities] twice, unlike the corresponding explicit
> [source].

Uh. I am not sure if that sort of mapping is a useful feature. More thinking
required.

> >> What is gained by [package require tcllib::testutilities] is primarily
> >> that one avoids hardcoding paths (relative paths, yes, but still
> >> hardcoded paths) into the test files.
> >
> > The price you are paying for this is that you now cannot run
> the testsuite
> > anymore if Tcllib is not installed because the package won't be
> found. Or
> > using the wrong helper package if some other version of Tcllib
> is installed.
>
> That's a very pessimistic view of the matter. The only case I can think
> of in which one would like to run the tcllib testsuite without having
> /any/ version of tcllib installed is "make test"-style activities:
> testing new software before actually installing it.

Your view seems to be for a person/role getting the sources of Tcllib,
building, testing, then installing.
Not sure what name to give that role, it is not a 'developer' I would say.
Maybe 'installer'.

To me 'make test' is an integral part of 'development' activities. I.e.
testing the software while working on it, making sure that changes do not
break things. Like I did, for example, yesterday when speeding up the
struct::stack Tcl code a bit. Would be bad if it became fast, but also
wrong. Proactive testing before committing changes, instead of waiting for
installers to tell me what I broke.

> In this situation,
> the environment is fairly well controlled, so providing an extra
> package is not a problem.

Hm ...



> Developers working on tcllib should be expected to at least have some
> version installed,

It is likely. Do I want to depend on that? It makes for too much coupling to
the outside in my mind. I prefer that development is in a sandbox, something
self-contained, or, if not possible, minimal dependencies.

> so unless you foresee several radical changes in the
> testutilities API, problems because the installed version is not that
> of CVS HEAD should be rare, and easily recognised when they eventually
> occur. What could be inconvenient is the transition period when such a
> package is being introduced into tcllib, but OTOH it would probably
> take a couple of years before any large portion of tests were rewritten
> to take advantage of it. ;-)


> > Running the testsuite of Tcllib without Tcllib installed is IMHO an
> > important use case during development.
>
> What is *lost* with the
>

[stmt1]
> source [file join \
> 	[file dirname [file dirname [file join [pwd] [info script]]]] \
> 	devtools testutilities.tcl]
>
> arrangement is the ability to run tests /without having all of the
> tcllib repository at hand/.

I believe you meant 'source directory' instead of 'repository'.

I think I am beginning to understand our differences of view, which are
fundamental.

I am assuming and expecting that development on a part of Tcllib, i.e.
something like 'modules/report' always happens in a sandbox containing the
whole of the Tcllib sources. Together with my drive to avoid external
dependencies I arrive at [stmt1], which reflects both.

You seem to have the case where you copy 'modules/report' somewhere else,
being standalone, and still wish to be able to test it. At which point you
need the installed Tcllib, to find dependencies on other packages in Tcllib,
like struct::matrix.

Ok, that sounds quite artificial ...

A less-artificial case seems to be that: You develop a new package FOO, and
you start outside of Tcllib, do the testing outside, and then later wish to
drop it into Tcllib and have the tests working without change.

In that case I would have actually started developing FOO from within a
Tcllib sandbox, having the assumed envirnment available from the beginning.


My main conclusion here is that I see the non-ability to run tests without
having all of the tcllib source directory at hand as no loss at all.

In a way, we both need and assume the existence Tcllib, always, we simply
choose different locations where to find it.
You assume to have it installed, and I go and assume that I have all the
sources.

I like my approach because I believe it gives me better control about the
test environment and less possibility of interference, despite having to put
up with the longish command to load all that environment.


> Wherever I have a foo.test, it is assumed
> that there is a ../devtools/testutilities.tcl to source.

Yes.

> There will be
> such a file if you've checked out all of tcllib or tcllib/modules from
> CVS, but at least I find it much more prudent to check out only "my
> own" module (especially if the project hierarchy in which it was
> originally developed doesn't match tcllib).

Reading this now, after writing all of the above I feel stupid. It seems I
just paraphrased everything written now, and could have spared me the
effort. OTOH, seeing my inference of usecase matching your description now
makes me fairly sure that I have some understanding of your thoughts.

We seem to have fundamentally different views on the process and system.


> > To re-enable execution you have to extend the auto_path,
>
> Which is quite feasible to do from the environment in the "make test"
> scenario, and unnecessary otherwise.
>
> > adding again the path information you do not want to the .test files.
>
> Of course it shouldn't be the responsibility of the .test files to
> provide this information! The main point of having the [package]
> mechanism is to allow code to call upon other code without knowing
> exactly where it is found!

True. However with testing, where I want to rigorously control the
environment I consider exactly that a disadvantage.

> > Even so it stays
> > brittle because anybody can intercept this by adding a higher
> version of the
> > helper package anywhere on the auto_path.

> If that is brittle, then it is so only because the development of the
> helper package is being mismanaged!

> A higher version of the package
> will only be picked if it has the same major version number as the one
> you require. The major version number must be incremented if the
> package is changed so that it is not backwards compatible. So how could
> newer versions be a problem?

I agree, they shouldn't be. However, can you bet on that? It is easy to
mismanage this, have a bug


Thinking further, seems that my thoughts are churning (in a good way).

For one of the, right now half-baked, ideas which this discussion just
tossed up into my conscious mind is that our Tcllib 'testutilities' are in
some sense actually special, they are not a regular Tcllib package we
install. But they are, also, something, which is very common, similar to
'tcltest'. In a way they are an extension of 'tcltest' ... Can it be made
(or is) generic enough tthat we should/could install it ?

If it is generic enough we can move it ouf Tcllib. Then my feelings about it
as package required dependency seem to change and allow that, as it would
then be something managed externally anyway (i.e. like tcltest). The
question is, how much of the testutilities code is tied to the sepcialities
of Tcllib ... Oh, another flash of an idea ... With it outside, have it
overide the 'package unknown' command, search the code, like Tcllib for
packages first, and ignore packages of the same name outside ...

... I think I better stop here ... let it settle and think/sleep about it
more ...

Note: The testutils are a conglomeration of several groups of commands, each
group could be their own package ... Make a set of higher level test
framework packages ... Ok, I really have to settle and meditate ...

--
	Andreas Kupries <and...@Ac...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122