simpletest-support — Help, advice, bugs and workarounds.

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

To get it to work on OSX, I think I had to install the pear package
DB_Sqlite_Tools - not 100% sure though, I do remember having to
wrangle around a bit to figure out the right package name. I really
should have written it down at that point :)

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Douglas Hubler <dhubler@...> writes:
> On ubuntu i installed this package
>  php-sqlite3

my mistake, 
  php5-sqlite

is the one that i believe i'm using

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Jakob Ketterl <jketterl@...> writes:
> On Saturday 08 November 2008 05:55:44 Douglas Hubler wrote:
> > I added
> >  ./package/generate_coverage.sh
> > which can used as is and as an example for others to follow
> >
> > I'd like to get this reviewed and merged into trunk.
> 
> i'd like to try this out, but onfurtunately i don't know what 
> package "DB/sqlite.php" is taken from. can anyone help me out?

On ubuntu i installed this package
 php-sqlite3

But i suspect this is the correct PECL package.

Package    Stable/(Latest) Local
SQLite     1.0.3 (stable)        SQLite database bindings

If you can confirm this, I'll add it to the documentation. Let me know your OS

Re: [Simpletest-support] Code coverage

[Jakob K. <jke...@ch...>]

On Saturday 08 November 2008 05:55:44 Douglas Hubler wrote:
> I added
>  ./package/generate_coverage.sh
> which can used as is and as an example for others to follow
>
> I'd like to get this reviewed and merged into trunk.

i'd like to try this out, but onfurtunately i don't know what 
package "DB/sqlite.php" is taken from. can anyone help me out?

TIA

Re: [Simpletest-support] Mock functions?

[Douglas H. <dh...@gm...>]

Douglas Hubler <dhubler@...> writes:
> I find mock functions useful 

i read thru this thread

http://article.gmane.org/gmane.comp.php.simpletest.general/2741

and i totally agree with the Noel and Chris but one thing I like about TDD is
that may code is cleaner after, but when i isolate _every_ single function, my
code becomes so fragmented it can becomes ugly again.

Here's an example of the test case me using php-mock-functions
 http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/set4d/tests/DrupalUpdateTest.php?revision=1.2&view=markup 

and here's the code i am testing 
 http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/set4d/DrupalUpdate.php?revision=1.1&view=markup

[Simpletest-support] Mock functions?

[Douglas H. <dh...@gm...>]

I find mock functions useful because many times I'm writing code that interfaces
with functions and not other objects.  I found I could mock functions with the
library
 http://code.google.com/p/php-mock-function/
without requiring runkit.  Instead I run my tests each in their own process and
control the include files and it works great.  

The php-mock-function library is very alpha and hasn't taken off.

Anyway, I just want to start a conversation on the subject. What are others 
using?

Reference
http://www.workhabit.com/labs/mock-function-testing-drupal

Re: [Simpletest-support] Code coverage

[Marcus B. <ma...@la...>]

Hi...

Douglas Hubler wrote:
> I'd like to get this reviewed and merged into trunk. 

I'll try to do this today or tomorrow.

> 
> I'd be willing to create some documentation, if someone can point me in the
> right direction.

It's currently written in a homebrew XML format :). It's mostly HTML
with a few extra tags that gets crunched by some XSLT. You should
probably add a page just before the last one (the independent browser).

Have a look and I'll explain in detail later.

yours, Marcus
-- 
Marcus Baker
ma...@la...

Re: [Simpletest-support] Code coverage

[Sebastian S. <ssc...@ch...>]

Since our team needs the same stuff, keep us informed. Maybe I can convince
someone that writing docs is a good thing. 

S. 


-----Ursprüngliche Nachricht-----
Von: Douglas Hubler [mailto:dh...@gm...] 
Gesendet: Samstag, 8. November 2008 05:56
An: sim...@li...
Betreff: Re: [Simpletest-support] Code coverage



Marcus Baker <marcus@...> writes:
> Douglas Hubler wrote: 
> > /simpletest/simpletest/branches/code_coverage/
updated

> >  - package as extension?
done

> >  - make dependency on smarty optional
> 
> Better to remove it altogether.
smarty in gone altogether. 


I added 
 ./package/generate_coverage.sh
which can used as is and as an example for others to follow

I'd like to get this reviewed and merged into trunk. 

I'd be willing to create some documentation, if someone can point me in the
right direction.


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Simpletest-support mailing list
Sim...@li...
https://lists.sourceforge.net/lists/listinfo/simpletest-support

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Marcus Baker <marcus@...> writes:
> Douglas Hubler wrote: 
> > /simpletest/simpletest/branches/code_coverage/
updated

> >  - package as extension?
done

> >  - make dependency on smarty optional
> 
> Better to remove it altogether.
smarty in gone altogether. 


I added 
 ./package/generate_coverage.sh
which can used as is and as an example for others to follow

I'd like to get this reviewed and merged into trunk. 

I'd be willing to create some documentation, if someone can point me in the
right direction.

Re: [Simpletest-support] Global variables between tests/setup

[Marcus B. <ma...@wo...>]

Hi...

Taras Diakiw wrote:
> I was wondering if SimpleTest changes global variables between unit tests. This is the only explanation of the $root_path variable being unitialised that I can think of, as my code does not explicitly unset it.

Could you send us a self contained failing test case showing this
behaviour? It should be easy to diagnose if I can replicate it.

yours, Marcus

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

status update

Marcus Baker <marcus@...> writes: 
> >  - package as extension?
> Extension at first if it;s possible.

I've finished this work, no hooks in core required.

> >  - make dependency on smarty optional
> Better to remove it altogether.

this is coming along, not quite ready yet but almost there.

Re: [Simpletest-support] Global variables between tests/setup

[Marcus B. <ma...@wo...>]

Hi...

Taras Diakiw wrote:
> I was wondering if SimpleTest changes global variables between unit tests. This is the only explanation of the $root_path variable being unitialised that I can think of, as my code does not explicitly unset it.

No. It tries to do the simplest thing, which is to allow test conflicts,
but at least make them obvious. It's then up to the developer to resolve
them, or to explicitely reset globals between tests.

Here, of course, you want to do the opposite of normal behaviour. You
actually want to maintain a global between tests (why?). This should be
fine. Something odd is happening.

> 
> If, in fact, SimpleTest *does* unset all of the global variables between unit tests, shouldn't it also reset the state of whether files have been included or not.

Unless each test is executed in it's own process, this is impossible.
Right now that kills performance, although it will in future be added as
an explicit option for particular test cases.

> 
> After the following line in the 'load' function in test_case.php gets run:
> 
> $existing_globals = get_defined_vars();
> include_once($test_file);
> 
> $existing_globals does not contain the $root_path variable..

This is to give the illusion that file scope variables will still be
visible in the test methods )I think - can't remember). It imports these
variables, but does not reset any other globals.

Could you see if your variable is visible in get_declared_vars().

> Taras Diakiw

yours, Marcus

[Simpletest-support] Global variables between tests/setup

[Taras D. <Tar...@pr...>]

Hi everyone,

I have a configuration file, which is required_once, and sets a global variable called 'root_path'.

However, when simpletest runs the third test in a test suite, root_path is unitialised. I am certain that root_path does, in fact, get set to something prior to this point, as I have set up a break point at the line which sets root_path (which gets hit).

config.php
-------------
$root_path = 'blah';

and then SimpleTest will load FooUnitTest as part of the 'All' Test Suite

FooUnitTest
---------------
require_once('config.php'); // doesn't get included... config.php is included as part of the Test Prolog
require_once($root_path.'Bar.php'); // fails because the $root_path global variable is unitialised at this stage

I was wondering if SimpleTest changes global variables between unit tests. This is the only explanation of the $root_path variable being unitialised that I can think of, as my code does not explicitly unset it.

If, in fact, SimpleTest *does* unset all of the global variables between unit tests, shouldn't it also reset the state of whether files have been included or not.

After the following line in the 'load' function in test_case.php gets run:

$existing_globals = get_defined_vars();
include_once($test_file);

$existing_globals does not contain the $root_path variable..

?

Taras Diakiw
Software Developer
Practical Law Company 
T: +44 (0) 20 7202 1200 ext 5629 
F: +44 (0) 20 7202 1211 
W: http://www.practicallaw.com


This e-mail from Practical Law Company (http://www.practicallaw.com) is subject to our terms of use (http://www.practicallaw.com/9-103-0884). Information about our companies: Practical Law Company Limited. Registered in England and Wales. Registered Number: 02889191. Registered Office: 19 Hatfields, London SE1 8DJ. Telephone: +44(0)20 7202 1200.  Practical Law Company, Inc. Incorporated in Delaware, USA.  Corporate Office: 747 Third Avenue, 36th Floor, New York, NY 10017. Telephone: +1(646) 562-3400.

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

Hi Douglas,

> Can describe this process or point me details. I am not familiar with
> simpletest extensions at all.

There's no 'formal' extension mechanism, basically it's just loosely
coupled classes that interact with the extensible parts of the core.

If you look in /extensions/ you'll see the existing code that is
bundled - the format is:

/ext_name.php      <-- hook class to include
/ext_name/            <-- supporting package files
/ext_name/tests/   <-- unit tests for the individual package

some of the older and smaller extensions are just a single PHP file though.

Regards,
Mark

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Marcus Baker <marcus@...> writes:
> Extension at first if it;s possible. If we need hooks or extra code
> we'll put them into the core, and even into HEAD.

Can describe this process or point me details. I am not familiar with
simpletest extensions at all.


> 
> >  - make dependency on smarty optional
> 
> Better to remove it altogether. Have something simple (HTML tables and a
> bit of styling) and add the pretty version as an extension. It must,
> must work out of the box for "Simple"Test. Not much of a UI style guide
> I admit, but I'm afraid I'm quite rigid about it :(.

I plan to implement something useful in just php and it can be the default
implementation. I will investigate the reporters in simpletest.


re:branch
I branched from trunk, I could not find 1.1 branch, which for me is good because
I don't have to merge twice. 

Re: [Simpletest-support] Code coverage

[Marcus B. <ma...@wo...>]

Hi...

Douglas Hubler wrote:
> Marcus Baker <marcus@...> writes:
> i put it in 
> /simpletest/simpletest/branches/code_coverage/

Cool.

>  - package as extension?

Extension at first if it;s possible. If we need hooks or extra code
we'll put them into the core, and even into HEAD.

>  - make dependency on smarty optional

Better to remove it altogether. Have something simple (HTML tables and a
bit of styling) and add the pretty version as an extension. It must,
must work out of the box for "Simple"Test. Not much of a UI style guide
I admit, but I'm afraid I'm quite rigid about it :(.

yours, Marcus

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Marcus Baker <marcus@...> writes:
> I suggest that I we fix autorun sharpish, you try it out, then we regroup.

i decided to separate function simpletest_autorun in autorun.php into 2 parts
and I was able to avoid exit call.  So autorun functionality is intact as it
was. Let me know if this is ok.


> 
> How near are you to getting this stuff into a branch?

i put it in 
/simpletest/simpletest/branches/code_coverage/


2 open issues
 - package as extension?
 - make dependency on smarty optional

Re: [Simpletest-support] Code coverage

[Marcus B. <ma...@wo...>]

Hi...

Douglas Hubler wrote:
> yes, this is the big item i could use help on
> 
> - autorun uses shutdown hook to run tests
> - coverage also uses a shutdown hook to know when to save coverage 
> - coverage has to happen after tests run, but tests call exit so coverage 
>   never happens.

OK, so some kind of post test hook might be needed. I'd rather look at
alternatives first.

If you had a reporter decorator, and autorun understood the concept,
then you could just tell auto run to inject it. Then your post test
stuff could all be in the paintFooter() of the decorator.

I suggest that I we fix autorun sharpish, you try it out, then we regroup.

How near are you to getting this stuff into a branch?

yours, Marcus

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Mark Rickerby <maetl@...> writes:
> I have a question about the change to the return value of the cli exit
> routine...
> what was the issue that led to you commenting this out?


yes, this is the big item i could use help on

- autorun uses shutdown hook to run tests
- coverage also uses a shutdown hook to know when to save coverage 
- coverage has to happen after tests run, but tests call exit so coverage 
  never happens.

Re: [Simpletest-support] Code coverage

[Douglas H. <dh...@gm...>]

Mark Rickerby <maetl@...> writes:
> However, I had a few teething problems with thousands of E_NOTICE
> errors being thrown from the coverage run:

y will fix this, I didn't have the logging level right when developing this.


> I do think we need to
> look at whether this is best packaged as an extension (especially the
> reporting and static file generation part of it)... I don't have a

let me know if you decide that, and how to go about this
 
> I would like to see more abstraction at the level of generating the
> coverage reports 

makes sense


> 
> The dependency on Sqlite is understandable, not sure sure about the
> Smarty - again this would be more appropriate as an extension, not in
> core.
also makes sense

> there are a lot of simpletest users who make heavy use of the
> web tester, which peeks at responses over HTTP, so doesn't operate on
> code in the local process scope and can't be xdebugged.

you can use this in http env., depending on your php setup.  If you can include
autocoverage.php somewhere that will get called, it will collect coverage.  
This is how i use it.  Permission and current working directory also have to 
be conducive

> 
> But good work, thanks for that... I'm enjoying playing with it...

thanks

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

Here's a spiral chart illustrating the levels of test coverage within
simpletest itself, starting with 100% coverage (invoker.php
apparently), and working its way down to the reporter.php (29%
coverage, due to the paint methods that don't get called). This may
not be the best way to visualize this, but I'm just getting started
with it :)

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

Sorry, I was referring to the code coverage patch... One of the only
changes it makes to core code is to comment out that return exit
value...

Re: [Simpletest-support] Code coverage

[Marcus B. <ma...@wo...>]

Hi...

Mark Rickerby wrote:
> what was the issue that led to you commenting this out?

Which file? I don't remember commenting it out. Strange. Maybe the
functionality was already covered by autorun, but then I'd have deleted
the old line. I don't know.

> 
> regards,
> Mark

yours, Marcus

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

Have been letting this loose, and so far it seems to be generating all
the results perfectly - I'm assuming that it's as accurate as xdebug
can be, and I like the way that the source code is presented with
green/red illustrating which parts of the code were executed and which
were not.

However, I had a few teething problems with thousands of E_NOTICE
errors being thrown from the coverage run:

Notice: Undefined offset: in /simpletest/trunk/coverage.php on line 290

which is this line:

$lineCoverage = self::lineCoverageCodeToStyleClass($coverage[$i]);

I'm reviewing the code now, so expect to have some more technical
feedback soon... but in general, it looks good - I do think we need to
look at whether this is best packaged as an extension (especially the
reporting and static file generation part of it)... I don't have a
problem with coverage hooks going in to the main simpletest core
(again, subject to code review), but I think this needs to be more
closely integrated with the existing paradigm we have for reporters
(which is subject to change as we move forward - more on that in a
later post).

I would like to see more abstraction at the level of generating the
coverage reports - by default I would prefer to see the data being fed
directly back to a TextReporter decorator or something similar, so
that it could be displayed directly in the stdout rather than dumped
to HTML files. I think it would be good to give users the option of
where they want to export the results, and keep things as minimal and
close to the existing way of doing things as possible.

The dependency on Sqlite is understandable, not sure sure about the
Smarty - again this would be more appropriate as an extension, not in
core. We want people to be able to download simpletest and run it on
any PHP server, without needing to stitch together dependencies, but
having said that, anyone who is using pear will have absolutely no
problems.

I've never been a fan of code coverage personally, but now I'm
starting to see the possibilities. I think one of the potential
benefits is simply having greater high level visibility of the spread
of a code-base. It help you to understand the bigger picture when you
can browse the source code through the perspective of what the tests
are doing.

Another thing to bear in mind, is that the mileage of this feature may
vary - there are a lot of simpletest users who make heavy use of the
web tester, which peeks at responses over HTTP, so doesn't operate on
code in the local process scope and can't be xdebugged.

But good work, thanks for that... I'm enjoying playing with it...

Regards,
Mark

Re: [Simpletest-support] Code coverage

[Mark R. <ma...@co...>]

I have a question about the change to the return value of the cli exit
routine...

     if (SimpleReporter::inCli()) {
-        exit($result ? 0 : 1);
+        //exit($result ? 0 : 1);
     }
 }

what was the issue that led to you commenting this out?

regards,
Mark