Thread: Re: [Simpletest-support] Code coverage (Page 2)

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi...

Mark Rickerby wrote:
> Cool, I'll take a look at it and give you some feedback.

Thanks Mark.

> 
> Thanks!
> Mark

yours, Marcus

Hi...

Douglas Hubler wrote:
> Marcus Baker <marcus@...> writes:
> i put it in 
> /simpletest/simpletest/branches/code_coverage/

Cool.

>  - package as extension?

Extension at first if it;s possible. If we need hooks or extra code
we'll put them into the core, and even into HEAD.

>  - make dependency on smarty optional

Better to remove it altogether. Have something simple (HTML tables and a
bit of styling) and add the pretty version as an extension. It must,
must work out of the box for "Simple"Test. Not much of a UI style guide
I admit, but I'm afraid I'm quite rigid about it :(.

yours, Marcus

Marcus Baker <marcus@...> writes:
> Douglas Hubler wrote: 
> > /simpletest/simpletest/branches/code_coverage/
updated

> >  - package as extension?
done

> >  - make dependency on smarty optional
> 
> Better to remove it altogether.
smarty in gone altogether. 

I added 
 ./package/generate_coverage.sh
which can used as is and as an example for others to follow

I'd like to get this reviewed and merged into trunk. 

I'd be willing to create some documentation, if someone can point me in the
right direction.

Since our team needs the same stuff, keep us informed. Maybe I can convince
someone that writing docs is a good thing. 

S. 

-----Ursprüngliche Nachricht-----
Von: Douglas Hubler [mailto:dh...@gm...] 
Gesendet: Samstag, 8. November 2008 05:56
An: sim...@li...
Betreff: Re: [Simpletest-support] Code coverage

Marcus Baker <marcus@...> writes:
> Douglas Hubler wrote: 
> > /simpletest/simpletest/branches/code_coverage/
updated

> >  - package as extension?
done

> >  - make dependency on smarty optional
> 
> Better to remove it altogether.
smarty in gone altogether. 

I added 
 ./package/generate_coverage.sh
which can used as is and as an example for others to follow

I'd like to get this reviewed and merged into trunk. 

I'd be willing to create some documentation, if someone can point me in the
right direction.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Simpletest-support mailing list
Sim...@li...
https://lists.sourceforge.net/lists/listinfo/simpletest-support

Hi...

Douglas Hubler wrote:
> I'd like to get this reviewed and merged into trunk. 

I'll try to do this today or tomorrow.

> 
> I'd be willing to create some documentation, if someone can point me in the
> right direction.

It's currently written in a homebrew XML format :). It's mostly HTML
with a few extra tags that gets crunched by some XSLT. You should
probably add a page just before the last one (the independent browser).

Have a look and I'll explain in detail later.

yours, Marcus
-- 
Marcus Baker
ma...@la...

On Saturday 08 November 2008 05:55:44 Douglas Hubler wrote:
> I added
>  ./package/generate_coverage.sh
> which can used as is and as an example for others to follow
>
> I'd like to get this reviewed and merged into trunk.

i'd like to try this out, but onfurtunately i don't know what 
package "DB/sqlite.php" is taken from. can anyone help me out?

TIA

Jakob Ketterl <jketterl@...> writes:
> On Saturday 08 November 2008 05:55:44 Douglas Hubler wrote:
> > I added
> >  ./package/generate_coverage.sh
> > which can used as is and as an example for others to follow
> >
> > I'd like to get this reviewed and merged into trunk.
> 
> i'd like to try this out, but onfurtunately i don't know what 
> package "DB/sqlite.php" is taken from. can anyone help me out?

On ubuntu i installed this package
 php-sqlite3

But i suspect this is the correct PECL package.

Package    Stable/(Latest) Local
SQLite     1.0.3 (stable)        SQLite database bindings

If you can confirm this, I'll add it to the documentation. Let me know your OS

Douglas Hubler <dhubler@...> writes:
> On ubuntu i installed this package
>  php-sqlite3

my mistake, 
  php5-sqlite

is the one that i believe i'm using

To get it to work on OSX, I think I had to install the pear package
DB_Sqlite_Tools - not 100% sure though, I do remember having to
wrangle around a bit to figure out the right package name. I really
should have written it down at that point :)

On Tuesday 11 November 2008 02:32:59 Douglas Hubler wrote:
> Douglas Hubler <dhubler@...> writes:
> > On ubuntu i installed this package
> >  php-sqlite3
>
> my mistake,
>   php5-sqlite
>
> is the one that i believe i'm using

actually, you didn't make a mistake there. i had already installed php5-sqlite in a guess before, but that didn't work out. i've now additionally (yes you will need both packages) installed php-sqlite3 (how big a difference a few digits can cause ;-) ) and at least i got that dependency resolved. now i'll have to move to our dev server, where xdebug is available.

i also tried to install the DB_Sqlite_Tools from pear (from channel://pear.php.net/DB_Sqlite_Tools-0.1.6) but at least it didn't work out of the box. Maybe I'll have the time to get another look...

I'm running Ubuntu 8.04.01 Hardy.

Thank you :-)

I found two little bugs in the reporting templates (it's just missing echo 
statements). i'm including a patch :-)

Jakob Ketterl <jketterl@...> writes:
> I found two little bugs in the reporting templates (it's just missing echo 
> statements). i'm including a patch 
thanks, i committed these

Any progress on general review?

Hi...

Douglas Hubler wrote:
> Any progress on general review?

Sorry, been my birthday. I'm working on it as time allows.

yours, Marcus

Hi Douglas,

First of all thank you so much for this contribution !

Now for my remarks :

- there's not only a dependcy upon SQLlite but also on PEAR's DB/sqlite. 
Could we get rid of PEAR first ?

- there's a "test.php" file in the /coverage directory, I think it 
should be put into /coverage/test/

- the /test/sample-code.php should move up one level into something like 
/test/sample/code.php (that way we also remove the "-" in the filename, 
"_" are used all over the place in SimpleTest code base)

- while I'm at it, shouldn't we move /coverage/CoverageTemplates to 
/coverage/templates

I'll send over an other set of remarks once I have PEAR installed on my 
machine.

Yours,
	Perrick

Douglas Hubler wrote:
> Jakob Ketterl <jketterl@...> writes:
>> I found two little bugs in the reporting templates (it's just missing echo 
>> statements). i'm including a patch 
> thanks, i committed these
> 
> Any progress on general review?
> 
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Simpletest-support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpletest-support
> 
> 

Perrick Penet <perrick@...> writes:
> First of all thank you so much for this contribution !

thanks for the review

> - there's not only a dependcy upon SQLlite but also on PEAR's DB/sqlite. 
> Could we get rid of PEAR first ?

I installed sqlite from ubuntu package. Can you give advice, on what i can do to
remove PEAR, I'm not too familiar with this.

> - there's a "test.php" file in the /coverage directory, I think it 
> should be put into /coverage/test/

sure.  

I can reword this doc too, it doesn't make this clear
http://simpletest.sourceforge.net/en/writing_extensions.html

> 
> - the /test/sample-code.php should move up one level into something like 
> /test/sample/code.php (that way we also remove the "-" in the filename, 
> "_" are used all over the place in SimpleTest code base)

sure

> 
> - while I'm at it, shouldn't we move /coverage/CoverageTemplates to 
> /coverage/templates
sure

> 
> I'll send over an other set of remarks once I have PEAR installed on my 
> machine.
thank you

Douglas Hubler <dhubler@...> writes:
> Perrick Penet <perrick@...> writes:
> > - there's not only a dependcy upon SQLlite but also on PEAR's DB/sqlite. 
> > Could we get rid of PEAR first ?
> I installed sqlite from ubuntu package. Can you give advice, on what i can 
> do to remove PEAR, I'm not too familiar with this.

I'm still not sure what do here

> > - there's a "test.php" file in the /coverage directory, I think it 
> > should be put into /coverage/test/

fixed

> > - the /test/sample-code.php should move up one level into something like 
> > /test/sample/code.php

fixed

> > - while I'm at it, shouldn't we move /coverage/CoverageTemplates to 
> > /coverage/templates

fixed

I'd really like to merge this to trunk.  I can continue to make improvements on
trunk. I will merge trunk into this branch before trying to merge to trunk.

Hi...

Douglas Hubler wrote:
> I'd really like to merge this to trunk.  I can continue to make improvements on
> trunk. I will merge trunk into this branch before trying to merge to trunk.

I won't be able to look at it until after Xmas now. I'm going to do a
bit of a stint to get a tarball out around then as well. Hopefully we
can include the coverage stuff in the code for people to try out
(unofficially first).

yours, Marcus

Jakob Ketterl <jketterl@...> writes:
> On Tuesday 18 November 2008 06:13:16 Douglas Hubler wrote:
> > Jakob Ketterl <jketterl@...> writes:
> > > I found two little bugs in the reporting templates (it's just missing
> > > echo statements). i'm including a patch
> >
> > thanks, i committed these
> >
> > Any progress on general review?
> 
> yes, i've even managed to integrate it with phpundercontrol (using a special 
> reporter class i've hacked up myself).
> 
> however, i didn't use the concept of an external shell script to separate 
> coverage setup/collection from the actual test run

that's great, my goal was to give folks APIs for custom setups, i almost didn't
create shell script to encourage folks to do just what you did, but then i
thought we'd loose the folks that just wanted to follow a simple pattern.

> i've read in the source that uncovered files are scanned for from getcwd() 
> downwards (iirc) - so maybe there's a way to make that configurable?

yes, that directory really should be made configurable

> i've also had a look into what's inside the sqlite file, and from my point of 
> view the structure doesn't look that complicated, imho that's nothing a 
> serialized array couldn't handle (if it really needs to be stored to disk - 
> for my case it's not even necessary).

the tests i run take about 2+ hours and sqlite helps me perform data selection
without loading the entire coverage into memory.  Having said this, all this is
handled by CoverageDataHandler and so this could have alternate implementations

> i think it would be better to integrate this deeper into the simpletest main 
> code, but that's beyond the scope of "extension", if i'm correct...

For reporting, I see someone developing some unit test reports that integrate
the coverage data, i think someone even suggested that.  I like the fact that
they can run independently, but i think there are a lot of ways folks can create
optional ways to make them work together.  

For data collection, in my situation, i run simple tests from CLI, that in turn
make http requests to a local apache server, that i need to accumulate coverage
data for.  This request knows nothing about simpletest.

On another note

I'm starting the process of using this branch for generating
 http://acquia.com/files/test-results/index.html
This is a report that is generated on a nightly basis.

 (2nd post, gmane may have ate first)

Jakob Ketterl <jketterl@...> writes:
> On Tuesday 18 November 2008 06:13:16 Douglas Hubler wrote:
> > Any progress on general review?
> i didn't use the concept of an external shell script 

excellent, i almost didn't write shell script so folks would use the API

> i've read in the source that uncovered files are scanned for from getcwd() 
y, we can make that configurable

> i've also had a look into what's inside the sqlite file, and from my point of 
> view the structure doesn't look that complicated, imho that's nothing a 
> serialized array couldn't handle (if it really needs to be stored to disk - 
> for my case it's not even necessary).

in my case, i generate a lot of data and sqlite helps me from loading entire
coverage into memory. All of sqlite functionality is in CoverageDataHandler so
it can be replaces or we can have alternate implementations no problem

> i think it would be better to integrate this deeper into the simpletest main 
> code, but that's beyond the scope of "extension", if i'm correct...

extension can know about simpletest, simpletest probably shouldn't know
extension directly, however... the

for report data, yes, i could see some integration there.  

for coverage data, although i initiate test from simpletest, web tests make http
requests to my web server and i need to gather coverage data there. Nothing is
known about simpletest there.

On Tuesday 18 November 2008 06:13:16 Douglas Hubler wrote:
> Jakob Ketterl <jketterl@...> writes:
> > I found two little bugs in the reporting templates (it's just missing
> > echo statements). i'm including a patch
>
> thanks, i committed these
>
> Any progress on general review?

yes, i've even managed to integrate it with phpundercontrol (using a special 
reporter class i've hacked up myself).

however, i didn't use the concept of an external shell script to separate 
coverage setup/collection from the actual test run, and i didn't use 
autocoverage.php. this is because our code is separated from its tests, and 
simpletest itself also resides outside of the source tree. that way i can 
simply use a whitelist approach to tell cc what parts of the code it should 
generate statistics upon.

the only thing that doesn't work in that approach is the collection of 
uncovered files, so i've written a recursive function that will 
require_once() everything inside my source directories. not the best 
solution, i must admit (especially if there's executable code inside your php 
files) but it works for me atm.

i've read in the source that uncovered files are scanned for from getcwd() 
downwards (iirc) - so maybe there's a way to make that configurable?

i've also had a look into what's inside the sqlite file, and from my point of 
view the structure doesn't look that complicated, imho that's nothing a 
serialized array couldn't handle (if it really needs to be stored to disk - 
for my case it's not even necessary).

i think it would be better to integrate this deeper into the simpletest main 
code, but that's beyond the scope of "extension", if i'm correct...

Hi...

Jakob Ketterl wrote:
> i think it would be better to integrate this deeper into the simpletest main 
> code, but that's beyond the scope of "extension", if i'm correct...

Your instincts are probably right. Certainly the core will probably have
to add the appropriate hooks.

Any dependence on SQL lite would have to go for it to end up in the
core, and preferably even the file system as we cannot assume write
permissions (e.g. in cruise environments).

yours, Marcus

PHPCoverage [1] is trivial to integrate with Simpletest. My
test-runner contains the following:

...

require_once 'phpcoverage/CoverageRecorder.php';
require_once 'phpcoverage/reporter/HtmlCoverageReporter.php';

...

  $coverage = count($coverage_files) > 0;
  if ($coverage) {
    $coverage_reporter = new HtmlCoverageReporter("Code Coverage
Report", "", "report");
    $cov = new CoverageRecorder($coverage_files, array(), $coverage_reporter);
    $cov->startInstrumentation();
  }

  $result = $test->run($reporter);

  if ($coverage) {
    $cov->stopInstrumentation();
    $cov->generateReport();
    $coverage_reporter->printTextSummary();
  }

  exit($result ? 0 : 1);

Where $coverage_files is an array of filenames/paths that should be
reported for.

[1] http://developer.spikesource.com/wiki/index.php/Projects:phpcoverage

--
troels

On Wed, Apr 23, 2008 at 4:57 PM, Lars Olesen <lso...@gm...> wrote:
> Is it possible to do a report on code coverage using simpletest? If it
>  is, it would be possible to use simpletest with phpUnderControl and
>  cruisecontrol. It is pretty neat that you can get both a code coverage
>  report and a codesniffer report.
>
>  --
>  Lars Olesen
>  Gratis backup https://mozy.com/?code=9MYJPF
>
>  -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>  http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>  _______________________________________________
>  Simpletest-support mailing list
>  Sim...@li...
>  https://lists.sourceforge.net/lists/listinfo/simpletest-support
>

Hey Lars;

On Apr 23, 2008, at 9:57 AM, Lars Olesen wrote:

> Is it possible to do a report on code coverage using simpletest? If it
> is, it would be possible to use simpletest with phpUnderControl and
> cruisecontrol. It is pretty neat that you can get both a code coverage
> report and a codesniffer report.

There's no official code coverage yet.  There's some institutional  
resistance to it here 'cause it is a false meter (i.e., and good TDD  
project will actually have well over 100% code coverage).  People see  
"95%" and get all happy, when there's still major areas not covered.   
I guess its like any statistical data though, its only as good as the  
person interpreting it.

That said, it is on the road map as it is an oft requested feature.   
I have plans on doing it in PHPT as well, so hopefully there can be  
some cross pollination.  As Troel said in another email, it's really  
trivial to implement, the hard part is getting useful info out.  I've  
got some interesting ideas for PHPT since each test case gets  
executed in its own process I can show how many times a given line  
was exercised at least once during a test, but discussion of that's  
for another list :-)

-T

Hi...

Travis Swicegood wrote:
> There's some institutional  
> resistance to it here 'cause it is a false meter (i.e., and good TDD  
> project will actually have well over 100% code coverage).

Resistance has ceased :). If no one else were working on SimpleTest, 
I would write code code coverage myself before the SimpleTest2 release.

The order of roadmap events is linear because of the pessimistic 
assumption that I would be the only one working on the project. 
Clearly not the case, but I like to start project management from 
the bad side of the bell curve.

The only dependency inducing feature is the dispatch mechanism that 
enables plug-ins. Once that is done, the roadmap features can go in 
in any order. It then becomes simply a problem of finding someone to 
take up the cause.

yours, Marcus

On Wed, Apr 23, 2008 at 6:57 PM, Lars Olesen <lso...@gm...> wrote:
> Is it possible to do a report on code coverage using simpletest?

You may also get interested in limb_unit[1] which provides such a feature.

[1] - http://wiki.limb-project.com/doku.php?id=limb3:en:packages:tests_runner:limb_unit

-- 
Best regards, Pavel