You can subscribe to this list here.
2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
(3) |
Aug
(2) |
Sep
|
Oct
(2) |
Nov
(1) |
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2008 |
Jan
|
Feb
(2) |
Mar
|
Apr
|
May
(5) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(2) |
Dec
|
2009 |
Jan
|
Feb
|
Mar
(1) |
Apr
(2) |
May
(1) |
Jun
|
Jul
|
Aug
(4) |
Sep
(6) |
Oct
(4) |
Nov
(5) |
Dec
(7) |
2010 |
Jan
(2) |
Feb
(3) |
Mar
|
Apr
|
May
(2) |
Jun
(3) |
Jul
(1) |
Aug
(5) |
Sep
(1) |
Oct
|
Nov
(6) |
Dec
|
2011 |
Jan
|
Feb
|
Mar
(5) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
2012 |
Jan
(9) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2013 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2014 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Len T. <len...@gm...> - 2014-03-06 23:26:36
|
Since the beginning of the jumble project we have been running a continuous integration tool in-house, but after moving to sourceforge it became difficult for external contributors to see details of problems that the CI system picked up. With the idea of making it easier for people to run the same checks as the CI tool, and also potentially moving to another CI tool, I have been having a play with Jenkins and enhancing the build.xml to run the same checks. The idea being we can either use a public Jenkins server (e.g. cloudbees), or set up an external jenkins server. In the main jumble directory, the default target for the build.xml now run a set of checks: including checkstyle, findbugs, spelling checks, junit tests, and javadoc building. Feel free to fix any of the problems that the are currently showing :-) Cheers, Len. |
From: Mark U. <ma...@cs...> - 2013-04-22 11:03:48
|
Len and everyone, I've finally managed to do a new release of Jumble (1.2.0). This incorporates the various minor improvements made over the last 2 years, plus a completely rewritten Eclipse plugin and update site. (It uses newer Eclipse APIs, and allows you to click on a failed mutation to go to the corresponding source code line). The download site is: https://sourceforge.net/projects/jumble/files/jumble/1.2.0 And the Eclipse update site is the same, but a nicer URL is just: http://jumble.sourceforge.net Comments welcome. My comment is that you would not *believe* how difficult it seems to be to create an Eclipse update site that is 'P2' compliant -- expect a few spurious repository warnings about missing P2 metadata during the install process. After a few people have tried it out, I'll announce it on slashdot or somewhere. Cheers Mark |
From: Mark U. <ma...@cs...> - 2013-04-22 11:00:29
|
Len and everyone, I've finally managed to do a new release of Jumble (1.2.0). This incorporates the various minor improvements made over the last 2 years, plus a completely rewritten Eclipse plugin and update site. (It uses newer Eclipse APIs, and allows you to click on a failed mutation to go to the corresponding source code line). The download site is: https://sourceforge.net/projects/jumble/files/jumble/1.2.0 And the Eclipse update site is the same, but a nicer URL is just: http://jumble.sourceforge.net Comments welcome. My comment is that you would not *believe* how difficult it seems to be to create an Eclipse update site that is 'P2' compliant -- expect a few spurious repository warnings about missing P2 metadata during the install process. After a few people have tried it out, I'll announce it on slashdot or somewhere. Cheers Mark |
From: Marcos L. <mtk...@ho...> - 2012-01-31 17:27:14
|
The class org.junit.ExpectedException is not found even when i use the -d flag to make it deferred and not use any kind of special class loading i have checked the jumble source code and i saw this array declaration on com.reeltwo.jumble.mutation.MutatingClassLoader: private static final String[] DEFERRED_PREFIXES = new String[] { "java.", "sun.reflect.", "junit.", "org.junit.", /* "com.sun.management.OperatingSystemMXBean" "javax.", // Too general to have deferred "com.sun", // Too general to have deferred */ }; with this code the class ExpectedException gets loaded by the class loader got by ClassLoader.getSystemClassLoader() i have tried changing the jumble source code so ExpectedException class doesn't get deferred and thus be able to be loaded with this loader in MutatingClassLoader: private final Repository mRepository; where Repository is org.apache.bcel.util.Repository and the change i made in the code was the next: private static final String[] DEFERRED_PREFIXES = new String[] { "java.", "sun.reflect.", /* "junit.", "org.junit.", "com.sun.management.OperatingSystemMXBean" "javax.", // Too general to have deferred "com.sun", // Too general to have deferred */ }; what i've got is a message like this "com.myCompany..ActionTest is not a test class." that is given by the jumble class com.reeltwo.jumble.fast.FastRunner in the method "checkClasses" I got this message because jumble can't identify (due to change in the jumble source) the @Test annotations as such in each of the methods of ActionTest ( and i dont know why because each test metod is annotated with @Test). The thing is i do not understand how can changing some code to make the junit classes to be loaded not as deferred classes can impact on jumble on a way it does not recognizes my test as a test anymore. I'have tried with the -d Flag for the org.junit.ExpectedException class but got no positive results Any idea? i will keep you posted if i find out something else Date: Sun, 29 Jan 2012 16:12:40 +1000 Subject: Re: [Jumble-devel] Possible Classloader problem with jumble From: ma...@cs... To: mtk...@ho... CC: jum...@li... Marcos, Yes, Jumble certainly has its own class loader. That is how it makes various bytecode mutations to the class that it loads. It delegates the loading of most other classes to the underlying class loader. But this area of Java is quite fragile, and it is easy for it to end up using a class loader that you didn't expect, or that has the wrong class path. I suggest that you download the source code from the SVN repository, and look at trunk/jumble/src/com/reeltwo/jumble/mutation/MutatingClassLoader.java to see what it is doing. Cheers Mark On 28 January 2012 04:27, Marcos Lambolay <mtk...@ho...> wrote: (first of all i wanted to notify that the confirmation mail for this list seems not working) hi, i am trying to get jumble into the projects i work on my job. i've run jumble on the classes of one of this projects and it performs the mutations on some classes but for some others it tells "test class is broken", despite i run those test classes with an ant task and they run ok. I've trying to know what is going wrong and started to run jumble in debug mode on eclipse for each one of the classes that jumble reported their tests to be broken. When i debug these classes, i get into junit class TestResult's instance fields so i can see the description of the errors and failures that junit reports when jumble run the test class, because jumble tells me "tests class is broken" but it doesn't tell why it says the tests class brokes so i need to debug to know what is happening. The issues and error details i've observed that are the result of the failed tests jumble does are some of the followings: (I cannot post certain pieces of code due to legal restrictions of the company i work for, but i will try to be the more explanatory that i can) For the class "Forwarder" i run jumble in debug mode and runs smoothly and makes the mutations perfectly. This is the case that surprises me the most, because this when run with the ant task jumble reports me "test class is broken", but in debug mode in eclipse runs ok. For the class Action, and many others, in the debug mode an error occurs when the test class are run and when inspected the errors descriptions, the error is java.lang.ClassNotFoundException: org.junit.rules.ExpectedException. I have that class on the junit jar and it's path is available for the eclipse debug mode and for the ant task. Also, this class is in a jar of the selenium framework with its dependencies I pressume that it could be that the class loader can't decide what version of the class to load. But i'm not sure. For the class Module, i do the same thing: inspect the error message that is available on the TestResult junit class when the tests is run by jumble and finishes executing. For this class i get: java.lang.LinkageError: loading constraint violation: loader "com/reeltwo/jumble/mutation/MutatingClassLoader@5b8d5b8d" previously initiated loading for a different type with name "org/hamcrest/Description" defined by loader "sun/misc/Launcher$AppClassLoader@7fce7fce" This error got me astonished. It makes me think that Jumble has its own class loader and maybe it somehow can be the source of some of my troubles. I really will appreciate any kind of help, Marcos ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Jumble-devel mailing list Jum...@li... https://lists.sourceforge.net/lists/listinfo/jumble-devel |
From: Mark U. <ma...@cs...> - 2012-01-29 06:12:48
|
Marcos, Yes, Jumble certainly has its own class loader. That is how it makes various bytecode mutations to the class that it loads. It delegates the loading of most other classes to the underlying class loader. But this area of Java is quite fragile, and it is easy for it to end up using a class loader that you didn't expect, or that has the wrong class path. I suggest that you download the source code from the SVN repository, and look at trunk/jumble/src/com/reeltwo/jumble/mutation/MutatingClassLoader.java to see what it is doing. Cheers Mark On 28 January 2012 04:27, Marcos Lambolay <mtk...@ho...> wrote: > (first of all i wanted to notify that the confirmation mail for this list > seems not working) > > hi, i am trying to get jumble into the projects i work on my job. > i've run jumble on the classes of one of this projects and it performs the > mutations on some classes but for some others it tells "test class is > broken", despite i run those test classes with an ant task and they run ok. > > I've trying to know what is going wrong and started to run jumble in debug > mode on eclipse for each one of the classes that jumble reported their > tests to be broken. > When i debug these classes, i get into junit class TestResult's instance > fields so i can see the description of the errors and failures that junit > reports when jumble run the test class, because jumble tells me "tests > class is broken" but it doesn't tell why it says the tests class brokes so > i need to debug to know what is happening. The issues and error details > i've observed that are the result of the failed tests jumble does are some > of the followings: > > (I cannot post certain pieces of code due to legal restrictions of the > company i work for, but i will try to be the more explanatory that i can) > > For the class "Forwarder" i run jumble in debug mode and runs smoothly and > makes the mutations perfectly. This is the case that surprises me the most, > because this when run with the ant task jumble reports me "test class is > broken", but in debug mode in eclipse runs ok. > > For the class Action, and many others, in the debug mode an error occurs > when the test class are run and when inspected the errors descriptions, the > error is java.lang.ClassNotFoundException: > org.junit.rules.ExpectedException. > I have that class on the junit jar and it's path is available for the > eclipse debug mode and for the ant task. > Also, this class is in a jar of the selenium framework with its > dependencies > I pressume that it could be that the class loader can't decide what > version of the class to load. But i'm not sure. > > For the class Module, i do the same thing: inspect the error message that > is available on the TestResult junit class when the tests is run by jumble > and finishes executing. For this class i get: > java.lang.LinkageError: loading constraint violation: loader > "com/reeltwo/jumble/mutation/MutatingClassLoader@5b8d5b8d" previously > initiated loading for a different type with name "org/hamcrest/Description" > defined by loader "sun/misc/Launcher$AppClassLoader@7fce7fce" > This error got me astonished. It makes me think that Jumble has its own > class loader and maybe it somehow can be the source of some of my troubles. > > I really will appreciate any kind of help, > Marcos > > > ------------------------------------------------------------------------------ > Try before you buy = See our experts in action! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-dev2 > _______________________________________________ > Jumble-devel mailing list > Jum...@li... > https://lists.sourceforge.net/lists/listinfo/jumble-devel > > |
From: Marcos L. <mtk...@ho...> - 2012-01-27 18:27:37
|
(first of all i wanted to notify that the confirmation mail for this list seems not working) hi, i am trying to get jumble into the projects i work on my job. i've run jumble on the classes of one of this projects and it performs the mutations on some classes but for some others it tells "test class is broken", despite i run those test classes with an ant task and they run ok. I've trying to know what is going wrong and started to run jumble in debug mode on eclipse for each one of the classes that jumble reported their tests to be broken. When i debug these classes, i get into junit class TestResult's instance fields so i can see the description of the errors and failures that junit reports when jumble run the test class, because jumble tells me "tests class is broken" but it doesn't tell why it says the tests class brokes so i need to debug to know what is happening. The issues and error details i've observed that are the result of the failed tests jumble does are some of the followings: (I cannot post certain pieces of code due to legal restrictions of the company i work for, but i will try to be the more explanatory that i can) For the class "Forwarder" i run jumble in debug mode and runs smoothly and makes the mutations perfectly. This is the case that surprises me the most, because this when run with the ant task jumble reports me "test class is broken", but in debug mode in eclipse runs ok. For the class Action, and many others, in the debug mode an error occurs when the test class are run and when inspected the errors descriptions, the error is java.lang.ClassNotFoundException: org.junit.rules.ExpectedException. I have that class on the junit jar and it's path is available for the eclipse debug mode and for the ant task. Also, this class is in a jar of the selenium framework with its dependencies I pressume that it could be that the class loader can't decide what version of the class to load. But i'm not sure. For the class Module, i do the same thing: inspect the error message that is available on the TestResult junit class when the tests is run by jumble and finishes executing. For this class i get: java.lang.LinkageError: loading constraint violation: loader "com/reeltwo/jumble/mutation/MutatingClassLoader@5b8d5b8d" previously initiated loading for a different type with name "org/hamcrest/Description" defined by loader "sun/misc/Launcher$AppClassLoader@7fce7fce" This error got me astonished. It makes me think that Jumble has its own class loader and maybe it somehow can be the source of some of my troubles. I really will appreciate any kind of help, Marcos |
From: Mark U. <ma...@cs...> - 2012-01-09 11:14:27
|
Nicolas, Great. Can you put the Apache 2.0 licence header at the top of each of your files, with your name as author, and then send me a copy of them all? Thanks Mark On 6 January 2012 10:55, Nicolas Rusconi <nic...@gm...> wrote: > Found a couple of issues while testing it in jenkins, but now is working > in both, > Hudson 2.2.0 and Jenkins 1.446 > Let me know how do you want to proceed. > > Thanks > Nicolas.- > > > On Thu, Jan 5, 2012 at 5:34 PM, Nicolas Rusconi <nic...@gm... > > wrote: > >> Mark, >> >> I was just testing it under Jenkins when I got your email :) >> so I will need a little more time to tests it, but the idea is to make it >> work on both. >> runs ok in Hudson 2.2.0 >> >> Yes, I have no problem on using apache 2.0, as long as it's open source >> it's ok with me. >> >> Nicolas.- >> >> On Thu, Jan 5, 2012 at 5:23 PM, Mark Utting <ma...@cs...>wrote: >> >>> Nicolas, >>> >>> Yes, these would be very useful to add to the Jumble distribution. >>> >>> Do you know if the Hudson plugin also works with Jenkins? >>> >>> Are you willing to put these files under the Apache 2.0 open source >>> license so we >>> can include them into the Jumble distribution? >>> >>> Thanks >>> Mark >>> >>> >>> On 6 January 2012 00:00, Nicolas Rusconi <nic...@gm...>wrote: >>> >>>> Hi Jumble devs, >>>> >>>> In the last coupleof weeks I created 2 tools around jumble to be more >>>> productive for me. >>>> The first one is an ant task to be able to run jumble on all your >>>> classes, or a set of classes >>>> and generates a single xml report with all the scores and points. >>>> You can take a llok at it here >>>> https://github.com/nicolasrusconi/JumbleAnt >>>> >>>> There is the code and a small example on how to use it. >>>> I think it would be great to have that shipped with the jumble jar so >>>> is easier for people to use it. >>>> what do you think? >>>> >>>> The other tool is a Hudson plugin that, using the information generated >>>> in the xml report created by the ant task, >>>> generates a graph with the average project score as well as a per >>>> package/class score report >>>> here are some screenshots you can take a look at >>>> >>>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot1.png >>>> >>>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot2.png >>>> >>>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot3.png >>>> >>>> >>>> Let me know what you think, and if you have any suggestions or >>>> improvements. >>>> Thanks for building jumble >>>> >>>> Nicolas.- >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a >>>> complex >>>> infrastructure or vast IT resources to deliver seamless, secure access >>>> to >>>> virtual desktops. With this all-in-one solution, easily deploy virtual >>>> desktops for less than the cost of PCs and save 60% on VDI >>>> infrastructure >>>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox >>>> _______________________________________________ >>>> Jumble-devel mailing list >>>> Jum...@li... >>>> https://lists.sourceforge.net/lists/listinfo/jumble-devel >>>> >>>> >>> >> > |
From: Nicolas R. <nic...@gm...> - 2012-01-06 00:56:04
|
Found a couple of issues while testing it in jenkins, but now is working in both, Hudson 2.2.0 and Jenkins 1.446 Let me know how do you want to proceed. Thanks Nicolas.- On Thu, Jan 5, 2012 at 5:34 PM, Nicolas Rusconi <nic...@gm...>wrote: > Mark, > > I was just testing it under Jenkins when I got your email :) > so I will need a little more time to tests it, but the idea is to make it > work on both. > runs ok in Hudson 2.2.0 > > Yes, I have no problem on using apache 2.0, as long as it's open source > it's ok with me. > > Nicolas.- > > On Thu, Jan 5, 2012 at 5:23 PM, Mark Utting <ma...@cs...>wrote: > >> Nicolas, >> >> Yes, these would be very useful to add to the Jumble distribution. >> >> Do you know if the Hudson plugin also works with Jenkins? >> >> Are you willing to put these files under the Apache 2.0 open source >> license so we >> can include them into the Jumble distribution? >> >> Thanks >> Mark >> >> >> On 6 January 2012 00:00, Nicolas Rusconi <nic...@gm...>wrote: >> >>> Hi Jumble devs, >>> >>> In the last coupleof weeks I created 2 tools around jumble to be more >>> productive for me. >>> The first one is an ant task to be able to run jumble on all your >>> classes, or a set of classes >>> and generates a single xml report with all the scores and points. >>> You can take a llok at it here >>> https://github.com/nicolasrusconi/JumbleAnt >>> >>> There is the code and a small example on how to use it. >>> I think it would be great to have that shipped with the jumble jar so is >>> easier for people to use it. >>> what do you think? >>> >>> The other tool is a Hudson plugin that, using the information generated >>> in the xml report created by the ant task, >>> generates a graph with the average project score as well as a per >>> package/class score report >>> here are some screenshots you can take a look at >>> >>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot1.png >>> >>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot2.png >>> >>> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot3.png >>> >>> >>> Let me know what you think, and if you have any suggestions or >>> improvements. >>> Thanks for building jumble >>> >>> Nicolas.- >>> >>> >>> ------------------------------------------------------------------------------ >>> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex >>> infrastructure or vast IT resources to deliver seamless, secure access to >>> virtual desktops. With this all-in-one solution, easily deploy virtual >>> desktops for less than the cost of PCs and save 60% on VDI infrastructure >>> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox >>> _______________________________________________ >>> Jumble-devel mailing list >>> Jum...@li... >>> https://lists.sourceforge.net/lists/listinfo/jumble-devel >>> >>> >> > |
From: Nicolas R. <nic...@gm...> - 2012-01-05 20:35:03
|
Mark, I was just testing it under Jenkins when I got your email :) so I will need a little more time to tests it, but the idea is to make it work on both. runs ok in Hudson 2.2.0 Yes, I have no problem on using apache 2.0, as long as it's open source it's ok with me. Nicolas.- On Thu, Jan 5, 2012 at 5:23 PM, Mark Utting <ma...@cs...> wrote: > Nicolas, > > Yes, these would be very useful to add to the Jumble distribution. > > Do you know if the Hudson plugin also works with Jenkins? > > Are you willing to put these files under the Apache 2.0 open source > license so we > can include them into the Jumble distribution? > > Thanks > Mark > > > On 6 January 2012 00:00, Nicolas Rusconi <nic...@gm...>wrote: > >> Hi Jumble devs, >> >> In the last coupleof weeks I created 2 tools around jumble to be more >> productive for me. >> The first one is an ant task to be able to run jumble on all your >> classes, or a set of classes >> and generates a single xml report with all the scores and points. >> You can take a llok at it here >> https://github.com/nicolasrusconi/JumbleAnt >> >> There is the code and a small example on how to use it. >> I think it would be great to have that shipped with the jumble jar so is >> easier for people to use it. >> what do you think? >> >> The other tool is a Hudson plugin that, using the information generated >> in the xml report created by the ant task, >> generates a graph with the average project score as well as a per >> package/class score report >> here are some screenshots you can take a look at >> >> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot1.png >> >> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot2.png >> >> http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot3.png >> >> >> Let me know what you think, and if you have any suggestions or >> improvements. >> Thanks for building jumble >> >> Nicolas.- >> >> >> ------------------------------------------------------------------------------ >> Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex >> infrastructure or vast IT resources to deliver seamless, secure access to >> virtual desktops. With this all-in-one solution, easily deploy virtual >> desktops for less than the cost of PCs and save 60% on VDI infrastructure >> costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox >> _______________________________________________ >> Jumble-devel mailing list >> Jum...@li... >> https://lists.sourceforge.net/lists/listinfo/jumble-devel >> >> > |
From: Mark U. <ma...@cs...> - 2012-01-05 20:23:23
|
Nicolas, Yes, these would be very useful to add to the Jumble distribution. Do you know if the Hudson plugin also works with Jenkins? Are you willing to put these files under the Apache 2.0 open source license so we can include them into the Jumble distribution? Thanks Mark On 6 January 2012 00:00, Nicolas Rusconi <nic...@gm...> wrote: > Hi Jumble devs, > > In the last coupleof weeks I created 2 tools around jumble to be more > productive for me. > The first one is an ant task to be able to run jumble on all your classes, > or a set of classes > and generates a single xml report with all the scores and points. > You can take a llok at it here > https://github.com/nicolasrusconi/JumbleAnt > > There is the code and a small example on how to use it. > I think it would be great to have that shipped with the jumble jar so is > easier for people to use it. > what do you think? > > The other tool is a Hudson plugin that, using the information generated in > the xml report created by the ant task, > generates a graph with the average project score as well as a per > package/class score report > here are some screenshots you can take a look at > > http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot1.png > > http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot2.png > > http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot3.png > > > Let me know what you think, and if you have any suggestions or > improvements. > Thanks for building jumble > > Nicolas.- > > > ------------------------------------------------------------------------------ > Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex > infrastructure or vast IT resources to deliver seamless, secure access to > virtual desktops. With this all-in-one solution, easily deploy virtual > desktops for less than the cost of PCs and save 60% on VDI infrastructure > costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox > _______________________________________________ > Jumble-devel mailing list > Jum...@li... > https://lists.sourceforge.net/lists/listinfo/jumble-devel > > |
From: Nicolas R. <nic...@gm...> - 2012-01-05 14:00:08
|
Hi Jumble devs, In the last coupleof weeks I created 2 tools around jumble to be more productive for me. The first one is an ant task to be able to run jumble on all your classes, or a set of classes and generates a single xml report with all the scores and points. You can take a llok at it here https://github.com/nicolasrusconi/JumbleAnt There is the code and a small example on how to use it. I think it would be great to have that shipped with the jumble jar so is easier for people to use it. what do you think? The other tool is a Hudson plugin that, using the information generated in the xml report created by the ant task, generates a graph with the average project score as well as a per package/class score report here are some screenshots you can take a look at http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot1.png http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot2.png http://cloud.github.com/downloads/nicolasrusconi/JumbleHudsonPlugin/hudsonScreenshot3.png Let me know what you think, and if you have any suggestions or improvements. Thanks for building jumble Nicolas.- |
From: alexis _ <ale...@ho...> - 2011-09-22 20:24:32
|
great ! the updated jumble.jar does the trick ! can't wait to see that release finally out jumble rocks ! cheers alexis > Date: Thu, 22 Sep 2011 10:27:24 +1200 > From: len...@gm... > To: ale...@ho... > CC: jum...@li... > Subject: Re: [Jumble-devel] jumble 1.1.0 and junit 4.9 ExpectedException > > alexis _ wrote: > > I have the feeling Jumble uses a JUnit version embeded in jumble.jar instead of using my own junit.jar archive, > > therefore jumble may not support @Rule annotation and ExceptedException class (introduced in Junit 4.7 so it seems, i > > am using Junit 4.9) > > Yes, it does currently have embedded junit version 4.6. > > > > i may be wrong but it seems recompiling jumble with junit4.9 would do the job. but i am not sure as Java is not my > > favorite programming language :) > > > > Can someone provide me a hint on how to resolve this issue ? > > I updated the SVN version of jumble to use junit 4.9. > > You can get a replacement jumble.jar without the full hassle of > checking out and building from SVN via: > https://jumble.svn.sourceforge.net/svnroot/jumble/trunk/jumble/jumble.jar > > We are due to make a new release as there have been several > improvements since the 1.1.0 release. > > > Cheers, > Len. |
From: Len T. <len...@gm...> - 2011-09-21 22:27:40
|
alexis _ wrote: > I have the feeling Jumble uses a JUnit version embeded in jumble.jar instead of using my own junit.jar archive, > therefore jumble may not support @Rule annotation and ExceptedException class (introduced in Junit 4.7 so it seems, i > am using Junit 4.9) Yes, it does currently have embedded junit version 4.6. > i may be wrong but it seems recompiling jumble with junit4.9 would do the job. but i am not sure as Java is not my > favorite programming language :) > > Can someone provide me a hint on how to resolve this issue ? I updated the SVN version of jumble to use junit 4.9. You can get a replacement jumble.jar without the full hassle of checking out and building from SVN via: https://jumble.svn.sourceforge.net/svnroot/jumble/trunk/jumble/jumble.jar We are due to make a new release as there have been several improvements since the 1.1.0 release. Cheers, Len. |
From: alexis _ <ale...@ho...> - 2011-09-21 20:02:03
|
hello, first of all, thanks a lot for jumble. it just rocks ! thanks to this great piece of software, i discovered some vulnerabilities in my testing code. for instance, i was testing that a given exception was raised, but i wasn't testing the anything regarding the exception message. $ java -jar "lib\jumble.jar" -X -r -j -i -k -w -c . ILVParser Mutating ILVParser Tests: ILVParserTest Mutation points = 215, unit test time limit 2.94s M FAIL: ILVParser:25: CP[6] "Length can not be negative" -> "___jumble___" M FAIL: ILVParser:26: CP[8] 65535 -> 65536 .................................................. .................................................. ...............M FAIL: ILVParser:71: 16 -> 17 ....M FAIL: ILVParser:71: 16 -> 17 .................................................. ....M FAIL: ILVParser:111: 6 -> 7 M FAIL: ILVParser:111: 4 -> 5 M FAIL: ILVParser:111: removed array assignment M FAIL: ILVParser:112: 7 -> 8 .M FAIL: ILVParser:112: removed array assignment ...........M FAIL: ILVParser:116: 4 -> 5 .................... Score: 95% as a result i changed my testing code to use the @Rule annotation to test excepted exceptions. more precisely, i changed from @Test (expected=IllegalArgumentException.class) public void lengthArrayLengthCanNotBeNegative() { byte[] res = ilv.toLengthArray(-1); } to import org.junit.Rule; import org.junit.rules.ExpectedException; [...] @Rule public ExpectedException thrown = ExpectedException.none(); [...] @Test public void lengthArrayLengthCanNotBeNegative() throws IllegalArgumentException { thrown.expect(IllegalArgumentException.class); thrown.expectMessage("Length can not be negative"); byte[] res = ilv.toLengthArray(-1); } However, now when i run jumble i get $ java -jar "lib\jumble.jar" -X -r -j -i -k -w -c ".;\lib\junit-4.9.jar" --verbose ILVParser TEST FINISHED WITH ERROR: toLengthArray(ILVParserTest): org/junit/rules/ExpectedException java.lang.NoClassDefFoundError: org/junit/rules/ExpectedException at ILVParserTest.<init>(ILVParserTest.java:28) [...] Mutating ILVParser Tests: ILVParserTest Score: 0% (TEST CLASS IS BROKEN) Mutation points = 215 I have the feeling Jumble uses a JUnit version embeded in jumble.jar instead of using my own junit.jar archive, therefore jumble may not support @Rule annotation and ExceptedException class (introduced in Junit 4.7 so it seems, i am using Junit 4.9) i may be wrong but it seems recompiling jumble with junit4.9 would do the job. but i am not sure as Java is not my favorite programming language :) Can someone provide me a hint on how to resolve this issue ? thanks alex ps: i think my younger daugther just sent you guys an empty email :) |
From: Silvio R. <si...@ri...> - 2011-04-01 23:10:32
|
Hello, with a score of 96% Jumble gives following Error Message: M FAIL: RingBuffer:23: 125 (}) -> 126 (~) M FAIL: RingBuffer:24: 126 (~) -> 127 The code in this lines is: public RingBuffer(int capacity) { a = (Item[]) new Object[capacity]; } I absolutely can't get any meaning out of this error message, and I am requesting some help please :) Thx a lot Silvio |
From: Gregor B. <gre...@gm...> - 2011-03-29 10:06:36
|
hi, i solved the problem with an hint from an collegue to use an junit 3 test suite. public class RingBufferTest extends junit.framework.TestSuite { public static junit.framework.Test suite() { junit.framework.TestSuite suite= new junit.framework.TestSuite(); suite.addTest(new junit.framework.JUnit4TestAdapter(RingBufferTest1.class)); suite.addTest(new junit.framework.JUnit4TestAdapter(RingBufferTest2.class)); return suite; } } greetings gregor ----- Original Message ----- > From: "Sean A. Irvine" <sa...@xt...> > To: "Gregor Binder" <gre...@gm...> > Cc: Jum...@li... > Sent: Sunday, 27 March, 2011 9:52:37 PM > Subject: Re: [Jumble-devel] jumble with test suite > Hi, > > Easy part first: there is some documentation on the output format at > http://jumble.sourceforge.net/index.html > > A few of the links from there will give a little more detail about > the mutations performed. > > To use the @TestClass approach you need to put an annotation in your > source class indicating the names of the corresponding test classes. > > Something like: > > Foo.java > > @TestClass(value = {"FooTest", "AnotherTest"}) > public class Foo { > // ... class implementation > } > > FooTest.java and AnotherTest.java are JUnit test classes that test > the implementation of Foo. Of course, in general you should really > only have one test class per implementation class, and for performance > reasons, you want to make sure you only include test classes that > actually test Foo. > > You will also need jumble-annotations.jar in your class path when > you compile/run the source. > > I don't think this is properly documented anywhere, since it is not > yet part of a release. I believe, you'll need to work from the head > version for this to work. > > Sean. > > > Gregor Binder wrote: > > hi, > > > > thanks for your answer but this doesn't help. > > can you give me an extended example how to work with @TestClass? > > also on http://jumble.sourceforge.net/ i found nowhere an example > > how to interpret the jumble results :-( > > > > sorry for send this twice i forgot cc to the list. > > > > greetings > > gregor > > > > ----- Original Message ----- > >> From: "Sean A. Irvine" <sa...@xt...> > >> To: "Gregor Binder" <gre...@gm...> > >> Sent: Friday, 25 March, 2011 5:03:20 AM > >> Subject: Re: [Jumble-devel] jumble with test suite > >> Hi Gregor, > >> > >> I don't think you can use a suite directly. However, depending on > >> how > >> you are using Jumble, you can either simply list multiple test > >> classes > >> on the command line or list multiple test classes in the class to > >> be > >> tested with the @Test annotation: > >> > >> @TestClass(value = {"RingBuffer1Test", "RingBuffer2Test"}) > >> > >> Hope this helps, > >> Sean. > >> > >> Gregor Binder wrote: > >>> hi, > >>> > >>> i have a testsuite for my class that i like to test with jumble. > >>> with only one testclass it worked perfect but with the suite get > >>> the > >>> following error: > >>> ERROR: RingBufferTest is not a test class. > >>> > >>> the testsuite looks like: > >>> @RunWith(Suite.class) > >>> @Suite.SuiteClasses( { > >>> RingBufferTestMutation.class > >>> }) > >>> public class RingBufferTest { > >>> public static void AllTests() { > >>> } > >>> } > >>> > >>> how can i force jumble to use the testsuite? > >>> is there anywhere an example for that? > >>> > >>> greetings > >>> gregor > >>> > >>> ------------------------------------------------------------------------------ > >>> Enable your software for Intel(R) Active Management Technology to > >>> meet the > >>> growing manageability and security demands of your customers. > >>> Businesses > >>> are taking advantage of Intel(R) vPro (TM) technology - will your > >>> software > >>> be a part of the solution? Download the Intel(R) Manageability > >>> Checker > >>> today! http://p.sf.net/sfu/intel-dev2devmar > >>> _______________________________________________ > >>> Jumble-devel mailing list > >>> Jum...@li... > >>> https://lists.sourceforge.net/lists/listinfo/jumble-devel > >>> > > |
From: Sean A. I. <sa...@xt...> - 2011-03-27 20:06:00
|
Hi, Easy part first: there is some documentation on the output format at http://jumble.sourceforge.net/index.html A few of the links from there will give a little more detail about the mutations performed. To use the @TestClass approach you need to put an annotation in your source class indicating the names of the corresponding test classes. Something like: Foo.java @TestClass(value = {"FooTest", "AnotherTest"}) public class Foo { // ... class implementation } FooTest.java and AnotherTest.java are JUnit test classes that test the implementation of Foo. Of course, in general you should really only have one test class per implementation class, and for performance reasons, you want to make sure you only include test classes that actually test Foo. You will also need jumble-annotations.jar in your class path when you compile/run the source. I don't think this is properly documented anywhere, since it is not yet part of a release. I believe, you'll need to work from the head version for this to work. Sean. Gregor Binder wrote: > hi, > > thanks for your answer but this doesn't help. > can you give me an extended example how to work with @TestClass? > also on http://jumble.sourceforge.net/ i found nowhere an example how to interpret the jumble results :-( > > sorry for send this twice i forgot cc to the list. > > greetings > gregor > > ----- Original Message ----- >> From: "Sean A. Irvine" <sa...@xt...> >> To: "Gregor Binder" <gre...@gm...> >> Sent: Friday, 25 March, 2011 5:03:20 AM >> Subject: Re: [Jumble-devel] jumble with test suite >> Hi Gregor, >> >> I don't think you can use a suite directly. However, depending on how >> you are using Jumble, you can either simply list multiple test classes >> on the command line or list multiple test classes in the class to be >> tested with the @Test annotation: >> >> @TestClass(value = {"RingBuffer1Test", "RingBuffer2Test"}) >> >> Hope this helps, >> Sean. >> >> Gregor Binder wrote: >>> hi, >>> >>> i have a testsuite for my class that i like to test with jumble. >>> with only one testclass it worked perfect but with the suite get the >>> following error: >>> ERROR: RingBufferTest is not a test class. >>> >>> the testsuite looks like: >>> @RunWith(Suite.class) >>> @Suite.SuiteClasses( { >>> RingBufferTestMutation.class >>> }) >>> public class RingBufferTest { >>> public static void AllTests() { >>> } >>> } >>> >>> how can i force jumble to use the testsuite? >>> is there anywhere an example for that? >>> >>> greetings >>> gregor >>> >>> ------------------------------------------------------------------------------ >>> Enable your software for Intel(R) Active Management Technology to >>> meet the >>> growing manageability and security demands of your customers. >>> Businesses >>> are taking advantage of Intel(R) vPro (TM) technology - will your >>> software >>> be a part of the solution? Download the Intel(R) Manageability >>> Checker >>> today! http://p.sf.net/sfu/intel-dev2devmar >>> _______________________________________________ >>> Jumble-devel mailing list >>> Jum...@li... >>> https://lists.sourceforge.net/lists/listinfo/jumble-devel >>> > |
From: Gregor B. <gre...@gm...> - 2011-03-26 15:11:31
|
hi, thanks for your answer but this doesn't help. can you give me an extended example how to work with @TestClass? also on http://jumble.sourceforge.net/ i found nowhere an example how to interpret the jumble results :-( sorry for send this twice i forgot cc to the list. greetings gregor ----- Original Message ----- > From: "Sean A. Irvine" <sa...@xt...> > To: "Gregor Binder" <gre...@gm...> > Sent: Friday, 25 March, 2011 5:03:20 AM > Subject: Re: [Jumble-devel] jumble with test suite > Hi Gregor, > > I don't think you can use a suite directly. However, depending on how > you are using Jumble, you can either simply list multiple test classes > on the command line or list multiple test classes in the class to be > tested with the @Test annotation: > > @TestClass(value = {"RingBuffer1Test", "RingBuffer2Test"}) > > Hope this helps, > Sean. > > Gregor Binder wrote: > > hi, > > > > i have a testsuite for my class that i like to test with jumble. > > with only one testclass it worked perfect but with the suite get the > > following error: > > ERROR: RingBufferTest is not a test class. > > > > the testsuite looks like: > > @RunWith(Suite.class) > > @Suite.SuiteClasses( { > > RingBufferTestMutation.class > > }) > > public class RingBufferTest { > > public static void AllTests() { > > } > > } > > > > how can i force jumble to use the testsuite? > > is there anywhere an example for that? > > > > greetings > > gregor > > > > ------------------------------------------------------------------------------ > > Enable your software for Intel(R) Active Management Technology to > > meet the > > growing manageability and security demands of your customers. > > Businesses > > are taking advantage of Intel(R) vPro (TM) technology - will your > > software > > be a part of the solution? Download the Intel(R) Manageability > > Checker > > today! http://p.sf.net/sfu/intel-dev2devmar > > _______________________________________________ > > Jumble-devel mailing list > > Jum...@li... > > https://lists.sourceforge.net/lists/listinfo/jumble-devel > > |
From: Gregor B. <gre...@gm...> - 2011-03-22 17:33:47
|
hi, i have a testsuite for my class that i like to test with jumble. with only one testclass it worked perfect but with the suite get the following error: ERROR: RingBufferTest is not a test class. the testsuite looks like: @RunWith(Suite.class) @Suite.SuiteClasses( { RingBufferTestMutation.class }) public class RingBufferTest { public static void AllTests() { } } how can i force jumble to use the testsuite? is there anywhere an example for that? greetings gregor |
From: Tiago V. <ti....@gm...> - 2011-03-10 16:46:45
|
Hi, I am trying to use Jumble to assess the quality of my test suite. I am trying to use the eclipse plugin for that. I have ny class under test starting like this: @TestClass("com.puppycrawl.tools.checkstyle.api.FastStackTest") public class FastStack<E> implements Iterable<E> { ... } And when I run "Jumble Class" I get the following log, but no results: Mutating com.puppycrawl.tools.checkstyle.api.FastStack Tests: com.puppycrawl.tools.checkstyle.api.FastStackTest Score: 0% (NO TEST CLASS) Mutation points = 18 What seems to be the problem? Tiago Veloso ti....@gm... |
From: Antony T. <ant...@gm...> - 2010-11-22 22:13:40
|
Hi Sean, Apologies for the late response, I have been away. I am using the release version of Jumble (1_1_0). What I did was to run the Unit test against JumbleUtils that was provided in the Jumble source code itself against the problematic test class file directly (As opposed to using the Jumble Maven plugin build) and it worked. I don't think it's got to do with the JUnit4 Test detection being fragile in this case. I suspect that the problem lies with Maven... although I cannot explain why it is exhibiting such behaviour (Failing the JUnit Test class detection) when I'm using the Jumble jar file within the Jumble Maven plugin that I have created. It could be something to do with the dependencies, but I've removed as many dependencies as I could... I need to investigate this further. I will get back when I have more results or data. Thanks. Regards, Antony On Wed, Nov 17, 2010 at 10:43 PM, Sean A. Irvine <sa...@xt...> wrote: > Are you using the release version or the head version? The JUnit4 > Test detection used to be pretty fragile. It is possible, however, > that we still haven't got this right. If you can send me an > example of a problematic test class, I'll try it against the > version I have. > > Sean. > > Antony Teoh wrote: > >> Hi all, >> I started some development work on the Jumble Maven plugin and got it to >> run Jumble on all the test classes on a particular Maven project, but I have >> an issue now in which Jumble is not recognising the JUnit4 Test classes >> (annotations) and it is classifying them as not a valid Test class. (The >> debugger shows that It fails the isJUnit4TestClass(Class clazz) method under >> JumbleUtils.java) >> >> It worked before on the Jumble Eclipse plugin (When I wrote a script to >> batch execute jumble.jar Eclipse plugin on multiple class files) and >> recognised that particular Test class as a valid test class, so I'm >> perplexed as to why the maven jumble source is not classifying the Test >> class as a valid one (not picking up the @Test annotations correctly). >> >> Any help would be appreciated, and I would be willing to give further >> details on the problem if needed. >> >> Thanks. >> >> Regards, >> >> Antony >> >> >> ------------------------------------------------------------------------ >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Jumble-devel mailing list >> Jum...@li... >> https://lists.sourceforge.net/lists/listinfo/jumble-devel >> > > |
From: Sean A. I. <sa...@xt...> - 2010-11-17 22:43:10
|
Are you using the release version or the head version? The JUnit4 Test detection used to be pretty fragile. It is possible, however, that we still haven't got this right. If you can send me an example of a problematic test class, I'll try it against the version I have. Sean. Antony Teoh wrote: > Hi all, > I started some development work on the Jumble Maven plugin and got it to > run Jumble on all the test classes on a particular Maven project, but I > have an issue now in which Jumble is not recognising the JUnit4 Test > classes (annotations) and it is classifying them as not a valid Test > class. (The debugger shows that It fails the isJUnit4TestClass(Class > clazz) method under JumbleUtils.java) > > It worked before on the Jumble Eclipse plugin (When I wrote a script to > batch execute jumble.jar Eclipse plugin on multiple class files) and > recognised that particular Test class as a valid test class, so I'm > perplexed as to why the maven jumble source is not classifying the Test > class as a valid one (not picking up the @Test annotations correctly). > > Any help would be appreciated, and I would be willing to give further > details on the problem if needed. > > Thanks. > > Regards, > > Antony > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > ------------------------------------------------------------------------ > > _______________________________________________ > Jumble-devel mailing list > Jum...@li... > https://lists.sourceforge.net/lists/listinfo/jumble-devel |
From: henry <hen...@go...> - 2010-11-16 18:44:06
|
On Tue, 2010-11-16 at 13:34 +1300, Mark Utting wrote: > That's a nice approach. And it would be great for Jumble to be able > to report coverage results as well as mutation results. > > How much Jumble speedup are you seeing from this coverage-led mutation > testing? I've only checked comparative performance for a very limited sample of tests (and the measurement of plain jumble times was not very accurate), but the results were pretty decent Test case A = 186 mutations with 28% detection. Jumble : Approx 310 Seconds PIT : 186 seconds Test case B = 38 mutations with 97% detection Jumble : Approx 25 seconds PIT: 13 seconds I suspect jumble will still outperform PIT in some scenarios - particularly where the number of mutations is low, but PIT can be easily optimised for these scenarios. > > Which coverage tool are you using? > PIT embeds it's own pretty basic coverage tool based on code from http://code.google.com/p/javacoveragent/. This was chosen purely on the grounds of simplicity to work with. There's room for some improvement in picking tests if a more advanced coverage tool is swapped in - for example by prioritising tests that have higher branch coverage than others with equal line coverage. > > A second speedup is achieved by using the instrumentation api > to load the mutated class instead of creating a separate > classloader each > Isn't this a misuse of the instrumentation api? The Instrumentation > class docs say: "Since the changes are purely additive, these tools do > not modify application state or behavior. Examples of such benign > tools include monitoring agents, profilers, coverage analyzers, and > event loggers." > > But the mutations that Jumble does definitely do change the behaviour > of the class under test! I guess you are assuming that the > Instrumentation class currently does not check that the changes are > purely additive? But what if future Instrumentation classes do check > this? To be honest it's not something I've given much thought to. My reading of the above text in its full context is that these are suggested usages for the API rather than that other usages are precluded. There is certainly plenty of other code out there using the instrumentation api to modify class behaviour (off the top of my head java rebel and JMockit come to mind and I suspect some of the aspect oriented and dynamic jvm language people will also be taking this approach). I think it's very unlikely that a jvm would try and enforce a "no change to class behaviour" policy. Firstly I don't see what the motivation would be to do so, and secondly I don't see how you code it. I do however have a long and interesting history of being wrong about things. > > So you're talking about a mutation that is executed by test T1, but > doesn't cause T1 to fail. However, the mutation does corrupt/poison > some static fields in the class under test that then causes a later > test T2 to fail (even though T2 does not execute the mutation). Actually I was trying to express the opposite problem. Any static state will be setup by the unmutated class. The problem is therefore that later mutations that would have affected static state during initialisation won't, and hence won't be detected. I agree that this probably isn't a big issue, and it would be pretty straight forward to make it configurable. Cheers Henry |
From: Mark U. <ma...@cs...> - 2010-11-16 00:34:22
|
Henry, On 16 November 2010 12:08, henry <hen...@go...> wrote: > I've been playing around quite a lot with Jumble recently, integrating > it with a project I've been working for running test suites in parallel > and distributed across multiple machines. > > As part of this I've been looking at ways of speeding up Jumble. After > trying out a lot of fairly bad ideas I've now managed to achieve a > significant speedup that may be of interest, using what I'm calling > coverage lead mutation testing. > That's a nice approach. And it would be great for Jumble to be able to report coverage results as well as mutation results. How much Jumble speedup are you seeing from this coverage-led mutation testing? > > The idea is pretty simple. Before running the mutation tests perform a > code coverage analysis and create a map of which individual tests hit > each line of code in the class to be mutated. Only those tests that > actually hit the mutated line are then run. > > In the extreme case this means that for a mutation on a line with no > test coverage no tests will be run (compared with standard jumble which > must run all the tests). Even if the tests have 100% line coverage, this > approach should on average need to run fewer tests in order to detect a > mutation. > Which coverage tool are you using? > > A second speedup is achieved by using the instrumentation api to load > the mutated class instead of creating a separate classloader each time. > This avoids the overhead of classloading plus wasted cpu cycles due to > permgen errors. > Isn't this a misuse of the instrumentation api? The Instrumentation class docs say: "Since the changes are purely additive, these tools do not modify application state or behavior. Examples of such benign tools include monitoring agents, profilers, coverage analyzers, and event loggers." But the mutations that Jumble does definitely do change the behaviour of the class under test! I guess you are assuming that the Instrumentation class currently does not check that the changes are purely additive? But what if future Instrumentation classes do check this? > There is a disadvantage to this however - the system is unable to detect > mutations that would have altered any static state that persists between > individual tests. The simplest case of this (static initializers) can be > easily detected and dealt with, but detecting methods that are called > when constructing singletons etc is probably not practical. > So you're talking about a mutation that is executed by test T1, but doesn't cause T1 to fail. However, the mutation does corrupt/poison some static fields in the class under test that then causes a later test T2 to fail (even though T2 does not execute the mutation). That does sound pretty indirect - maybe this should be regarded as a *feature*? :-) ie. "it encourages a testing style where mutations must be detected directly, not via their side effects on later tests." Anyway, one could always turn this --coverage-led option off and run a full Jumble pass (more slowly) if you want to be sure of detecting these subtle kinds of mutations. So long as the limitations of the --coverage-led approach are explained clearly, I don't think this is a major problem. Cheers Mark |
From: henry <hen...@go...> - 2010-11-15 23:08:58
|
I've been playing around quite a lot with Jumble recently, integrating it with a project I've been working for running test suites in parallel and distributed across multiple machines. As part of this I've been looking at ways of speeding up Jumble. After trying out a lot of fairly bad ideas I've now managed to achieve a significant speedup that may be of interest, using what I'm calling coverage lead mutation testing. The idea is pretty simple. Before running the mutation tests perform a code coverage analysis and create a map of which individual tests hit each line of code in the class to be mutated. Only those tests that actually hit the mutated line are then run. In the extreme case this means that for a mutation on a line with no test coverage no tests will be run (compared with standard jumble which must run all the tests). Even if the tests have 100% line coverage, this approach should on average need to run fewer tests in order to detect a mutation. A second speedup is achieved by using the instrumentation api to load the mutated class instead of creating a separate classloader each time. This avoids the overhead of classloading plus wasted cpu cycles due to permgen errors. There is a disadvantage to this however - the system is unable to detect mutations that would have altered any static state that persists between individual tests. The simplest case of this (static initializers) can be easily detected and dealt with, but detecting methods that are called when constructing singletons etc is probably not practical. There is still room to improve the performance - tests are currently run in random order rather than in execution time order, and tests currently still execute after the threshold has been breached. If anyone is interested the code is available at http://code.google.com/p/pitestrunner/. It's still in a fairly raw state, so expect bugs, but I'd appreciate any feedback. Thanks Henry |