Thread: [Ctags-devel] how to run the regression tests
Brought to you by:
dhiebert
From: Elliott H. <en...@je...> - 2007-05-28 17:36:01
|
i saw a comment in the svn log suggesting that it's not obvious how to run the regression tests. there are a couple of problems i've come across. 1. the "testing.mak" makefile assumes that the "ctags" on your path is the version you want to test, and the "ctags.ref" on your path is the reference copy. i've edited my copy of this file so that CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. this lets me use my development version automatically, and avoids embarrassing accidents with other versions that might be on my path. 2. ctags 5.6 has several known infinite-loop bugs, and the Subversion repository contains test cases for these bugs. so you can't use 5.6 as your reference copy. every time i'm happy with some changes, i copy my "dctags" over my "ctags.ref". one thing that's awkward is that adding a test case and fixing a bug is likely to cause the regression test to "fail", but there's no way (except trust) to signal to others that this is intended and acceptable. (so someone needs to check the new Test/simple.py is correct.) --elliott |
From: Elias P. <el...@us...> - 2007-05-28 17:48:58
|
On Mon, 2007-05-28 at 10:35 -0700, Elliott Hughes wrote: > i saw a comment in the svn log suggesting that it's not obvious how > to run the regression tests. > > there are a couple of problems i've come across. > > 1. the "testing.mak" makefile assumes that the "ctags" on your path > is the version you want to test, and the "ctags.ref" on your path is > the reference copy. i've edited my copy of this file so that > CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. this > lets me use my development version automatically, and avoids > embarrassing accidents with other versions that might be on my path. Thanks for explaining this again. Is there a website with this explanation? If not, I think it would be a good idea to make one (just copy&paste your mail there), and then a note in README (it's where i searched for the info) like: For developer infomation, see <URL> > 2. ctags 5.6 has several known infinite-loop bugs, and the Subversion > repository contains test cases for these bugs. so you can't use 5.6 > as your reference copy. every time i'm happy with some changes, i > copy my "dctags" over my "ctags.ref". > > one thing that's awkward is that adding a test case and fixing a bug > is likely to cause the regression test to "fail", but there's no way > (except trust) to signal to others that this is intended and > acceptable. (so someone needs to check the new Test/simple.py is > correct.) > The python changes look good to me, the diffs only contain + lines for the newly recognized tags AFAICS, as it should be. -- Elias Pschernig <el...@us...> |
From: Darren H. <dhi...@us...> - 2007-05-30 03:33:03
|
On May 28, 2007, at 12:35 PM, Elliott Hughes wrote: > i saw a comment in the svn log suggesting that it's not obvious how > to run the regression tests. I posted a quick-start on this to the list as one of the first messages: <http://sourceforge.net/mailarchive/forum.php? thread_name=200609212235.51617.dhiebert% 40users.sourceforge.net&forum_name=ctags-devel> This was updated/corrected in the last message on this thread. > there are a couple of problems i've come across. > > 1. the "testing.mak" makefile assumes that the "ctags" on your path > is the version you want to test, and the "ctags.ref" on your path is > the reference copy. i've edited my copy of this file so that > CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. this > lets me use my development version automatically, and avoids > embarrassing accidents with other versions that might be on my path. I didn't count on other having multiple version of ctags in their path. I normally have "." first in my path, and since I am working and building ctags in my current directory, I never run into the problem you encountered. I don't see a way around your problem without the developer having to modify the makefile. > 2. ctags 5.6 has several known infinite-loop bugs, and the Subversion > repository contains test cases for these bugs. so you can't use 5.6 > as your reference copy. every time i'm happy with some changes, i > copy my "dctags" over my "ctags.ref". Yes, I've never been sure what to do about this. However, the quick- start I referred to above stated that copying over an updated ctags over ctags.ref was the intended procedure. > one thing that's awkward is that adding a test case and fixing a bug > is likely to cause the regression test to "fail", but there's no way > (except trust) to signal to others that this is intended and > acceptable. (so someone needs to check the new Test/simple.py is > correct.) Again, this was discussed in my quick-start. It depends upon the developer making the change to vet the changes for acceptance or rejection. If they are accepted, then that version of ctags becomes the reference standard (ctags.ref) for future checks. The regression test is really to detect changes in the output. Typical regression tests are hard to define here because we have a hard time defining what is "correct". Correct is the output desired, which may change as one learns more (i.e. changes the code to produce more desirable output). Therefore, our regression tests detect changes to the output which must be approved or rejected by the developer. The tests really isolate what changed to focus the examination. I am open to suggestions of a better way to do this. Darren -- Darren Hiebert http://darrenhiebert.com |
From: Elliott H. <en...@je...> - 2007-05-30 17:42:17
|
On 2007-05-29, at 20:32, Darren Hiebert wrote: > On May 28, 2007, at 12:35 PM, Elliott Hughes wrote: > >> i saw a comment in the svn log suggesting that it's not obvious how >> to run the regression tests. > > I posted a quick-start on this to the list as one of the first > messages: > > <http://sourceforge.net/mailarchive/forum.php? > thread_name=200609212235.51617.dhiebert% > 40users.sourceforge.net&forum_name=ctags-devel> > > This was updated/corrected in the last message on this thread. maybe we should check it in? i'm not sure anyone ever reads them, but there's a reasonable tradition of "HACKING" files. at least it would be something you could type from memory when someone next asks ;-) >> there are a couple of problems i've come across. >> >> 1. the "testing.mak" makefile assumes that the "ctags" on your path >> is the version you want to test, and the "ctags.ref" on your path is >> the reference copy. i've edited my copy of this file so that >> CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. this >> lets me use my development version automatically, and avoids >> embarrassing accidents with other versions that might be on my path. > > I didn't count on other having multiple version of ctags in their > path. I normally have "." first in my path, and since I am working > and building ctags in my current directory, I never run into the > problem you encountered. I don't see a way around your problem > without the developer having to modify the makefile. we could commit the "./" change. (or use GNU make fanciness to say "same directory as the makefile" rather than "current directory", if we want the freedom to cd in rules.) this would, as i understand you, make no difference to your PATH- with-. case and fix my paranoid-sysadmin no-.-in-PATH case, and fix other people's .-in-PATH-but-not-first cases. actually, it would probably save you a step because you wouldn't need to rename/link ctags to dctags. if you like, i can write the GNU make magic to say "./ctags if it exists, otherwise ./dctags". i think making "make test" work out of the box for as close to "everyone" as we can manage would be worthwhile. >> 2. ctags 5.6 has several known infinite-loop bugs, and the Subversion >> repository contains test cases for these bugs. so you can't use 5.6 >> as your reference copy. every time i'm happy with some changes, i >> copy my "dctags" over my "ctags.ref". > > Yes, I've never been sure what to do about this. However, the quick- > start I referred to above stated that copying over an updated ctags > over ctags.ref was the intended procedure. it wasn't the copying that i meant was awkward so much as the fact that sometimes i need a "ctags.ref" that i don't have, perhaps because someone else committed changes. (though that's partially self- inflicted by the fact that i run a nightly build. if i only ever manually updated, this wouldn't happen unless i'm forced to update to resolve a merge.) > I am open to suggestions of a better way to do this. i can't think of any, short of committing model answers. but there aren't many developers, and i think we're all running the tests, so it's not a big problem at the moment. not worth the hassle of changing this. --elliott |
From: Darren H. <dhi...@us...> - 2007-05-31 03:41:20
|
Elliot, Good points all around. Lest I make myself misunderstood, I did not intend that what I posted as a quick-start to be the "set in stone" way of doing things. It was intended more as a "state of the union" report in order to get folks going. That said, it has been my intent to completely redo the development makefile. This has become necessary now that the code is in SVN and that my primary machine is now a Mac. All the other issues you developers have raised shows how that is needed. This is my primary goal in order to make the next release. It (and its quirkiness) was OK when only I used it, but it needs to grow up now. Darren On May 30, 2007, at 10:36 AM, Elliott Hughes wrote: > On 2007-05-29, at 20:32, Darren Hiebert wrote: >> On May 28, 2007, at 12:35 PM, Elliott Hughes wrote: >> >>> i saw a comment in the svn log suggesting that it's not obvious how >>> to run the regression tests. >> >> I posted a quick-start on this to the list as one of the first >> messages: >> >> <http://sourceforge.net/mailarchive/forum.php? >> thread_name=200609212235.51617.dhiebert% >> 40users.sourceforge.net&forum_name=ctags-devel> >> >> This was updated/corrected in the last message on this thread. > > maybe we should check it in? i'm not sure anyone ever reads them, > but there's a reasonable tradition of "HACKING" files. at least it > would be something you could type from memory when someone next > asks ;-) > >>> there are a couple of problems i've come across. >>> >>> 1. the "testing.mak" makefile assumes that the "ctags" on your path >>> is the version you want to test, and the "ctags.ref" on your path is >>> the reference copy. i've edited my copy of this file so that >>> CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. >>> this >>> lets me use my development version automatically, and avoids >>> embarrassing accidents with other versions that might be on my path. >> >> I didn't count on other having multiple version of ctags in their >> path. I normally have "." first in my path, and since I am working >> and building ctags in my current directory, I never run into the >> problem you encountered. I don't see a way around your problem >> without the developer having to modify the makefile. > > we could commit the "./" change. (or use GNU make fanciness to say > "same directory as the makefile" rather than "current directory", > if we want the freedom to cd in rules.) > > this would, as i understand you, make no difference to your PATH- > with-. case and fix my paranoid-sysadmin no-.-in-PATH case, and fix > other people's .-in-PATH-but-not-first cases. actually, it would > probably save you a step because you wouldn't need to rename/link > ctags to dctags. if you like, i can write the GNU make magic to say > "./ctags if it exists, otherwise ./dctags". > > i think making "make test" work out of the box for as close to > "everyone" as we can manage would be worthwhile. > >>> 2. ctags 5.6 has several known infinite-loop bugs, and the >>> Subversion >>> repository contains test cases for these bugs. so you can't use 5.6 >>> as your reference copy. every time i'm happy with some changes, i >>> copy my "dctags" over my "ctags.ref". >> >> Yes, I've never been sure what to do about this. However, the >> quick-start I referred to above stated that copying over an >> updated ctags over ctags.ref was the intended procedure. > > it wasn't the copying that i meant was awkward so much as the fact > that sometimes i need a "ctags.ref" that i don't have, perhaps > because someone else committed changes. (though that's partially > self-inflicted by the fact that i run a nightly build. if i only > ever manually updated, this wouldn't happen unless i'm forced to > update to resolve a merge.) > >> I am open to suggestions of a better way to do this. > > i can't think of any, short of committing model answers. but there > aren't many developers, and i think we're all running the tests, so > it's not a big problem at the moment. not worth the hassle of > changing this. > > --elliott > > -- Darren Hiebert http://darrenhiebert.com |
From: Elliott H. <en...@je...> - 2007-06-04 16:04:58
|
> That said, it has been my intent to completely redo the development makefile. > This has become necessary now that the code is in SVN and that my primary > machine is now a Mac. i've been building from Subversion on a Mac since there was a Subversion repository, and on Mac OS X for years before that. what's the problem? or do you want to automatically build universal binaries on Mac OS? i've used something like this on my PowerPC Macs: CFLAGS="-isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc" make see also: http://developer.apple.com/technotes/tn2005/tn2137.html i've not yet seen a project that automatically builds universal binaries out of the box, though it would be pretty cool to do so. --elliott On 2007-05-30, at 20:41, Darren Hiebert wrote: > Elliot, > > Good points all around. Lest I make myself misunderstood, I did not > intend that what I posted as a quick-start to be the "set in stone" > way of doing things. It was intended more as a "state of the union" > report in order to get folks going. > > That said, it has been my intent to completely redo the development > makefile. This has become necessary now that the code is in SVN and > that my primary machine is now a Mac. All the other issues you > developers have raised shows how that is needed. This is my primary > goal in order to make the next release. It (and its quirkiness) was > OK when only I used it, but it needs to grow up now. > > Darren > > On May 30, 2007, at 10:36 AM, Elliott Hughes wrote: > >> On 2007-05-29, at 20:32, Darren Hiebert wrote: >>> On May 28, 2007, at 12:35 PM, Elliott Hughes wrote: >>> >>>> i saw a comment in the svn log suggesting that it's not obvious how >>>> to run the regression tests. >>> >>> I posted a quick-start on this to the list as one of the first >>> messages: >>> >>> <http://sourceforge.net/mailarchive/forum.php? >>> thread_name=200609212235.51617.dhiebert% >>> 40users.sourceforge.net&forum_name=ctags-devel> >>> >>> This was updated/corrected in the last message on this thread. >> >> maybe we should check it in? i'm not sure anyone ever reads them, >> but there's a reasonable tradition of "HACKING" files. at least it >> would be something you could type from memory when someone next >> asks ;-) >> >>>> there are a couple of problems i've come across. >>>> >>>> 1. the "testing.mak" makefile assumes that the "ctags" on your path >>>> is the version you want to test, and the "ctags.ref" on your >>>> path is >>>> the reference copy. i've edited my copy of this file so that >>>> CTAGS_TEST is "./dctags" and "./ctags.ref" is my reference copy. >>>> this >>>> lets me use my development version automatically, and avoids >>>> embarrassing accidents with other versions that might be on my >>>> path. >>> >>> I didn't count on other having multiple version of ctags in their >>> path. I normally have "." first in my path, and since I am >>> working and building ctags in my current directory, I never run >>> into the problem you encountered. I don't see a way around your >>> problem without the developer having to modify the makefile. >> >> we could commit the "./" change. (or use GNU make fanciness to say >> "same directory as the makefile" rather than "current directory", >> if we want the freedom to cd in rules.) >> >> this would, as i understand you, make no difference to your PATH- >> with-. case and fix my paranoid-sysadmin no-.-in-PATH case, and >> fix other people's .-in-PATH-but-not-first cases. actually, it >> would probably save you a step because you wouldn't need to rename/ >> link ctags to dctags. if you like, i can write the GNU make magic >> to say "./ctags if it exists, otherwise ./dctags". >> >> i think making "make test" work out of the box for as close to >> "everyone" as we can manage would be worthwhile. >> >>>> 2. ctags 5.6 has several known infinite-loop bugs, and the >>>> Subversion >>>> repository contains test cases for these bugs. so you can't use 5.6 >>>> as your reference copy. every time i'm happy with some changes, i >>>> copy my "dctags" over my "ctags.ref". >>> >>> Yes, I've never been sure what to do about this. However, the >>> quick-start I referred to above stated that copying over an >>> updated ctags over ctags.ref was the intended procedure. >> >> it wasn't the copying that i meant was awkward so much as the fact >> that sometimes i need a "ctags.ref" that i don't have, perhaps >> because someone else committed changes. (though that's partially >> self-inflicted by the fact that i run a nightly build. if i only >> ever manually updated, this wouldn't happen unless i'm forced to >> update to resolve a merge.) >> >>> I am open to suggestions of a better way to do this. >> >> i can't think of any, short of committing model answers. but there >> aren't many developers, and i think we're all running the tests, >> so it's not a big problem at the moment. not worth the hassle of >> changing this. >> >> --elliott >> >> > > -- > Darren Hiebert > http://darrenhiebert.com > > > |
From: Darren H. <dhi...@us...> - 2007-06-08 02:59:22
|
The problems have more to do with the fact that the developer makefile in use for ctags was based upon my own Linux-based environment. For example, it contains targets that still use CVS to make releases, as well as one of the tests was generating tags for the Linux kernel tree, which was assumed to be where it is placed on Fedora Linux. I didn't mean building universal binaries, since I don't distribute any (or at least haven't until now). And, oh, how I wish that valgrind was ported to the Mac! I see that its web site lists it as high priority to do so (for the Intel Mac). Maybe I need to build support for linking against dmalloc into the Makefile for those developer not on Linux. Darren On Jun 4, 2007, at 11:04 AM, Elliott Hughes wrote: > > That said, it has been my intent to completely redo the > development makefile. > > This has become necessary now that the code is in SVN and that my > primary > > machine is now a Mac. > > i've been building from Subversion on a Mac since there was a > Subversion repository, and on Mac OS X for years before that. > what's the problem? or do you want to automatically build universal > binaries on Mac OS? i've used something like this on my PowerPC Macs: > > CFLAGS="-isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 - > arch ppc" make > > see also: > > http://developer.apple.com/technotes/tn2005/tn2137.html > > i've not yet seen a project that automatically builds universal > binaries out of the box, though it would be pretty cool to do so. > > --elliott > > On 2007-05-30, at 20:41, Darren Hiebert wrote: > >> Elliot, >> >> Good points all around. Lest I make myself misunderstood, I did >> not intend that what I posted as a quick-start to be the "set in >> stone" way of doing things. It was intended more as a "state of >> the union" report in order to get folks going. >> >> That said, it has been my intent to completely redo the >> development makefile. This has become necessary now that the code >> is in SVN and that my primary machine is now a Mac. All the other >> issues you developers have raised shows how that is needed. This >> is my primary goal in order to make the next release. It (and its >> quirkiness) was OK when only I used it, but it needs to grow up now. -- Darren Hiebert http://darrenhiebert.com |
From: Elliott H. <en...@je...> - 2007-06-08 05:26:21
|
On Thu, 2007-06-07 at 21:59 -0500, Darren Hiebert wrote: > The problems have more to do with the fact that the developer > makefile in use for ctags was based upon my own Linux-based > environment. For example, it contains targets that still use CVS to > make releases, as well as one of the tests was generating tags for > the Linux kernel tree, which was assumed to be where it is placed on > Fedora Linux. ah, i wondered why the funny location. Debian/Ubuntu use /usr/src/linux-source-*/. > I didn't mean building universal binaries, since I don't distribute > any (or at least haven't until now). > > And, oh, how I wish that valgrind was ported to the Mac! I see that > its web site lists it as high priority to do so (for the Intel Mac). you could always use BootCamp or Parallels to run Ubuntu in a VM. assuming Valgrind isn't slow enough already ;-) > Maybe I need to build support for linking against dmalloc into the > Makefile for those developer not on Linux. on Mac OS you just need to set the right environment variables. try: MallocHelp=1 ./dctags or "man malloc". --elliott > Darren > > On Jun 4, 2007, at 11:04 AM, Elliott Hughes wrote: > > > > That said, it has been my intent to completely redo the > > development makefile. > > > This has become necessary now that the code is in SVN and that my > > primary > > > machine is now a Mac. > > > > i've been building from Subversion on a Mac since there was a > > Subversion repository, and on Mac OS X for years before that. > > what's the problem? or do you want to automatically build universal > > binaries on Mac OS? i've used something like this on my PowerPC Macs: > > > > CFLAGS="-isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 - > > arch ppc" make > > > > see also: > > > > http://developer.apple.com/technotes/tn2005/tn2137.html > > > > i've not yet seen a project that automatically builds universal > > binaries out of the box, though it would be pretty cool to do so. > > > > --elliott > > > > On 2007-05-30, at 20:41, Darren Hiebert wrote: > > > >> Elliot, > >> > >> Good points all around. Lest I make myself misunderstood, I did > >> not intend that what I posted as a quick-start to be the "set in > >> stone" way of doing things. It was intended more as a "state of > >> the union" report in order to get folks going. > >> > >> That said, it has been my intent to completely redo the > >> development makefile. This has become necessary now that the code > >> is in SVN and that my primary machine is now a Mac. All the other > >> issues you developers have raised shows how that is needed. This > >> is my primary goal in order to make the next release. It (and its > >> quirkiness) was OK when only I used it, but it needs to grow up now. > > -- > Darren Hiebert > http://darrenhiebert.com > > > -- Elliott Hughes, http://www.jessies.org/~enh/ |