Bitcoin

Although Jenkins may not be the best system, we already have jenkins and
pull-tester (which is a dumb python script I wrote to test all incoming
pull requests from github).  

They both run the same set of scripts, namely those at
https://github.com/TheBlueMatt/test-scripts (its pretty basic right now,
but since it is on github, I was hoping someone would find the
inspiration to add to it).

I dont really care if we keep using jenkins, but I figure we might as
well keep all the tests in one place?

Anyway, I'm all for more testing (I'm always complaining about how we
need more tests for stuff...).

Matt

On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
> Hi All,
> 
> After the failure to get any real testing done for the 0.7 release (all
> of which is my fault) I have decided to rejig things.
> 
> I am heavily into test driven development, and I have a strong
> background in requirements management, and automation.
> 
> I want to leave bettermeans behind, maybe we might be able to keep the
> voting aspect of it, and link it into mantis.
> 
> So, what I have been doing over the past few weeks is developing a
> rudimentary requirements set, basic requirement tracking, tests to
> prove/stress the requirements.
> 
> The next most important thing is to get release/acceptance tests done -
> these primarily focus on new stuff doesnt break old (ie lose a wallet,
> etc) and needs no special requirements.
> 
> To this end I have installed various opensource applications (mantis,
> salomeTMF, bugzilla, etc) and am currently evaluating the best workflow
> process.
> 
> This was a much longer post, but decided against it. :)
> 
> So, what I want to know is who wants to be a part helping me out with
> all this? I am finalising the workflow flow diagrams and they should be
> ready for inspection soon.
> 
> Anyone interested in helping out/reviewing processes? even if it is just
> some encouragement, it is all greatly appreciated.
> 
> Drop me an email if you want access to the current setup and help me
> review the different software for the bitcoin workflow process.
> 
> cheers,
> 
> steve

on 09/25/2012 02:32 PM steve said the following:
> Anyone interested in helping out/reviewing processes? even if it is just
> some encouragement, it is all greatly appreciated.

not enough time in the day for me to seriously help out, but since you
asked, here's some encouragement. :) more testing == good.

This is definitely worth doing and I wish you every encouragement.

For my part I'm working on a different area of the Bitcoin ecosystem and
that is taking up all my time so I can only cheer you on from the sidelines.

On 25 September 2012 21:49, Daniel F <nanotube@...> wrote:

> on 09/25/2012 02:32 PM steve said the following:
> > Anyone interested in helping out/reviewing processes? even if it is just
> > some encouragement, it is all greatly appreciated.
>
> not enough time in the day for me to seriously help out, but since you
> asked, here's some encouragement. :) more testing == good.
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@...
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

I think it's a great initiative Steve,

On Tue, Sep 25, 2012 at 10:41 PM, Matt Corallo <bitcoin-list@...> wrote:

> They both run the same set of scripts, namely those at
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right now,
> but since it is on github, I was hoping someone would find the
> inspiration to add to it).

The pulltester and Jenkins are really useful, I think it's best to
keep these and improve them if needed.

> I dont really care if we keep using jenkins, but I figure we might as
> well keep all the tests in one place?

Which brings me to another issue: we desperately need a page that
links all the developer resources for bitcoin dev:

- link to my doxygen docs (https://dev.visucore.com/bitcoin/doxygen/)
- bluematt's pulltester and jenkins
- any testing dashboards Steve is going to add

But also:

- how to check out the source code, how to find the build instructions
for your platform
- contribution guidelines
- where to ask w/ development problems (this mailing list, #bitcoin-dev)
- where to begin, which bugs to solve first
...

Maybe on bitcoin.org, but at least easy to find for people that want
to contribute, not just some deeply nested wiki page. Ie, something
like

https://www.mozilla.org/en-US/contribute/
http://www.ogre3d.org/developers
...

I'm willing to write this. But I know these kinds of proposals always
end in a big discussion about what should be and what should not be on
bitcoin.org, however we should be a bit pragmatic here.

Wladimir

on 09/26/2012 01:49 AM Wladimir said the following:

> I'm willing to write this. But I know these kinds of proposals always
> end in a big discussion about what should be and what should not be on
> bitcoin.org, however we should be a bit pragmatic here.

May I suggest a page bitcoin.org/developers, that links to a wiki page
of developer resources?
That way there's an easy link from the main site, but the content is
readily editable and expandable.

On Wednesday, September 26, 2012 11:41:13 AM Daniel F wrote:
> on 09/26/2012 01:49 AM Wladimir said the following:
> > I'm willing to write this. But I know these kinds of proposals always
> > end in a big discussion about what should be and what should not be on
> > bitcoin.org, however we should be a bit pragmatic here.
> 
> May I suggest a page bitcoin.org/developers, that links to a wiki page
> of developer resources?
> That way there's an easy link from the main site, but the content is
> readily editable and expandable.

The front page already has wiki links. Adding a direct link to a developer 
resources page there would probably make sense.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Matt,

Glad to have another ninja onboard :)

On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).

I have never heard of jenkins before.  I need to do some more digging.
is this the right thing?

https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin

Mantis on the other hand, I know exceptionally well.  I hate
duplication of work/data unless absolutely necessary.  I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)

So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?

Adding to this doesnt make sense.  Each one of these reporting methods
is for a different thing.  I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns].  Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.

The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.

> 
> They both run the same set of scripts, namely those at 
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).

I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases.  have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.

This is the outline of the testing that I setup for 0.7

https://secure.bettermeans.com/projects/4256/wiki

> 
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?

Yes, I would love to unify all build testing and testcases into one
place.  I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)

2 - Acceptance based testcases - these are testcases that should be
run for every build.  Check out the General Acceptance Tests in the
wiki link for examples and testcases

3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)

These three things alone are a massive task. and they still wont cover
everything.  I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])

Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.

> 
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).

Nice, I love testing.  I think we will get on :)

And I would rather go for interoperability between testing rather than
rewriting it all.

Cheers,

steve

> 
> Matt
> 
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>> 
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>> 
>> I am heavily into test driven development, and I have a strong 
>> background in requirements management, and automation.
>> 
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>> 
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>> 
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>> 
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>> 
>> This was a much longer post, but decided against it. :)
>> 
>> So, what I want to know is who wants to be a part helping me out
>> with all this? I am finalising the workflow flow diagrams and
>> they should be ready for inspection soon.
>> 
>> Anyone interested in helping out/reviewing processes? even if it
>> is just some encouragement, it is all greatly appreciated.
>> 
>> Drop me an email if you want access to the current setup and help
>> me review the different software for the bitcoin workflow
>> process.
>> 
>> cheers,
>> 
>> steve
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT
BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb
M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP
y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/
xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI
bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4=
=W5xI
-----END PGP SIGNATURE-----

Steve,

> So, currently there are 4 potential places for bugs to be reported
> 1 - jenkins (and unit tests)
> 2 - git
> 3 - mailing list
> 4 - forum (bitcointalk...)
> 5? - is there still the ability to add bugs via sourceforge?

Currently github is the authoritative place to report issues. When
someone reports a bug on the mailing list, IRC or forum, they are
generally asked to make a github issue (or, someone else makes the
issue for them). Failed tests are generally also reported on github,
by the pull tester.

We currently have 232 issues, mostly classified into categories such
as "Bug", "Improvement", "GUI", "Wallet", and so on.

Also it's easy to refer to github issues in commits with #123, with
automatic linking.

I'm not sure it is worth the effort to move to another system
(especially if you need a another login etc...). But I'm probably
misunderstanding what you're trying to do.

Wladimir

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26/09/2012 13:49, Wladimir wrote:
> Steve,

Hi Wladimir,

> 
>> So, currently there are 4 potential places for bugs to be 
>> reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 
>> - forum (bitcointalk...) 5? - is there still the ability to add 
>> bugs via sourceforge?
> 
> Currently github is the authoritative place to report issues. When
>  someone reports a bug on the mailing list, IRC or forum, they are
>  generally asked to make a github issue (or, someone else makes the
>  issue for them). Failed tests are generally also reported on 
> github, by the pull tester.

excellent, that makes things much easier.

> 
> We currently have 232 issues, mostly classified into categories 
> such as "Bug", "Improvement", "GUI", "Wallet", and so on.
> 
> Also it's easy to refer to github issues in commits with #123, with
> automatic linking.
> 
> I'm not sure it is worth the effort to move to another system 
> (especially if you need a another login etc...). But I'm probably 
> misunderstanding what you're trying to do.

I think you might be misunderstanding a little. I am not trying to
replace the current system, I need to make sure that what I do will be
compatible with it (seamlessly so for the developer). I do not want this
to generate extra work for the development team.

However testing is a lot more than just bug reporting, dont get me
wrong bug reports are important, but so is running a testcase and that
testcase passing, especially if that testcase is linked to the proof
of a requirement. I am trying to develop a qa environment that is
conducive to testing and will allow the testers to shine in all their
glory :) and we need different tools and methodologies.

Git is too developer centric to be useful for organising testing. -
however there is a large amount of software that is compatible with
git, so the core development team only ever need to work with git.

The linking between a bug, the requirement, the fix, the retest, and
updating of regression testplan is vital. So is the ability to
organise testing campaigns and assigning tests, work units and test
relevant docs/scripts/ideas, etc.

I hope this clears things up a bit?

Cheers,

steve

> 
> Wladimir
> 

- -----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/
2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN
ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg
G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/
ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq
x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0=
=iHI1
- -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip
yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv
EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8
oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P
9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP
yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc=
=yP7N
-----END PGP SIGNATURE-----

Running a concurrent Mantis tracker would be confusing and fragment the
development pathway. We have an issue tracker; it's on github.

What's being talked about here are two separate things. Jenkins is a
continuous integration system. It can be configured to run the suite of
unit tests, regression tests, and any kind of automated functional tests
for every commit on github and every pull request.

Github is our issue tracker. Github, and only github, is where new issues
should be reported (unless it's security related, in which case an email
should be sent to the core devs directly).

Certainly developers should be responsible for making sure that regression
tests for bugs they fix make it either into the unit tests or Matt's
functional test repository. QA should hold them accountable for that
(re-opening tickets for bugs that have been fixed but without regression
tests).

The other thing we're talking about is coordinated release testing--getting
release candidates in the hands of actual users and making sure that issues
are reported. This is something that can't be automated as the point really
is to pick up on things that the testing suite missed. You sound more
qualified than me for coming up with a process, but in the end discovered
issues should be reported to github, the final repository of issues that
hold up Gavin from doing a release.

Just my 0.002BTC
Mark

On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@...> wrote:
>
> I think you might be misunderstanding a little. I am not trying to
> replace the current system, I need to make sure that what I do will be
> compatible with it (seamlessly so for the developer). I do not want this
> to generate extra work for the development team.
>
> However testing is a lot more than just bug reporting, dont get me
> wrong bug reports are important, but so is running a testcase and that
> testcase passing, especially if that testcase is linked to the proof
> of a requirement. I am trying to develop a qa environment that is
> conducive to testing and will allow the testers to shine in all their
> glory :) and we need different tools and methodologies.
>
> Git is too developer centric to be useful for organising testing. -
> however there is a large amount of software that is compatible with
> git, so the core development team only ever need to work with git.
>
> The linking between a bug, the requirement, the fix, the retest, and
> updating of regression testplan is vital. So is the ability to
> organise testing campaigns and assigning tests, work units and test
> relevant docs/scripts/ideas, etc.
>
> I hope this clears things up a bit?
>
> Cheers,
>
> steve
>

On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach <mark@...> wrote:
> Certainly developers should be responsible for making sure that regression
> tests for bugs they fix make it either into the unit tests or Matt's
> functional test repository. QA should hold them accountable for that
> (re-opening tickets for bugs that have been fixed but without regression
> tests).

As a goal or general principle, this is agreeable.

But slavish attention to this will only get ignored.  There is finite
developer resources, and regression tests for certain types of bugs,
like prickly P2P network interaction bugs or RPC API bugs, could
potentially involve many days or weeks of coding, to sufficiently
simulate the environment.  The ability to easily, automatically and
programmatically reproduce certain classes of bugs is simply out of
reach right now, and nobody is going to shut down development to fix
that problem.

We should move towards this direction, yes, but bitcoin test cases are
not always going to be as easy as writing (say) a compiler testcase.

We can always use the help of a few good QA coders:  simulating a P2P
environment and checking the RPC API are two examples of very
complicated problems that -can- be automated for testing... with a lot
of work.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26/09/2012 17:06, Mark Friedenbach wrote:
> Running a concurrent Mantis tracker would be confusing and fragment
> the development pathway. We have an issue tracker; it's on github.

I think you misunderstand what I am proposing.

QA needs more than just an issue tracker. i have yet to find any
opensource software that integrates testcases, nor any method for
generating testplans, nor any method for linking testcases and plans
to requirements, that work with git.

We will need software for this (as well as workflow software) and it
is much easier to integrate this into mantis/bugzilla. They are both
so much more functional than Git.

both mantis and bugzilla have full two way functionality with git.

> 
> What's being talked about here are two separate things. Jenkins is
> a continuous integration system. It can be configured to run the
> suite of unit tests, regression tests, and any kind of automated
> functional tests for every commit on github and every pull
> request.

well 3 but okay.  Jenkins integrates both with mantis (and therefore a
testsuite, etc) and with git. I do not see why anything should be any
different.  again I am not trying to change any current process, just
develop some new ones.

> 
> Github is our issue tracker. Github, and only github, is where new
> issues should be reported (unless it's security related, in which
> case an email should be sent to the core devs directly).

You will only ever receive bug reports via git. How they are entered
should not be of concern.

There will be no space in mantis/zilla for bugs that are not related
to testcases.

> 
> Certainly developers should be responsible for making sure that
> regression tests for bugs they fix make it either into the unit
> tests or Matt's functional test repository. QA should hold them
> accountable for that (re-opening tickets for bugs that have been
> fixed but without regression tests).

I feel very strongly that developers should not do regression testing
or any signoff testing on their own code. QA should do the testing. I
am 50/50 if they should write the testcases. the QA process should
make things easier for the dev team, not generate more work for them.

> 
> The other thing we're talking about is coordinated release
> testing--getting release candidates in the hands of actual users
> and making sure that issues are reported. This is something that
> can't be automated as the point really is to pick up on things that
> the testing suite missed. You sound more qualified than me for
> coming up with a process, but in the end discovered issues should
> be reported to github, the final repository of issues that hold up
> Gavin from doing a release.

All the core development team will still use git. the extra software
is needed by test.

(And the third point was coming up with a suite of tests for 3rd party
developers to test their interoperability - this will having nothing
to do with git, or mantis. But the solution should be compatible with
mantis/zilla)

> 
> Just my 0.002BTC Mark
> 
> On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@...> wrote:
>> 
>> I think you might be misunderstanding a little. I am not trying
>> to replace the current system, I need to make sure that what I do
>> will be compatible with it (seamlessly so for the developer). I
>> do not want this to generate extra work for the development
>> team.
>> 
>> However testing is a lot more than just bug reporting, dont get
>> me wrong bug reports are important, but so is running a testcase
>> and that testcase passing, especially if that testcase is linked
>> to the proof of a requirement. I am trying to develop a qa
>> environment that is conducive to testing and will allow the
>> testers to shine in all their glory :) and we need different
>> tools and methodologies.
>> 
>> Git is too developer centric to be useful for organising testing.
>> - however there is a large amount of software that is compatible
>> with git, so the core development team only ever need to work
>> with git.
>> 
>> The linking between a bug, the requirement, the fix, the retest,
>> and updating of regression testplan is vital. So is the ability
>> to organise testing campaigns and assigning tests, work units and
>> test relevant docs/scripts/ideas, etc.
>> 
>> I hope this clears things up a bit?
>> 
>> Cheers,
>> 
>> steve
>> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYz8CAAoJEFvEB9dQFvtQjkMH/Apa95IRh21mfNIuyK8kOSdt
55tLoT9a6DFyF1IPTgjHQlPN/A0JCPy/p2rIEEL7XzWpCMu1zU8BzBNmJsxGAZJG
C0ue1eDEywKNFEMPTgQdebC2MbNSfUBA6lGJ5vijQlcXKoIuiV/LS7IMYh57T4u1
6Tc/SGypGe8kBLuFTihmIGH5uFS6arNGlcGgh+HRn+O4jKiAcw06lIoKh7S9Rj5e
bmkimvOfproCIZeNQfSJH1BfYZaVVsJ1ouVI7ch6ytFpKsZ622zYF0Iq3042kEEp
Fyqh9pDDNTJ/dwbyFpTx0WaxZySdZfZmQOCxFCAeLaCpop/nKeUnW5fy3i0sYno=
=rfHO
-----END PGP SIGNATURE-----

There are test cases that can be automated. That's Jenkins, and those will
be run automagically.

Then there are tests that cannot be automated; things like "Does the GUI
look OK on all of the platforms that we support (Windows XP/2000/Vista/7/8,
Ubuntu/Debian blah with window managers foo and bar, OSX 10.5/6/7/8)."

Thanks to Matt, we're doing great with automated functional test cases (can
always do better, of course).

We're failing on simple, boring stuff like making sure we actually run on
all of the platforms that we say we run on BEFORE final release. That is
where I think a QA team can add a lot of value.

Steve: I'm worried you're over-designing The Process. A release acceptance
test plan could be nothing more than a step-by-step checklist on a wiki
page, Google Doc, or Drobox shared folder...

-- 
--
Gavin Andresen

On Wed, 2012-09-26 at 13:28 +0100, steve wrote:
> Hi Matt,
> 
> Glad to have another ninja onboard :)
> 
> On 25/09/2012 21:41, Matt Corallo wrote:
> > Although Jenkins may not be the best system, we already have
> > jenkins and pull-tester (which is a dumb python script I wrote to
> > test all incoming pull requests from github).
> 
> I have never heard of jenkins before.  I need to do some more digging.
> is this the right thing?
> 
> https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin
For a mantis plugin, sure, I guess...
> 
> Mantis on the other hand, I know exceptionally well.  I hate
> duplication of work/data unless absolutely necessary.  I will check
> jenkins out (just out of interest what is it actually meant to do? the
> website implies framework, but not what its for)
Jenkins currently just runs the test script after each new commit to
bitcoin (and provides binaries to anyone who wants them), so its pretty
basic (though jenkins has way more features than we use).  The bitcoin
one lives at http://jenkins.bluematt.me/
> 
> > 
> > They both run the same set of scripts, namely those at 
> > https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> > now, but since it is on github, I was hoping someone would find
> > the inspiration to add to it).
> 
> I will check it out. I wrote a very basic script that wikified the
> changelog,
We currently keep a changelog at https://en.bitcoin.it/wiki/Changelog (I
went back and added tons of logs a while back and it got updated, though
0.7 seems to be missing...) anyway, automating that would be nice...
> and linked to the changes and created wiki pages for the
> testcases.  
Having more info on that changelog page would be nice.
> have you seen the stuff I put on bettermeans? bits keep
> vanishing then re appearing.
I have been meaning to catch up with the various attempts at better
bitcoin testing that have started up a few times, but I keep never
getting around to it...
> 
> This is the outline of the testing that I setup for 0.7
> 
> https://secure.bettermeans.com/projects/4256/wiki
> 
> > 
> > I dont really care if we keep using jenkins, but I figure we might
> > as well keep all the tests in one place?
> 
> Yes, I would love to unify all build testing and testcases into one
> place.  I am still on the fence as to including unit tests into this.
> However I do see 3 distinct type of testcases
Even if unit tests are considered separate, having it all run in one
huge test script makes it quite easy to implement new things (like
pull-tester) which test some arbitrary bitcoind commit in the same way
as every other tester.  
> 1 - requirements based testcases (requirements based off the current
> block chain rules - these are edge cases and known interoperability
> issues)
The BitcoinjBitcoindComparisonTool.jar file which is run as a part of
the test scripts tries to hit as many block acceptance edge cases as
possible (I'm sure I missed a ton, but it hits a lot too).  I've also
been pushing alternate implementation implementors to use it to test
their own implementations.
> 
> 2 - Acceptance based testcases - these are testcases that should be
> run for every build.  Check out the General Acceptance Tests in the
> wiki link for examples and testcases
> 
> 3 - Testcases for reference implementations of things (like multisig -
> i see these working like the /test folder when you install a new perl
> module)
> 
> These three things alone are a massive task. and they still wont cover
> everything.  I would like to get the workflow so that people can
> sponsor or donate to a specific campaign (eg a new feature is
> implemented, people want it tested so can donate just for that
> campaign [developing testcases, structure, requirements, etc])
> 
> Once this is done, I will get to do some exciting stuff (like writing
> fuzzers, automation, etc) unfortunately I do not know python, only perl.
As far as I'm concerned more test cases are more test cases, it may get
unwieldy to maintain, but at least we'd have more test cases :)

In terms of general testing strategies, I really prefer to script it
all, jenkins is quite nice in that it can have slave workers using a
different OS which run their own tests and then report back to the main
jenkins instance.  Getting a real Windows slave to run the installer and
test that thoroughly as well as basic Mac things (I know OSX uses a very
different build system...) would be nice (though I dont really have time
to write all those tests...)

re: GUI testing is hard: I've heard Qt's unit test framework is really
powerful and can even include things like click scripting and analysis
of the current views (though, I agree, its still no doubt hard).  

Matt

On Wed, Sep 26, 2012 at 8:53 PM, Matt Corallo <bitcoin-list@...> wrote:
> Jenkins currently just runs the test script after each new commit to
> bitcoin (and provides binaries to anyone who wants them), so its pretty
> basic (though jenkins has way more features than we use).  The bitcoin
> one lives at http://jenkins.bluematt.me/

Jenkins is excellent at cycling through tests,  while additional
external tools may bring some value they're not required.  It's also
essential to automate all tests that we really care are run— with our
small active development group and volunteer contributors the only
tests we can count on being run are the automated ones. Automated
tests included with the software— or at least the source— are also the
only way to have a good chance of catching gnarly platform
interactions.

I think more than talking about testing I think we need is actual
testing. Code coverage from the current tests (e.g. bitcoin-test and a
testnet sync) is very unimpressive, and while coverage isn't some
magical silver bullet and does not, by itself, mean the tests are good
flaws in uncovered code can't be detected by the tests.  We also lack
simple testing cycle documentation for people interested in testing
manually to walk through, etc. I think all the meta discussion is not
very useful until we actually have more substance to put into it.
Otherwise I fear we're just building an airport by painting stripes
and waiting for the planes to land...

If someone wants to help and would like a list of some of the testing
I think would be useful, ping me off-list and I can blast some
suggestions. But I assume that anyone who actually wants to work on
this isn't short of ideas, and at this point "work on what interests
you, report what interesting thing you accomplish or discover" is
probably a perfectly fine level of coordination.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Gavin,

Sorry for the delayed response, I wanted to take a couple of days to
reflect on your email.

On 26/09/2012 19:09, Gavin Andresen wrote:


And their are other methods too.



The GUI::Test package for perl will allow this to be greatly
automated. (I have done this before on the localisation of photoshop.)



this why we need detailed testscripts and plans.  so we know what has
and hasnt been done. The more boring the task the more work that needs
to go into testcase development.  This is the area I see as my
greatest failing last time.  I have a large number of virtual machines
and should have at least this work.  But we need very detailed
testcases.  with decent testplans just downloading the software,
syncing the block chain, syncing an existing wallet, rescanning the
blockchain and verifying the balance would cover a large number of
tests.  The idea behind having lots of very specific testcases is you
get to see what tests have not been run.



I understand your concern, however I have taken a couple of days to
reflect on this and I still strongly feel that in order to make sure
that this sticks, and is still useful in 1 years time we need to lay
proper foundations. Those foundations are not word documents,
spreadsheets, etc.  they are selecting the right tools for the job.

We can gain so much benefit from using 3rd party software.
(bettermeans would rock if it wasnt rotting)

I am sure you could do your coding work just using vi, but an sdk
makes it much easier and allows you to work in a more productive manner.

I have had a couple of off list emails with some testers and they also
feel that it is very important to make sure we have a sound foundation
(mantis is so much more than just a bug reporting tool, I see the bug
reporting functionality as secondary to the main test run
functionality - but it doesnt have to be mantis based, we do need
workflow and testcase software though - and proper software for this
is much better than just a massive google doc.) however I am checking
out some other software that has been recommended.  It will be very
hard to change 'the process' once we have something we are used too
(just look at the current resistance) I promise nothing will change
for the dev team.  But test does need other tools, and processes.

If you feel that strongly that I am going about this the wrong way, I
am happy to step back and let someone else sort it out (I will still
do all the testing I possibly can). I would feel that this would be a
real shame and we have the chance to setup requirements to
functionality to tests all with traceability. why not do it right from
the start?

I will open up my vps' somepoint over the next few days and you can
see what I mean. I will setup a fake git project, and sort out the
interactions.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t
w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2
TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8
TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4
2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7
6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo=
=ekQ/
-----END PGP SIGNATURE-----

On Sun, Sep 23, 2012 at 8:12 AM, Mike Hearn <mike@...> wrote:
> Has anyone got long term longs that contain the pool size and timestamps?
>
> Unfortunately I forgot to enable timestamps in the logs for my own
> nodes (the privacy benefit of disabling this by default is
> questionable, imho). But just looking at the general trends and
> cross-checking against my own memory it definitely seems that there
> are more and more pending transactions that don't get cleared into
> blocks.
>
> One of my nodes now routinely has 4000 transactions in the mempool.
> Blocks typically clear only a few hundred at most, which is what you'd
> expect given current transaction rates (around 300 per ten minute
> interval). So what are the other pending transactions doing and why
> aren't they getting drained out of the mempool?

Yeah, my public nodes currently have 2200+  Over time, it gets
cluttered naturally due to the disconnect between what miners mine and
what relayers relay.

I've long argued that all mempool implementations should limit the
lifetime of any TX to a specific number of blocks.  Rationale:
- bitcoin clients retransmit until TX is confirmed
- provides a deterministic lifetime for a TX; if you KNOW a TX will
disappear 144 blocks (24 hours) after you stop transmitting, then it
is probably safe to initiate recovery procedures and perhaps revise
the transaction
- prevents zombie TXs from littering memory... they hang around,
wasting resources, but never get confirmed

No one has strenuously argued against this, so I suppose it is down to
writing a patch, and coming up with a good number we (as a network)
can agree upon.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@...

On Sun, Sep 23, 2012 at 4:30 PM, Jeff Garzik <jgarzik@...> wrote:
> Yeah, my public nodes currently have 2200+  Over time, it gets
> cluttered naturally due to the disconnect between what miners mine and
> what relayers relay.

Right, this disconnect is why simple scalar measures of mempool size
aren't terribly informative.

There are bursts of weird transactions (e.g. someone was flooding zero
value txn a few weeks ago; before that there were some enormous series
of double-spend induced orphans), and other sustained loads that quite
a few miners are intentionally excluding.

> No one has strenuously argued against this, so I suppose it is down to
> writing a patch, and coming up with a good number we (as a network)
> can agree upon.

Sounds good— my only concern is that nodes will repeat their own
transactions but not the unconfirmed parents. So being more aggressive
can turn otherwise valid transactions into orphans.

Would there be value in an archive-mempool which is only checked when
you receive an orphan transaction?

I would point out that you can't _KNOW_ a txn will disappear. Someone
else could happily reannounce it. (I know you know this; but it's good
to be clear on that point when we talk about it!)

> Sounds good— my only concern is that nodes will repeat their own
> transactions but not the unconfirmed parents.

Nodes repeat wallet transactions and any previous transactions that
are not yet included in the chain (see
CWalletTx::RelayWalletTransaction). So I don't think it's an issue.

(ok, bitcoinj clients don't do that, they just announce their own transactions)

On 9/23/12, Jeff Garzik <jgarzik@...> wrote:
> - provides a deterministic lifetime for a TX; if you KNOW a TX will
> disappear 144 blocks (24 hours) after you stop transmitting, then it
> is probably safe to initiate recovery procedures and perhaps revise
> the transaction
> - prevents zombie TXs from littering memory... they hang around,
> wasting resources, but never get confirmed

I don't understand. Can the chain enforce this number?
Why can't clients delete all those transactions right now?

On 9/23/12, Gregory Maxwell <gmaxwell@...> wrote:
> There are bursts of weird transactions (e.g. someone was flooding zero
> value txn a few weeks ago; before that there were some enormous series
> of double-spend induced orphans), and other sustained loads that quite
> a few miners are intentionally excluding.

Why clients store transactions that don't obey the current rules of
the chain at all?

On Tue, Sep 25, 2012 at 1:34 PM, Jorge Timón <timon.elviejo@...> wrote:
> On 9/23/12, Jeff Garzik <jgarzik@...> wrote:
>> - provides a deterministic lifetime for a TX; if you KNOW a TX will
>> disappear 144 blocks (24 hours) after you stop transmitting, then it
>> is probably safe to initiate recovery procedures and perhaps revise
>> the transaction
>> - prevents zombie TXs from littering memory... they hang around,
>> wasting resources, but never get confirmed
>
> I don't understand. Can the chain enforce this number?
> Why can't clients delete all those transactions right now?

This is discussion about transactions which are not in the chain yet.

> On 9/23/12, Gregory Maxwell <gmaxwell@...> wrote:
>> There are bursts of weird transactions (e.g. someone was flooding zero
>> value txn a few weeks ago; before that there were some enormous series
>> of double-spend induced orphans), and other sustained loads that quite
>> a few miners are intentionally excluding.
>
> Why clients store transactions that don't obey the current rules of
> the chain at all?

The double spending transaction is not stored— which is, in fact, the
problem which creates these huge chain. When a transaction depending
on the doublespend is received we do not know its parent (because we
dropped it because it was a rule violation) so we keep it around as an
orphan hoping its parent arrives.

The software could maintain a cache of rejected txids to consult for
orphan txn's parents, but it would need to be dropped any time there
is a reorg so I don't know how useful it would be.