From: Julian S. <js...@ac...> - 2008-12-30 09:46:04
|
On Tuesday 30 December 2008 00:30:48 Nicholas Nethercote wrote: > I'm thinking about trying to improve the regtests, they would be much more > useful if more reliable and less noisy. Yes. That would be great. > One thing I want to do is work out a good way to attach the diffs from the > failing tests so that people can investigate problems without having to ask > the usual "what went wrong with this test?" That would also be very helpful. I've noticed that the most common reason for regtests to fail on different distros is due to trivial differences in stack tracebacks, particularly the addition/loss of intermediate frames due to differences in inlining. So the test appears to fail for a completely spurious reason, when in fact the functionality it was trying to test actually works correctly. What would help greatly is a more flexible way of specifying the acceptable (expected) tracebacks. Something exactly like the current suppression syntax, allowing frame-level wildcards and function/object-name-level-wildcards would be ideal. So I guess that would mean writing a custom C program to do the check, rather than just using diff. J |