Re: [dacapobench-researchers] Build problems using beta2

[David M. <dam...@mc...>]

Hi Steve,
I've solved these issues. See below for details.

On Thu, Dec 10, 2009 at 5:43 PM, Steve Blackburn
<Ste...@an...> wrote:
> Hi David,
>
>> fop fails to build because the unit tests fail. I tried building fop
>> outside of dacapo, which gave me the same result. While the junit
>> summaries claim that no test failed, I can see exceptions about fop
>> not being able to find some font files. I believe the actual build
>> process works just fine, but the unit tests are run by default, which
>> make the report a build error to ant. Is anyone else having similar
>> issues? I haven't been able to find much by googling on this issue.
>
> I'm not aware of this problem.
>
> FYI, you can always see the results of our 12-hourly regression builds
> here:
>        http://dacapo.anu.edu.au/regression/sanity/latest.html
>
> I assume this issue is sensitive to your particular environment.  If
> your problems are specific to fop (and not the dacapo framework), I
> suggest you follow it up with the maintainers of fop.   If it turns
> out to be a problem with the way we're building fop, please let us
> know.   I'm not aware of any such issue.

The issue here is that the unit tests require an additional download
to run. I had to download a tar.gz from
  http://offo.sourceforge.net/hyphenation/index.html
and place the fop-hyph.jar into fop's lib directory. Only then do the
unit tests complete successfully. I suspect that on your regression
machine these files are somewhere on the class path. Shouldn't the
harness download that file?

>> jython fails to verify its checksum.
>
> You mean that the jython svn checkout does not match the stored md5
> checksum.   I don't know why this would be so.   I've not seen such a
> problem before.   What is your environment (OS, java version, etc)?

Ok, I figured it out. The problem is that ant's fileset leaves the
order unspecified. But since a fileset is used to create the
concatenated file, the order of the contents can vary from machine to
machine, which makes the checksum be different.

>> P.S. I noticed that the checksum verification process is rather
>> involved: Extract the tar, concat all the contents and save the result
>> to disk, then create the checksum of the concatenated file.
>> Essentially everything is read and written twice. I don't know ant
>> well enough to improve this, but maybe the 'totalproperty' parameter
>> of the checksum task could help? This is not critical, but I thought
>> it was worth mentioning.
>
> I'm not aware of any better approach.  If you know of a better way,
> we'd be glad to hear :-)
>
> Sorry I don't have more insight than this.   If there are any others
> reading the list who have experienced anything like this, your input
> would be appreciated.

Attached is a patch to util.xml which fixes both problems:
- It sorts the list of files
- It computes the checksum directly from the extracted files

The patch does not change the signature of the macro, but note that
the .MD5 files in beta2 are not valid any more and need to get
regenerated. I added some echo output which can be used to copy the
checksum from.

There is one smaller inefficiency with my patch, the checksum target
creates .MD5 files for every file which gets verified. This is
completely useless, because the macro compares the totalproperty to
the one stored on disk, but I could not find a way to turn this
behavior off. It should still be faster than the previous
implementation.

~David