Re: [dacapobench-researchers] DaCapo and SPECjvm2008

[Steve B. <Ste...@an...>]

Hi Xiaoming,

On 20/11/2008, at 6:15 PM, xiaoming gu wrote:
> Hi, all. I'm wondering what's the relation/difference between DaCapo  
> and SPECjvm2008 with the news that Sunflow and Derby have been  
> incorporated or are going to be incorporated into DaCapo. The above  
> two benchmarks are already in SPECjvm2008, at least the same name.

Good question.

Part of the answer to your question is that DaCapo and SPEC set out to  
solve different problems (see our OOPSLA and CACM paper for more on  
this).  DaCapo is focused on the needs of researchers.  SPEC's goals  
include the provision of a level playing field for product  
comparison.  Both are important, but they are quite different  
objectives.

Also, SPEC chose to draw their benchmarks from a number of sources, so  
jvm2008 does include overlap with DaCapo, Scimark and jvm98.  However,  
the overlap is probably superficial because the specific  
configurations of the workloads differ.  I was in communication with  
people from SPEC in 2005 and 2006 while they were putting together  
jvm2008 and provided them with information on DaCapo, its composition,  
our future plans, and our methodology.

Another point of difference is that jvm2008 has configured its  
benchmarks as throughput workloads (they iterate for a fixed period,  
rather than perform a fixed amount of work).   This can be a useful  
approach when evaluating throughput-oriented workloads (such as  
transaction servers), but has a number of drawbacks, which we discuss  
in our OOPSLA paper.

Finally, DaCapo is intended to be dynamic, constantly updated.   We've  
been slow to put out our second release (overdue now), but if you take  
a peak at subversion you'll see that it has been constantly moving  
forward, with new workloads and other refinements.   We hope to make  
them available in the form of a release soon!

> I checked the paper on CACM and found SPECjvm2008 is not mentioned.  
> Would you give me some clues? Thanks.

Yes, we wrote the paper before SPECjvm2008 was released.  We submitted  
the final version of our CACM paper in April, while SPECjvm2008 was  
released on May 7, 2008.

Thanks,

--Steve