Menu
▾
▴
[dacapobench-researchers] DaCapo 9.10-beta0 available
|
From: Steve B. <Ste...@an...> - 2009-09-12 05:04:35
|
We are pleased to announce the beta release of the forthcoming major release of the DaCapo benchmark suite. We apologize if you receive this email more than once. Please note that this BETA release is NOT SUITABLE AS A RESEARCH TOOL. The purpose of the release is to invite researchers to evaluate the it and give us feedback (see links below for mailing list and bug trackers). Please be sure to note known bugs and limitations (see end of release notes) before sending in feedback. This will be the first of a number of beta releases. Please check the web site for updates. We welcome your feedback and thank many of you for the contributions you've made already. Best wishes, Steve Blackburn, ANU John Zigman, ANU Kathryn McKinley, UT Austin (On behalf of the DaCapo project). ----- Key information: 1. Source and binary downloads are available here: https://sourceforge.net/projects/dacapobench/files/dacapobench/9.10-beta0 2. Release notes are available below and here: http://www.dacapo-bench.org/RELEASE_NOTES.txt 3. Bug tracker and feature requests here: http://sourceforge.net/projects/dacapobench/develop http://sourceforge.net/tracker/?group_id=172498&atid=861957 http://sourceforge.net/tracker/?group_id=172498&atid=861960 4. Mailing list here: https://sourceforge.net/mail/?group_id=172498 5. Regression results here: http://dacapo.anu.edu.au/regression/sanity/latest/index.html http://dacapo.anu.edu.au/regression/perf/head.html ----- dacapo-9.10-beta0 RELEASE NOTES 2009-09-10 ************************************************************************ * IMPORTANT: This beta release is NOT SUITABLE AS A RESEARCH TOOL. * * * * Please wait for the full release before using the suite for research * * purposes. The benchmarks in this beta release are: * * * * 1) not fully tuned, * * 2) not fully evaluated, * * 3) known to have various limitations and bugs, and * * 4) subject to change without notice. * * * ************************************************************************ This is the first beta release of the anticipated 9.10 release of the DaCapo benchmark suite. These notes are structured as follows: 1. Purpose 2. Overview 3. Usage 4. Changes 5. Known problems and limitations 1. Purpose ---------- This beta release exists specifically (and only) to allow community feedback and contribution to the upcoming major release, and to allow JVM vendors to test against the suite prior to the final release. We strongly encourage the community to explore and evaluate these benchmarks and to send feedback to the dacapo research group via the mailing list and/or bug tracker. Please check the list of known issues (below) before sending feedback. mailing list: dac...@li... (subscribe via: http://sourceforge.net/mail/?group_id=172498) bug tracker: http://sourceforge.net/tracker/?group_id=172498&atid=861957 or navigate to it from http://sourceforge.net/projects/dacapobench/develop We are particularly interested in feedback on the following: o The decision to add benchmarks "avrora", "batik", "derby", "sunflow", "tomcat", "tradebeans", and "tradesoap" (see 4.1 below). o The decision to "retire" benchmarks "antlr", "bloat", "chart", and "hsqldb" (see 4.2 below). o The tuning / running time of each of the benchmarks in their default settings. o The structure and behavior of the tradebeans, tradesoap and tomcat client/server benchmarks. o Any issues associated with building, running and usability of the suite. 2. Overview ----------- The DaCapo benchmark suite is slated to be updated every few years. The 9.10 release will be the first major update of the suite, and is strictly incompatible with previous releases: new benchmarks have been added, old benchmarks have been removed, all other benchmarks have been substantially updated. It is for this reason that in any published use of the suite, the version of the suite must be explicitly stated. The release sees the retirement of a number of single-threaded benchmarks (antlr, bloat and chart), the replacement of hsqldb by derby, the addition of six completely new benchmarks, and the upgrade of all other benchmarks to reflect the current release state of the applications from which the benchmarks were derived. These changes are consistent with the original goals of the DaCapo project, which include the desire for the suite to remain relevant and reflect the current state of deployed Java applications. Each of these benchmarks is tested for both performance* and correctness nightly. Results are available here: o performance: http://dacapo.anu.edu.au/regression/perf/head.html o sanity: http://dacapo.anu.edu.au/regression/sanity/latest/ * tradebeans and tradesoap are not yet running performance tests 3. Usage -------- 3.1 Downloading o Download the binary jar and/or source zip from: https://sourceforge.net/projects/dacapobench/files/ o Access the source from subversion via svn co https://dacapobench.svn.sourceforge.net/svnroot/dacapobench dacapobench 3.2 Running o Run a benchmark: java -jar <dacapo-jar-name>.jar <benchmark> o For usage information, run with no arguments. 3.3 Building o You must have a working, recent version of ant installed. Change to the benchmarks directory and then run: ant -p for instructions on how to build. 4. Changes ---------- 4.1. Additions avrora: AVRORA is a set of simulation and analysis tools in a framework for AVR microcontrollers. The benchmark exhibits a great deal of fine-grained concurrency. The benchmark is courtesy of Ben Titzer (Sun Microsystems) and was developed at UCLA. batik: Batik is an SVG toolkit produced by the Apache foundation. The benchmark renders a number of svg files. derby: Derby is an in-memory database benchmark, using the Derby database produced by the Apache foundation, and executing the pseudojdbc workload previously used by hsqldb. Derby replaces hsqldb. sunflow: Sunflow is a rendering system for photo-realistic images based on raytracing engine. tomcat: Tomcat uses the Apache Tomcat servelet container to run some sample web applications. tradebeans: Tradebeans runs the Apache daytrader workload "directly" (via EJB) within a Geronimo application server. Daytrader is derived from the IBM Trade6 benchmark. tradesoap: Tradesoap is identical to the tradebeans workload, except that client/server communications is via soap protocols (and the workloads are reduced in size to compensate the substantially higher overhead). Tradebeans and tradesoap were added as a pair specifically to allow researchers to analyze overheads associated with the widely used soap protocol. 4.2. Deletions antlr: Antlr is single threaded and highly repetitive. The most recent version of jython uses antlr; so antlr remains represented within the DaCapo suite. bloat: Bloat is not as widely used as our other workloads and the code exhibited some pathologies that were arguably not representive or desirable in a suite that was to be representive of modern Java applications. chart: Chart was repetitive and used a framework that appears not to be as widely used as most of the other DaCapo benchmarks. The Batik workload has some similarities with chart (both are render vector graphics), but is part of a larger heavyly used framework from Apache. hsqldb: Hsqldb has been replaced by derby, which runs the same workload but uses a much more widely used database engine. 4.3. Updates All other benchmarks have been updated to reflect the latest release of the underlying application (with the exception of eclipse, where the update to 3.5 is underway). 4.4. Other Notable Changes The packaging of the DaCapo suite has been completely re-worked and the source code is entirely re-organized. 5. Known Issues --------------- Please consult the bug tracker for a complete and up to date list of known issues (http://sourceforge.net/tracker/?group_id=172498&atid=861957 ). DaCapo is an open source community project. We welcome all assistance in addressing bugs and shortcomings in the suite. A few notable high priority issues are listed here: 5.1 Concurrency We intend to conduct a concurrency audit, analyzing and documenting the level of concurrency in each of the workloads. The per-benchmark documentation available at the commandline (with the -i switch) should report the level of threading for each benchmark. Although most benchmarks exhibit significant concurrency, we have made the conscious decision to continue to include a few single-threaded benchmarks. We do this because as long as there exist a significant number of single threaded applications in popular use, the single-threaded performance of a JVM is important. 5.2 Trade The trade benchmarks currently do not reliably run beyond one or two iterations (and consequently don't appear in our peformance regression results since those regressions perform 10 iterations). The benchmarks are known to suffer from memory leaks and database deadlocks. 5.3 Eclipse We are in the process of updating eclipse, but have not yet completed this, so at present eclipse runs the same version as it did in the 2006 release. 5.4 Derby We intend to move to using a TCPC workload rather than PseudoJDBC for derby. We also wish to remove derby's build-time dependency on a 1.4 JVM. 5.5 Tomcat We intend to make the tomcat workload more interesting. Performacne results show that tomcat currently has a remarkably flat warm-up curve when compared to other benchmarks. |