sablevm-developer

[Sablevm-developer] summary of JGF benchmark problems

[Chris P. <chr...@ma...>]

Hi,

I'm also having problems in the switch interpreter on a uniprocessor 
with the following benchmarks in the JGF suites:

single-threaded, section2

FFTBenchSizeC
LUFactBenchSizeA

single-threaded, section3

MolDynBenchSizeB
RayTracerBenchSize[AB]

multi-threaded, section1

SyncBench (we talked about this already)

multi-threaded, section2

LUFactBenchSizeA

multi-threaded, section3

RayTracerBenchSize[AB]

================

The errors reported, with the exception of SyncBench, are some 
application-specific "validation error".

Anyway, I just thought I'd post a summary of the problems I've encountered.

If you're curious, and you want some (basic) scripts (that almost 
definitely need modifying) to run these benchmarks, after extracting, 
look in chris/sandbox/sablevm/src/spmt/benchmarks/ for transform_grande, 
transform_thread_grande, run_grande, run_thread_grande, and 
create_JGF_report.

Cheers,
Chris

Re: [Sablevm-developer] summary of JGF benchmark problems

[Grzegorz B. P. <ga...@de...>]

W liście z pon, 02-02-2004, godz. 23:29, Chris Pickett pisze: 
> I'm also having problems in the switch interpreter on a uniprocessor 
> with the following benchmarks in the JGF suites:

Do you mean that you encounter problems with these benchmarks
using current staging version of SableVM? or your modified version?

> Anyway, I just thought I'd post a summary of the problems I've encountered.
Good! Though what do you plan to find solutions to them? ;-)

> If you're curious, and you want some (basic) scripts (that almost 
> definitely need modifying) to run these benchmarks, after extracting, 
> look in chris/sandbox/sablevm/src/spmt/benchmarks/ for transform_grande, 
> transform_thread_grande, run_grande, run_thread_grande, and 
> create_JGF_report.
Just out of curiosity. All of these benchmarks are under LGPL and thus
are in our SVN, right? (or have you obtained special permission?)

Anyway - if these benchmarks fail on current staging then it'd
definitely be something I'd like to add to the massive testing
machinery I am working on/with right now. Well... in fact even if
they dont I'd like to do it too.

Cheers,

				Grzegorz B. Prokopski

-- 
Grzegorz B. Prokopski <ga...@de...>
Debian GNU/Linux      http://www.debian.org
SableVM - LGPLed JVM  http://www.sablevm.org
Why SableVM ?!?       http://devel.sablevm.org/wiki/WhySableVM

Re: [Sablevm-developer] summary of JGF benchmark problems

[Chris P. <chr...@ma...>]

Grzegorz B. Prokopski wrote:

> W li=B6cie z pon, 02-02-2004, godz. 23:29, Chris Pickett pisze:=20
>=20
>>I'm also having problems in the switch interpreter on a uniprocessor=20
>>with the following benchmarks in the JGF suites:
>=20
>=20
> Do you mean that you encounter problems with these benchmarks
> using current staging version of SableVM? or your modified version?
>=20

With my (shouldn't be broken) merge of SableVM into the switch=20
interpreter.  You're right ... I should test this against a vanilla=20
staging ... but I'm pretty confident that I haven't actually made any=20
changes against the plain switch interpreter (all the spmt stuff is=20
#ifdef'd away).  And I don't really have time to do it just now (I have=20
just finished running all the benchmarks I have, started on Friday, and=20
they didn't even all terminate -- the overhead of my spmt thing is=20
incredible when it's not actually doing anything).

>=20
>>Anyway, I just thought I'd post a summary of the problems I've encounte=
red.
>=20
> Good! Though what do you plan to find solutions to them? ;-)

Well, I still need help (suggestions) with JGFSyncBench problem (I=20
believe there's some kind of multithreading bug lurking in SableVM).=20
But ... I can't really look into things much more right now, actually=20
(conference deadlines), so I thought I'd let other people know, in case=20
you're interested (you are, so it's good!).  I'm actually not going to=20
rely on JGF for submitted work because of these issues, but it doesn't=20
matter, because I'm not having problems with SPECjvm98 and JOlden.


>>If you're curious, and you want some (basic) scripts (that almost=20
>>definitely need modifying) to run these benchmarks, after extracting,=20
>>look in chris/sandbox/sablevm/src/spmt/benchmarks/ for transform_grande=
,=20
>>transform_thread_grande, run_grande, run_thread_grande, and=20
>>create_JGF_report.
>=20
> Just out of curiosity. All of these benchmarks are under LGPL and thus
> are in our SVN, right? (or have you obtained special permission?)

I didn't actually put the benchmarks in the repository.  You have to=20
download them from the JGF site.  They're distributed with source code,=20
but no special license info apart from standard copyright.  That said,=20
if you make a patch for a benchmark, I'm sure the authors would consider =

applying it.  I'm not sure why they're not (L)GPL'd, actually, but the=20
authors can probably tell you more.  I also have (again, very basic)=20
scripts to run other benchmarks, such as SPECjvm98, in that directory=20
(Bruno has a version packaged for Ashes2).  You'll see (it's barely=20
commented, but it's obvious from the scripts how to arrange things ...=20
you just unpack the tarballs into my benchmarks/ directory, into folders =

called "grande" and "thread_grande").

> Anyway - if these benchmarks fail on current staging then it'd
> definitely be something I'd like to add to the massive testing
> machinery I am working on/with right now. Well... in fact even if
> they dont I'd like to do it too.

They're also very performance oriented, with a focus on scientific=20
computing (I think).  The benchmarks in section1 of each suite are all=20
micro-benchmarks, reporting frequencies of various things, so you can=20
compare VM's (the switch interpreter does not look good against Sun's=20
interpreter-JIT).

Definitely worth downloading and playing around with, from a "make=20
SableVM robust and fast" point of view, more than the other benchmarks=20
suites I've been using.  I think you might have even more problems with=20
the inlined threading engine.

Cheers,
Chris

Re: [Sablevm-developer] summary of JGF benchmark problems

[Chris P. <chr...@ma...>]

>> Do you mean that you encounter problems with these benchmarks
>> using current staging version of SableVM? or your modified version?
>>
> 
> With my (shouldn't be broken) merge of SableVM into the switch 
> interpreter.  You're right ... I should test this against a vanilla 
> staging ... but I'm pretty confident that I haven't actually made any 
> changes against the plain switch interpreter (all the spmt stuff is 
> #ifdef'd away).

Or you can just forget about it and wait for me to test it again on the 
current staging, to really confirm that it's 100% not my fault.  None of 
this is urgent for me.  Sorry if I gave the impression that it was. 
Rather, just a bug report, which I'd like to help fix.

Chris