Benchmarking

I have done some benchmarking, using different compilers to build the same runtime environment. It showed some interesting results.

There is a small demo-application in JewelScript's SVN repository. It's called "JILRunOnly". I made this to show developers how to embed JewelScript in any application. It's a simple command-line program written in ANSI-C, so it can be compiled on any platform using any compiler.

Having just added a new project for Microsoft's VS Express 2013 to the repository, I came up with the idea to compare the outputs of all compilers I have access to. The compilers used are TDM-GCC 4.8.1 (via DevC++ 5.9.2), Microsoft VS Express 2008 and VS Express 2013.

I made fully optimized release builds of "JILRunOnly", targeting 32-bit in VS 2008 and 32/64-bit in the other two compilers. Then I ran a synthetic benchmarking script and compared their execution times and virtual machine instructions per second.

Here are the results:

alloc
loop
fib
ack
IPS

TDM-GCC 32
3.287 s
1.594 s
0.922 s
0.42 s
91.18 M

TDM-GCC 64
3.105 s
1.513 s
0.905 s
0.421 s
95.52 M

VS2008 32
2.98 s
1.466 s
0.952 s
0.483 s
96.55 M

VS2013 32
3.619 s
2.356 s
1.138 s
0.609 s
73.38 M

VS2013 64
2.98 s
1.373 s
0.982 s
0.422 s
98.63 M

Benchmark descriptions:

alloc: Allocates, initializes and destroys 10 million script objects. Basically it measures memory allocation, deallocation and copying performance.

loop: Simply runs through six nested loops, each of them counting to 18, resulting in a total number of 34,012,224 iterations. This measures raw byte-code execution performance.

fib: Recursively calculates the Fibonacci sequence for N=32, resulting in 7,049,155 recursions. This mainly measures function call and variable stack performance.

ack: Recursively calculates the Ackerman sequence for M=3 N=8, resulting in 2,785,999 recursions. Measures function call and variable stack performance as well.

IPS: The average number of executed virtual machine instructions per second calculated by the script.

Conclusions:

The arithmetic / recursive benchmarks perform better with the GCC output, while the memory / iteration based benchmarks perform better with the Microsoft compiler. The 32-bit output of the VS 2013 compiler is a different story though. I have no idea why the 32-bit build would be so much slower than the 64-bit build of the same compiler, or the 32-bit build from VS 2008 for that matter.

It seems building 32-bit applications with VS 2013 is not recommended. If you have both VS 2008 and 2013 installed, I would recommend using VS 2008 to build 32-bit versions and VS 2013 for 64-bit versions.
link