Re: [sensor-dev] Results of very simple performance test + reported Jamon values are false

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Age,

How did you calculate the factor of 4?

1M iterations with start-stop:
Sensor: 937 ms.
JAMon: 1594 ms.

So, Sensor time = 59% of JAMon, so Sensor is 41% faster than JAMon.

Or am I missing something?

Ciao,
Jeroen.

Age Mooy wrote:

>What !!??
>
>That is a ridiculously large difference. Can everyone else please also
>run the performance test and mention which OS/JDK they use ? It's hard
>to believe the linux VM is THAT much slower.
>
>Age
>
>PS
>YES !!! 4 times as fast as Jamon !! (waves hands in the air :)
>
>
>
>On 3/24/06, Jeroen Borgers <jbo...@xe...> wrote:
>  
>
>>Hi Age,
>>
>>Okay, I've installed the svn Eclipse plugin and checked out Sensor and
>>achieved to get it running in Eclipse. :-)
>>
>>I run the jUnit test in Eclipse of SimplePerformanceTest. The second
>>time I tested it, I got:
>>
>>Total time (parent timer): 937 ms.
>>Number of hits (test timer): 1000000.
>>Total time (test timer)    : 125 ms.
>>Average time (test timer)  : 0 ms.
>>
>>So, this is even better than JAMon!
>>1.067.000 calls/s.
>>
>>Not bad!
>>
>>How much time did it take on your laptop?
>>If it is so much slower for you, I would consider switching to another OS!
>>
>>Jeroen.
>>
>>Age Mooy wrote:
>>
>>    
>>
>>>I ran the Jamon performance test (twice) as it was specified on the
>>>Jamon page. I took the two "Full Factory" results and looked at the
>>>same number you looked at (the "it took" number. But on my Inspiron
>>>9300 powerhouse laptop it consistently took a full 15000 (!) ms !
>>>That's 10 times as long as your test... so something is really weird
>>>here.
>>>
>>>This could theoreticaly be a hard peformance difference between linux
>>>(me) and windows (tou) but it's very hard to believe.
>>>
>>>But we can test this. I checked in a special unit test called
>>>SimplePerformanceTest last night. It's excluded from the normal Maven
>>>unit test list but you can run it seperately with Maven or with
>>>eclipse. I ran it with eclipse and got the same performance as the
>>>Jamon 2 tests on my machine. It prints some simple lines to System.out
>>>after running the 100000 iterations.
>>>
>>>Could you please run the SimplePerformance test on your machine and
>>>see what you get ?
>>>
>>>Age
>>>
>>>
>>>On 3/20/06, Jeroen Borgers <jbo...@xe...> wrote:
>>>
>>>
>>>      
>>>
>>>>Hi Age,
>>>>
>>>>Sounds great, but I am not convinced.
>>>>
>>>>How did you test the JAMon performance?
>>>>The JAMon page refers to com.jamonapi.TestClassPerformance, it also uses
>>>>a monitor around the test with the loop (of 1.000.000 iterations),
>>>>the last code block:
>>>>       timingMon.start();
>>>>       test.factoryMonitor();
>>>>       log(timingMon.stop()); :
>>>>
>>>>Gives me the next output:
>>>>----
>>>>Full Factory TimingMonitor()- uses cached version so doesn't create
>>>>child monitors
>>>>   Monitor mon=MonitorFactory.start('pages.admin');
>>>>   mon.stop();
>>>>JAMon Label=pages.admin, Units=ms.: (Hits=2000000.0, Avg=7.0E-5,
>>>>Total=140.0, Min=0.0, Max=16.0, Active=0.0, Avg Active=1.0, Max
>>>>Active=1.0, First Access=Mon Mar 20 10:28:02 CET 2006, Last Access=Mon
>>>>Mar 20 10:28:05 CET 2006)
>>>>It took 1594 ms.
>>>>----
>>>>The JAMon label Avg=7.0E-5 is bogus because of the clock resolution as
>>>>you point out. However, the time all 1.000.000 iterations took is: 1594 ms.
>>>>This means 627.000 calls per second on my 5160 laptop with jdk 1.4.2.
>>>>
>>>>So, I would say the numbers on the JAMon page are correct (405.000
>>>>calls/s for jdk1.4). Agree?
>>>>
>>>>Ciao,
>>>>Jeroen.
>>>>
>>>>PS. I haven't had time to time Sensor yet.
>>>>
>>>>Age Mooy wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Hi
>>>>>
>>>>>I did some extremely rudimentary performance overhead tests using the
>>>>>following code (snippet):
>>>>>
>>>>>============================
>>>>>
>>>>>parentTimer.start();
>>>>>
>>>>>for (int i = 0; i < 1000000; i++) {  // or 500000
>>>>>timer.start();
>>>>>timer.stop();
>>>>>}
>>>>>
>>>>>parentTimer.stop();
>>>>>
>>>>>=============================
>>>>>
>>>>>I consistently got performance of around 68000 start/stop operations
>>>>>per second (the average time for 500000 iterations was 7.4 seconds and
>>>>>the average time for 1000000 iterations was 14.6 seconds)
>>>>>
>>>>>I measured the total time of the parent timer since the time
>>>>>resolution of the VM/OS is not accurate enough in the 0 - 10 ms range.
>>>>>The total time of the timer that was being started and stopped in the
>>>>>loop consistently reports values about 3 times as fast, which would
>>>>>lead to performance of about 200000 iterations per second.
>>>>>
>>>>>Funnily enough that last number is prety close to the performance
>>>>>listed on the Jamon 2.0 page so I ran the Jamon 2.0 performance tests
>>>>>and got almost exactly the same performance as Sensor. Sensor was
>>>>>slightly faster on a 1.4 jdk.
>>>>>
>>>>>This means that the performance listed on the Jamon 2.0 website is not
>>>>>correct since they did not take the VM/OS timing resolution into
>>>>>account.
>>>>>
>>>>>Conclusion: Sensor does not seem to have any performance problems
>>>>>relative to Jamon and performs similar or better than Jamon 2.0.
>>>>>
>>>>>Age
>>>>>N�HS^�隊X���'���u��<�ڂ�.���y�"��*m�x%jx.j���^�קvƩ�X�jب�ȧ��m�ݚ���v&��קv�^�+����j�Z���{az���^��h��஋�n���)�{h�����ا�׫�+h�(m�����Z��jY�w��ǥrg�y$���Oxḝn�mj��^��'����z������x%��Rz{(�ׯzZ)z�b��,���y�+��޶m����+-��.�ǟ�����+-��b�ا~��z{(�ׯzZ)er==
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>--
>>>>drs. Jeroen Borgers
>>>>Senior Consultant, SCJP, SCEA
>>>>Xebia IT Architects BV, Utrechtseweg 49, 1213 TL Hilversum, The Netherlands, www.xebia.com
>>>>office: +31(0)35-5381921, mobile: +31(0)6-30128951, jbo...@xe...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>--
>>drs. Jeroen Borgers
>>Senior Consultant, SCJP, SCEA
>>Xebia IT Architects BV, Utrechtseweg 49, 1213 TL Hilversum, The Netherlands, www.xebia.com
>>office: +31(0)35-5381921, mobile: +31(0)6-30128951, jbo...@xe...
>>
>>
>>
>>    
>>

-- 
drs. Jeroen Borgers
Senior Consultant, SCJP, SCEA
Xebia IT Architects BV, Utrechtseweg 49, 1213 TL Hilversum, The Netherlands, www.xebia.com
office: +31(0)35-5381921, mobile: +31(0)6-30128951, jbo...@xe...