I have been doing some prelimary testing calling rules through the REST API. At first glance it seems significantly slower than I would expect. Rules that run lighting fast through benchmarking in the studio take maybe 1000 times as long to execute through the Webservice API or REST API. I am using just the demo environment at the moment but wonder if people more experienced than me could share their experience on REST API performance. What sort of speed should I be expecting and how do I tune tomcat and openl tablets for rule execution performance?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for interesting with OpenL Tablets. Sorry, but we need additional information about your performance tests. How it was done and could you provide results?
For example if your rule is very simple and calculation time is about 0.01ms, but with webservice API and REST API response time is about 10ms. It is OK.
Best Regards, Marat Kamalov.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am using the demo system with no changes or tuning. If I run a benchmark against the DriverAgeType rule with a single test case I get a timing of 0.000536 ms (see attached screenshot). If I exercise the same rule using the REST API the same rule takes about half a second. By your suggestion it should take about half a millisecond (which would be OK). So the REST API is adding a much higher overhead than I was expecting and I am hoping can be overcome by tuning. I have used the bat file below and CURL to run the test.
ECHO Start Time %STARTTIME%
ECHO End Time %ENDTIME%
rem convert STARTTIME and ENDTIME to centiseconds
set /A STARTTIME=(1%STARTTIME:~0,2%-100)360000 + (1%STARTTIME:~3,2%-100)6000 + (1%STARTTIME:~6,2%-100)100 + (1%STARTTIME:~9,2%-100)
set /A ENDTIME=(1%ENDTIME:~0,2%-100)360000 + (1%ENDTIME:~3,2%-100)6000 + (1%ENDTIME:~6,2%-100)100 + (1%ENDTIME:~9,2%-100)
rem calculate duration
set /A DURATION=%ENDTIME%-%STARTTIME%
I have looked at your script, so I can say that this script adds big deviation in measurement. I found that the script brings 13 ms delay (130 centiseconds) without a request to a REST service. When I ran the script against the remote REST service, it shown 18 ms (180 centiseconds). So a real request delay is about 5ms (18 ms - 13 ms). I think that 13 ms is time of parsing .BAT file + executing FOR and ECHO statements + running curl command + parsing and validating curl params.
I also measured that REST service using SoapUI. And it shown the following results:
Threads
avg, ms
requests per second
1
1.33
369
5
1.83
809
10
2.36
1157
20
2.74
1767
50
9.41
2386
100
32.8
2522
200
75.0
2566
500
74.3
2611
1000
74.8
2563
Best Regards, Yury Molchan.
Last edit: OpenL Tablets 2015-05-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you. I agree I get a much better performance of the demo REST service using SOAPUI. However my tests are still showing slower results than your own by a factor of 10. For instance I get an average of 13ms for 1 thread and 36 tps. I am on a reasonably powerful processor (i7-4600) so suspect my configuration of either SOAPUI or Tomcat is not as tuned as yours. Any suggestions?
UPDATE * I can see my antivirus working overtime and this is affecting my results. Let me investigate before answering this post.
Last edit: Steve Lane 2015-05-12
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As a final post on this subject I can confirm that I can get very good performance through the REST API that matches your experience. SOAPUI is a good tool to measure this. For anyone else doing this type of testing I would recommend checking your antivirus settings as this can affect results.
OpenL Tablets is an excellent product. Thank you for your help.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been doing some prelimary testing calling rules through the REST API. At first glance it seems significantly slower than I would expect. Rules that run lighting fast through benchmarking in the studio take maybe 1000 times as long to execute through the Webservice API or REST API. I am using just the demo environment at the moment but wonder if people more experienced than me could share their experience on REST API performance. What sort of speed should I be expecting and how do I tune tomcat and openl tablets for rule execution performance?
Hi,
Thank you for interesting with OpenL Tablets. Sorry, but we need additional information about your performance tests. How it was done and could you provide results?
For example if your rule is very simple and calculation time is about 0.01ms, but with webservice API and REST API response time is about 10ms. It is OK.
Best Regards, Marat Kamalov.
Hi Marat,
I am using the demo system with no changes or tuning. If I run a benchmark against the DriverAgeType rule with a single test case I get a timing of 0.000536 ms (see attached screenshot). If I exercise the same rule using the REST API the same rule takes about half a second. By your suggestion it should take about half a millisecond (which would be OK). So the REST API is adding a much higher overhead than I was expecting and I am hoping can be overcome by tuning. I have used the bat file below and CURL to run the test.
ECHO OFF
set STARTTIME=%TIME%
FOR /L %%A IN (1,1,100) DO (
curl -H "Content-Type:application/json" -X POST -d "{ \"age\": 20, \"gender\": \"Male\" }" http://localhost:8080/webservice/REST/Example3-AutoPolicyCalculation/DriverAgeType
echo ... %%A
)
set ENDTIME=%TIME%
ECHO Start Time %STARTTIME%
ECHO End Time %ENDTIME%
rem convert STARTTIME and ENDTIME to centiseconds
set /A STARTTIME=(1%STARTTIME:~0,2%-100)360000 + (1%STARTTIME:~3,2%-100)6000 + (1%STARTTIME:~6,2%-100)100 + (1%STARTTIME:~9,2%-100)
set /A ENDTIME=(1%ENDTIME:~0,2%-100)360000 + (1%ENDTIME:~3,2%-100)6000 + (1%ENDTIME:~6,2%-100)100 + (1%ENDTIME:~9,2%-100)
rem calculate duration
set /A DURATION=%ENDTIME%-%STARTTIME%
ECHO %DURATION% centiseconds
Hi Stive,
I have looked at your script, so I can say that this script adds big deviation in measurement. I found that the script brings 13 ms delay (130 centiseconds) without a request to a REST service. When I ran the script against the remote REST service, it shown 18 ms (180 centiseconds). So a real request delay is about 5ms (18 ms - 13 ms). I think that 13 ms is time of parsing .BAT file + executing FOR and ECHO statements + running curl command + parsing and validating curl params.
I also measured that REST service using SoapUI. And it shown the following results:
Best Regards, Yury Molchan.
Last edit: OpenL Tablets 2015-05-07
Hi Yury,
Thank you. I agree I get a much better performance of the demo REST service using SOAPUI. However my tests are still showing slower results than your own by a factor of 10. For instance I get an average of 13ms for 1 thread and 36 tps. I am on a reasonably powerful processor (i7-4600) so suspect my configuration of either SOAPUI or Tomcat is not as tuned as yours. Any suggestions?
UPDATE * I can see my antivirus working overtime and this is affecting my results. Let me investigate before answering this post.
Last edit: Steve Lane 2015-05-12
Hi Yury,
As a final post on this subject I can confirm that I can get very good performance through the REST API that matches your experience. SOAPUI is a good tool to measure this. For anyone else doing this type of testing I would recommend checking your antivirus settings as this can affect results.
OpenL Tablets is an excellent product. Thank you for your help.