Compared WS Stacks

• Axis 1.4 http://ws.apache.org/axis/
• Axis2 1.5.1 http://ws.apache.org/axis2/
• CXF 2.2.6 http://cxf.apache.org/

Missing important players

• Metro https://metro.dev.java.net/

Benchmark design

The benchmark is designed to test binary transfer. If you have big XML payloads streaming parser based solutions like Axis2 or CXF will probably outperform Axis 1.4 to a greater degree.

All web services are developed using the Java first approach. A WSDL is generated from the following interface:

public interface QueryService {
  String query(String query);
  byte[] getNTuples(int tuples, String requestID);		
}

Each stack has its own WSDL <-> Java tools. These tools produce slightly different WSDL descriptions, but all use

<xsd:element name="return" type="xsd:base64Binary" />

for the return element type of the getNTuples operation.

• When query is invoked, a tuple producing thread starts putting tuples into a bounded (1000 elements) ArrayBlockingQueue?.
• RequestID is returned (I was planning to test this with concurrent users - maybe I will one day).
• Tuple schema (int, int, double, double, string80, date ).
• In total 100,000 tuples are produced (about 16MB).

• When getNTuples is invoked, tuples are read from the queue and serialised to a binary format (using Externalizable like approach).
• The fetch size is specified by the tuples argument (bad name choice).
• Serialised tuples are written to a ByteArrayOutputStream and the result of ByteArrayOutputStream.toByteArray() is returned. This is different for JAX-RS where tuples are streamed directly to the response output stream.

• Client code uses auto-generated stubs (JAX-RS client uses direct WebClient?).
• The entire 100,000 result set is retrieved for each run.
• Several runs with increasing fetch size (tuples returned per request) are performed.
• Tuples are deserialised on the client side and their string representations saved to a file.

JAX-RS annotations:

	@GET
	@Produces("text/plain")
	public String query() { ... }

	@GET
	@Path("/{requestID}/{tuples}")
	@Produces("application/octet-stream")
	public StreamingOutput getNTuples(
			@PathParam("requestID") final String requestID,
			@PathParam("tuples") final Integer tuples) { ... }

h2. Results

• all timings averaged from 5 runs, after a warm-up run
• Axis2 runs out of memory on the server for the fetch size 50000 with default (128m) heap size - works when increased to 512m
• Axis 1.4 runs out of memory on client and server for the fetch size 50000 with default heap - works with 512m
• experiments run on my MacBook? Pro laptop with Spotify running (far from controlled environment)
• webapps deployed on Tomcat 6.0.24 (tested with 5.5.28 and there was no significant difference)

Table 1: RAW numbers. Times in seconds. Function of the fetch size.

	Axis 1.4	Axis 2	CXF JAX-WS	CXF JAX-WS MTOM	CXF JAX-RS
10	19.7	27.0	11.8	13.0	12.4
50	6.7	8.4	4.7	4.5	4.2
100	4.9	5.8	3.5	3.2	3.0
500	3.5	3.6	2.4	2.2	1.9
1000	3.4	3.2	2.3	2.0	1.7
2000	3.2	3.1	2.2	1.9	1.6
5000	4.5	3.1	2.3	2.0	1.5
50000	3.5	3.3	2.5	1.9	1.4
100000	3.8	3.3	2.6	2.0	1.4

Figure 1: Plotted results. Time( fetchSize ).

Discussion

• CXF wins the race on all fronts.
• Axis based stacks seem to be memory hungry - the entire result set - when saved to file is about 16m - one would expect 128m to be enough. CXF seems to be leaner.
• Switching on MTOM in CXF is trivial (checked with tcpmon if SOAP messages use attachments). MTOM optimisation in CXF seems to work for all base64Binary elements (no need to change WSDL).

  <jaxws:properties>
    <entry key="mtom-enabled" value="true" />
  </jaxws:properties>
  

• Benefits of using attachments are visible for larger fetch sizes. Efficiency of MTOM could be improved. My data structures do not support easy streaming via the DataHandler object. It is possible to develop such a stream based pipe. I suspect we could get close to JAX-RS performance with proper server side streaming for MTOM attachments.
• With JAX-RS we are able to connect TupleOutputStream with the response stream and therefore achieve proper data streaming. Binary data transfer with RESTful web services is the most efficient, but it could be matched by WS with MTOM if implemented properly.

Conclusion

If you are starting from scratch - use CXF:

• it performs well and is less memory hungry than Axis
• it uses modern configuration management with Spring dependency injection
• it supports most of the WS-* standards
• it supports RESTful web services with JAX-RS

If you are currently using Axis (OGSA-DAI case) or Globus (which is modified Axis) consider switching to CXF. It supports all important industrial standards. In OGSA-DAI we should consider dropping support for Axis 1.4 and Globus 4.2 and focus on WS-* presentation layer and maybe add RESTful presentation layer - both backed by the same web service stack. It also brings us in line with the JEE industry standards.

Axis 1.x IS DEAD. Last release was in 2006! We could be a bit more responsive to the changing world :).