Comparative Performance modified by Brian Burton

Brian Burton — Sun, 02 Mar 2014 22:55:15 -0000

Let's start off by stating the obvious. Mutable collections are faster than immutable ones. That's a basic fact of life. Consider a hash table for example. A mutable hash map can simply replace a value in an internal array in response to a put() call while an immutable one has to create a number of new objects to build a new version of the map reflecting the change. So, yes, mutable collections are faster.

The real questions are: how much faster are mutable collections and will you really notice the difference. Based on benchmark runs a JImmutableHashMap is about 2-3 times slower than a HashMap but is about 1.5x faster than a TreeMap. Unless your application spends most of its time CPU bound updating collections you probably won't notice much of a difference using an immutable collection.

Here is a sample run using the org.javimmutable.collections.hash.TimingComparison benchmark program that comes with javimmutable-collections. The program uses a random number generator to create sequences of puts, gets, and deletes. The program can be tweaked in a variety of ways but the primary setting is the number of loops (operations) to perform. The program repeats the same series of random operations on a TreeMap, a JImmutables.map(), and a JImmutables.array(). These test runs were performed on a MacBook Pro with heap settings -Xms384m -Xmx512m.

-- subset of results for 250k ops run using TreeMap averages include many other runs
java map adds 74847 removes 24925 gets 150228 size 74814 elapsed 80
jimm map adds 74847 removes 24925 gets 150228 size 74814 elapsed 57
jimm ary adds 74847 removes 24925 gets 150228 size 74814 elapsed 50

java map adds 75044 removes 24846 gets 150110 size 75013 elapsed 80
jimm map adds 75044 removes 24846 gets 150110 size 75013 elapsed 55
jimm ary adds 75044 removes 24846 gets 150110 size 75013 elapsed 50

java map adds 75093 removes 24995 gets 149912 size 75050 elapsed 81
jimm map adds 75093 removes 24995 gets 149912 size 75050 elapsed 55
jimm ary adds 75093 removes 24995 gets 149912 size 75050 elapsed 51

java avg: 79.6  hash avg: 55.6  array avg: 50.0

As you can see the TreeMap completed 250,000 operations in about 80 milliseconds on average winding up with a map containing 75k elements. JImmutables.map() completed the same operations in 55 ms and JImmutables.array() in 50 ms. Using the same program a HashMap averages around 21 ms.

Increasing the number of operations to 500k (double) the average times are 52 ms for HashMap, 200 ms for TreeMap, 128 ms for JImmutables.map() and 118 ms for JImmutables.array(). The resulting collections contain approximately 150k elements. For the same run JImmutables.sortedMap() takes approximately 375 ms.

Cranking the number of operations up some more (1.5 million) we get these times: HashMap 262 ms, JImmutables.map() 530 ms, JImmutables.array() 476 ms. The final collection contained 448k elements. For these I used heap settings -Xms384m -Xmx768m.

-- subset of results for 1.5 million ops run using HashMap averages include many other runs
java map adds 449849 removes 149741 gets 900410 size 448534 elapsed 269
jimm map adds 449849 removes 149741 gets 900410 size 448534 elapsed 521
jimm ary adds 449849 removes 149741 gets 900410 size 448534 elapsed 466

java map adds 449889 removes 149428 gets 900683 size 448581 elapsed 267
jimm map adds 449889 removes 149428 gets 900683 size 448581 elapsed 517
jimm ary adds 449889 removes 149428 gets 900683 size 448581 elapsed 465

java map adds 450106 removes 149794 gets 900100 size 448768 elapsed 265
jimm map adds 450106 removes 149794 gets 900100 size 448768 elapsed 522
jimm ary adds 450106 removes 149794 gets 900100 size 448768 elapsed 462

java avg: 260.0  hash avg: 505.9  array avg: 456.2

From these test runs with maps it's clear that java.util.HashMap is wicked fast as would be expected but the fully immutable alternatives are still within a factor of 2-3 of it even for collections as large as 448k elements. Saying something is 2-3 times slower than something else sounds bad but keep in mind that this test performed one and a half million operations, wound up with a collection of 448k elements, and the difference in execution time between the best mutable and immutable versions was only about 245 milliseconds! In exchange for that quarter second your program would have all the benefits of immutability including:

elimination of the need for locking or any potential for lock contention impacting performance
elimination of the need for defensive copying or any potential for different parts of the program using the same map changing the data unexpectedly and breaking other parts of the program
improvements in maintainability from ability to reason about how and when collections might change

If your program is CPU bound and works with enormous data structures it might require the use of mutable data structures. However for most programs switching to fully immutable data structures would have no noticeable effect on performance but would provide major benefits in the design and maintainability of the system.

Recent changes to Comparative Performance

Comparative Performance modified by Brian Burton