Back to Top

Monday, March 07, 2011

Microbenchmarking and you

Crossposted from the Transylvania JUG website.

Microbenchmarking is the practice of measuring the performance characteristics (like CPU, memory or I/O) of a small piece of code to determine which would be better suited for a particular scenario. If I could offer but one advice on this, it would be this: don't. It is too easy to get it wrong and bad advice resulting from bad measurement is like cancer.

If you don't want to take my first advice, here is my second advice: if you really want to do microbenchmarking watch this talk by Joshua Bloch: Performance Anxiety and use a framework like caliper, which I present below.

caliper is a Java framework written at Google for doing Java microbenchmarks as correctly as possible. To use, first you have to build it (there are no prebuild jars yet, nor is it present in the central Maven repository, sorry):

svn checkout caliper
cd caliper

Now you can start writing your benchmark. Benchmarks are written in a style similar to the JUnit3 tests:

  • you have to extend the class
  • your methods must conform to the public void timeZZZZ(int reps) signature
  • you can override the setUp and tearDown methods to implement initialization / finalization

Below is a simple example (taken from the caliper homepage):

public class MyBenchmark extends SimpleBenchmark {
  public void timeMyOperation(int reps) {
    for (int i = 0; i < reps; i++) {

To run this you have multiple possibilities:

  • Use the caliper script included in the code distribution (this is a SH script, so it won't work under Windows):
    ~/projects-personal/caliper/build/caliper-0.0/caliper --trials 10 org.transylvania.jug.espresso.shots.d20110306.MyBenchmark
    you can also execute the script without parameters to get a list and description of command line parameters.
  • Run it from your favorite IDE. You need to add the following libraries: allocation.jar, caliper-0.0.jar. The main class is and the parameters are the same you would pass to the caliper runner
  • Add a main method to your test class which would contain the following:
    public static void main(String... args) throws Exception {
      Runner.main(MyBenchmark.class, args);

By default caliper outputs an easy to understand text output. You have also the option to publish the benchmark as a nice HTML page (see this page for example). The publication is done trough a Google AppEngine app and is publicly available to anyone (a caveat to remember). For more information see the caliper questions on StackOveflow. You might also be interested in the java performance tunning website if you need to perform such tasks.


  1. Anonymous8:23 PM

    And for a piece of Perl code, what do you recommend? "Benchmark" module? (

  2. @Anonymous - yes, the Benchmark module looks good. Although I have not used it personally, the author list gives me great confidence that it works "as advertised".

    Also, Perl is purely interpreted (not JIT-ed - at least not in v5) and with a simple ref-counting GC, so you should be able to get more consistent results than with something as advanced as Java.

  3. Coming back after several years :-), but I remembered this blogpost when reading a summary of YAPC::EU: apparently the new recommended benchmarking module in Perl is Dumbbench.