, , , , , ,

SunSpider Benchmark: Heralded, maligned and misunderstood benchmark

If you read the comments to smartphone reviews, you will find comments maligning the use of the SunSpider benchmark. There is nothing wrong with running SunSpider benchmark on a mobile phone. It is the way it is used. Using the SunSpider benchmark to declare a phone as the fastest in the land is erroneous.

SunSpider benchmark results are used out of context

SunSpider is a “benchmark (that) tests the core JavaScript language only, not the DOM or other browser APIs. It is designed to compare different versions of the same browser, and different browsers to each other.”  The SunSpider benchmark is designed as a web browser benchmark. It was never intended by its developers to be used to compare the performance of different mobile phones. The SunSpider benchmark is also specifically a JavaScript speed test. It is not a test to determine how fast a web page will load.

JavaScript is used in a web browser to create image rollovers or those online for calculators you see embedded in web pages. Not all websites use JavaScript, especially mobile websites. Using the SunSpider benchmark as a basis for determining smartphone performance, is using one small aspect of smartphone to determine which is the fastest.

Still, does this mean that it is not a good test of smartphone performance?

The SunSpider benchmark is not a hardware stress test

I conducted some SunSpider 1.0.1 tests from an Apple iPhone 5 and Samsung Galaxy S4 i9505. Here are the SunSpider benchmark results I obtained from these two phones: (Lower is better.)

Apple iPhone 5

  • Safari – 699.7 ms
  • Chrome – 3684.4 ms

Samsung Galaxy S4 i9505

  • Stock Webkit Browser – 1041.4 ms
  • Chrome –  1146.1 ms
iOS-Safari
Apple iPhone 5 with Safari
iPhone-Chrome
Apple iPhone 5 with Chrome
i9505-webkit
Samsung Galaxy S4 i9505 with Stock Webkit Browser
i9505-Chrome
Samsung Galaxy S4 i9505 with Chrome

Okay, I will leave it with you to try to make heads or tails of these results. I should note, despite the disparity in the SunSpider 1.0.1 benchmarks on the iPhone, I really cannot tell much of a difference when browsing the web. Both web browsers seem fast. If anything, I get the impression that Chrome is actually faster. On the other hand, it is easy enough to understand if we accept that SunSpider really is neither a hardware stress test nor a Web page loading test. What you are really testing here is the efficiency of different browsers in rendering JavaScript. That is really all the SunSpider benchmark is testing.

SunSpider optimization is not a bad thing

Some reviewers have started saying that SunSpider is not a useful smartphone benchmark because it has become the target of optimization. There is nothing wrong with smartphone manufacturers, or more specifically web browser developers, in optimizing for better SunSpider benchmark performance. That is why the benchmark was created in the first place.

Better SunSpider benchmark performance is an indicator of better JavaScript performance. Take it for that and nothing more. It really was never intended to be a benchmark of smartphone performance or Web browser page loading time.