Friday, 14 June 2013

Android Browser Benchmark Comparison (June 2013) - Chrome vs Firefox vs Opera vs Next

Due to the openness of the Android platform, the end user of an Android device has a wealth of browsers to choose from.
Although many of the users would be happy by simply going with the default Chrome browser that comes installed by default in every new device, there are other sounding names in the market that may appeal to the power users due to their ability to sync with other desktop browsers of choice for instance.
So, let's see how the main 3 browsers on the Android plus the new comer - Next Browser - compete against each other.

The Browsers

The Tests

  • Sunspider: SunSpider is a benchmark suite that aims to measure JavaScript performance on tasks relevant to the current and near future use of JavaScript in the real world, such as encryption and text manipulation. The suite further attempts to be balanced and statistically sound.
  • Peacekeeper: Online speed test by Futuremark, mainly using rendering, mathematical and memory operation.
  • Octane: JavaScript test suite by Google, used to optimize Google Chrome web browser.
  • Kraken: A test suite by Mozilla, and according to them: More than Sunspider, V8, and Dromaeo, Kraken focuses on realistic workloads and forward-looking applications.
  • RoboHornet: RoboHornet is a benchmark designed around performance pain points real web developers care about. Designed to be as independent as possible, might be the future big test that all browsers should care about.
  • Wirple BMark: Performance test for HTML5 3D applications. Tests performance in both Canvas3D and WebGL.
  • HTML5 Test: The HTML5 test score is an indication of how well your browser supports the upcoming HTML5 standard and related specifications.
  • Acid3 Test: Acid3 test is a web test page from the Web Standards Project that checks a web browser's compliance with elements of various web standards, particularly the Document Object Model (DOM) and JavaScript.

The Methodology

All the test where carried out in an Nexus 7 device running Android 4.2. No other services where running except for the browser and the test itself. The tests that don't give a statistically significance for their results, where run several times and the results averaged in the final score.


SunSpider JavaScript 1.0 Benchmark

SunSpider is a JavaScript benchmark developed by Apple's WebKit team since 2007. The main focus of the test is to carry out the so called real-world tests. This is, the test aims to benchmark problems that your browser has to solve many times during your browsing sessions. Things like text manipulation or encryption.

There is a clear winner in this test, Firefox with Chrome coming out in the end. Firefox manages to be 20% faster than Chrome in this one.
Being that this test is made by an independent entity to the these browsers, it has an important meaning for this test.
Winner: Firefox (+1)
Looser: Chrome, Next (-1)

Futuremark's Peacekeeper Benchmark

The Peacekeeper test, is developed by the Finish company Futuremark, dedicated to producing benchmarks both for the home and business sectors. The test caries out mainly rendering, mathematical and memory operations.

There is no clear winner of the test, but Opera comes out on top, closely followed by Chrome. There is a clear looser tough, Firefox, very distanced from the rest of the pack with a score of only 63% of the winner.
Winner: Opera, Chrome (+1)
Looser: Firefox (-1)

Octane v1 Benchmark

This is Google's benchmark and the one they actively use to test their new releases of Chrome. We would expect Chrome to fair better than the other browsers in this one, but let's see the results.

Chrome does indeed come ahead, but by an insignificant margin compared to Opera. The Next browser is the clear looser of the test fairing a score of 80% of the winner.
Winner: Opera, Chrome (+1)
Looser: Next (-1)

Kraken 1.1 Benchmark

Kraken is for Mozilla what SunSpider is for Apple. A very throughout test designed to benchmark some real world test scenarios. We would expect Firefox to come out on top in this one, since it's one of the choice tests carried out during this browser's production.

The winner here, is not Firefox like expect, it's actually Chrome followed very closely by Opera. Firefox doesn't fare that well, but the clear looser is Next, taking 15% more time to finish the test.
Winner: Chrome, Opera (+1)
Looser: Next (-1)

RoboHornet RH-A1 Benchmark

This is a new benchmark behind worked out by Google's team. It's still in alpha state but it's made in a way that both contributors and developers can vote what they would like to see tested in order to make an independent and modular browser test platform.

In the future we expect this test to become one of the main ones for every benchmark. For now we must put it in context by comparing it with the other ones.

Surprisingly Next comes way out on top in this test. I remind you that these are average results taken from a number of tests, so this is not just a statistical glitch you are seeing.
Chrome comes out in the end with only 50% of the score of our winner here but Opera doesn't fare that much better.
The real looser of the test is actually Firefox that wasn't able to carry out the test successfully so in here we present the results for running only the part of the tests that Firefox was able to carry out.
Winner: Next (+1)
Looser: Firefox (-1)

Wirple BMark

Wirple is not a test we must be overly concerned for the present time on mobile browsers. It tests 3D browser rendering in both Canvas3D and WebGL, something that we still don't expect mobile devices to excel when running application within a browser. Still, for the sake of comparison, where are the results.

The fact that all browsers where able to cary out the test is already a victory in this category. Chrome is the fastest one, followed closely by Next Browser. Opera and Firefox come out in the end.
Next Browser, although able to finish the test, showed some font display problems during the test.
Winner: Chrome (+1)
Looser: Opera, Firefox (-1)

HTML5 Test

This test, unlike all those before so far, is not made to measure performance but to measure compatibility with the latest HTML5 standards. Basically is a test determined to check if your browser will be able to correctly display the information it interprets from the server.

In the fist place we have Opera. Firefox and Chrome fail a bit behind but they still manage to keep up fine. Next on the other side of the picture, gets quite a low score in the test and is the real looser here.
Winner: Opera (+1)
Looser: Next (-1)

Acid3 Test

Just like the HTML5 test, the Acid3 test aim is not to measure the speed of execution, but the accuracy of that execution. Acid3 test checks the browser's compliance with DOM and Javascript.

All the browsers managed to finish the test with a score of 100%, but Firefox displayed rendering warnings during the test (an important point since the test is a measure of the scores but also of the perceived visual accuracy). As such, Firefox is really the only looser here.
Winner: Opera, Chrome, Next (+1)
Looser: Firefox (-1)


So, let's take a look at the final score:
  • Chrome: 4
  • Firefox: -3
  • Opera: 4
  • Next: -2

And the winner is are:
Chrome & Opera

When we begun this test on the Android platform, we might have been lead to think that Chrome would obviously come out on top and then the real battle would be for the 2nd place between the other contenders.

Actually the picture is quite different.

The new contender, Next Browser lagged behind in most of the tests, but it's actually quite capable of carrying out most of the tasks assigned to it and was able to carry out practically all the tests - except for a small glitch in the Wirple BMark - without any problems. This browser has quite a few nice add-ons that are great for mobile browsing and tablet use.

Firefox was in fact the most problematic of the browsers since it wasn't able to finish the RoboHornet test and had some complications passing the Acid3 test. Being our main desktop browser, we really wanted a good reason to use it in the Android platform in order for syncing convenience, but the fact is that Firefox needs quite a re-work on the mobile front in order to be able to compete against the other browsers

Opera was actually very close to Chrome in most of the tests. Being this their first release since moving to the WebKit platform, these are very good news. So, at the present time, we believe it's just a matter of personal preference of which browser you decide to use. For heavy desktop users of Chrome the answer might be simple, but for the ones using some other browser or - even better - Opera's own desktop products, you should probably install Opera on your Android device and give it a run.

We are actually quite curious to see what will happen in the future as Opera matures their WebKit platform browser and what will happen to Next. Will it just fade like many of the newcomers before it, or be able to be a viable alternative in the mobile browsing world.