A better benchmark: Benchmarking Browsers with Real Websites

July 3, 2009 · by Haavard

We are all used to all the artificial JavaScript benchmarks out there by now. What they have in common is that they all test small parts of JavaScript, and none of them are testing real-life performance.

Now someone has spoken up, and pointed out that JavaScript-only benchmarks are a bad idea:

The problem with browser benchmarks today is that they are extremely synthetic. A majority of them are Javascript only because its easier to test.

Better yet, they created a benchmark which measures the actual loading time of different sites in various browsers. This means that the benchmarks do not rely on network speed, nor are they narrowly focusing on JavaScript.

The results will probably vary depending on what sites you choose to test, but the average loading times in this particular test should put to rest the misconception that Opera has somehow fallen behind when it comes to performance.

Results:

Opera
Safari
Chrome
Firefox
IE

I don't think Carakan is going to have much of an impact on these more realistic benchmarks. But people will probably continue to quote the flawed JS-only benchmarks in the future, so it will be nice to beat everyone else there too.

Readers' comments 12

Left under the original article, as archived.

João EirasFriday, July 3, 2009 11:26:32 AM
Originally posted by haavard:

so it will be nice to beat everyone else there too.

Nice statement
Joonas LehtolahtiFriday, July 3, 2009 11:41:06 AM
I have to agree on my own experience. While I have seen that currently Opera's JavaScript performance is not on par with other browsers (Firefox and Chrome both seem to be much better in this regard), the overall browsing experience still feels smoother with Opera. This is especially on sites that focus on lots of content and less JavaScript.

Can't wait for Carakan so that Opera goes back on top with JavaScript too.
David BloomFriday, July 3, 2009 12:41:38 PM
To be fair, this guy's tests aren't very realistic either. They don't factor in network latency, which would benefit Chrome and IE8 because they can parse ahead to speculatively download external resources.
Rafael LuikFriday, July 3, 2009 1:09:45 PM
That post is awesome!
Opera Rules!
HaavardFriday, July 3, 2009 1:09:52 PM
It seems that he specifically wanted to leave out network factors. Those can be extremely variable. He also addresses it in the FAQ:
Other features like DNS look-ahead (likely cached) or different networking engines probably costs microseconds compared to the milliseconds it takes for a page to load.
Yeni SetiawanFriday, July 3, 2009 2:38:33 PM
Agree, we can't judge a browser's performance just based on javascript test.
David BloomFriday, July 3, 2009 3:24:47 PM
Originally posted by haavard:

It seems that he specifically wanted to leave out network factors. Those can be extremely variable.

Not if you test it right. There are proxies out there that can simulate latency and limited bandwidth in a consistent manner. These factors could even have been simulated by simply using a PHP script that sleeps to delay the response.

Edit: Really, I'd like to see a test of multiple browsers against multiple network connection and system performance factors, so users can see what is the fastest browser for a slow computer on a fast connection, a fast computer on a high-speed but also high-latency connection (sattelite?), etc.
Khaled KhalilSaturday, July 4, 2009 5:47:15 AM
i just tested the benchmark (with the same sites) by my own, and got even weirder Opera/FF ratio.
AMD Athelon Neo 64 1.6 GHz 2 GB memory
LinuxMint 7 64 (kernel 2.6.28-11)
Opera 10.00 Beta Build 4464 (Site Average: 89.7 ms)
Konqueror 4.2.2 (Site Average: 194.4 ms)
FireFox 3.0.11 (Site Average: 285.4 ms)

to be fair, keep in mind my experience was applied on latest Opera snapshot, while for Firefox and Konqueror they are latest stable release.
Khaled KhalilSaturday, July 4, 2009 5:55:42 AM
Originally posted by haavard:

I don't think Carakan is going to have much of an impact on these more realistic benchmarks
as long it will not be worse it will be more than welcomed by users, Opera is already excellent in CSS and DOM, most of the claims about its downsides goes for JS, so if it could be handled as good as CSS, that of course make difference.
João EirasSunday, July 5, 2009 1:29:55 PM
iPhone OS 3 on iPod touch
http://www.pcmag.com/article2/0,2817,2349677,00.asp
And despite the much faster JavaScript performance, web pages load slightly slower in OS 3.0 than in 2.2.

Which means, javascript performance is hardly significant most of the times.
okeSunday, July 26, 2009 5:29:23 PM
@Haavard
I know Opera does not give dates for proposed releases but can you at least give us some estimates when we might start to see action on Carakan? (this year still, first half 2010?)

I mean the latest news on Carakan was 5 Feruary only.
Cutting SpoonSaturday, August 22, 2009 9:39:27 PM
Well Opera ran great on my sister's laptop. 9.64 she's got.

800MHz Coppermine Celeron.
128MB RAM.

No skipping on YouTube, and with VLC plugin she could play 720p content in the browser.

Funny Opera's recommended requirement is Firefox's minimum. And it still runs on practically no memory. Even win98/32MB won't kill a person.