Developers › A Journal from Behind the Trenches
Beware of misleading comments from Microsoft on JavaScript test suite "test262" and IE10
I notice that yet another "benchmark" blog post from Microsoft is making the rounds on the internet.This time they are bragging about having the highest score for JavaScript test suite "test262".
As one might expect, the impression people are getting when reading the blog is not one that matches reality. The results are misleading unless you read the "fine print", which seems to escape most people who comment on it. You see, while the IE10 preview appears to come out on top, the results are not what most people think they are.
A high score does not show overall compliance. It only shows whether you pass the currently available tests or not. This is a flashback to Microsoft's blog post on HTML5 compliance last year.
Here's what the actual test suite page says (emphasis mine):
test262 is not yet complete. It is still undergoing active development. Some portions of the ES5 specification have very complete test coverage while other portions of the specification have only partial test coverage. Some tests may be invalid or may yield false positive or false negative results. A perfect passing score on test262 does not guarantee that a JavaScript implementation perfectly supports ES5. Because tests are being actively added and modified, tests results from different days or times may not be directly comparable.
Summary:
- The test suite is not complete
- It is in active development
- It does not cover all parts of the specification
- There may be invalid tests
- Tests may yield false positive or false negative results
- A perfect score does not imply a perfect ES5 implementation
- Tests are being added and removed all the time
If you read the Microsoft blog post carefully, they do mention that it is a work in progress. But the way it is being reported on it's obvious that the message people read was "IE10 is the most compliant". The blog post also reveals that Microsoft submitted thousands of tests (which they should receive praise for), so it does not seem to be coincidental that they are reporting on it now.
In other words: While Microsoft briefly mentions in "fine print" that the test is not complete, it seems that most people are misled, and think the tests results say something that they don't.
Readers' comments 26
Left under the original article, as archived.
About this article
| Published in | A Journal from Behind the Trenches |
|---|---|
| Date | July 1, 2011 |
| Author | Haavard |
| Tags | test262, javascript, ecmascript, standards, microsoft, ie10, internet explorer |
| Original | Internet Archive |
Martin KadlecFriday, July 1, 2011 10:50:41 AM
João DavidFriday, July 1, 2011 10:52:33 AM
João EirasFriday, July 1, 2011 11:18:13 AM
Originally posted by BS-Harou:
+1
Michael A. Puls IIFriday, July 1, 2011 12:22:28 PM
Originally posted by BS-Harou:
Yeh, that's why a lot of the tests fail in Opera.
andrew walkerFriday, July 1, 2011 12:25:16 PM
Originally posted by BS-Harou:
And doesn't appear to be at all active on ES Discuss
andrew walkerFriday, July 1, 2011 12:28:55 PM
Charles SchlossFriday, July 1, 2011 2:20:30 PM
lots of them have no error
pause resume button might be buggy
now it is randomly stopingnow the test just broke itselfEricJHFriday, July 1, 2011 2:31:05 PM
There's lies, goddamn lies and communication strategies ....
MyOpera team, please fix this!Friday, July 1, 2011 3:25:38 PM
Originally posted by BS-Harou:
Quoted for Truthery. I'm surprised haavard didn't mention that Opera's the only modern browser with gaping holes in their es5 support.http://kangax.github.com/es5-compat-table/
Charles SchlossFriday, July 1, 2011 3:29:40 PM
JimFriday, July 1, 2011 5:26:42 PM
Originally posted by BS-Harou:
Originally posted by Chas4:
Wasn't Opera bragging about their Sputnik score before?
Oh btw Sputnik was integrated into test262 http://code.google.com/p/sputniktests/
We need a JS overhaul for multi threading, among many other updates like the webm/webp codecs, sqlite version update, and zlib/openssl license text needs updating in opera:about if your using the newer versions.
Ragnar IronbloodSaturday, July 2, 2011 7:21:55 AM
Opera 11.50 results:
Total: 5246
Succeeded: 5166
Failed: 80
Chromium 13 results:
Total: 5246
Succeeded: 5115
Failed: 131
Firefox 5.0 results:
Total: 5246
Succeeded: 5065
Failed: 181
all three browsers are fresh installs.
Ragnar IronbloodSaturday, July 2, 2011 8:07:04 AM
Opera 11.50 RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 322.9ms +/- 0.8%
--------------------------------------------
3d: 45.4ms +/- 1.7%
cube: 12.8ms +/- 2.4%
morph: 17.7ms +/- 2.0%
raytrace: 14.9ms +/- 2.7%
access: 41.7ms +/- 1.6%
binary-trees: 3.6ms +/- 10.3%
fannkuch: 21.2ms +/- 1.4%
nbody: 10.4ms +/- 3.5%
nsieve: 6.5ms +/- 5.8%
bitops: 15.1ms +/- 3.5%
3bit-bits-in-byte: 1.7ms +/- 20.3%
bits-in-byte: 2.5ms +/- 15.1%
bitwise-and: 2.0ms +/- 16.8%
nsieve-bits: 8.9ms +/- 2.5%
controlflow: 4.7ms +/- 7.3%
recursive: 4.7ms +/- 7.3%
crypto: 21.0ms +/- 2.3%
aes: 12.6ms +/- 2.9%
md5: 5.0ms +/- 0.0%
sha1: 3.4ms +/- 10.9%
date: 40.8ms +/- 1.4%
format-tofte: 18.8ms +/- 2.4%
format-xparb: 22.0ms +/- 1.5%
math: 29.7ms +/- 2.5%
cordic: 8.1ms +/- 5.0%
partial-sums: 16.1ms +/- 1.4%
spectral-norm: 5.5ms +/- 6.8%
regexp: 14.0ms +/- 0.0%
dna: 14.0ms +/- 0.0%
string: 110.5ms +/- 0.8%
base64: 10.8ms +/- 2.8%
fasta: 19.5ms +/- 1.9%
tagcloud: 34.0ms +/- 1.0%
unpack-code: 23.5ms +/- 1.6%
validate-input: 22.7ms +/- 2.1%
Firefox 5.0 RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 324.1ms +/- 0.7%
--------------------------------------------
3d: 53.8ms +/- 1.4%
cube: 21.6ms +/- 3.2%
morph: 9.5ms +/- 4.0%
raytrace: 22.7ms +/- 1.5%
access: 49.4ms +/- 2.2%
binary-trees: 6.0ms +/- 5.6%
fannkuch: 25.1ms +/- 2.5%
nbody: 7.6ms +/- 4.9%
nsieve: 10.7ms +/- 3.2%
bitops: 22.8ms +/- 1.3%
3bit-bits-in-byte: 1.0ms +/- 0.0%
bits-in-byte: 12.0ms +/- 0.0%
bitwise-and: 2.9ms +/- 7.8%
nsieve-bits: 6.9ms +/- 3.3%
controlflow: 3.8ms +/- 7.9%
recursive: 3.8ms +/- 7.9%
crypto: 24.1ms +/- 2.9%
aes: 13.4ms +/- 2.8%
md5: 6.6ms +/- 5.6%
sha1: 4.1ms +/- 5.5%
date: 41.5ms +/- 2.5%
format-tofte: 28.1ms +/- 1.4%
format-xparb: 13.4ms +/- 6.3%
math: 30.6ms +/- 1.6%
cordic: 7.3ms +/- 4.7%
partial-sums: 15.0ms +/- 0.0%
spectral-norm: 8.3ms +/- 4.2%
regexp: 18.2ms +/- 1.7%
dna: 18.2ms +/- 1.7%
string: 79.9ms +/- 1.2%
base64: 5.6ms +/- 6.6%
fasta: 16.5ms +/- 2.3%
tagcloud: 23.2ms +/- 1.9%
unpack-code: 25.7ms +/- 1.3%
validate-input: 8.9ms +/- 2.5%
Chromium 13 RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 376.0ms +/- 2.3%
--------------------------------------------
3d: 51.0ms +/- 11.7%
cube: 16.0ms +/- 15.2%
morph: 16.5ms +/- 2.3%
raytrace: 18.5ms +/- 24.0%
access: 43.0ms +/- 5.2%
binary-trees: 3.4ms +/- 10.9%
fannkuch: 24.2ms +/- 5.9%
nbody: 7.9ms +/- 5.1%
nsieve: 7.5ms +/- 19.2%
bitops: 49.3ms +/- 8.2%
3bit-bits-in-byte: 3.1ms +/- 7.3%
bits-in-byte: 7.9ms +/- 2.9%
bitwise-and: 26.6ms +/- 1.4%
nsieve-bits: 11.7ms +/- 35.6%
controlflow: 4.6ms +/- 15.0%
recursive: 4.6ms +/- 15.0%
crypto: 23.3ms +/- 3.8%
aes: 11.8ms +/- 9.8%
md5: 6.1ms +/- 6.7%
sha1: 5.4ms +/- 6.8%
date: 56.7ms +/- 1.5%
format-tofte: 19.3ms +/- 3.9%
format-xparb: 37.4ms +/- 2.4%
math: 31.0ms +/- 5.4%
cordic: 9.3ms +/- 10.9%
partial-sums: 17.1ms +/- 3.1%
spectral-norm: 4.6ms +/- 25.6%
regexp: 13.5ms +/- 6.7%
dna: 13.5ms +/- 6.7%
string: 103.6ms +/- 0.9%
base64: 6.8ms +/- 4.4%
fasta: 17.6ms +/- 2.1%
tagcloud: 25.7ms +/- 1.9%
unpack-code: 35.9ms +/- 1.7%
validate-input: 17.6ms +/- 2.1%
Charles SchlossSaturday, July 2, 2011 2:33:05 PM
AsiresSunday, July 3, 2011 8:13:00 AM
Charles SchlossSunday, July 3, 2011 3:13:16 PM
Originally posted by Asires:
still will depend on the hardware and how updated the drivers are, I have seen a 100 ms difference on sunspider between an older and new machine, using the same browser
ChrisSunday, July 3, 2011 9:03:44 PM
Originally posted by toyotabedzrock:
Really? You mean this, where the Sputnik creators are being praised for publishing the test like that, and where there is no bragging whatsoever?And of course the difference here, where Sputnik was not made by Opera at all, while test262 had a significant number of tests recently submitted by Microsoft.
So are you going to apologize now?
eliotcougarMonday, July 4, 2011 12:51:27 PM
Most of the other so called "errors" I see are:
1) Not throwing an exception where it should be thrown — this increase JS engine performance, BTW...
2) Errors which are, if fixed, may cause a lot of loose code to fail unexpectedly on a lot of random websites...
UPD: 1000+ errors are caused by not implemented "Strict Mode"... This test is so not uniform... Some features give you 1000 points and others only 10-20...
Martin KadlecMonday, July 4, 2011 6:14:29 PM
not true, plenty of modern websites and apps are using it. They often provide ES3 fallback but that is not good solution.
Constantine VesnaFriday, July 8, 2011 3:36:33 PM
also:
sputnik is testing ES3 compliance, and it is a part of test262 suite.
sunspider, v8 and kraken test js speed, they are not relevant to this discussion.
Hallvord R. M. SteenThursday, July 21, 2011 4:12:31 PM
They will be..
Martin KadlecThursday, July 21, 2011 5:30:06 PM
Originally posted by hallvors:
Not true, Opera supports Array extras from ES5, and some other things like Array.isArray, Date.prototype.toISOString, String.prototype.trim, etc.
Constantine VesnaThursday, July 21, 2011 7:59:03 PM
Originally posted by hallvors:
Now that's a good news ! We eagerly await
Matheus S. BitencourtSunday, August 7, 2011 9:33:28 PM
ChirpieTuesday, August 9, 2011 8:20:07 PM
Originally posted by macusking:
Complete and utter nonsense. Opera is now faster than any version before it, and one of the fastest browsers on the market.
Good job. You were fooled by Google's aggressive marketing.
Matheus S. BitencourtSunday, September 4, 2011 6:27:06 PM
Originally posted by Chirpie: