Developers › A Journal from Behind the Trenches

Microsoft's IE9 standards tests vs. reality

March 17, 2010 · by Haavard

The good news is that Internet Explorer 9 supports SVG!

The bad news is that Microsoft's standards support table could mislead people into thinking that IE9 is more standards compliant than other browsers.

Since I'm already talking about SVG, let's look at that as an example. If you read the description on their page, you will notice that it doesn't actually show SVG compliance as such. It shows how each browser does when running the 31 tests Microsoft created, when even the SVG 1.1 Tiny test suite has more than 150 different tests.

CodeDread has a published list detailing SVG support in different browsers. As you can see, IE9 still does poorly compared to other browsers. So while Microsoft's own page would give you the impression that IE9 has excellent SVG support, that is not the reality.

It's great that IE9 will support SVG, but I think Microsoft's page is rather misleading. Let's hope they are planning to make use of the full test suite at some point.

In conclusion: Microsoft should update the page to make it more obvious what it actually shows. It currently gives the false impression that IE9 beats all other browsers at standards compliance.

Not cool, Microsoft. Not cool.

Readers' comments 50

Left under the original article, as archived; 1 page of comments could not be recovered.

eliotcougarWednesday, March 17, 2010 1:30:55 PM
Ha!.. Did anyone have any doubts?..
Daniel AleksandersenWednesday, March 17, 2010 1:43:19 PM
Isn’t all their marketing like this?
Krio LythWednesday, March 17, 2010 1:58:18 PM
Unfortunately, all SVG test pages are unavailable now at Microsoft Test Center. We can't even say now if their tests were broken or not.

However, if we assume that their tests stick to standards, it's "clean" marketing - no lying while presenting good sides and concealing ugly ones.
FransWednesday, March 17, 2010 2:12:37 PM
I was already wondering how they came up with their values.
Brian HuismanWednesday, March 17, 2010 2:46:06 PM
Even though they spun their results to compare other browsers to the short list of things IE9 does support, it is good to see them releasing yet another test suite. More test suites are always good news. Let's get the other browsers moving on the tests that IE has cherry-picked for this list.
FransWednesday, March 17, 2010 2:51:37 PM
Originally posted by Brian Huisman:

Let's get the other browsers moving on the tests that IE has cherry-picked for this list.

Assuming they are correct. I recall seeing one or two wrong CSS2.1 tests when I looked over those Microsoft-submitted testcases and I only randomly clicked through a few.
RijkWednesday, March 17, 2010 3:31:10 PM
@Frans: the numbers here are more manageable, so I suppose these tests will get reviewed quickly enough. For a moment I thought they were *only* summitting tests where at least one competitor fails, but they aren't that nasty

BTW, IE9 does pass all the test in the automatic Selectors text, which is very nice.
sirnh1Wednesday, March 17, 2010 4:04:00 PM
Originally posted by haavard:

but I think Microsoft's page is rather misleading.

It's Microsoft, what else did you expect?

(I would really like to see test results, from test not done by microsoft, because all I have is windows XP and that one doesn't support IE9 ..., so thank your for the link to codedread )
PlazzmexWednesday, March 17, 2010 4:40:43 PM
People. I think we need to share this post as for much people as we can. Post it on your blogs, on your websites, forums etc..
LuchioWednesday, March 17, 2010 6:57:19 PM
Well, they are friendly enough to provide test cases and tell you where each browser fails. Aside from the doubtful marketing standpoint (releasing only tests that they pass!), I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.
prd3Wednesday, March 17, 2010 8:21:24 PM
Originally posted by Luchio:

I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.

The official testsuite already does that. Also, Microsoft's tests contain several errors. So no, Microsoft's tests are not useful. If they had bothered to let other people check them first, perhaps. But right now it's just wrong marketing garbage.
Cutting SpoonWednesday, March 17, 2010 9:31:58 PM
"This is how many tests (in each set) our browser will pass"

...or that's the message *I* took away from it.

http://ie.microsoft.com/testdrive/Default.html

Opera 10.51 RC1 handles most of that crap easily. On an N280 netbook running XP, in Power-Saver settings, with 10.10 running more than a hundred tabs in the background. They've got a ways to go.
d4rkn1ghtWednesday, March 17, 2010 9:48:10 PM
Originally posted by mgillespie:

It's Microsoft, the most corrupt corporation on this planet, they will do what they need to do to ensure domination and control in every sector they trade in.

And don't forget that some people still swear by that serious flawed company. They love to put down on Opera, Apple, Linux, or any other company that truly had innovated.
AbandonShip?Wednesday, March 17, 2010 10:32:37 PM
Thanks for 100% clearness, haavard.

At the first moment I saw the chart, I was shocked,
but I thought later, this this can not be true, Opera "was"
the the browser with the best SVG support.
Now I feel reassured.
Some things seems to never change at Micro$oft.

Edit: I hope the people will spread it.
Daniel HendrycksWednesday, March 17, 2010 10:55:05 PM
Can they just be honest, once?
AtTheOperaThursday, March 18, 2010 1:12:10 AM
Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.
David BruantThursday, March 18, 2010 1:26:55 AM
My impression is that they are trying to have their Acid test. The same way everyone was pointing at IE for their ridiculous score on Acid3, IE's trying to point at everyone else by submitting tests on which other major browsers fail.
Their dark green "100% line" is a very efficient communication idea.
Daniel HendrycksThursday, March 18, 2010 1:47:30 AM
Originally posted by AtTheOpera:

Do you do anything other than complain?

He is informing people that Microsoft is lying.
Originally posted by AtTheOpera:

Microsoft is truly making positive strides

True, but they should not lie about it.
AtTheOperaThursday, March 18, 2010 2:03:08 AM
The tabs below provide details and links to each of the tests we submitted to the various W3C working groups to help the web become more interoperable.

Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.
Daniel HendrycksThursday, March 18, 2010 2:16:46 AM
Originally posted by AtTheOpera:

It doesn't claim to be all tests or the best tests, but ones that MS submitted.

But, it certainly was quite a diminuendo. Don't worry, Haavard was just disliking the fact that they used tests created by themselves to make it look 5x better than what it is. Kinda like Chrome and the V8 benchmark. And, obviously, he did not like that they did not implement Theora.
Cutting SpoonThursday, March 18, 2010 2:37:04 AM
Originally posted by AtTheOpera:

Not the emphasis on "test we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.

Oddly, your typo makes your statement "correct". There was NOT real emphasis on the submission component, at a glance it looks like the inclusive listing of all important tests. Why? Well, most people seem not to even know what a "browser" is...

Pity that Apple's flawed xServe comparison page is no longer up.
prd3Thursday, March 18, 2010 10:44:52 AM
Originally posted by AtTheOpera:

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.

So it's OK for Microsoft to post misleading claims, but not to point out when they do so?

If Microsoft is making such positive strides, why can't they do it without lying?

What good do positive strides to if they are going to keep lying to people?

It's like one step forward, two steps back.

You know as well as everyone else that the page is blatantly misleading, and now lots of people think IE9 has better support for open standards than other browsers.

Why are you defending liars and attacking people who expose liars?
prd3Thursday, March 18, 2010 10:46:47 AM
Originally posted by AtTheOpera:

Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.

Never mind the fact that there are huge errors in their tests... If Microsoft had been honest, they would have run browsers through the full existing testsuites instead of cherry-picking a tiny amount of tests they optimized IE for.

Why didn't they optimize IE for the existing tests instead? Why did they have to write their own (with multiple errors)?

We both know the answer to that.
FransThursday, March 18, 2010 10:53:40 AM
Originally posted by AtTheOpera:

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.

Complaining that Opera didn't support border-radius while it supported much more useful CSS3 selectors near-perfectly for ages is chopping down. Pointing out that Microsoft is making misleading statements is chopping down? Hah!
Originally posted by AtTheOpera:

Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.

It's more like the small print than "the emphasis."

By the way, those CSS3 selector tests (and various others) are very inaccessible behind iframes etc. so frankly I can't be bothered to look at them. However, I'm quite inclined to agree that at least some of them are probably nonsense as prd3 says (I assume he checked them or some such? prd3?). Like I said, a bunch of those CSS2.1 ones sure were.
Mağruf ÇolakoğluThursday, March 18, 2010 11:03:49 AM
http://www.operaturkiye.net/microsoftun-ie9-icin-svg-testi-ve-bazi-gerceklerhttp://files.myopera.com/Tamil/Smilies/Happy.gif -
Charles SchlossThursday, March 18, 2010 12:55:40 PM
Back to their old tricks

Tho I wonder how they maintain 4 different versions of IE at the same time? 5 tru 8
LuchioThursday, March 18, 2010 2:04:53 PM
Originally posted by prd3:

Originally posted by Luchio:

I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.
The official testsuite already does that.

For SVG, yes, but there are other subjects such as CSS that are tested there.

Originally posted by prd3:

Also, Microsoft's tests contain several errors.

Not trying to defend Microsoft here, but could you give specifics? Or do you just assume that the tests are wrong because they come from the MS Factory?
FransThursday, March 18, 2010 2:09:15 PM
Anybody with Vista SP2 or higher, how does IE9 do on this test?

Oh, and this one?
icareThursday, March 18, 2010 6:15:47 PM
There's something strange with their test page. With the SVG, for instance, they say Opera fails at "Default values of 'x', 'y', 'width', and 'height' attibutes for the 'svg' tag" and at the "Zero-length paths and 'stroke-linecap' attribute" and indeed on their test page, it fails.
Now, if I'm not wrong the first one refers to chapter 5.3, yet the official test, on the w3.org website seems to work pretty well.
The second one seems to refer to chapter 11.4 and yet the official test seems to work pretty well again.
Now, I don't know a lot about this coding stuff so maybe I'm all wrong, maybe I didn't see something but I'd like someone more experimented to tell me if I'm wrong or Microsoft did "forgot" something in their tests.
FransThursday, March 18, 2010 6:39:45 PM
I don't think the test you linked to for the first one tests the same thing? Anyway, given Fx' and Chromium's behavior this does indeed seem to be a bug in Opera, though I can't seem to find the relevant spec describing the behavior.

I think the second one is more related to pathLength here? http://www.w3.org/TR/SVG11/paths.html#PathElement
FransThursday, March 18, 2010 8:48:14 PM
http://www.glazman.org/weblog/dotclear/index.php?post/2010/03/18/Microsoft-you-should-be-ashamed

Someone took a look at some of the CSS tests.
Daniel HendrycksThursday, March 18, 2010 9:34:52 PM
Originally posted by Luchio:

Not trying to defend Microsoft here, but could you give specifics?

Coding errors. There are invalid pages according to the W3C Validator.
FransThursday, March 18, 2010 9:36:43 PM
Originally posted by Daniel James Hendrycks:

Coding errors. Live invalid pages according to the W3C Validator.

I thought they took the SVG Validator down?
Cutting SpoonFriday, March 19, 2010 6:48:03 AM
Let's see if they become the first to release a "slow" hardware-accelerated interface.
AdrianPMSaturday, March 20, 2010 12:51:26 AM
Just when I was getting enthusiastic about IE 9 and its makers' new commitment with REAL web development, this comes to light...

Personal memo: Don't be too enthusiastic with new IE release announcements, ever again.
FransSaturday, March 20, 2010 8:53:38 AM
Originally posted by Keldian.-:

ust when I was getting enthusiastic about IE 9 and its makers' new commitment with REAL web development, this comes to light...

Personal memo: Don't be too enthusiastic with new IE release announcements, ever again.

In all fairness, they are improving leaps and bounds. I just wish they wouldn't lie about how much they improved.
KirillSaturday, March 20, 2010 9:11:19 AM
http://blogs.msdn.com/ie/archive/2010/03/19/the-internet-explorer-testcenter-welcome-page-clarification-corrections.aspx
FransSaturday, March 20, 2010 9:16:42 AM
That is good news, though I really don't see the use for the table. I would think it should be fairly easy for any browser vendor to pick a number of tests they pass (or write new ones and make sure they pass, same difference), consequently perhaps do some extra testing to increase the number of tests failed by other browsers and then publish it. IE9 passing all of its own testcases is meaningless and I can only think of intent to mislead or lack of thought as the reason behind publishing it.
Cutting SpoonSaturday, March 20, 2010 9:19:43 AM
Well, it keeps their name in the press.
FransSaturday, March 20, 2010 9:25:12 AM
I'd say that falls under intent to mislead, though probably no worse than most marketing when interpreted like that.
icareSaturday, March 20, 2010 7:00:31 PM
Originally posted by Frenzie:

I don't think the test you linked to for the first one tests the same thing? Anyway, given Fx' and Chromium's behavior this does indeed seem to be a bug in Opera, though I can't seem to find the relevant spec describing the behavior.

The microsoft URL contain the words "/chapter_05.3.svg" and "/chapter_11.4.svg" at the end. That's why I thought it was refering to this and this (plus the word "stroke-linecap" in the second test). But you're probably right, if Fx and Chrome pass the test, it must be something in Opera...
Thanks for your attention.
PozsonyiMarkSaturday, March 20, 2010 11:33:35 PM
Microsoft?
Failsoft?

They can't hide the truth from the experts
DaringineerSunday, March 21, 2010 2:06:42 AM
Originally posted by codedread:

IE9 Preview 1 is the first Microsoft browser to natively support any form of SVG. Let's hope they achieve parity with Firefox 1.5 by the time it is released.
coolfigaroMonday, March 22, 2010 9:16:40 PM
While I agree that at first glance the IE Testing Center page does make the IE9 preview look more stellar than it might be, I don't think it was that egregious.

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.

The page has good text describing how these were specific ambiguities that MS wanted to clarify in the specs. Their preview is intended for developers. When a developer takes a look at a page describing standards (or really anything technical for that matter), he or she should know better than to just look at the graphics. I sincerely hope you don't take that approach to reading the standards themselves. That's how you end up with crap like this: http://imgur.com/oMgGG.png

Bottom line: if you really like web standards, then you should be encouraging the work they're doing to catch up and to clarify/improve the standards.
Cutting SpoonMonday, March 22, 2010 9:40:38 PM
1) Whose site put forth that test? I fail to spy an address.

2) A cropped graph? Not nearly so bad as Shankland's manipulated graphs on cNet a while ago. (Cut the first 300 out of one, showed only Dromaeo DOM for Dromaeo total)
FransMonday, March 22, 2010 10:36:25 PM
Originally posted by coolfigaro:

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.

Yeah, the original page showed other browsers doing far worse than Opera. He didn't capture quite what kind of an incorrect image the table gives. As I already said, saying "we pass our own tests" is either a clever marketing ploy (misleading without really saying anything untrue) or lack of thought.

Originally posted by coolfigaro:

I sincerely hope you don't take that approach to reading the standards themselves. That's how you end up with crap like this: http://imgur.com/oMgGG.png

What about crap like this? What about Opera's DOM compliance compared to IE's? It's nice that they're moving forward, but the notion that IE is better in anything but a few meager details is ludicrous.
prd3Tuesday, March 23, 2010 7:29:59 AM
Originally posted by coolfigaro:

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.

How does the illustration image attached to the blog post misrepresent the situation?

It doesn't, and of course you know that, but you couldn't resist attacking the messenger, could you?

The fact is that the page falsely gives the impression

You are dishonestly relying on people to read the fine print to understand what the page is all about, but the proof is in the pudding: Hardly anyone got it, which is why "IE9 is more standards compliant than other browsers" was spreading across the web.

The page has good text describing how these were specific ambiguities that MS wanted to clarify in the specs.

It had a wall of text, a fine print, that most people didn't bother to read. And Microsoft knows this.

Bottom line: if you really like web standards, then you should be encouraging the work they're doing to catch up and to clarify/improve the standards.

If only they could stop lying.
JimFriday, March 26, 2010 2:26:56 PM
Originally posted by Frenzie:

I was already wondering how they came up with their values.

The same way banks come up with there own tests. It's literaly like telling a college student to write the questions for there own test.
beritakuWednesday, April 14, 2010 1:30:37 AM
Is this comparison table cart legit and creditable?
Daniel HendrycksWednesday, April 14, 2010 1:48:32 AM
Originally posted by beritaku:

Is this comparison table cart legit and creditable?

Which one? The one where it shows IE9 having 100% is not "legit".