Improving Interoperability - the Story of a Bug

March 30, 2011 · by Karl Dubost

Open The Web is an initiative from Opera to encourage Web developers to use Web standards for ensuring interoperability across Web browsers (and Web clients as large). We publish articles. We participate in conferences. We contact Web sites owners that have interoperability issues. Opera users kindly report issues through Opera Bug Report Wizard. We sometimes group similar issues in a work package.

My favorite package has been “HTML served as XHTML package” (OTW-6223). Some Web sites (including high profile Web sites) send to some Web clients HTML content with the mime type application/xhtml+xml. The parsing rules for XHTML and HTML are different. When sending application/xhtml+xml, site owners have to be sure that their content is, at least, well-formed XML. I have written about this issue in Wrong To Be Right. We noticed that the issue was often happening with sites using Microsoft IIS and ASP.net.

Starbucks was listed under OTW-6642 (created on 2010-12-20) in the package OTW-6223. The site was sending to Opera users HTML content with application/xhtml+xml. We tried to contact the site owners without too much success. This is one of the issues that we, the Web community, should try to address: a universal reporting system for Web sites interoperability. The biggest challenge being to find the right channel and appropriate level of communications. A system too open might lead to bashing when too closed might be ignored.

Yesterday I closed OTW-6642 with joy. The issue has been solved by a Starbucks Software Engineer, Rohan Singh @rohansingh. He noticed the blog post and left a comment

Hey Karl,

Thanks for the heads up about this happening on Starbucks.com. We tracked it down to an issue in the ASP.NET Browser Capabilities functionality.

As it turns out, if you supply ASP.NET with a browsercaps file (database of browser capabilities), it sets the response’s Content-Type to the user agent’s preferred Content-Type.

Of course, this doesn’t make a lot of sense, since HTML isn’t going to magically change into XHTML or WML or anything else simply because it’s the preferred type.

Anyway, we’ve worked around this behavior and are testing the fix. Should be out soon.

Thanks again!

He then solved the issue on Starbucks Web sites and then wrote a blog post to explain the ASP.Net feature which created the issue. Not only it was sending the wrong information to Opera but also to clients such as wget. My favorite quote of his blog post is (emphasis is mine):

The simplest fix, of course, is to get rid of any *.browser files you may be using in your application. I understand redirecting to a mobile version of your site for mobile browsers or the like, but basing any major functionality on guesses about the user’s browser is a great path to future pain.

Rohan provided a fix by either getting rid of any *.browser files or at least remove the instances of preferrendRenderingMime.

Respect to Rohan Singh, really. It will also help us contact and eventually fix other sites exhibiting the same behavior such as: McAfee, Excalibur, MGM Grand, SpanAir, Adidas, etc.

If you are a site owner, please contact us and/or fix it. If you are a Web user having this issue, please use the Opera Bug Report Wizard to notify us.

Readers' comments 9

Left under the original article, as archived.

Charles SchlossWednesday, March 30, 2011 1:39:32 PM
Ronit KumarWednesday, March 30, 2011 1:54:30 PM
lucideerSunday, April 3, 2011 6:42:28 PM
Originally posted by Rohan Singh:

it sets the response’s Content-Type to the user agent’s preferred Content-Type.

Obviously this is perfectly silly behaviour on Microsoft's part, but is XHTML really Opera's (and no-one else's) preferred content-type? Opera's Accept: seems to me (in my limited understanding of q values) to list text/html and application/xhtml+xml with equal preference.
Karl DubostThursday, April 7, 2011 1:18:57 PM
It doesn't matter, equal preference or lower preference. It's a negotiation between two parties. One is saying:

"I can process this and that, and these are my preferences."

(It doesn't say you have to send me this! when there are choices). The server then is deciding and replies:

"Hmm, ok. I'll send you this."

The issue is that sometimes the server lies about what it really sends. Basically if you send XHTML (application/xhtml+xml), you *really* have to send XHTML. :)
lucideerThursday, April 7, 2011 8:45:52 PM
Don't worry - I'm well aware of the issue, no need to explain. That wasn't my question though.

What I'm wondering is, if Opera says "I'll take html or xhtml with *EQUAL* preference" and Firefox says "I'll take html or xhtml with *EQUAL* preference", why does Opera get a different header than Firefox gets.
TurinSaturday, April 9, 2011 5:48:16 PM
Has Microsoft been contacted about this issue? Singh's workaround is in the end just that and will probably--read almost definitely--not be noticed or implemented by all problematic sites.
Karl DubostTuesday, April 19, 2011 2:59:24 PM
Opera gets a different content because of the server, see the initial blog post.

@Turin: yes.
Unregistered userWednesday, April 20, 2011 11:14:50 PM
Yuhong Bao writes: Good thing Opera supports reparsing broken XHTML pages as HTML
lucideerSaturday, April 23, 2011 5:12:32 PM
Originally posted by karlcow:

Opera gets a different content because of the server, see the initial blog post.

The thing that was confusing me was *why* the server was sending Opera a different content-type. In the initial blog-post you have a quote from Rohan Singh saying:

Originally posted by rohan:

it turns out, if you supply ASP.NET with a browsercaps file (database of browser capabilities), it sets the response’s Content-Type to the user agent’s preferred Content-Type

This confused me because application/xhtml+xml is NOT Opera's preferred content-type - so Rohan's statement is false or mis-informed. Your other blog-post here makes the issue far clearer though:

Originally posted by karlcow:

we can be sure, the server does user agent sniffing.

So I doubt contacting Microsoft will really help - they're obviously deliberately targeting user-agents to make sites break in Opera.