Developers › A Journal from Behind the Trenches
Browser sniffing gone wrong (again): Cars.com
If you open cars.com in Opera, all you will get is a bunch of jumbled characters. Is this another bug in Opera? No, it's browser sniffing again. And not just basic user agent string sniffing either.
Using Opera's built in "Mask as Firefox", which completely changes Opera's user agent string will in fact not fix the problem. You will need something which completely hides Opera from the server. I am using Proxomitron combined with Andrew Gregory's custom config file for faking browser IDs.
So even if a site is still broken when masking as Firefox, the site could actually be digging a lot deeper to identify the browser. If you want to be absolutely sure that it is not a bug in Opera, you may want to use a local proxy which rewrites HTTP requests on the fly to make sure that the server isn't doing anything it shouldn't.
To this day, browser discrimination remains probably Opera's main compatibility problem. If only sites could at least send what they are sending to Firefox, a lot of more sites would start working.
Using Opera's built in "Mask as Firefox", which completely changes Opera's user agent string will in fact not fix the problem. You will need something which completely hides Opera from the server. I am using Proxomitron combined with Andrew Gregory's custom config file for faking browser IDs.
So even if a site is still broken when masking as Firefox, the site could actually be digging a lot deeper to identify the browser. If you want to be absolutely sure that it is not a bug in Opera, you may want to use a local proxy which rewrites HTTP requests on the fly to make sure that the server isn't doing anything it shouldn't.
To this day, browser discrimination remains probably Opera's main compatibility problem. If only sites could at least send what they are sending to Firefox, a lot of more sites would start working.
Readers' comments 50
Left under the original article, as archived; 1 page of comments could not be recovered.
About this article
| Published in | A Journal from Behind the Trenches |
|---|---|
| Date | October 6, 2008 |
| Author | Haavard |
| Original | Internet Archive |
João EirasMonday, October 6, 2008 1:20:11 PM
Remco LantingMonday, October 6, 2008 1:43:49 PM
[Edit] On closer inspection, this might be confusing Opera:
Transfer-Encoding: gzip, chunked
Vary: Accept-Encoding
Content-Encoding: deflate
Robin ZalekMonday, October 6, 2008 2:29:44 PM
Edit: Scratch that, this problem seems to exist on a quantum level. As any attempt I make to debug it just changes the results for me.
Andrew GregoryMonday, October 6, 2008 4:21:28 PM
Removing "deflate" from the "Accept-Encoding" header results in "Content-Encoding" of "gzip" instead of "deflate", which works around whatever server-side problem they have.
I would have thought "Mask as Firefox" would work. Since the JS component of my cfg file can't have any affect (since the page isn't decoded correctly), it must be the User-Agent string. The one in my config is rather old. Hmm, I've just tried the old UA in Fiddler2 (I don't use Proxo any more) and that didn't change anything.
Just weird.
Kyle BakerMonday, October 6, 2008 6:19:50 PM
João EirasTuesday, October 7, 2008 12:14:59 AM
Originally posted by Andrew Gregory:
How about moving deflate to the end of the header, and leaving gzip before ?
I saw another website that broke due to the same thing.
InvestorTuesday, October 7, 2008 2:57:52 AM
If using a Proxy eg. http://www.quickbrowsing.com/ takes care of browsersniffing, then Opera should make a deal with them... Have a *Open With Proxy* option on menu.
( www.cars.com opens with quickbrowsing, but not sure site works as intended, though)
HaavardTuesday, October 7, 2008 7:51:57 AM
FavTuesday, October 7, 2008 9:02:17 AM
There are like hundreds of thousands of those proxy sites. It's a simple script (free). So not sure why you say they should make a deal (unless its your site). I had few of them as well...
Unregistered userTuesday, October 7, 2008 10:34:53 AM
HaavardTuesday, October 7, 2008 12:31:17 PM
Anyway, I'm not sure if this is actually a browser sniffing issue after all, but I have seen quite a few sites where using Opera's masking feature is not sufficient. So the tip about using Proxomitron (or other local proxies) is still valid.
Andrew GregoryTuesday, October 7, 2008 3:24:26 PM
And now for a little bit of interesting trivia. The "Transfer-Encoding" the site uses for this page is "gzip, chunked". According to a book I have here, "chunked" is the only encoding allowed by HTTP 1.1. It's an old book, so it could be out of date. Anyway, I changed the "TE" header Opera sends out from "deflate, gzip, chunked, identity, trailers" to "deflate, chunked, identity, trailers" (i.e. dropped gzip). The site didn't use any Transfer-Encoding at all, and it worked.
And then while I was testing all this, Opera suddenly stopped sending HTTP 1.1 headers and started sending HTTP 1.0 requests, with no "TE" header at all. The site responded with no "Transfer-Encoding" of its own, and the page proceeded to work as expected. No idea what's happened there
I wonder if the "gzip, chunked" is OK, and if it is, does Opera support it? Or if the site is using it correctly?
João EirasTuesday, October 7, 2008 7:39:03 PM
Originally posted by anonymous:
Now xoo, go away.
Charles SchlossTuesday, October 7, 2008 8:00:59 PM
I also sent this in the message:
Why Open the Web?
Despite the connecting purpose of the Web, it is not entirely open to all of its users. When used correctly, HTML documents can be displayed across platforms and devices. However, many devices are excluded access to Web content.
http://my.opera.com/community/openweb/info/
Charles SchlossWednesday, October 8, 2008 6:40:08 PM
Good afternoon,
I'm sorry I can't provide five reasons, but one reason Cars.com does not support Opera is its improper rendering of JavaScript and CSS. In our quest to provide the greatest experience for our users, we must choose which browsers to support. We hope that you will continue to enjoy our site on a supported browser.
Sincerely,
Blake Washer
Cars.com Consumer Support
Kyle BakerWednesday, October 8, 2008 7:01:54 PM
Charles SchlossWednesday, October 8, 2008 7:16:27 PM
Are you sure because after about haveing the site open for about 5 hours the site shows back up and does not give me the srambled mess.
Web Specifications Supported in Opera 9:
http://www.opera.com/docs/specs/opera9/
Browser sniffing gone wrong (again): Cars.com:
http://my.opera.com/haavard/blog/2008/10/06/browser-sniffing-gone-wrong-again-cars-com?cid=6178770
Charlie Schloss
Why Open the Web?
Despite the connecting purpose of the Web, it is not entirely open to all of its users. When used correctly, HTML documents can be displayed across platforms and devices. However, many devices are excluded access to Web content.
http://my.opera.com/community/openweb/info/
Now waiting for the response...
Charles SchlossThursday, October 9, 2008 3:42:30 AM
http://my.opera.com/Chas4/albums/showpic.dml?album=540739&picture=8439760
its cars.com working after the site being open for about 4 hours then a page refresh
FataLThursday, October 9, 2008 5:52:54 PM
Charles SchlossThursday, October 9, 2008 5:58:04 PM
HaavardThursday, October 9, 2008 7:02:56 PM
Unregistered userFriday, October 10, 2008 5:04:44 PM
HaavardFriday, October 10, 2008 6:01:00 PM
Maybe I can get you in touch with one of our people. They've done some analysis, and I would guess that the fix is relatively simple.
Unregistered userFriday, October 10, 2008 9:16:42 PM
Alexey FeldgendlerSaturday, October 11, 2008 8:42:42 PM
Unregistered userSunday, October 19, 2008 8:13:53 PM
Shwetank DixitThursday, October 30, 2008 5:00:50 AM
The problem caused by Opera sending a different (and perfectly valid) set of HTTP headers on reload (once it knows the server) than other browsers. The server screws up when receiving those headers. It may be a good idea to check the response of the server when recieving a request from Opera Vs other browsers. If they can make the server give the same response to Opera that it gives to other browsers, then the problem could be solved.
Andrew GregoryThursday, October 30, 2008 9:55:20 AM
I've done some further investigation using Fiddler.
The pertinent header sent to Opera in this case is:
Transfer-encoding: gzip, chunked
I've found that if the "gzip" is changed to "deflate", then Opera will decode the reply. i.e. the cars.com server is not correctly indicating the compression method it has used.
Firefox works because it does not claim to support compression in the transfer. Opera claims to support compression in the transfer because it does support it, but if the server decides to compress the data, Opera then needs to be told the correct compression method!
It's buggy server code that has probably never been tested because no other browser claims to support transfer compression.
Unregistered userWednesday, November 26, 2008 9:39:24 PM
Unregistered userSaturday, December 13, 2008 6:00:41 AM
Charles SchlossSaturday, December 13, 2008 6:36:16 AM
Errors found while checking this document as HTML 4.01 Transitional!Result: 50 Errors, 1 warning(s)
Encoding: iso-8859-1
Root Element: låì%Û
I opened the page in Opera 10 Alpha 1 it opened fine then about 30 seconds later back to the scranbled code
http://files.myopera.com/Chas4/files/Opera%2010%20alpha%201%20on%20cars.com.tiff -
Charles SchlossMonday, December 15, 2008 8:29:48 PM
http://files.myopera.com/Chas4/albums/540739/Opera%2010%20alpha%201%20on%20carscom.png -
it is something later on on the code that Opera does not like
Andrew GregoryWednesday, December 17, 2008 2:58:05 AM
Charles SchlossWednesday, December 17, 2008 3:22:03 AM
Andrew GregoryWednesday, December 17, 2008 10:03:34 AM
Charles SchlossWednesday, December 17, 2008 8:14:46 PM
toward the bottom of the scrambled page is bold scabled code
öÖé3¨^’󳑙ꣂÕA×Á™:A/ÀeL„²xß;¦÷zo,Ôg;åÇV©|½GÕ*ï#uGnë¼~õrMÈ yÝоþÜÉ 'Tûï$Œ Dƒ@´cauê@¼ûþxp–]ýÕj9‚»Pž÷9œÆéfL} ·ZÈG¹‹ÉÒ’Å 1^m+�ºJ–ÚI+†–úf&e½ J¨Ÿ·ü›N!ÈÌÔ‹U:Fƒõ‹zXÞÌ ÃÞl
can you find anything in Opera Dragonfly in the Error Console? while at cars.com?
Charles SchlossWednesday, December 17, 2008 8:40:48 PM
http://files.myopera.com/Chas4/albums/540739/cars%20963.png -
I try looking for a new car send here:
http://www.cars.com/go/configurator/styles.jsp?nclp=true&zc=81120&rd=&aff=national&makeid=47&modelid=2916&myid=&match=&dlid=&orgid=&aff=national
scrambled code again
Andrew GregoryThursday, December 18, 2008 5:38:54 AM
Charles SchlossWednesday, February 4, 2009 3:30:17 AM
Site is still very badly messed up
http://files.myopera.com/Tamil/Smilies/Bang.gif -
edit: sent feed back
http://www.cars.com/go/about/us.jsp?aff=national§ion=H&content=cont
Charles SchlossWednesday, February 4, 2009 7:22:36 PM
Case 01427743: Your inquiry to Cars.com [ ref:00D3WKx.50046sVbA:ref ]
Good afternoon,
Thank you for contacting us. Unfortunately, we are not able to support the Opera browser at this time. Please use one of our supported browsers, Apple Safari or Mozilla Firefox, to access Cars.com.
Sincerely,
Blake Washer
Cars.com Consumer Support
Andrew GregoryThursday, February 5, 2009 1:14:56 AM
Charles SchlossThursday, February 5, 2009 1:39:59 AM
funny that the drone mentioned the IE is not supported
Unregistered userSaturday, February 13, 2010 5:08:47 AM
Charles SchlossSaturday, February 13, 2010 5:20:47 AM
tbutlerFriday, February 19, 2010 7:09:42 AM
To demonstrate this concisely, I used wget with multiple --header arguments to reproduce the problem outside of Opera. There was one wrinkle with this. Wget, or at least the version I have (1.11.4) specifies HTTP/1.0 in the GET request. And the cars.com server only compresses the result if it says HTTP/1.1. Since wget doesn't provide an argument to change this, I used a binary editor to modify the executable, changing the string to 1.1. After that, I get this result:
$ ./cget.sh
--2010-02-19 01:51:02-- http://www.cars.com/go/index.jsp
Resolving www.cars.com... 74.119.98.141
Connecting to www.cars.com|74.119.98.141|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16516 (16K) [text/html]
Saving to: `cars.reply'
100%[==========================================================================================>] 16,516 --.-K/s in 0.08s
2010-02-19 01:51:03 (194 KB/s) - `cars.reply' saved [16516/16516]
$ file cars.reply
cars.reply: gzip compressed data, from Unix
$ mv cars.reply cars.reply.gz
$ gunzip cars.reply.gz
$ file cars.reply
cars.reply: gzip compressed data, from Unix
$ mv cars.reply cars.reply.gz
$ gunzip cars.reply.gz
$ file cars.reply
cars.reply: ISO-8859 HTML document text, with very long lines, with CRLF, LF line terminators
$
And cget.sh is just a script to make editing the wget command a bit easier. The interesting thing is that if you drop the TE header, it fixes the problem. The reply is still compressed, but only once.
$ cat cget.sh
#!/usr/bin/bash
./wget \
--header="Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0" \
--header="TE: deflate, gzip, chunked, identity, trailers" \
-O cars.reply http://www.cars.com/go/index.jsp
tbutlerFriday, February 19, 2010 7:28:39 AM
below, but only if you have a version of wget that specifies HTTP/1.1 in the GET request. Note that gunzip is run on the data twice.
$ ./wget --header="Accept-Encoding: gzip" --header="TE: gzip" -O cars.reply http://www.cars.com/
--2010-02-19 02:24:33-- http://www.cars.com/
Resolving www.cars.com... 74.119.98.141
Connecting to www.cars.com|74.119.98.141|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.cars.com/go/index.jsp [following]
--2010-02-19 02:24:33-- http://www.cars.com/go/index.jsp
Reusing existing connection to www.cars.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 16518 (16K) [text/html]
Saving to: `cars.reply'
100%[==========================================================================================>] 16,518 --.-K/s in 0.04s
2010-02-19 02:24:33 (374 KB/s) - `cars.reply' saved [16518/16518]
$ gunzip -c < cars.reply | gunzip -c > cars.reply.html
$ file cars.reply.html
cars.reply.html: ISO-8859 HTML document text, with very long lines, with CRLF, LF line terminators
John DaviesTuesday, April 27, 2010 7:53:50 PM
then it works with an out-of-the-box binary.
Charles SchlossFriday, June 4, 2010 10:02:55 PM
It is the mobile site:
http://m.cars.com/
http://www.pickuptrucks.com/html/stories/international/page1.html does the same thing as the main cars.com site
Cars.com contact page:
http://m.cars.com/contactus.jsp
Charles SchlossSunday, June 6, 2010 11:33:46 PM
Case 01926656: Your Cars.com Inquiry [ ref:00D3WKx.5004At7kg:ref ] [email protected] ([email protected])
Greetings,
Thank you for the response. While we do strive to work within all browsers, Opera currently isn’t a browser which our web site functions properly. However, we are currently looking to address this issue. Until this is resolved, we would like to suggest that you access our web site using a more industry standardized browser (i.e Internet Explorer, Mozilla FireFox, or Safari.) Please let us know if you have any questions.
Thank you,
Christopher Almodovar
Cars.com Consumer Support
We value your feedback – Please click below to answer four quick questions about your recent experience with your Cars.com Support Specialist. Thank you! http://cars.force.com/supportsurvey?caseid=5004000000At7kg
I did mention this post in the email I sent (had to use the m.cars.com to send it tho)
JimWednesday, June 9, 2010 3:43:35 PM