Developers › A Journal from Behind the Trenches

Yet another unscientific report on browser security from NSS Labs

July 20, 2011 · by Haavard

It's that time of the year again.

NSS Labs publishes yet another report on socially-engineered malware, and Microsoft's browser incidentally wins it with a huge margin again. Strange, isn't it?

There's no need for me to reiterate all the problems with the report in this post. My post on the subject from back in 2009 is still mostly valid, so feel free to take a look.

The report is still equally unverifiable, poorly put together, and seemingly "tampered" with to boost IE's score. Does anyone really believe that IE is not only better, but far superior to any of the other browsers?

If they do, they really should consider the major issues with the report.

Update 2011-07-21: An anonymous/undercover NSS Labs employee denied that the report was sponsored by Microsoft. Although they have a history of posting misleading claims, I edited the post to avoid it taking away the focus from the many problems of the report itself, such as the poor methodology.

Readers' comments 17

Left under the original article, as archived.

Charles SchlossWednesday, July 20, 2011 9:59:10 PM
http://files.myopera.com/Tamil/Smilies/Bang.gif -

I would say IE is much worse of a browser to use and the blame is on Microsoft, becuase they got the high market share they became the only target

IE 6 still has over 20 security holes (yes IE 6 is still supported it dies when XP does in 2014)
TreeGoThursday, July 21, 2011 4:28:43 AM
Sounds like sour apples, a little ... not sure. Does IE do anything well?

IE9 seems pretty stable and nice here, actually.
earth01Thursday, July 21, 2011 7:12:30 AM
No matter. Opera has been, is and will always be the best Internet suite.
It is not just a browser, it is our Opera.

PS: the last snapshot is very good Congratulations !
ChrisThursday, July 21, 2011 9:08:30 AM
Originally posted by TreeGo:

Does IE do anything well?
That's not the question, now is it? The question is if a report with such obvious flaws is trustworthy.

Do you REALLY think IE blocks nearly everything and all other browsers nearly nothing? Is that REALLY believable to you?
TreeGoThursday, July 21, 2011 9:37:07 AM
Originally posted by Slamdex:

Originally posted by TreeGo:
Does IE do anything well?
That's not the question, now is it? The question is if a report with such obvious flaws is trustworthy. Do you REALLY think IE blocks nearly everything and all other browsers nearly nothing? Is that REALLY believable to you?

Perhaps ... does NSS Labs and ZDNet and ComputerWeekly and the many other techy websites that are reporting this finding do this without destroying their own reputations if there is not something to this?

I am a 'browser' ... not a techie-sort-of-person ... and like many others of us who read something like this, we tend to believe it, I think. And apparently many 'techies' believe it as well.

I'd like to think Opera is better than IE9 at this sort of thing because, in general, I prefer Opera for many other reasons, but I'm willing to accept less flattering assessments of Opera in comparison to IE if it is warranted.
Ben TudballThursday, July 21, 2011 10:05:06 AM
Their report states Opera 10 on page 10. It is suppose to state Opera 11. Misleading.

@TreeGo: Yep, I suspect everyone agrees also, if it is warranted, including Opera. It is their lack of indepth detail and the excluded results among other things that make it difficult to assess the merit of their data and if their findings are warranted.
ChrisThursday, July 21, 2011 10:05:54 AM
Originally posted by TreeGo:

Perhaps ... does NSS Labs and ZDNet and ComputerWeekly and the many other techy websites that are reporting this finding do this without destroying their own reputations if there is not something to this?
Tech "journalists" don't usually bother to dig deeper, and just regurgitate press releases. They get things wrong all the time, and yet are never held accountable for that.

Why would tech sites get a bad reputation by regurgitating just another piece of disinformation when they aren't getting it for regurgitating all sorts of other nonsense?

Mind you, not all tech journalists are that crappy. Some actually called NSS Labs on their BS.

Originally posted by TreeGo:

I am a 'browser' ... not a techie-sort-of-person ... and like many others of us who read something like this, we tend to believe it, I think. And apparently many 'techies' believe it as well.
Yes, that's exactly what Microsoft and NSS Labs are counting on. It means that they can easily get away with lies and disinformation.

I'd like to think Opera is better than IE9 at this sort of thing because, in general, I prefer Opera for many other reasons, but I'm willing to accept less flattering assessments of Opera in comparison to IE if it is warranted.
I don't know if Opera is better or worse at this. All I know that the NSS Labs report isn't a valid way to find out.

I mean, come on. Do you really think it's realistic that IE catches just about every single piece of malware while all other browsers barely catch any?
Vikram PhatakFriday, July 22, 2011 5:21:07 PM
I don't know who posted from NSS Labs as an "anonymous user", but I (CTO at NSS Labs) am stating categorically that Microsoft did NOT pay NSS Labs to produce this report. Nor did any other vendor. It says so clearly in the first page of the report. If Opera had come out on top, I expect you would be promoting the results in much the same way Microsoft is doing now.

Also, NSS Labs has been completely transparent with regards to the test methodology and has reached out to Opera on numerous occasions for feedback. If you are going to claim NSS Labs test results are incorrect, what block rate vs. malware do you think Opera should have achieved?
ChrisFriday, July 22, 2011 9:12:03 PM
Originally posted by vphatak:

If Opera had come out on top, I expect you would be promoting the results in much the same way Microsoft is doing now.
I doubt that Opera would be promoting a shitty piece of pseudoscience like that. What you dishonestly "forgot" to mention is that Microsoft did fund these reports earlier. Now I guess they are funding you indirectly so as to avoid scrutiny?

Also, NSS Labs has been completely transparent with regards to the test methodology and has reached out to Opera on numerous occasions for feedback.
Lies. You refused to even share the URL list. Go ahead and disprove that, I dare you!

If you are going to claim NSS Labs test results are incorrect, what block rate vs. malware do you think Opera should have achieved?
Who gives a crap? This is about your crappy piece of pseudoscientific nonsense.
Cutting SpoonSaturday, July 23, 2011 2:55:09 AM
So the program is considered to have intercepted an event if it displays a modal dialog? Weak. Microsoft's filtering is often opted-out by end users, just like UAC. Most common reason is too many popups and/or false positives.

People also have this nasty tendency to click "OK" when presented with a box on their screen. Site reputation warnings are ignored for very large hosts like MegaUpload anyway.

Unfortunately the system also does not protect against non-program malware.

In short, this was a cherry-picked test with incorrect validation criteria. Use of screenshots, dissertation of testing setup and feature discussion reads out like an IE9 banner ad.

Does anyone trust NSS to test anything? Where does the money come from? You can't operate a large network and hire a bunch of technicians only to produce very short papers which are free to download....
GonzaloSaturday, July 23, 2011 3:12:10 PM
IE was always so backward in features and security that I gave it up altogether many years ago. So I can't even tell if it is a good product now. When I can't use Opera (site compatibility) I go for Firefox.
Constantine VesnaSaturday, July 23, 2011 5:28:22 PM
Opera never blocked a single malware site for me*, while IE9 says "Dude, this file is dangerous, i will delete it, ok ?" for 99% of .exe files...

Not sure which approach is more "secure", but microsoft's one is clearly more "protective".

* - NO browser was ever able to detect and block real-life malware, which i did run into.
ChrisSaturday, July 23, 2011 9:32:11 PM
Originally posted by c69:

IE9 says "Dude, this file is dangerous, i will delete it, ok ?" for 99% of .exe files
Blocking nearly all .exe files regardless is just pure stupidity.

It's like a browser blocking 99% of all sites and then bragging about how secure it is.
Charles SchlossSunday, July 24, 2011 4:09:19 AM
Originally posted by Slamdex:

Blocking nearly all .exe files regardless is just pure stupidity.

Its kinda easy to get around that, very simple social engineering can bypass it almost 100%
Charles SchlossThursday, August 18, 2011 2:56:19 PM
A note on malware detection performance
http://www.morbo.org/2011/08/note-on-malware-detection-performance.html

I think this relates to your post
Constantine VesnaWednesday, November 2, 2011 10:49:39 PM
http://yourbrowsermatters.org/

MS had made an impressive site (from marketing / propaganda standpoint) about browser security, where they compare IE9 , Chrome and Firefox - and IE9 wins.

Opera is not mentioned at all. Even while we have all seen the certificate hack in september, which affected every browser - except Opera.

This is really a shame, counting the fact that Opera is so much more secure than Firefox, at very least If there are some things for Opera to learn from Microsoft - "marketing" should be among first.

p.s.: because Chrome is "faster", IE9 is "more secure", Firefox "has firebug, adblock and noscript" .. And Opera "was some wierd browser with ads, back in 2002".
Cutting SpoonThursday, November 3, 2011 2:50:33 AM
Interesting that you should mention that site. Try masking as Firefox. Hehe, or mask as Internet Explorer.

A tenpenny site that's not even worth my two cents.