A few results from the TLS Prober

Highlights

• One year ago today since the TLS renegotiation-vulnerability became publicly known
• Currently 37,3% of servers are patched. With the same patchrate, we will have to wait another 2 year before all servers are patched
• Microsoft IIS, Redhat & CentOS have higher patch coverage than other server vendors, mainly due to having released patches early
• Debian has still not released a patch for their stable branch
• Servers with Extended Validation certificates are less likely to be patched than servers without(!)
• More certificate statistics

Background

A year ago (November 4th, 2009) news broke about the TLS Renego vulnerabilty, a vulnerabilty in the TLS protocol's mechanism to renegotiate the parameters of a secure connection. This permitted an attacker to, among other things, inject requests into an HTTPS connection, before tricking the client into putting its "stamp of approval" and credentials on the requests, opening up to a number of security problems.

A solution that added new functionality to all exisiting versions of the SSL and TLS protocol, RFC 5746, was eventually published in February, and a number of vendors, including Opera, have released patched versions of their TLS implementations to repair the problem.

However, the nature of the problem, and the fix, implies that to complete the fix all clients and all servers do not just have to be patched, they must also eventually refuse to establish connections with unpatched implementations.

The timing of such a policy change is, to put it mildly, a sensitive matter. Too early and you "break the net", too late and you leave users and services vulnerable for an unnecessarily long period. This situation is not helped by the fact that the IETF punted on the possibility to include a "patch-by" date in the RFC, instead leaving the switch-over policy decision to the application vendors (effectively the browser vendors).

To make good decisions about such a policy change, we need to know how many servers have been patched. The only way to get such information is to actually test servers to determine whether they have been patched, or not.

The solution to this, for our part, as described in a number of articles by Audun^1,2,3 was the "TLSProber", a scanner tool that I started developing while we and several browser vendors were exploring possible policies surrounding the Renego problem and TLS generally.

The "TLSProber" is based on TLSLite, a public domain TLS implementation written in Python by Trevor Perrin, which I extended to support TLS 1.2, several TLS extensions including the Renego RI-extension (RFC 5746), and also instrumented to generate several types of "bad" handshakes in order to test for several known problems¹ with SSL and TLS servers. Testing of several more extensions are planned.

This utility was then extended with a database system built using Django, a Python-based web- and database-development framework, so that we could easily record all the information gathered for later analysis.

At present this tool is performing a weekly test based on a list of 2.1 million hostnames derived from the Alexa top million list, 2 weeks of Opera Mini logs and sites visited by Opera's MAMA spider.

In total, about 380,000 unique IP addresses that respond to at least an SSL v3.0 handshake are tested each week.

In addition to the Renego test, the prober tests TLS protocol support, cipher suite support, and standards compliance.

There are a couple others doing similar testing, Ivan Ristic's SSLlabs being one of the larger ones.

Current Renego Patch status

At present the patch coverage is 37.2% among the tested servers, 8 months after the specification for the patches became available.

Given this patch rate, we are looking at a minimum 16 months, a total of 2 years, until we reach full coverage. In my opinion that is 12-18 months too long.

One group of unpatched sites that I consider problematic is the sites using Extended Validation (EV) certificate, a group comprising 3.16% of the sites sampled. While an EV certificate only vouches for the identity of the site, one way to look at it is that these sites are concerned enough about their
reputation that they purchased a more expensive certificate with more detailed information that is also checked much more thoroughly, and which is given a preferential treatment in the browser UI (the "green bar"). Personally, as mentioned before, I think that such sites should be held to higher standards than normal sites. I would expect such websites to make sure their servers were as fully patched as possible. Yet, I've discovered that rather than having a significantly higher patch coverage than the general population (I would have expected them to be at above 70% by now), they have a slightly lower (0.75 percentage points, 36.5% vs. 37.25%) Renego patch coverage than the general server population!



A problem that has been discussed before, is that 81% of these servers have some kind of non-compliance problem: typically the server refuse to negotiate correctly with a client that specify the TLS protocol's major version 4 (all current versions use 3 as the major version). If we ever encounter a situation that, for some reason, require TLS implementations to use a new major version number in the protocol, 81% of the secure web is going to break until the servers are upgraded (Ouch!)

In addition, a small fraction of the servers (0.2%) do not tolerate clients presenting other TLS 1.x versions, including the next version of TLS, TLS 1.3, which is not yet defined. These servers may cause problems even earlier than the above mentioned 78%, but they are currently a more manageable problem. There may be a bit of uncertainty about these servers, as the results may be false
positives, or the prober may be testing a site with a mix of patched an unpatched servers (0.1% of sites had a mix of these server this week).



When looking closer at the patch coverage we find that servers using some OS variants have a higher patch coverage than others.

Specifically, Microsoft, Red Hat and CentOS servers are significantly better patched (53%, 45%, and 41%, repectively) than others. Debian servers, for example, have a coverage of less than 4%(!).

There is some uncertainty to these numbers, since many web servers are configured to not give precise details about which server software and OS is being used; some do not provide any details at all.

There are also TLS server vendors whose names are not included in the data, since their servers function as as front ends (accelerators) of the registered servers, so the wrong vendor name get registered.

Debian does not seem to be willing to update the current stable releases with the patch for the Renego problem, perhaps due to a mistaken belief that the original hotfix that disabled server-side support for Renegotiation is enough (it is not). They have patched the next stable release, Squeeze, though, so I recommend that all administrators running Debian-powered secure servers should upgrade to that version ASAP.

I am also aware that at least one front end vendor isn't planning to release an update until April 2011, 14(!) months after the specification was released, not even a hotfix.

These two vendors are what has kept Opera's own secure servers from being patched , and although we still can't patch many of the servers, we will now patch some of the servers, and investigate workarounds for the others.

Microsoft, on the other hand, despite all their regression testing, which , according to my information, they may have done even more thoroughly this time than they usually do, got their patch out in about 6 months.



(Note: "Windows" in the above graph refer to Apache installations on Windows)



TLS protocol coverage

Currently the SSL/TLS protocol coverage shows the following picture:

• SSL v3-only servers are about 1.5% of the tested servers (down from 1.6% when our tests started)
• Among Renego-patched servers, SSL v3-only are about 0.24%. This is down from 0.5% early in the tests, which indicates that SSL v3-only servers are finally being phased out completely.
• TLS 1.2 servers now have a population in the testset of 11-12 servers (up from 2 in July, an increase of 450-500%)
• Intolerance of TLS 1.x clients was found among 3% in the sampled servers

This also means that more than 98% of the secure web is currently using TLS 1.0, a 10-year-old protocol.

Site certificates

Recently I have been doing some research on the certificates used by websites, looking at some of the aspects that interest me.

One of the minor interesting points I discovered was that while most sites send at most 4 certificates in the handshake, there was some (<1500) that sent more. One of them sent 17(!) certificates, most of them unrelated to the site certificate (which only needed two, as I recall) and even duplicates of some of the others. This does not really represent a problem, unless the site has high traffic and the owner pays by the megabyte, as each certificate is about 1-2 KB each.

Keysizes

• 2 of 3 sites uses 1024-bit RSA certificates
• 1 of 3 sites uses 2048-bit RSA certificates
• 0.1% use other RSA keysizes.
• No site certificate was using a key shorter than 1023 bits (there are a few 512 bit Roots in the full sample, though)

Wildcard certificates

Given that the IETF is about to publish an updated specification for hostname matching in certificates, I decided to take a look at some special variations:

• 37 sites had names with a wildcard in a non-leaf label of the servername, that is a name of the form "foo.*.example.com", which is not allowed by the current information RFC specification for HTTP over TLS (They were originally allowed by a specification published by Netscape).
• 25 sites had wildcard names that just covered a limited number of servers, e.g. "foo*.example.com", which are allowed.

Debian weak keys

A few years ago it was discovered that a modification done to Debian's version of the OpenSSL random generator, caused the random generator to be anything but random. The result was that RSA keys generated by affected Debian systems were very predicatable, and could be broken with a minimum of effort (less than 32768 tries needed).

In the sample of websites tested by the TLSProber this week, 1490 sites had site certificates with a key that is on the Debian-provided blacklist, 729 of them still have not expired, and one of them (a "homegrown" one) is valid until the Year 4746. At present the revocation status of these certificates has not been tested.

Most of these certificates are, however, what I call "homegrown", meaning that they are either self-signed, or issued by a Root generated by the local system administrator, and not certificates issued by a trusted CA. Such certificates will trigger certificate warnings for most users, and should not present a problem for careful users, but may be threat to those users that frequently visit the sites using the certificate.

While I noticed some certificates issued by well known CA's, these were relatively few and far between. Among the well known CA's the one that seemed to have the most, probably 100+, was CAcert.org, a CA that is AFAIK not included in any major browser's rootstore.



Credits: Audun Mathias Øygard provided some of the graphs in this article.