Apples, Oranges and Vulnerability Metrics

NOTE:  I am not asserting that my vulnerability analysis demonstrates that Windows is more secure.  Rather, I frequently hear and read Linux advocates making unsupported assertions to the opposite that Linux is inherently more secure than Windows.  The “unsupported” part of that bothers me, so I check for myself.  What I keep finding is that Linux distributions have more vulnerabilities, more serious vulnerabilities and the data does not support the assertions of security superiority for Linux and Open Source software.

Earlier this week, I posted a brief summary analysis of vulnerability data for Windows and Red Hat server (Windows vs Linux (Red Hat) – Server – 1st Half 2006), as well as Windows and Red Hat workstation products (Windows vs Linux (Red Hat) – Workstation – 1st Half 2006). 

Having worked on security analysis of this sort for a couple of years now, I expected (and my expectations were fulfilled) to get a common objection which I call the “apples to apples” objection.  It has a few variations, but the basic argument is that a Windows Product and a Linux Distribution are not equivalent, so the comparison is flawed.  I don’t agree that it is flawed, but I do believe that it is important to understand the differences and what the implications might be.

So, in line with my blog charter, I want to look at the “apples to apples” issue from several angles and see what we can see.

Vendor Defined Product

My first consideration is what is defined as a server or workstation product by the vendor.  I don’t define that, nor do the customers making purchases, it is defined by the respective vendors.  There are business reasons for doing so.  Red Hat could, for example, ship a Red Hat Enterprise Linux server product without OpenOffice, media player, etc.  They choose to do it this way because there is some business advantage – one possibility here is that they can assert that you get more value, features and applications for their subscription price.  It is a similar case for Microsoft, who includes full Active Directory with Windows Server as opposed to Red Hat, who licenses their enterprise directory separately.

So, my first perspective is that the vendors have to take the bad with the good.  If they market an Enterprise product including a bunch of apps, then they have to also take the heat if those apps have vulnerabilities.

Role-based Comparisons

A second way consideration for “apples to apples” would be to compare vendor product configured to server particular roles.  When someone says “hey, the numbers don’t apply, because I can take my modular Linux system and configure it without many of those components.”  Alternatively, “my web server doesn’t have gnome or firefox installed, so your numbers are misleading.”  I grant a certain amount of validity to these objections – modularity is a good principle for security, a good development philosophy.

The problem I have with this argument is that the folks that raise this objection typically stop there.  They assume a minimally built Linux role based server will have vastly fewer vulnerabilities and exposures than a Windows one.  I personally like to test assumptions, which is what we did when we sponsored some role-based comparisons by Security Innovation, LLC (SI) and later, by Dickerson Technologies, LLC.  Security Innovation performed two studies covering the previous 12 months, Role Comparison Report: Web Server Role and Role Comparison Report: Database Server Role.  (I talked about the Dickerson study in a previous blog entry, JeffOS EAL4+ Secure System).

It turns out that a minimal deployed Red Hat Enterprise Linux system has over 250 packages installed, not counting role-required applications.  It also turns out that if you do install a minimal LAMP server using Red Hat 3 (and not including gnome, firefox, etc), it still had more vulnerabilities, more severe vulnerabilities and higher days-of-risk than all Windows Server 2003 components plus SQL 2000 SP3, as illustrated in this chart from the study.  Another interesting finding in the database study was that SI had to enable a lot of components beyond the minimum, including the graphical interface and browser, in order to deploy Oracle successfully per Oracle guidelines – so the assumption of a minimal build doesn’t necessarily always hold.

So, I could do my analysis this way as well and it would perhaps some of those “apples to apples” objectors.  There are a couple of reasons I don’t.  First and foremost is that it is a ton of work to figure out what packages are in a given role, and then filter for the most common roles, etc.  I’ll leave that to the analysts and security firms to do in their bigger studies.  For the second reason, read on.

Union of Many Roles

How many businesses go through a vendor/OS/product selection process and then use the selected OS in exactly one role?  Two roles?  Three?  I am sure it happens, and if so, I declare those the exception that proves the rule.  When I think about it, I think it is much more likely that a server OS will be used it multiple ways – Web server, database server, application server, mail server, firewall, file server, print server, and so on.  Over a period of time, fixes may apply to only one server role (e.g. Apache), all roles (kernel), or none of the roles (e.g. KDE, when gnome is installed).  However, it seems reasonable that as more roles are utilized in a network, the union of those roles becomes closer and closer to the set of all components.  This means that the set of fixes that best approximates what the network, as a whole, would need over time approaches the full set of fixes (and vulnerabilities).

I can think of some obvious exceptions.  A company may choose not to install the MP3 player and OpenOffice on any servers, in which case those could be excluded.  However, a Windows-based environment could similarly choose to lock down Media player and restrict IIS installations.  The point is, we can’t predict those details, so looking at the full set becomes the most reasonable predictable representation of potential vulnerabilities and fixes at a broad level.

All Vendor Products

There is one final viewpoint on the “apples to apples” objection that I can think of, and that is to include all Microsoft products into a measurement along with Windows.  So, if we look at fixes to Windows Server plus Office plus SQL plus Exchange and so on, then some argue that this might be a more “apples to apples” comparison with a Linux distribution.  I haven’t done it yet, nor started gathering the necessary data, but the idea has merit.  In fact, I’m willing to do it at least once to see how it turns out.  And you know what?  My money would be on an outcome that once again does not support the assertion that Linux and Open Source are superior.

A Final Comment

I just recently read Security is not like parmesan cheese by Alun Jones (no relation) and I think it is pertinent to this discussion.  He’s basically saying if one system is in fact more “inherently secure” than the other, then there must be some identifiable development process that is granting that superiority.  On one hand, Microsoft is four years along since they “acknowledged” their security problem and began working to make improvements.  On the other hand, you have a Novell VP asserting that Linux is “very secure” (here):

Levy: Security is an important consideration for any business computing initiative. The Linux desktop is a very secure environment. Of course, the Linux operating system is also quite secure, for it has a strong UNIX heritage, and it has been in development for more than 10 years.

I’ve never understood how folks keep a straight face while making these statements. 

Assuming there were a 12 step program for improving security, none of the Linux advocates appear to be able to get past step 1, where you admit the problem applies to you.  Until that happens, I predict “more of the same” for Linux security.

Best Regards ~ Jeff


About the Author
Jeff Jones

Principal Cybersecurity Strategist

Jeff Jones a 27-year security industry professional that has spent the last decade at Microsoft working with enterprise CSOs and Microsoft's internal teams to drive practical and measurable security improvements into Microsoft products and services. Additionally, Jeff analyzes vulnerability trends Read more »