The First Step on the Road to More Secure Software is admitting you have a Problem

Hi, Michael here.

I am always bemused when Jeff Jones performs in-depth security vulnerability analysis and reports his findings, not because of the content of his findings, but because of the incredible arm-chair commentary that follows.

Jeff and I have seen and heard it all:

  • “This is FUD”
  • “Yeah, but it’s not an apples to apples comparison”
  • “How can you believe this guy? He works for Microsoft!”
  • “What would Microsoft know about security?”
  • “For his next trick…”
  • “That chart really hits home the fact that statistics can be used to prove any side of any argument”
  • “Of course he says Windows is the best, that’s what he’s paid to do.”
  • “Counting vulnerabilities is a natural way to measure security. If you’re a retard.”
  • “The other big reason linux is more secure is many black hats LOVE open source principles”
  • “Can someone please slap MSoft in the teeth”
  • “I can’t actually remember a time when my mac needed a patch to fix a security hole.”

You get the picture. I could keep going, but I have a blog post to write!

So let’s ignore raw stats for a moment, let’s not compare RedHat to Mac OSX to Ubuntu to Windows Vista, because let’s face it, no-one can agree on any measurement of security without getting knotted up. So let’s just ignore the comparison stuff. Measuring security is a real challenge, and while we may debate the merits of vulnerability counts, right now it’s the only concrete metric we have.

When Bill Gates released his Trustworthy Computing Memo in 2002, many people thought it was just a marketing stunt. It was not a marketing stunt: BillG edicts are always taken very seriously inside Microsoft. In fact, I will go one step further; the only way you make big changes in a large software company is when the boss says you have to do so. So why did Bill send the memo to all Microsoft employees? It was simple, he (and the entire senior management team for that matter) recognized Microsoft faced a problem that needed solving; the company needed to shore up the security of its products. So Bill sent his memo to get the ball rolling.

Now let’s go back to Jeff’s recent analysis. Cover up the Mac OS X and Linux stats for a moment so you can only see the Windows XP SP2 and Windows Vista bars. Windows Vista has had fewer security vulnerabilities than Windows XP SP2. Conventional wisdom (which is often wrong, especially when it becomes urban legend) tends to suggest that the more lines of code you have the more bugs you have. That might very well be true, and Windows Vista is certainly larger than Windows XP SP2; yet right now, we are on track for an approximately 50% reduction in vulnerabilities compared to Windows XP SP2. Think about that figure for a moment: about a 50% reduction (and that does not account for the reduction in vulnerability severity) despite the increase in code size.

So if Windows Vista has more code than Windows XP SP2, why are we seeing a reduction in vulnerabilities? Simple: the SDL! Microsoft decided to change its development practices to enforce greater security discipline. The only way you reduce security vulnerabilities is by focusing on improving code security, design security, reducing attack surface, education, tracking evolving threats, mandatory use of tools, banning known bad functionality, better compilers, better linkers, better libraries etc etc. And that is what the SDL is all about and what our team is laser-focused on.

The reason you’re seeing a reduction in vulnerabilities across major Microsoft products is simple:

  • Microsoft recognized it needed to improve security.
  • Bill said so (as did the rest of senior management)
  • Our group swung into action and helped the rest of the company come up to speed on security issues.
  • The Microsoft development processes changed to adopt the SDL

You improve security by focusing on security. Not by wishing on a star. Not by believing age-old myths about “given enough eyeballs…. blah blah.” If the “eyeballs” mantra were true, we’d have very few open source security bugs. But there are plenty of open source security bugs found after products ship. Hmmm, this would seem to raise some interesting question on the validity of the “enough eyeballs” belief given these hard facts.

Now let’s go back to Jeff’s chart for a moment. Cover the Windows columns and look at the other columns. However you want to skew or spin it, that’s a lot of security vulnerabilities that needed fixing once a product had shipped. Admit it. Come on; admit it, that’s a lot of bugs. I don’t care how big a Linux distro is, or how many IM clients Ubuntu ships with, or the merits of UAC vs su. That’s a lot of security vulnerabilities!

Now ask yourself this question – how many people involved in the development of these other products have you heard say, “Wow, we have a lot of security bugs, we really should do something systematic to fix this problem.” I’ll be very happy to be proved wrong, but all I hear is crickets. I see no-one else in the industry standing up and saying, “Let’s fix this.”

I just hear emotion, excuses and dogma.

At Microsoft, BillG’s memo was a “we need to fix this” memo, and we are now seeing results, but not perfection. There will be no perfection, because no software is 100 percent secure, but progress is being made across all Microsoft products, not just Windows, because of the SDL.

Let me close with a story. A few years ago I spoke to some senior technical people from a large financial organization about software security. After visiting Microsoft they were off to visit another operating system vendor. I won’t name names. The financial company was very interested in our early results, and they were encouraged by what they saw because of the SDL. I asked the most senior guy in the room to ask the other company one very simple question, “What are they doing to improve the security of their product? And by that I mean, what are they doing to reduce the chance security vulnerabilities will creep into the product in the first place? And they cannot use the word ‘Microsoft’ in the reply.” Two weeks later, the guy phoned me and said his company would buy Microsoft products and nothing from the other company. I asked him why. He said because all they could do was make up excuses (see the list at the start for examples!) rather than admit to having numerous critical security vulnerabilities and no process to reduce their ingress.

Ok, one more comment! I would love to see others in the industry stand up and admit there is a problem that needs solving and start doing something about it. I really, really would, because we need to secure the entire computing ecosystem. Comparing numbers is interesting, but what really matters is this: is progress being made? At Microsoft the answer is “yes” but only because BillG realized there was a problem to be solved and that is what led to the birth of the SDL.

Join the conversation

  1. CGomez

    If you are just making excuses, then you aren’t letting secure code be part of your process.

    Numbskulls who just blindly believe MSFT products can’t possibly be getting better miss one key element.

    MSFT has gone to great expense to change the culture, provide training, and HIRE NEW PEOPLE with experience in producing secure software.

    That last point can not be emphasized enough.

    MSFT is not just an organization with the same 40,000 developers (number made up) since 1981.  It changes, grows, adds and subtracts.  People join the company, leave, re-join… leave again, and maybe re-join by having their new company purchased.

    That means MSFT has added security experience to its roster over the last ten years, and if you aren’t a blind MSFT bigot, you can see the results.

  2. ericfitz

    You forgot one of my personal favorites:

    "The vulnerability counts are lower because Micro$oft is hiding all the extra fixes in their huge patches and not reporting them".

  3. Igor Levicki

    DISCLAIMER: I am not a Linux user, I am not anti-Microsoft. I just analyze things carefully.

    Comparison of Vista with XP SP2 is flawed.

    First, both operating systems share the same codebase. Many post SP2 (and perhaps even pre-SP3) fixes have been included into first Vista RTM build. If all those fixes were backported to XP it would be as secure as Vista without the annoying part (UAC).

    Second, you are seeing 50% reduction on exactly what number of exposed Vista machines?

    Has that 50% figure been normalized to account for much wider (and longer) exposure of Windows XP?

    As for IM clients, they don’t run under admin account on Linux, neither they are part of OS kernel team’s responsibility. In Linux if something is not secure you can fix it yourself, wait for a patch, or uninstall it and use something else.

    In Windows, single Internet Explorer can infect whole computer even if it is not used for surfing because its components are used in email, help, and office applications. You can’t uninstall it, or fix it yourself.

    As for UAC .vs. su, your argument clearly shows that Microsoft still doesn’t grasp the concept of security.

    Security doesn’t mean nagging the user to find out whether he consents with this or that and thus shifting the blame on the user in case of a problem.

    Proper security model must ask for credentials (not for consent!) and must do it sparingly. Microsoft security model is simply flawed.

    As for "Let’s fix it", it is just a publicity stunt. Open-source developers do not need to proclaim what they are going to do in order to be able to do it.

    Moreover, you used a logical fallacy to convice that company to buy your product — you basically said "yes, we have security issues but they have them too". You have also (wrongly) suggested that they do not admit the problem of (in)security, and that only you do.

    Security issues of others should be irrelevant when someone is considering your company as an option. The only relevant thing is whether your option is secure and stable enough for their purpose. I wonder what kind of support and security bugfixing they will get once Microsoft phases out the OS or the application version you sold them.

    For Microsoft, security was an after-thought. That is why you are now beating your chest and why you act all surprised how nobody else talks about it.

    Finally, if BillG was the only person amongst ~40,000 people in Microsoft to realize that Windows is a glaring security black hole when everyone and their grandmother knew it, then I am afraid to think what will happen if and when he leaves permanently.

  4. sdl

    Igor – I agree with very little of what you said!

    Sure there are fixes in Vista made because of SP2 hindsight, but there are a lot of bugs that DON’T affect Vista because we made so many important wholesale code changes. We also added SAL annotations to Vista code that helped us track down bugs. I think analyzing XP vs Vista is perhaps the most honest comparison because the code is similar.

    The point about this being a publicitiy stuff is again incorrect. And the open source guys DO need some direction to strengthen their code. One guy can’t do it.

    As for "You have also (wrongly) suggested that they do not admit the problem of (in)security, and that only you do." Show me some text ANYWHERE stating from <some guy at Software Shop A> stating that <Software Shop A> has security bugs. What you said sounds like only Microsoft has security bugs!

  5. TF_kj

    Michael, great post. I like the bullets:

    * Microsoft recognized it needed to improve security.

    * Bill said so (as did the rest of senior management)

    * Our group swung into action and helped the rest of the company come up to speed on security issues.

    * The Microsoft development processes changed to adopt the SDL

    I respect the process changes that you guys have implemented. Great to see Msoft participate at BlackHat too.

    There always will be vuln in your code, but you guys have made progress. Congrats.

    Couple other things:

    1. How come it took Bill so long to address the glaring security problems in Microsoft’s products and development processes?

    2. UAC has gotta go.

  6. TF_kj

    Sorry, one last question that I forgot:

    3. How many vulnerabilities are fixed silently in patch updates? Does anyone at Microsoft record patched vulnerabilities that are not publicly reported?

  7. Igor Levicki

    I posted a reply yesterday but seeing it is not up, it seems there is some censorship going on here.

  8. sdl

    Responding to Igor – the only posts we screen are spam, I don’t see any reply from you listed in the blog logs. We encourage open and objective dialog.

Comments are closed.