Interview with Patchguard Architect Forrest Foltz (Windows Vista x64 Security – Patchguard follow up)

Here I am doing my thing, looking at some of the security improvements in Windows Vista x64 (see pt1 and pt2), when all of a sudden, Patchguard seems to be hot news.

 

[NOTE:  Readers, if you need more details on Patchguard, start with my previous post Windows Vista x64 Security – Pt 2 – Patchguard.]

 

So, leveraging my coveted super power of “walking down the hall”, I tracked down the Windows Architect for Kernel Patch Protection, Forrest Foltz, to get his perspective on Patchguard, what benefit it has and how it came about (basically following my charter of trying to look at security topics from multiple angles).

This is Forrest –>

 

Jeff: Forrest, what exactly is it you do for the Windows team in your role as an architect?

 

Forrest:  When I first arrived on the NT kernel team in ’98 from another Windows OS group the name of which I’m no longer permitted to utter, I joined a small set of folks who were defining Windows64 and performing our 64-bit port to the Alpha processor.  Since then I’ve been designing and implementing or helping to implement new support and features into the OS, mainly focusing on kernel and HAL work: PAE, the AMD64 port, NX, patchguard, kernel/hal enlightenments for our upcoming hypervisor, etc.  Most recently I’ve been [doing really, really cool future stuff that Jeff is editing out].  In addition to helping get Vista shipped, of course!

 

Jeff: That seems like a wide scope, how does that work?

 

Forrest:   Pretty darn well.  I am honored to work closely with a fellow by the name of Dave Cutler, who I suspect nominates me for various and sundry tasks that coincidentally are projects that I find both interesting and challenging.

 

Jeff:  Isn’t it true that are actually the leader of a secret anti-competitive team that thinks about ways of locking competitors out, so Microsoft products has a better chance with customers?

 

Forrest: (silence, puzzled look…)

 

Jeff:  Just kidding.  Seriously, I want to ask you some questions about Patchguard.  Who first thought of the idea of Patchguard?

 

Forrest:  This was back in the middle of the AMD64 product cycle (Windows 2003 Server SP1) and everybody was thinking about security, viruses, root kits and that sort of thing.  Dave Cutler came into my office and talked about how great it would be if we could lock the kernel down in some way, and one thing led to another.  When you first ship an OS for a new architecture, there’s a huge opportunity in many areas: no backward compatibility issues!  On AMD64 all code running in the Windows kernel must be compiled natively, so if we wanted to lock the kernel down somehow it was clear that this was the time to do it and the platform to do it on.

 

 

Jeff:  Wait a second, isn’t Patchguard a new feature in Windows Vista?

 

Forrest:  No, we shipped the first version in 64-bit Windows Server 2003 SP1 and then in 64-bit Windows XP, so it has been part of shipping product for nearly 18 months now, and vendors has access to builds several months before that.

 

Jeff:  I had the impression from some stuff I’ve read in the news that 3rd-party software vendors were surprised by Patchguard in Vista.  Do you know when you first started working with development partners and let them know about Patchguard?

 

Forrest:  We posted our kernel protection policy up on MSDN a few months before one of the AMD64 beta releases for WS2003 SP1 / Windows XP64.  That post combined with the Patchguard-enabled AMD64 beta generated a little feedback – mostly from folks who were building instrumentation, kernel debuggers and the like.  Some figured out the “right” way to do what they were trying to do, and others we worked with to architect new interfaces.  In a few cases, there really wasn’t any sane way to accomplish what they were doing with kernel patching.

 

Jeff:  You may not know this one, but do you know how long the kernel team has been working with AV vendors to add defined interfaces?

 

Forrest:  For at least as long as I’ve been on this team, which is probably over eight years now, but I’m sure the history goes back much further.  Certainly I recall working with AV folks before that on the aforementioned, unmentionable OS.

 

Jeff:   That’s consistent with my recollection from my days at McAfee as well.   So, what problem is Patchguard solving that exists in previous operating systems?

 

Forrest:  In basic operating system architecture, there’s this line in the sand between the interfaces and the implementation.  The OS presents interfaces to be used by external code.  Subsequent releases of the OS are obliged to present the same interfaces (there are rare exceptions, such as when APIs are deprecated) but the implementation behind those interfaces may vary significantly as we are able to take advantage of new algorithms and hardware. 

 

In an OS like Windows NT there’s hardware protection between application code and kernel code that prevents applications from partying on our implementation (in stark contrast to the MS-DOS days, where apps would modify the OS to their heart’s content).  However, we also support a kernel environment that is extensible by third-party kernel code (e.g. drivers) and there is no physical protection there, so some ISVs feel free to do the same sorts of things they used to do back in the MS-DOS days – party on code and data that are a part of the operating system’s implementation.  This code and data is what we normally reserve the right to modify and enhance in order to move the platform forward, but we become limited in our ability to do that to the extent that third-party code makes assumptions about implementation based on the implementation of previous releases of the OS.

 

Generalizing, this is just not good for reliability or performance, and to make matters worse, there are often big problems in the way these kernel patches were applied.  For example we found many that didn’t take the possibility of a multiprocessor environment into account.

 

But the biggest threat, in my mind, was that of kernel-mode malware – root kits, viruses and the like.  x86 root kit code almost invariably modifies kernel code or other key kernel structures, and we wanted a way to quickly determine that such malware was present in the system and bring things to an immediate halt if so.  And, of course, application compatibility with root kits is clearly not a concern.

 

Windows ain’t MS-DOS.

 

Jeff:  Okay, I get that.  I just finished reading the Uninformed paper that identifies ways to bypass Patchguard.  So, what do you say to the argument Patchguard already been invalidated as a security benefit? 

 

Forrest:  Well, first, let me point out that Skape and Skywing’s paper assume that bypass attempts are coming from kernel-mode code.  That is an important distinction that most people tend to gloss over.  There are other scenarios that are important to security that aren’t touched by their research results.

 

Jeff:  I’m made that same point myself, actually.  What about bypassing from kernel mode then? 

 

Forrest:  Unfortunately on most existing x64 hardware there aren’t any more privileged protection levels available.  Applications run at ring 3, the kernel runs at ring 0, and there isn’t a “ring -1″ within which to implement something like Patchguard, so we have to rely on obfuscation to do our work.  The problem with that, of course, is that the black hat community is smart.  We think the obfuscation is pretty good, but they will figure it out. 

 

There are a couple of things that support the Patchguard security benefit though.  First, with kernel-mode code signing, we can limit the amount of random, unknown drivers that get loaded and can potentially bypass Patchguard.  That’s a real blow to Trojan-type rootkit installers.  The only drivers loaded will be signed by a Certificate that has been issued to a partner that is actively working and cooperating with us.  That means if we detect it, we can track back to the responsible team.

 

Additionally, we can monitor the Patchguard-targeted work going on in the black hat community, and release a Patchguard update from time to time that will thwart existing Patchguard attacks, if necessary.  That’s the short-term plan going forward.

 

Jeff:  So, basically it provides security protection against stealth attacks from unknown sources and it helps discourage partner development teams from bad behavior.  What about in the long-term?

 

In the long-term, enforcing Patchguard helps us prepare for the possibility of leveraging hypervisor technology, which really does run in a sort of “ring -1″.  When a hypervisor becomes ubiquitous, we have the possibility of moving the Patchguard implementation into that ring, correcting the fundamental assumption that underlies the Skape and Skywing analysis.

 

Basically, if we didn’t implement Patchguard now, then legitimate ISVs would begin relying on their ability to patch the kernel for their own purposes… then, when we shipped the more securable hypervisor-based implementation, our options to build in more robust hardware security might be limited.  So, there is security value today, but equally important is the preserving our ability to improve security even more in the future.

 

Jeff:  What about “known good” applications such as Antivirus and Antispyware programs?  Shouldn’t you make an exception for those?

 

Forrest:  It’s a good question, but what is “known good”?  I don’t know how to define that.  One thing we’ve learned is that good intentions aren’t enough, as the other Skywing paper demonstrates with respect to Anti-virus products.  It sounds appealing, but I don’t see how anyone outside of the kernel team can understand the design well enough to change it safely and then duplicate our test, compatibility and release processes to a degree that meets our release requirements – and incidentally, the level of quality customers want.

 

Jeff:  So, that’s it then.  Nobody can extend the kernel but your team?

 

Forrest:  That’s another misunderstanding, actually, since we do provide methods for extending the kernel.  We have a process for ISVs to build and sign drivers and get them to load and run on 64-bit Vista.  Those drivers can use the documented interfaces to extend kernel functionality in ways that are designed into the system and accounted for.  The Windows filtering platform (read about it here) for firewalls is a good example of this.

 

So, that’s another important distinction that’s been lost in the noise.  We’re working to ensure there are defined ways to extend the kernel, which then allows us to threat model, plan, test, etc, as we need to do.

 

Jeff:  What about Microsoft security products?  Do they have a way around Patchguard – after all, they’re part of Microsoft too?

 

Forrest:  Absolutely not… our “out of band” (e.g. doesn’t necessarily ship with the OS) code can’t use undocumented interfaces, and a Patchguard backdoor would certainly be undocumented.  This was one of the supposed “concerns” of certain ISVs raised early on, but that’s sort of silly.  Everyone here knows that reverse engineer folks would discover such a mechanism about 5 seconds after one of our products shipped with same, so we’d be doubly embarrassed: first the backdoor would be exposed to the world, and second, everybody would start using it.  I don’t even talk to our own security product folks — I mean, I’m sure they’re nice guys, but to me, they’re just like any other ISV developing on “my” platform.

 

Jeff:  Well, Forrest, let me say thanks.  I really appreciate your time.  You are one of the few people who even could’ve answered some of these questions, plus the information about the hypervisor possibilities in the future is really interesting.  And personally, I am feeling just a little bit more secure on my 64-bit home machine.

 

About the Author
Jeff Jones

Principal Cybersecurity Strategist

Jeff Jones a 27-year security industry professional that has spent the last decade at Microsoft working with enterprise CSOs and Microsoft's internal teams to drive practical and measurable security improvements into Microsoft products and services. Additionally, Jeff analyzes vulnerability trends Read more »