Enhancements to /GS in Visual Studio 11

As mentioned in previous posts, there are some interesting changes afoot regarding security in Visual Studio 11. Here is the next installment of the series by Tim Burrell outlining more of the work done by Security Science and the talented folks on the Visual Studio team…


Microsoft is actively developing Visual Studio 11 and continually looking for ways to improve security-related functionality in the software. We previously noted that we are updating the on-by-default /GS compiler switch, which provides protection against some memory safety bugs such as buffer overflows. This post will provide additional information on those changes.

You may recall that /GS buffer overrun protection places a cookie on the stack between local variables and critical security-critical metadata such as the return address.

The integrity of the GS cookie is checked at the end of the function, prior to the return address being used to return to the caller; if the cookie has been corrupted then execution is terminated rather than carrying on and transferring control to a now suspect return address in memory.

Note that this kind of protection is designed to catch the traditional overflow scenario – i.e. modification of consecutive bytes – and this is indeed by far the most common type of stack corruption bug. However it does not protect a scenario such as:

If the attacker can control the value of ‘n’ above then he can corrupt a single TCHAR character, leaving any GS cookie untouched:

In reviewing those Microsoft Security Response Center (MSRC) cases due to stack-based corruption that were not covered by the existing /GS mechanism, we noted one error that stood out as being more common than others: misplaced null terminators. A typical code sequence might be something like:


The ManipulateString() function correctly writes data within the bounds of the string ‘buf’– but fails to keep track of the final length ‘cch’ of the resulting string. The instruction that null-terminates the string could therefore write outside the bounds of the string buffer without corrupting the GS cookie.

Compile the code above using the Visual Studio 11 Developer Preview tools and you will see that the generated code includes an extra check:

The compiler has inserted range validation code for the null-terminating instruction to guard against an out-of-bounds write to memory, roughly equivalent to:

A couple of questions arising from this are:

1.       “What is the __report_rangecheckfailure() function?”

2.       “When/how often does this range validation happen?”

The __report_rangecheckfailure() is similar to the existing __report_gsfailure() function; it just terminates the program to prevent further execution in a state that we know is about to become untrustworthy. We will come back to this in more detail in a later post.

With respect to how often such range validation happens, it is targeted precisely at the code pattern for which there is historical data indicating the highest risk of a bug being present, namely an assignment to a single array element where:

          The array element size is 1 or 2 bytes, i.e. typically a string.

          The value being written is zero, i.e. to catch the null terminator case.

          The array is declared to be of fixed known size (note that this could be a local or global array so not restricted to the stack).

In addition, for the compiler to be able to insert the instruction guarding against a range violation, it needs to know the size of the array. So an additional requirement in Visual Studio 11 Developer Preview is that the array assignment instruction involves an array of locally and statically declared size. By means of illustration, the following would not lead to a range check being inserted:

As always this is a trade-off. By targeting these extra checks as described above, Visual Studio 11 by default provides extra protection for a limited set of bugs that history tells us are the most common kind of stack-corruption bugs not covered previously by /GS, while minimizing performance and codesize impact by keeping the number of such checks low overall.

And of course /GS continues to provide the familiar cookie-based protection against traditional stack overflows.

The /GS compiler switch is one of many security enhancements being looked at for Visual Studio 11 and is but one small part of the Security Development Lifecycle (SDL) process and methodology for developing secure software, which includes much more than just using specific compiler switches – read more and find additional resources related to SDL here.

Tim Burrell, MSEC Security Science.


About the Author
Dave Ladd

Principal Security Group Program Manager

Join the conversation

  1. Anonymous

    The last example should be "extern char buf[];" – without the *.

    Looks useful.

  2. sdl

    Thanks Simon – good catch! Now corrected 🙂

  3. Anonymous

    char a[100];

    std::fill(a + n, a + m, 0);

    Range checking occurs (m – n) times?

  4. Anonymous

    @Mitsuha: I would guess not, because the std::fill implementation assigns to an iterator (pointer, not array), and that it doesn't assign constant zero (but a value supplied as an argument).

    What I would like to know is whether there will be a debug flag that turns on array bounds checking for *all* assignments. It seems easy enough to implement given that most of the work is already done and would help catch bugs before shipping, which is always good.

  5. Anonymous

    You keep using this phrase "stack overflow".  I do not think it means what you think it means.

    The vulnerability being addressed is properly called "overrun of a stack buffer".

    Actually, I am interested in how the__report_gsfailure function performs under conditions of stack overflow (or any other SEH exception).  Is it safe to perform SEH exception handler search and stack unwinding after a buffer overrun has occurred corrupting stack frames?

  6. Anonymous

    @David: Hmm….

    I want to know whether assignments to element of std::array<> is checked or not.

  7. sdl


    The range checking under /GS (on-by-default for shipping code) is only applied to an assignment to a single array element under the conditions noted above

    so you are correct that range checking would not be performed in the std::fill example that you describe. We don’t expect to provide any further bounds checking of this form in this release. However it is an area of on-going and active work.

    Ref helping catch bugs before shipping: /RTC (msdn.microsoft.com/…/8wtf2dfz.aspx) is useful for inserting extra runtime checks to help with this – though it is not designed to be used for the final shipping build. And of course CodeAnalysis (blogs.msdn.com/…/code-analysis-for-all.aspx), compiler warnings, and IDE extensions (blogs.msdn.com/…/banned-apis-and-extending-the-visual-studio-2010-editor.aspx) are also all great tools to help flag bugs early on. Beyond Visual Studio, I would also encourage taking a look at the SDL tools available for each stage of the development cycle (http://www.microsoft.com/…/tools.aspx).

    @Ben Voigt

    Yes, by “stack overflow” here we are meaning “stack-based buffer overflow”, or equivalently “overrun of a stack buffer” as you note – apologies for the lax terminology.

    About safety of SEH search and stack unwinding: the assumption is that if a program is calling __report_gsfailure() then it has reached an unexpected state; any SEH metadata on the stack (the case for 32-bit programs) is therefore treated as potentially untrusted. So __report_gsfailure() terminates the process directly without searching for or calling any SEH handlers.

    On a related note, an attacker may try to corrupt and trigger use of SEH metadata before a security violation is detected: you can read in more detail about how SAFESEH and SEHOP can help in preventing the exploitation of SEH overwrites. (blogs.technet.com/…/preventing-the-exploitation-of-seh-overwrites-with-sehop.aspx)

  8. Anonymous

    @SDL Team

    Thank you for clarifying. I now understand that inline-function does not cause unexpected range checking.

Comments are closed.