One of the most common security vulnerabilities in Windows software is the buffer overrun exploit. It works by feeding a well crafted data stream into a program that uses a stack based buffer without correctly checking the length of the data stream. By writing past the end of the buffer the overwrite can:

1. Store malicious assembler instructions in the stack’s memory pages
2. Change the current function’s return address on the stack so that the malicious instructions are executed instead of the original calling code hwne the function returns.

This technique for injecting code can be used to take control of the current process and possibly the whole PC if the program is running as a user with admin rights.

The problem originates from the fact that Intel’s early x86 processors would enforce the READ and WRITE flags on memory pages; but not the EXECUTE flag that is used for executable code pages. The memory pages used by a program’s stack never have the EXECUTE flag and step 2) above should fail because an attempt is being made to execute READ/WRITE data.

By the time Intel added the NX flag to enforce execution protection, many carelessly written programs relied on being able to execute code in a non-EXECUTE page. So when Microsoft added DEP (Data Execution Protection) in Windows XP SP2, they were unable to turn it on globally because of the large number of third party applications and add-ons that would fail.

Of course, there is one program in particular that would benefit from DEP – Internet Explorer. It’s used by more than half a billion people worldwide and can be subjected to buffer overrun attacks when visiting malicious or hacked web sites. Unfortunately, only a tiny percentage of the IE users around the world have DEP enabled. 

On Windows XP you cannot turn on DEP in Internet Explorer through the user interface, even if you turn on DEP globally like this in the Control panel System applet:

There are certain programs that are always excluded from DEP on Windows XP for compatibility reasons. IE is one of these programs because so many third party add-ons and OCXs failed to work correctly with DEP when Service Pack 2 was released.

Incidentally, HttpWatch works with DEP and is built with the /NXCOMPAT flag to indicate that it can safely be used when DEP is enabled.

On Windows Vista you can enable DEP in IE 7 by using this checkbox:

The checkbox cannot only be modified if you run IE 7 as the administrator:

So why doesn’t everyone do this and why isn’t it the default? That’s because there’s one major non compliant add-on that nearly everyone installs – the Sun Java runtime. When it’s installed and you have DEP enabled, IE 7 raises a DEP error if you visit a site that uses a Java applet:

Sun has acknowledged the problem, but the high priority bug has been open since April 2007:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6545701

Once this has been fixed the way is clear for DEP to be enabled for most IE users on Windows Vista.

In the meantime, IE 7 on Vista still offers much better protection against buffer overruns that Windows XP. That is because of two new features:

• Protected Mode – by default IE 7 runs in a low rights mode even if you are logged in as an administrator. A hijacked instance of IE therfore has limited access to other programs and data on the PC.
• Address Space Layout Randomization (ASLR) – on Windows Vista important system DLLs are placed at random locations in each process. The hacker can no longer rely on system functions being at known locations and would have to make a guess. Any incorrect guess causes the hijacked program to crash preventing continued execution of the malicious code.