
What does it all mean?
It was probably mid 2003 after the effects of Sapphire, Nimda and Code Red that Intel and AMD both started working with Microsoft to patch the x86 architecture itself. The inherent flaw is two fold; for one, malicious code should not write past the end of an array like line from fingerd, and second the computer should not continue to happily execute the machine code on the stack.
For those not familiar with the stack, here is a quick illustration of how memory looks to an x86 based machine. All memory is divided up into several segments; the stack, heap, and code (or text). Segments are then further divided into pages. Generally, a program executes and runs from the "code" segment. The heap stores dynamically loaded memory, static variables, global variables - anything that need exist during the duration of the program's executable life. The stack is the wild west of memory; function parameters, general variables and arrays end up here. Memory on the stack is reused; once we have processed a function whatever junk it left over in memory resides there.
In the fingerd example from the previous page, writing more than 512 characters to the array begins to overwrite components of the program. When a program begins to load an array, a special byte after the array (called a return address) tells the computer to go back to the code segment and continue to run the program. Unfortunately if a malicious input overwrites the return address with an address in the stack, the computer will execute the code at the location designated, rather than the code the return address pointed to.
NX attempts to correct the buffer overflow issue by disallowing the computer from executing memory in specific pages of the stack. Thus if a malicious input overwrites the portion of the stack and the return address with a clever piece of machine code, NX signals the kernel to panic and the program is terminated.
Thus, some issues are addressed with buffer overflows. It makes it much harder for a malicious input to load a payload into memory with any astounding effects, since we cannot execute some pages of memory on the stack. What NX doesn't stop are buffer overflows that do not rely on their own payload. A malicious input can still overwrite the return address and direct exectution to some other code segment that can cause nefarious deeds - such as a function that creates a file before the user is correctly identified as having the rights to do so. The program might also utilize shared libraries and external system functions like execve, which would give the input identical privileges to the user executing the program. A clever return address combined with initialization of some parameters on the stack would result in execve replicating a payload without machine code.
This is not to say that NX protection is not a step in the right direction. In fact, NX/XD is a good first step to locking down the x86 architecture, as long as it's adopted correctly. OpenBSD and the Execshield projects have made the largest progress with implementing non-executable writable pages and other features, if only in software. However, NX does not completely eliminate buffer overflow exploits, and thus far it has only caused more problems than it has solved with Windows SP2.
Special thanks to D.J. Bernstein, who provided the technical background for the majority of this primer.
1) "As the program starts flipping through the code in its array (which is actually called a stack), it reaches the function gets()."
I think you've mixed a couple of concepts together here. The code is *not* in the stack. They're two different sections of memory.
I would reword this as:
"As the program starts executing the code of the main routine, it reaches the function gets()."
2) "And since line exists inside a stack in memory, writing past the end of line actually starts to overwrite critical pieces of the program!"
This may be more of a nit: I would say that it will overwrite data critical to the program. The code proper is not in the stack, and so you can't actually overwrite the *code* when you go off the end of 'line'.
I'll understand if you don't accept this quibble.
3) "When a program begins to load an array, a special byte after the array (called a return address) tells the computer to go back to the code segment and continue to run the program."
I don't believe the return address is actually a byte, as that would only support an address with an 8-bit range. Isn't it a four byte value?
-- Andyvan