Bugs like the one shown above are commonly exploiting using the so-called stack smashing attack. To understand how it works, one has to take a look at the function call convention e.g. on Linux/ix86.
When invoking a function, the caller will first push all arguments onto the stack (in the case of strings, the address of the character array is pushed, not the string itself). Pushing something on the stack means that the stack pointer register (esp on ix86) is decremented by the size of that item, and the item is stored at the memory location pointed to by the stack pointer.
Then, the caller executes the call instruction, which pushes the current program counter register onto the stack, pointing to the next byte after the call instruction (called the return address). It then loads the program counter with the address specified by the instruction, i.e. the address of the function.
The called function usually fiddles around with the ebp register, too, but I'll ignore that for clarity's sake right now. Finally, it decrements the stack pointer so that there's enough room on the stack for all local (automatic) variables.
When the function is done with whatever stuff it's supposed to do, it will readjust the stack pointer register so that it points to where it did right upon entry (this is where ebp usually comes in), and execute the ret (for return) statement. This will pop the next address off the stack (which happens to be the return address), and load it into the program counter. This causes the processor to resume execution at the instruction following the call instruction.
Your typical deadly buffer overflow occurs when your application stores data in a local (i.e. automatic) buffer, and writes past the end of the buffer. If the attacker can make the function accept enough data, it will overwrite the memory location holding the return address with data given by the attacker. When the function finally executes the ret statement, it will not return to the calling function anymore, but jump to the location given by the attacker.
In the original RTM worm exploit, as well as many of the ``canned''
exploits being used on the bugtraq mailing list, the data overflowing
the stack buffer contains ``shell code'', i.e. a machine instruction
sequence that will execute ``/bin/sh -c "evil command",''
and the fake return address jumps to the stack location where the shell
code begins:
The prefix of NOP (no-op) instructions as well as repeating the address several times makes the exploit more robust. Usually the exact stack layout differs slightly even between minor releases from a single vendor, because it's very compiler dependent. So you may not know exactly how many bytes there are between the end of the buffer variable, and the stack location of the return address. Hence you repeat the jump address several times, and provide an initial runway of NOP instructions to allow for this sort of fuzz.