US Patent 7945953 Method to identify buffer overflows and RLIBC attacks

A method and system detect buffer overflows and RLIBC attacks by determining if a critical call initiating function is a “potential threat”. In one embodiment, a critical call initiating function is considered a potential threat if the value of the return address of the critical call initiating function points to a location in memory between the location of the highest Thread Environment Block (TEB) or Process Environment Block (PEB) and the location of the lowest Thread Environment Block (TEB) or PEB. In another embodiment, a critical call initiating function making a call to a predefined critical operating system function is considered a potential threat if the value of the return address of the critical call initiating function points to the beginning of a new function with a zero offset.