Shadow Stack Comparisons
Comparing a call stack in system memory with a shadow call stack maintained by the processor to determine unauthorized shellcode activity.
How it works
This technique compares the call stack stored in system memory with the shadow call stack maintained in the cache memory of the processor. Mismatches between the two are compared since a return oriented programming attack may only be able to control or spoof the call stack and not the shadow call stack. Mismatches are counted and if the number of mismatches exceeds a certain threshold it is an indication of unauthorized activity and a security response action is performed.
If the threshold for detecting a stack anomaly is low, it may not detect a return-oriented attack with just one gadget, such as a return-to-libc or return-to-plt attack. Additionally, this technique may not detect JOP (Jump-oriented programming), as the return instruction is not executed.
Threat detection for return oriented programming
This patent describes a technique for detecting shellcode security exploits. A call stack of a computing device is compared with a shadow call stack maintained by a processor of the computing device since a return oriented program may only be able to control or spoof the call stack and not the shadow call stack. Mismatches between the two are counted and if the number of mismatches exceeds a certain threshold it is an indication of malicious activity and a security response action is performed.