System Call Analysis
Definition
Analyzing system calls to determine whether a process is exhibiting unauthorized behavior.
How it works
System calls are APIs between a user application and the operating system [1].
By analyzing a process's use of these APIs, it is, in some cases, possible to ascertain whether a program is exhibiting unauthorized behavior, including trying to escalate its privileges.
Gathering System Calls
A common method to capture system calls is to use kernel APIs to hook [2] a process's system call invocations.
The Linux system call ptrace tracks other system calls in a process and allows their alteration; this is made use of by GDB. strace utilizes ptrace and will print to stdout each system call invoked. Other applications record this data in local or remote databases.
The log entry for each system call, which may reference additional information such as the date and time, and the process tree for the process which made the system call, is relayed, in real time or post-facto, to an analysis module which consults a catalog or model to determine whether the distribution matches a known-good or known-bad pattern.
Analysis
System calls are analyzed with a variety of methods. Some analytics look for specific sequences of instructions, others may apply statistical methods to identify abnormal behavior. Sequences of instructions can be abstracted into conceptually higher order user activities, for example:
- An attacker executes many system calls in a short period of time, with several sequences which could be used to escalate privileges.
- Getting the contents from a URL, writing to a new file, and then executing the same file.
- A ransomware program which either uses a loop or creates many threads to: read a specified file, encrypt its contents, create an output file with a similar name to the original file, and delete the unencrypted original.
Considerations
- Duplicative or extraneous system calls may be added to malware to defeat analytics.
- Malware could replace API hooking instructions to allow system calls to be made without being monitored.
- A model built from a training set of system calls and related data may not be updated fast enough to detect new threats.
[1] Syscalls
[2] Hooking
Technique Subclasses
There are 2 techniques in this category, System Call Analysis.
| Name | ID | Definition | Synonyms |
|---|---|---|---|
| System Call Analysis | D3-SCA | Analyzing system calls to determine whether a process is exhibiting unauthorized behavior. | |
| - File Creation Analysis | D3-FCA | Analyzing the properties of file create system call invocations. |
References
The following references were used to develop the System Call Analysis knowledge-base article.
(Note: the consideration of references does not imply specific functionality exists in an offering.)
CAR-2020-05-001: MiniDump of LSASS
CAR-2021-05-011: Create Remote Thread into LSASS
CAR-2019-08-001: Credential Dumping via Windows Task Manager
CAR-2013-10-002: DLL Injection via Load Library
Deterministic method for detecting and blocking of exploits on interpreted code
MITRE Comments
This patent describes a technique for monitoring API calls. During execution of interpreted code the observed API calls are validated against a whitelist of API calls for that interpreted code file. Action is taken if the observed API call is not in accordance with the list.
Hardware-assisted system and method for detecting and analyzing system calls made to an operting system kernel
MITRE Comments
This patent describes a technique for monitoring system calls to detect malicious software code. A system call monitoring module operates at the kernel level and traps system calls. Monitoring data includes:
- information about the path to the file to be accessed by a system call.
- the memory address or range of addresses to be accessed by a system call.
- the context for the thread within operating system that will be interrupted by a system call.
- the type of system call information about the socket that is being used by system call in order to send or receive data.
- the history of system calls in order to monitor for specific sequences of system calls.
- the frequency or periodicity of a particular system call or set of systems calls.
Captured system call data is analyzed using data analysis algorithms such as machine learning algorithms, artificial intelligence algorithms, pattern recognition algorithms, or other known data analysis techniques. An alert is generated if it is likely that the system call was generated by malicious software code.
Malware detection in event loops
MITRE Comments
The patent describes determining if a sequence of events associated with a process are associated with malware. Based on the relative frequency of events, a loop within a sequence of events is located and a distribution of the events within the loop is determined. The distribution of events is then compared against a catalog of distributions to determine if it is associated with malware.
Post sandbox methods and systems for detecting and blocking zero-day exploits via api call validation
MITRE Comments
The patent describes a technique for monitoring API calls. Executable binary code of an application is first disassembled and scanned for API calls. Based on the recorded API calls, a rule list is generated. Software hooks are placed in the code for monitoring API calls during program execution and then each API call is validated using the generated rule list to permit or deny execution of API calls.
Rules are created that specify the type and location of the API call. For example, data collected for an application can show an API call to libc at address 0x43e0 and an API call by libc at address 0xlfb47. Accordingly, two rules are generated. The first rule specifies the location type and target of the API call at address 0x43e0, as well as the return address. The second rule is for the API call to the kernel and states the target address, return address, instruction, and target type.