Abstract
This paper presents a proposed approach called VAIL System Call Monitor (VSCM) to detect and thwart previously unknown code injection attacks. The idea is based on the fact that any process needs to correctly invoke CreateProcess() system calls, otherwise child-process creation will fail. VSCM intercepts and verifies CreateProcess() system call invocations from a monitored process. In case an unknown executable is detected in the first parameter of a call, this indicates its maliciousness. In response, VSCM encrypts that parameter value to render the call invalid, thereby thwarting adversaries' attacks by preventing the operating system from loading and executing the new malicious child process. VSCM runs in a microkernel-based virtual machine in order to achieve two-fold advantages: (1) isolate security-critical information from probable adversaries' attacks; and (2) exploit security-related and performance-related advantages associated with thin virtual machine monitors. The expected effectiveness of VSCM is high since it is circumvention-proof, and precise in extracting the normal behavior of applications chosen to be monitored.