Handling Evasive Malware

Members and Collaborators


Malware analysis relies heavily on the use of virtual machines and debuggers for functionality and for safety. There are subtle differences in operation between virtual machines and physical machines, and in operation with and without a debugger. Contemporary malware checks for these differences to detect that it is being analyzed, and may abort or modify its behavior.

An added challenge for malware analysis is the fact that malware relies heavily on the use of network for its operation. When network use is prohibited during analysis, malware does not exhibit any useful behaviors, which can be analyzed. But allowing unrestricted communication by malware is dangerous, because it may be a part of an active attack.

Our work focuses on three efforts:

  1. Cardinal Pill Testing - a modification of Red Pill Testing that aims to enumerate the differences between a given VM an a physical machine, through carefully designed tests.
  2. Apate - a WinDbg plug-in to detect malware evasion and circumvent it
  3. Fantasm - a framework for live malware analysis, while guaranteeing safety to Internet hosts.

Cardinal Pill Testing

Cardinal Pill Testing is a modification of Red Pill Testing, which aims to enumerate the differences between a given VM and a physical machine, through carefully designed tests. Cardinal Pill Testing finds five times more pills by running fifteen times fewer tests than Red Pill Testing. In our work we further examine the causes of pills and find that, while the majority of them stem from the failure of virtual machines to follow CPU design specifications, a significant number stem from under-specification of the effects of certain instructions by the Intel manual. This leads to divergent implementations in different CPU and virtual machine architectures. Cardinal Pill Testing successfully enumerates differences that stem from the first cause, but only exhaustive testing or an understanding of implementation semantics can enumerate those that stem from the second cause. Finally, we sketch a method to hide pills from malware by systematically correcting their outputs in the virtual machine.


Apate is an extension to WinDbg, which detects and defeats 70 attack vectors, which can be used by malware to detect debuggers' presence. Apate does this by performing: (1) just-in-time disassembling based on singlestepping, (2) careful monitoring of the debuggee’s execution and, when needed, modification of the debuggee’s states to hide the debugger’s presence. Apate outperforms other debugger-hiding technologies by a wide margin, addressing 58%–465% more attack vectors.


Fantasm is a framework for live malware experimentation under partial network containment. Fantasm runs on a testbed with full Internet access, and carefully constrains this access to achieve productive malware analysis, and minimize risk to outside hosts. Fantasm makes decisions on which communications to drop, allow into the Internet or impersonate - intercept and answer itself - based on its estimate of the communication's importance to the ongoing malware operation and based on its estimate of the risk this communication may pose to the Internet.

Software and Datasets


This material is based upon work supported by the Department of Homeland Security, and Space and Naval Warfare Systems Center, San Diego, under Contract No. N66001-10-C-2018. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of Homeland Security for the Space and Naval Warfare Systems Center, San Diego.