A Survey of Stealth Malware Attacks, Mitigation Measures, and Steps Toward Autonomous Open World Solutions
---
paper_title: A data mining framework for building intrusion detection models

paper_content:

There is often the need to update an installed intrusion detection system (IDS) due to new attack methods or upgraded computing environments. Since many current IDSs are constructed by manual encoding of expert knowledge, changes to IDSs are expensive and slow. We describe a data mining framework for adaptively building Intrusion Detection (ID) models. The central idea is to utilize auditing programs to extract an extensive set of features that describe each network connection or host session, and apply data mining programs to learn rules that accurately capture the behavior of intrusions and normal activities. These rules can then be used for misuse detection and anomaly detection. New detection models are incorporated into an existing IDS through a meta-learning (or co-operative learning) process, which produces a meta detection model that combines evidence from multiple models. We discuss the strengths of our data mining programs, namely, classification, meta-learning, association rules, and frequent episodes. We report on the results of applying these programs to the extensively gathered network audit data for the 1998 DARPA Intrusion Detection Evaluation Program.
---
paper_title: Taxonomy and Survey of Collaborative Intrusion Detection

paper_content:

The dependency of our society on networked computers has become frightening: In the economy, all-digital networks have turned from facilitators to drivers; as cyber-physical systems are coming of age, computer networks are now becoming the central nervous systems of our physical world—even of highly critical infrastructures such as the power grid. At the same time, the 24s7 availability and correct functioning of networked computers has become much more threatened: The number of sophisticated and highly tailored attacks on IT systems has significantly increased. Intrusion Detection Systems (IDSs) are a key component of the corresponding defense measures; they have been extensively studied and utilized in the past. Since conventional IDSs are not scalable to big company networks and beyond, nor to massively parallel attacks, Collaborative IDSs (CIDSs) have emerged. They consist of several monitoring components that collect and exchange data. Depending on the specific CIDS architecture, central or distributed analysis components mine the gathered data to identify attacks. Resulting alerts are correlated among multiple monitors in order to create a holistic view of the network monitored. This article first determines relevant requirements for CIDSs; it then differentiates distinct building blocks as a basis for introducing a CIDS design space and for discussing it with respect to requirements. Based on this design space, attacks that evade CIDSs and attacks on the availability of the CIDSs themselves are discussed. The entire framework of requirements, building blocks, and attacks as introduced is then used for a comprehensive analysis of the state of the art in collaborative intrusion detection, including a detailed survey and comparison of specific CIDS approaches.
---
paper_title: Intrusion detection and Big Heterogeneous Data: a Survey

paper_content:

Intrusion Detection has been heavily studied in both industry and academia, but cybersecurity analysts still desire much more alert accuracy and overall threat analysis in order to secure their systems within cyberspace. Improvements to Intrusion Detection could be achieved by embracing a more comprehensive approach in monitoring security events from many different heterogeneous sources. Correlating security events from heterogeneous sources can grant a more holistic view and greater situational awareness of cyber threats. One problem with this approach is that currently, even a single event source (e.g., network traffic) can experience Big Data challenges when considered alone. Attempts to use more heterogeneous data sources pose an even greater Big Data challenge. Big Data technologies for Intrusion Detection can help solve these Big Heterogeneous Data challenges. In this paper, we review the scope of works considering the problem of heterogeneous data and in particular Big Heterogeneous Data. We discuss the specific issues of Data Fusion, Heterogeneous Intrusion Detection Architectures, and Security Information and Event Management (SIEM) systems, as well as presenting areas where more research opportunities exist. Overall, both cyber threat analysis and cyber intelligence could be enhanced by correlating security events across many diverse heterogeneous sources.
---
paper_title: Intrusion detection systems: A survey and taxonomy

paper_content:

This paper presents a taxonomy of intrusion detection systems that is then used to survey and classify a number of research prototypes. The taxonomy consists of a classification first of the detection principle, and second of certain operational aspects of the intrusion detection system as such. The systems are also grouped according to the increasing difficulty of the problem they attempt to address. These classifications are used predictively, pointing towards a number of areas of future research in the field of intrusion detection.
---
paper_title: Anomaly-based network intrusion detection: Techniques, systems and challenges

paper_content:

The Internet and computer networks are exposed to an increasing number of security threats. With new types of attacks appearing continually, developing flexible and adaptive security oriented approaches is a severe challenge. In this context, anomaly-based network intrusion detection techniques are a valuable technology to protect target systems and networks against malicious activities. However, despite the variety of such methods described in the literature in recent years, security tools incorporating anomaly detection functionalities are just starting to appear, and several important problems remain to be solved. This paper begins with a review of the most well-known anomaly-based intrusion detection techniques. Then, available platforms, systems under development and research projects in the area are presented. Finally, we outline the main challenges to be dealt with for the wide scale deployment of anomaly-based intrusion detectors, with special emphasis on assessment issues.
---
paper_title: Android Security: A Survey of Issues, Malware Penetration, and Defenses

paper_content:

Smartphones have become pervasive due to the availability of office applications, Internet, games, vehicle guidance using location-based services apart from conventional services such as voice calls, SMSes, and multimedia services. Android devices have gained huge market share due to the open architecture of Android and the popularity of its application programming interface (APIs) in the developer community. Increased popularity of the Android devices and associated monetary benefits attracted the malware developers, resulting in big rise of the Android malware apps between 2010 and 2014. Academic researchers and commercial antimalware companies have realized that the conventional signature-based and static analysis methods are vulnerable. In particular, the prevalent stealth techniques, such as encryption, code transformation, and environment-aware approaches, are capable of generating variants of known malware. This has led to the use of behavior-, anomaly-, and dynamic-analysis-based methods. Since a single approach may be ineffective against the advanced techniques, multiple complementary approaches can be used in tandem for effective malware detection. The existing reviews extensively cover the smartphone OS security. However, we believe that the security of Android, with particular focus on malware growth, study of antianalysis techniques, and existing detection methodologies, needs an extensive coverage. In this survey, we discuss the Android security enforcement mechanisms, threats to the existing security enforcements and related issues, malware growth timeline between 2010 and 2014, and stealth techniques employed by the malware authors, in addition to the existing detection methods. This review gives an insight into the strengths and shortcomings of the known research methodologies and provides a platform, to the researchers and practitioners, toward proposing the next-generation Android security, analysis, and malware detection techniques.
---
paper_title: Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

paper_content:

In network intrusion detection research, one popular strategy for finding attacks is monitoring a network's activity for anomalies: deviations from profiles of normality previously learned from benign traffic, typically identified using tools borrowed from the machine learning community. However, despite extensive academic research one finds a striking gap in terms of actual deployments of such systems: compared with other intrusion detection approaches, machine learning is rarely employed in operational "real world" settings. We examine the differences between the network intrusion detection problem and other areas where machine learning regularly finds much more success. Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively. We support this claim by identifying challenges particular to network intrusion detection, and provide a set of guidelines meant to strengthen future research on anomaly detection.
---
paper_title: Review: Intrusion detection by machine learning: A review

paper_content:

The popularity of using Internet contains some risks of network attacks. Intrusion detection is one major research problem in network security, whose aim is to identify unusual access or attacks to secure internal networks. In literature, intrusion detection systems have been approached by various machine learning techniques. However, there is no a review paper to examine and understand the current status of using machine learning techniques to solve the intrusion detection problems. This chapter reviews 55 related studies in the period between 2000 and 2007 focusing on developing single, hybrid, and ensemble classifiers. Related studies are compared by their classifier design, datasets used, and other experimental setups. Current achievements and limitations in developing intrusion detection systems by machine learning are present and discussed. A number of future research directions are also provided.
---
paper_title: Survey on Android Rootkit

paper_content:

Rootkit can stealthily modify the code and data of operating system to achieve malicious goals,and it has long been a problem for computers.With increasingly equipped with operating systems,smart phones are as vulnerable to many of the same threats as computers.This paper analyzed the implementation of a SMS-based kernel-level Rootkit on Android system,described the attack behaviors of it,and put forward three effective detections such as EPA.
---
paper_title: Copilot - a coprocessor-based kernel runtime integrity monitor

paper_content:

Copilot is a coprocessor-based kernel integrity monitor for commodity systems. Copilot is designed to detect malicious modifications to a host's kernel and has correctly detected the presence of 12 real-world rootkits, each within 30 seconds of their installation with less than a 1% penalty to the host's performance. Copilot requires no modifications to the protected host's software and can be expected to operate correctly even when the host kernel is thoroughly compromised - an advantage over traditional monitors designed to run on the host itself.
---
paper_title: The design and implementation of tripwire: a file system integrity checker

paper_content:

At the heart of most computer systems is a file system. The file system contains user data, executable programs, configuration and authorization information, and (usually) the base executable version of the operating system itself. The ability to monitor file systems for unauthorized or unexpected changes gives system administrators valuable data for protecting and maintaining their systems. However, in environments of many networked heterogeneous platforms with different policies and software, the task of monitoring changes becomes quite daunting. Tripwire is tool that aids UNIX system administrators and users in monitoring a designated set of files and directories for any changes. Used with system files on a regular (e.g., daily) basis, Tripwire can notify system administrators of corrupted or altered files, so corrective actions may be taken in a timely manner. Tripwire may also be used on user or group files or databases to signal changes. This paper describes the design and implementation of the Tripwire tool. It uses interchangeable “signature” (usually, message digest) routines to identify changes in files, and is highly configurable. Tripwire is no-cost software, available on the Internet, and is currently in use on thousands of machines around the world.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Rootkits: Subverting the Windows Kernel

paper_content:

"It's imperative that everybody working in the field of cyber-security read this book to understand the growing threat of rootkits." --Mark Russinovich, editor, Windows IT Pro / Windows & .NET Magazine"This material is not only up-to-date, it defines up-to-date. It is truly cutting-edge. As the only book on the subject, Rootkits will be of interest to any Windows security researcher or security programmer. It's detailed, well researched and the technical information is excellent. The level of technical detail, research, and time invested in developing relevant examples is impressive. In one word: Outstanding." --Tony Bautts, Security Consultant; CEO, Xtivix, Inc."This book is an essential read for anyone responsible for Windows security. Security professionals, Windows system administrators, and programmers in general will want to understand the techniques used by rootkit authors. At a time when many IT and security professionals are still worrying about the latest e-mail virus or how to get all of this month's security patches installed, Mr. Hoglund and Mr. Butler open your eyes to some of the most stealthy and significant threats to the Windows operating system. Only by understanding these offensive techniques can you properly defend the networks and systems for which you are responsible." --Jennifer Kolde, Security Consultant, Author, and Instructor"What's worse than being owned? Not knowing it. Find out what it means to be owned by reading Hoglund and Butler's first-of-a-kind book on rootkits. At the apex the malicious hacker toolset--which includes decompilers, disassemblers, fault-injection engines, kernel debuggers, payload collections, coverage tools, and flow analysis tools--is the rootkit. Beginning where Exploiting Software left off, this book shows how attackers hide in plain sight. "Rootkits are extremely powerful and are the next wave of attack technology. Like other types of malicious code, rootkits thrive on stealthiness. They hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can't easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a backdoor into the target machine. "Understanding the ultimate attacker's tool provides an important motivator for those of us trying to defend systems. No authors are better suited to give you a detailed hands-on understanding of rootkits than Hoglund and Butler. Better to own this book than to be owned." --Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software (2004) and Building Secure Software (2002), both from Addison-Wesley"Greg and Jamie are unquestionably the go-to experts when it comes to subverting the Windows API and creating rootkits. These two masters come together to pierce the veil of mystery surrounding rootkits, bringing this information out of the shadows. Anyone even remotely interested in security for Windows systems, including forensic analysis, should include this book very high on their must-read list." --Harlan Carvey, author of Windows Forensics and Incident Recovery (Addison-Wesley, 2005)Rootkits are the ultimate backdoor, giving hackers ongoing and virtually undetectable access to the systems they exploit. Now, two of the world's leading experts have written the first comprehensive guide to rootkits: what they are, how they work, how to build them, and how to detect them. Rootkit.com's Greg Hoglund and James Butler created and teach Black Hat's legendary course in rootkits. In this book, they reveal never-before-told offensive aspects of rootkit technology--learn how attackers can get in and stay in for years, without detection.Hoglund and Butler show exactly how to subvert the Windows XP and Windows 2000 kernels, teaching concepts that are easily applied to virtually any modern operating system, from Windows Server 2003 to Linux and UNIX. Using extensive downloadable examples, they teach rootkit programming techniques that can be used for a wide range of software, from white hat security tools to operating system drivers and debuggers.After reading this book, readers will be able to Understand the role of rootkits in remote command/control and software eavesdropping Build kernel rootkits that can make processes, files, and directories invisible Master key rootkit programming techniques, including hooking, runtime patching, and directly manipulating kernel objects Work with layered drivers to implement keyboard sniffers and file filters Detect rootkits and build host-based intrusion prevention software that resists rootkit attacksVisit rootkit.com for code and programs from this book. The site also contains enhancements to the book's text, such as up-to-the-minute information on rootkits available nowhere else.
---
paper_title: Modern Operating Systems

paper_content:

For software development professionals and computer science students, Modern Operating Systems gives a solid conceptual overview of operating system design, including detailed case studies of Unix/Linux and Windows 2000. What makes an operating system modern? According to author Andrew Tanenbaum, it is the awareness of high-demand computer applications--primarily in the areas of multimedia, parallel and distributed computing, and security. The development of faster and more advanced hardware has driven progress in software, including enhancements to the operating system. It is one thing to run an old operating system on current hardware, and another to effectively leverage current hardware to best serve modern software applications. If you don't believe it, install Windows 3.0 on a modern PC and try surfing the Internet or burning a CD. Readers familiar with Tanenbaum's previous text, Operating Systems, know the author is a great proponent of simple design and hands-on experimentation. His earlier book came bundled with the source code for an operating system called Minux, a simple variant of Unix and the platform used by Linus Torvalds to develop Linux. Although this book does not come with any source code, he illustrates many of his points with code fragments (C, usually with Unix system calls). The first half of Modern Operating Systems focuses on traditional operating systems concepts: processes, deadlocks, memory management, I/O, and file systems. There is nothing groundbreaking in these early chapters, but all topics are well covered, each including sections on current research and a set of student problems. It is enlightening to read Tanenbaum's explanations of the design decisions made by past operating systems gurus, including his view that additional research on the problem of deadlocks is impractical except for "keeping otherwise unemployed graph theorists off the streets." It is the second half of the book that differentiates itself from older operating systems texts. Here, each chapter describes an element of what constitutes a modern operating system--awareness of multimedia applications, multiple processors, computer networks, and a high level of security. The chapter on multimedia functionality focuses on such features as handling massive files and providing video-on-demand. Included in the discussion on multiprocessor platforms are clustered computers and distributed computing. Finally, the importance of security is discussed--a lively enumeration of the scores of ways operating systems can be vulnerable to attack, from password security to computer viruses and Internet worms. Included at the end of the book are case studies of two popular operating systems: Unix/Linux and Windows 2000. There is a bias toward the Unix/Linux approach, not surprising given the author's experience and academic bent, but this bias does not detract from Tanenbaum's analysis. Both operating systems are dissected, describing how each implements processes, file systems, memory management, and other operating system fundamentals. Tanenbaum's mantra is simple, accessible operating system design. Given that modern operating systems have extensive features, he is forced to reconcile physical size with simplicity. Toward this end, he makes frequent references to the Frederick Brooks classic The Mythical Man-Month for wisdom on managing large, complex software development projects. He finds both Windows 2000 and Unix/Linux guilty of being too complicated--with a particular skewering of Windows 2000 and its "mammoth Win32 API." A primary culprit is the attempt to make operating systems more "user-friendly," which Tanenbaum views as an excuse for bloated code. The solution is to have smart people, the smallest possible team, and well-defined interactions between various operating systems components. Future operating system design will benefit if the advice in this book is taken to heart. --Pete Ostenson
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Rootkits: Subverting the Windows Kernel

paper_content:

"It's imperative that everybody working in the field of cyber-security read this book to understand the growing threat of rootkits." --Mark Russinovich, editor, Windows IT Pro / Windows & .NET Magazine"This material is not only up-to-date, it defines up-to-date. It is truly cutting-edge. As the only book on the subject, Rootkits will be of interest to any Windows security researcher or security programmer. It's detailed, well researched and the technical information is excellent. The level of technical detail, research, and time invested in developing relevant examples is impressive. In one word: Outstanding." --Tony Bautts, Security Consultant; CEO, Xtivix, Inc."This book is an essential read for anyone responsible for Windows security. Security professionals, Windows system administrators, and programmers in general will want to understand the techniques used by rootkit authors. At a time when many IT and security professionals are still worrying about the latest e-mail virus or how to get all of this month's security patches installed, Mr. Hoglund and Mr. Butler open your eyes to some of the most stealthy and significant threats to the Windows operating system. Only by understanding these offensive techniques can you properly defend the networks and systems for which you are responsible." --Jennifer Kolde, Security Consultant, Author, and Instructor"What's worse than being owned? Not knowing it. Find out what it means to be owned by reading Hoglund and Butler's first-of-a-kind book on rootkits. At the apex the malicious hacker toolset--which includes decompilers, disassemblers, fault-injection engines, kernel debuggers, payload collections, coverage tools, and flow analysis tools--is the rootkit. Beginning where Exploiting Software left off, this book shows how attackers hide in plain sight. "Rootkits are extremely powerful and are the next wave of attack technology. Like other types of malicious code, rootkits thrive on stealthiness. They hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can't easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a backdoor into the target machine. "Understanding the ultimate attacker's tool provides an important motivator for those of us trying to defend systems. No authors are better suited to give you a detailed hands-on understanding of rootkits than Hoglund and Butler. Better to own this book than to be owned." --Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software (2004) and Building Secure Software (2002), both from Addison-Wesley"Greg and Jamie are unquestionably the go-to experts when it comes to subverting the Windows API and creating rootkits. These two masters come together to pierce the veil of mystery surrounding rootkits, bringing this information out of the shadows. Anyone even remotely interested in security for Windows systems, including forensic analysis, should include this book very high on their must-read list." --Harlan Carvey, author of Windows Forensics and Incident Recovery (Addison-Wesley, 2005)Rootkits are the ultimate backdoor, giving hackers ongoing and virtually undetectable access to the systems they exploit. Now, two of the world's leading experts have written the first comprehensive guide to rootkits: what they are, how they work, how to build them, and how to detect them. Rootkit.com's Greg Hoglund and James Butler created and teach Black Hat's legendary course in rootkits. In this book, they reveal never-before-told offensive aspects of rootkit technology--learn how attackers can get in and stay in for years, without detection.Hoglund and Butler show exactly how to subvert the Windows XP and Windows 2000 kernels, teaching concepts that are easily applied to virtually any modern operating system, from Windows Server 2003 to Linux and UNIX. Using extensive downloadable examples, they teach rootkit programming techniques that can be used for a wide range of software, from white hat security tools to operating system drivers and debuggers.After reading this book, readers will be able to Understand the role of rootkits in remote command/control and software eavesdropping Build kernel rootkits that can make processes, files, and directories invisible Master key rootkit programming techniques, including hooking, runtime patching, and directly manipulating kernel objects Work with layered drivers to implement keyboard sniffers and file filters Detect rootkits and build host-based intrusion prevention software that resists rootkit attacksVisit rootkit.com for code and programs from this book. The site also contains enhancements to the book's text, such as up-to-the-minute information on rootkits available nowhere else.
---
paper_title: Detours: Binary Interception of Win32 Functions

paper_content:

Innovative systems research hinges on the ability to easily instrument and extend existing operating system and application functionality. With access to appropriate source code, it is often trivial to insert new instrumentation or extensions by rebuilding the OS or application. However, in today's world of commercial software, researchers seldom have access to all relevant source code. ::: ::: We present Detours, a library for instrumenting arbitrary Win32 functions on x86 machines. Detours intercepts Win32 functions by re-writing target function images. The Detours package also contains utilities to attach arbitrary DLLs and data segments (called payloads) to any Win32 binary. ::: ::: While prior researchers have used binary rewriting to insert debugging and profiling instrumentation, to our knowledge, Detours is the first package on any platform to logically preserve the un-instrumented target function (callable through a trampoline) as a subroutine for use by the instrumentation. Our unique trampoline design is crucial for extending existing binary software. ::: ::: We describe our experiences using Detours to create an automatic distributed partitioning system, to instrument and analyze the DCOM protocol stack, and to create a thunking layer for a COM-based OS API. Micro-benchmarks demonstrate the efficiency of the Detours library.
---
paper_title: Rootkits: Subverting the Windows Kernel

paper_content:

"It's imperative that everybody working in the field of cyber-security read this book to understand the growing threat of rootkits." --Mark Russinovich, editor, Windows IT Pro / Windows & .NET Magazine"This material is not only up-to-date, it defines up-to-date. It is truly cutting-edge. As the only book on the subject, Rootkits will be of interest to any Windows security researcher or security programmer. It's detailed, well researched and the technical information is excellent. The level of technical detail, research, and time invested in developing relevant examples is impressive. In one word: Outstanding." --Tony Bautts, Security Consultant; CEO, Xtivix, Inc."This book is an essential read for anyone responsible for Windows security. Security professionals, Windows system administrators, and programmers in general will want to understand the techniques used by rootkit authors. At a time when many IT and security professionals are still worrying about the latest e-mail virus or how to get all of this month's security patches installed, Mr. Hoglund and Mr. Butler open your eyes to some of the most stealthy and significant threats to the Windows operating system. Only by understanding these offensive techniques can you properly defend the networks and systems for which you are responsible." --Jennifer Kolde, Security Consultant, Author, and Instructor"What's worse than being owned? Not knowing it. Find out what it means to be owned by reading Hoglund and Butler's first-of-a-kind book on rootkits. At the apex the malicious hacker toolset--which includes decompilers, disassemblers, fault-injection engines, kernel debuggers, payload collections, coverage tools, and flow analysis tools--is the rootkit. Beginning where Exploiting Software left off, this book shows how attackers hide in plain sight. "Rootkits are extremely powerful and are the next wave of attack technology. Like other types of malicious code, rootkits thrive on stealthiness. They hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can't easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a backdoor into the target machine. "Understanding the ultimate attacker's tool provides an important motivator for those of us trying to defend systems. No authors are better suited to give you a detailed hands-on understanding of rootkits than Hoglund and Butler. Better to own this book than to be owned." --Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software (2004) and Building Secure Software (2002), both from Addison-Wesley"Greg and Jamie are unquestionably the go-to experts when it comes to subverting the Windows API and creating rootkits. These two masters come together to pierce the veil of mystery surrounding rootkits, bringing this information out of the shadows. Anyone even remotely interested in security for Windows systems, including forensic analysis, should include this book very high on their must-read list." --Harlan Carvey, author of Windows Forensics and Incident Recovery (Addison-Wesley, 2005)Rootkits are the ultimate backdoor, giving hackers ongoing and virtually undetectable access to the systems they exploit. Now, two of the world's leading experts have written the first comprehensive guide to rootkits: what they are, how they work, how to build them, and how to detect them. Rootkit.com's Greg Hoglund and James Butler created and teach Black Hat's legendary course in rootkits. In this book, they reveal never-before-told offensive aspects of rootkit technology--learn how attackers can get in and stay in for years, without detection.Hoglund and Butler show exactly how to subvert the Windows XP and Windows 2000 kernels, teaching concepts that are easily applied to virtually any modern operating system, from Windows Server 2003 to Linux and UNIX. Using extensive downloadable examples, they teach rootkit programming techniques that can be used for a wide range of software, from white hat security tools to operating system drivers and debuggers.After reading this book, readers will be able to Understand the role of rootkits in remote command/control and software eavesdropping Build kernel rootkits that can make processes, files, and directories invisible Master key rootkit programming techniques, including hooking, runtime patching, and directly manipulating kernel objects Work with layered drivers to implement keyboard sniffers and file filters Detect rootkits and build host-based intrusion prevention software that resists rootkit attacksVisit rootkit.com for code and programs from this book. The site also contains enhancements to the book's text, such as up-to-the-minute information on rootkits available nowhere else.
---
paper_title: Operating System Interface Obfuscation and the Revealing of Hidden Operations

paper_content:

Many software security solutions--including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors--rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel's system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.
---
paper_title: Modern Operating Systems

paper_content:

For software development professionals and computer science students, Modern Operating Systems gives a solid conceptual overview of operating system design, including detailed case studies of Unix/Linux and Windows 2000. What makes an operating system modern? According to author Andrew Tanenbaum, it is the awareness of high-demand computer applications--primarily in the areas of multimedia, parallel and distributed computing, and security. The development of faster and more advanced hardware has driven progress in software, including enhancements to the operating system. It is one thing to run an old operating system on current hardware, and another to effectively leverage current hardware to best serve modern software applications. If you don't believe it, install Windows 3.0 on a modern PC and try surfing the Internet or burning a CD. Readers familiar with Tanenbaum's previous text, Operating Systems, know the author is a great proponent of simple design and hands-on experimentation. His earlier book came bundled with the source code for an operating system called Minux, a simple variant of Unix and the platform used by Linus Torvalds to develop Linux. Although this book does not come with any source code, he illustrates many of his points with code fragments (C, usually with Unix system calls). The first half of Modern Operating Systems focuses on traditional operating systems concepts: processes, deadlocks, memory management, I/O, and file systems. There is nothing groundbreaking in these early chapters, but all topics are well covered, each including sections on current research and a set of student problems. It is enlightening to read Tanenbaum's explanations of the design decisions made by past operating systems gurus, including his view that additional research on the problem of deadlocks is impractical except for "keeping otherwise unemployed graph theorists off the streets." It is the second half of the book that differentiates itself from older operating systems texts. Here, each chapter describes an element of what constitutes a modern operating system--awareness of multimedia applications, multiple processors, computer networks, and a high level of security. The chapter on multimedia functionality focuses on such features as handling massive files and providing video-on-demand. Included in the discussion on multiprocessor platforms are clustered computers and distributed computing. Finally, the importance of security is discussed--a lively enumeration of the scores of ways operating systems can be vulnerable to attack, from password security to computer viruses and Internet worms. Included at the end of the book are case studies of two popular operating systems: Unix/Linux and Windows 2000. There is a bias toward the Unix/Linux approach, not surprising given the author's experience and academic bent, but this bias does not detract from Tanenbaum's analysis. Both operating systems are dissected, describing how each implements processes, file systems, memory management, and other operating system fundamentals. Tanenbaum's mantra is simple, accessible operating system design. Given that modern operating systems have extensive features, he is forced to reconcile physical size with simplicity. Toward this end, he makes frequent references to the Frederick Brooks classic The Mythical Man-Month for wisdom on managing large, complex software development projects. He finds both Windows 2000 and Unix/Linux guilty of being too complicated--with a particular skewering of Windows 2000 and its "mammoth Win32 API." A primary culprit is the attempt to make operating systems more "user-friendly," which Tanenbaum views as an excuse for bloated code. The solution is to have smart people, the smallest possible team, and well-defined interactions between various operating systems components. Future operating system design will benefit if the advice in this book is taken to heart. --Pete Ostenson
---
paper_title: Detecting Kernel-Level Rootkits Using Data Structure Invariants

paper_content:

Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.
---
paper_title: Detecting stealth software with Strider GhostBuster

paper_content:

Stealth malware programs that silently infect enterprise and consumer machines are becoming a major threat to the future of the Internet. Resource hiding is a powerful stealth technique commonly used by malware to evade detection by computer users and anti-malware scanners. In this paper, we focus on a subclass of malware, termed "ghostware", which hide files, configuration settings, processes, and loaded modules from the operating system's query and enumeration application programming interfaces (APIs). Instead of targeting individual stealth implementations, we describe a systematic framework for detecting multiple types of hidden resources by leveraging the hiding behavior as a detection mechanism. Specifically, we adopt a cross-view diff-based approach to ghostware detection by comparing a high-level infected scan with a low-level clean scan and alternatively comparing an inside-the-box infected scan with an outside-the-box clean scan. We describe the design and implementation of the Strider GhostBuster tool and demonstrate its efficiency and effectiveness in detecting resources hidden by real-world malware such as rootkits, Trojans, and key-loggers.
---
paper_title: Metamorphic worm that carries its own morphing engine

paper_content:

Metamorphic malware changes its internal structure across generations, but its functionality remains unchanged. Well-designed metamorphic malware will evade signature detection. Recent research has revealed techniques based on hidden Markov models (HMMs) for detecting many types of metamorphic malware, as well as techniques for evading such detection. A worm is a type of malware that actively spreads across a network to other host systems. In this project we design and implement a prototype metamorphic worm that carries its own morphing engine. This is challenging, since the morphing engine itself must be morphed across replications, which imposes restrictions on the structure of the worm. Our design employs previously developed techniques to evade detection. We provide test results to confirm that this worm effectively evades signature and HMM-based detection, and we consider possible detection strategies. This worm provides a concrete example that should prove useful for additional metamorphic detection research.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Evaluation of Android Anti-malware Techniques against Dalvik Bytecode Obfuscation

paper_content:

Popularity and growth of Android mobile devices has paved the way for exploiting popular apps using various Dalvik byte code transformation methods. Testing the antimalware techniques against obfuscation identifies the need of proposing effective detection methods. In this paper, we explore the resilience of anti-malware techniques against transformations for Android. The Proposed approach employs variable compression, native code wrapping and register renaming, in addition to already implemented transformations on Dalvik byte code. Evaluation results indicate low resilience of the antimalware detection engines against code obfuscation. Furthermore, we evaluate resilience of Androguard's code similarity and AndroSimilar's robust statistical feature signature against code obfuscated malware.
---
paper_title: Code mutation techniques by means of formal grammars and automatons

paper_content:

The paper describes formalization of existing code mutation techniques widely used in a viruses (polymorphism and metamorphism) by means of formal grammars and automatons. New model of metamorphic viruses and new classification of this type of viruses are suggested. The statement about undetectable viruses of this type is proved. In that paper are shown iterative approach toward construct complex formal grammars from the simplest initial rules for building metamorphic generator. Also there are some samples of applied usage of formal grammar model. The experiment for system call tracing of some viruses and worms is described. Possibility of using system call sequences for viruses detecting is shown.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Hunting for metamorphic

paper_content:

As virus writers developed numerous polymorphic engines, virus scanners became stronger in their defense against them. A virus scanner which used a code emulator to detect viruses looked like it was on steroids compared to those without an emulator-based scanning engine. Nowadays, most polymorphic viruses are considered boring. Even though they can be extremely hard to detect, most of today’s products are able to deal with them relatively easily. These are the scanners that survived the DOS polymorphic days. For some of the scanners DOS polymorphic viruses meant the ‘end of days’. Other scanners died with the macro virus problem. For most products the next challenge to take is 32-bit metamorphosis. Metamorphic viruses are nothing new. We have seen them in DOS days, though some of them, like ACG, already used 32-bit instructions. The next step is 32-bit metamorphosis under Windows environments. Virus writers already took the first step in that direction. In this paper the authors will examine metamorphic engines to provide a better general understanding of the problem that we are facing. The authors also provide detection examples of some of the metamorphic viruses.
---
paper_title: DroidAnalyst: Synergic App Framework for Static and Dynamic App Analysis

paper_content:

Evolution of mobile devices, availability of additional resources coupled with enhanced functionality has leveraged smartphone to substitute the conventional computing devices. Mobile device users have adopted smartphones for online payments, sending emails, social networking, and stores the user sensitive information. The ever increasing mobile devices has attracted malware authors and cybercriminals to target mobile platforms. Android, the most popular open source mobile OS is being targeted by the malware writers. In particular, less monitored third party markets are being used as infection and propagation sources. Given the threats posed by the increasing number of malicious apps, security researchers must be able to analyze the malware quickly and efficiently; this may not be feasible with the manual analysis. Hence, automated analysis techniques for app vetting and malware detection are necessary. In this chapter, we present DroidAnalyst, a novel automated app vetting and malware analysis framework that integrates the synergy of static and dynamic analysis to improve accuracy and efficiency of analysis. DroidAnalyst generates a unified analysis model that combines the strengths of the complementary approaches with multiple detection methods, to increase the app code analysis. We have evaluated our proposed solution DroidAnalyst against a reasonable dataset consisting real-world benign and malware apps.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Copilot - a coprocessor-based kernel runtime integrity monitor

paper_content:

Copilot is a coprocessor-based kernel integrity monitor for commodity systems. Copilot is designed to detect malicious modifications to a host's kernel and has correctly detected the presence of 12 real-world rootkits, each within 30 seconds of their installation with less than a 1% penalty to the host's performance. Copilot requires no modifications to the protected host's software and can be expected to operate correctly even when the host kernel is thoroughly compromised - an advantage over traditional monitors designed to run on the host itself.
---
paper_title: A Virtual Machine Introspection Based Architecture for Intrusion Detection

paper_content:

Today’s architectures for intrusion detection force the IDS designer to make a difficult choice. If the IDS resides on the host, it has an excellent view of what is happening in that host’s software, but is highly susceptible to attack. On the other hand, if the IDS resides in the network, it is more resistant to attack, but has a poor view of what is happening inside the host, making it more susceptible to evasion. In this paper we present an architecture that retains the visibility of a host-based IDS, but pulls the IDS outside of the host for greater attack resistance. We achieve this through the use of a virtual machine monitor. Using this approach allows us to isolate the IDS from the monitored host but still retain excellent visibility into the host’s state. The VMM also offers us the unique ability to completely mediate interactions between the host software and the underlying hardware. We present a detailed study of our architecture, including Livewire, a prototype implementation. We demonstrate Livewire by implementing a suite of simple intrusion detection policies and using them to detect real attacks.
---
paper_title: Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

paper_content:

In network intrusion detection research, one popular strategy for finding attacks is monitoring a network's activity for anomalies: deviations from profiles of normality previously learned from benign traffic, typically identified using tools borrowed from the machine learning community. However, despite extensive academic research one finds a striking gap in terms of actual deployments of such systems: compared with other intrusion detection approaches, machine learning is rarely employed in operational "real world" settings. We examine the differences between the network intrusion detection problem and other areas where machine learning regularly finds much more success. Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively. We support this claim by identifying challenges particular to network intrusion detection, and provide a set of guidelines meant to strengthen future research on anomaly detection.
---
paper_title: Rootkits: Subverting the Windows Kernel

paper_content:

"It's imperative that everybody working in the field of cyber-security read this book to understand the growing threat of rootkits." --Mark Russinovich, editor, Windows IT Pro / Windows & .NET Magazine"This material is not only up-to-date, it defines up-to-date. It is truly cutting-edge. As the only book on the subject, Rootkits will be of interest to any Windows security researcher or security programmer. It's detailed, well researched and the technical information is excellent. The level of technical detail, research, and time invested in developing relevant examples is impressive. In one word: Outstanding." --Tony Bautts, Security Consultant; CEO, Xtivix, Inc."This book is an essential read for anyone responsible for Windows security. Security professionals, Windows system administrators, and programmers in general will want to understand the techniques used by rootkit authors. At a time when many IT and security professionals are still worrying about the latest e-mail virus or how to get all of this month's security patches installed, Mr. Hoglund and Mr. Butler open your eyes to some of the most stealthy and significant threats to the Windows operating system. Only by understanding these offensive techniques can you properly defend the networks and systems for which you are responsible." --Jennifer Kolde, Security Consultant, Author, and Instructor"What's worse than being owned? Not knowing it. Find out what it means to be owned by reading Hoglund and Butler's first-of-a-kind book on rootkits. At the apex the malicious hacker toolset--which includes decompilers, disassemblers, fault-injection engines, kernel debuggers, payload collections, coverage tools, and flow analysis tools--is the rootkit. Beginning where Exploiting Software left off, this book shows how attackers hide in plain sight. "Rootkits are extremely powerful and are the next wave of attack technology. Like other types of malicious code, rootkits thrive on stealthiness. They hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can't easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a backdoor into the target machine. "Understanding the ultimate attacker's tool provides an important motivator for those of us trying to defend systems. No authors are better suited to give you a detailed hands-on understanding of rootkits than Hoglund and Butler. Better to own this book than to be owned." --Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software (2004) and Building Secure Software (2002), both from Addison-Wesley"Greg and Jamie are unquestionably the go-to experts when it comes to subverting the Windows API and creating rootkits. These two masters come together to pierce the veil of mystery surrounding rootkits, bringing this information out of the shadows. Anyone even remotely interested in security for Windows systems, including forensic analysis, should include this book very high on their must-read list." --Harlan Carvey, author of Windows Forensics and Incident Recovery (Addison-Wesley, 2005)Rootkits are the ultimate backdoor, giving hackers ongoing and virtually undetectable access to the systems they exploit. Now, two of the world's leading experts have written the first comprehensive guide to rootkits: what they are, how they work, how to build them, and how to detect them. Rootkit.com's Greg Hoglund and James Butler created and teach Black Hat's legendary course in rootkits. In this book, they reveal never-before-told offensive aspects of rootkit technology--learn how attackers can get in and stay in for years, without detection.Hoglund and Butler show exactly how to subvert the Windows XP and Windows 2000 kernels, teaching concepts that are easily applied to virtually any modern operating system, from Windows Server 2003 to Linux and UNIX. Using extensive downloadable examples, they teach rootkit programming techniques that can be used for a wide range of software, from white hat security tools to operating system drivers and debuggers.After reading this book, readers will be able to Understand the role of rootkits in remote command/control and software eavesdropping Build kernel rootkits that can make processes, files, and directories invisible Master key rootkit programming techniques, including hooking, runtime patching, and directly manipulating kernel objects Work with layered drivers to implement keyboard sniffers and file filters Detect rootkits and build host-based intrusion prevention software that resists rootkit attacksVisit rootkit.com for code and programs from this book. The site also contains enhancements to the book's text, such as up-to-the-minute information on rootkits available nowhere else.
---
paper_title: Modern Operating Systems

paper_content:

For software development professionals and computer science students, Modern Operating Systems gives a solid conceptual overview of operating system design, including detailed case studies of Unix/Linux and Windows 2000. What makes an operating system modern? According to author Andrew Tanenbaum, it is the awareness of high-demand computer applications--primarily in the areas of multimedia, parallel and distributed computing, and security. The development of faster and more advanced hardware has driven progress in software, including enhancements to the operating system. It is one thing to run an old operating system on current hardware, and another to effectively leverage current hardware to best serve modern software applications. If you don't believe it, install Windows 3.0 on a modern PC and try surfing the Internet or burning a CD. Readers familiar with Tanenbaum's previous text, Operating Systems, know the author is a great proponent of simple design and hands-on experimentation. His earlier book came bundled with the source code for an operating system called Minux, a simple variant of Unix and the platform used by Linus Torvalds to develop Linux. Although this book does not come with any source code, he illustrates many of his points with code fragments (C, usually with Unix system calls). The first half of Modern Operating Systems focuses on traditional operating systems concepts: processes, deadlocks, memory management, I/O, and file systems. There is nothing groundbreaking in these early chapters, but all topics are well covered, each including sections on current research and a set of student problems. It is enlightening to read Tanenbaum's explanations of the design decisions made by past operating systems gurus, including his view that additional research on the problem of deadlocks is impractical except for "keeping otherwise unemployed graph theorists off the streets." It is the second half of the book that differentiates itself from older operating systems texts. Here, each chapter describes an element of what constitutes a modern operating system--awareness of multimedia applications, multiple processors, computer networks, and a high level of security. The chapter on multimedia functionality focuses on such features as handling massive files and providing video-on-demand. Included in the discussion on multiprocessor platforms are clustered computers and distributed computing. Finally, the importance of security is discussed--a lively enumeration of the scores of ways operating systems can be vulnerable to attack, from password security to computer viruses and Internet worms. Included at the end of the book are case studies of two popular operating systems: Unix/Linux and Windows 2000. There is a bias toward the Unix/Linux approach, not surprising given the author's experience and academic bent, but this bias does not detract from Tanenbaum's analysis. Both operating systems are dissected, describing how each implements processes, file systems, memory management, and other operating system fundamentals. Tanenbaum's mantra is simple, accessible operating system design. Given that modern operating systems have extensive features, he is forced to reconcile physical size with simplicity. Toward this end, he makes frequent references to the Frederick Brooks classic The Mythical Man-Month for wisdom on managing large, complex software development projects. He finds both Windows 2000 and Unix/Linux guilty of being too complicated--with a particular skewering of Windows 2000 and its "mammoth Win32 API." A primary culprit is the attempt to make operating systems more "user-friendly," which Tanenbaum views as an excuse for bloated code. The solution is to have smart people, the smallest possible team, and well-defined interactions between various operating systems components. Future operating system design will benefit if the advice in this book is taken to heart. --Pete Ostenson
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: Detecting stealth software with Strider GhostBuster

paper_content:

Stealth malware programs that silently infect enterprise and consumer machines are becoming a major threat to the future of the Internet. Resource hiding is a powerful stealth technique commonly used by malware to evade detection by computer users and anti-malware scanners. In this paper, we focus on a subclass of malware, termed "ghostware", which hide files, configuration settings, processes, and loaded modules from the operating system's query and enumeration application programming interfaces (APIs). Instead of targeting individual stealth implementations, we describe a systematic framework for detecting multiple types of hidden resources by leveraging the hiding behavior as a detection mechanism. Specifically, we adopt a cross-view diff-based approach to ghostware detection by comparing a high-level infected scan with a low-level clean scan and alternatively comparing an inside-the-box infected scan with an outside-the-box clean scan. We describe the design and implementation of the Strider GhostBuster tool and demonstrate its efficiency and effectiveness in detecting resources hidden by real-world malware such as rootkits, Trojans, and key-loggers.
---
paper_title: An architecture for specification-based detection of semantic integrity violations in kernel dynamic data

paper_content:

The ability of intruders to hide their presence in compromised systems has surpassed the ability of the current generation of integrity monitors to detect them. Once in control of a system, intruders modify the state of constantly-changing dynamic kernel data structures to hide their processes and elevate their privileges. Current monitoring tools are limited to detecting changes in nominally static kernel data and text and cannot distinguish a valid state change from tampering in these dynamic data structures. We introduce a novel general architecture for defining and monitoring semantic integrity constraints using a specification language-based approach. This approach will enable a new generation of integrity monitors to distinguish valid states from tampering.
---
paper_title: Copilot - a coprocessor-based kernel runtime integrity monitor

paper_content:

Copilot is a coprocessor-based kernel integrity monitor for commodity systems. Copilot is designed to detect malicious modifications to a host's kernel and has correctly detected the presence of 12 real-world rootkits, each within 30 seconds of their installation with less than a 1% penalty to the host's performance. Copilot requires no modifications to the protected host's software and can be expected to operate correctly even when the host kernel is thoroughly compromised - an advantage over traditional monitors designed to run on the host itself.
---
paper_title: Detecting Kernel-Level Rootkits Using Data Structure Invariants

paper_content:

Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.
---
paper_title: Copilot - a coprocessor-based kernel runtime integrity monitor

paper_content:

Copilot is a coprocessor-based kernel integrity monitor for commodity systems. Copilot is designed to detect malicious modifications to a host's kernel and has correctly detected the presence of 12 real-world rootkits, each within 30 seconds of their installation with less than a 1% penalty to the host's performance. Copilot requires no modifications to the protected host's software and can be expected to operate correctly even when the host kernel is thoroughly compromised - an advantage over traditional monitors designed to run on the host itself.
---
paper_title: A survey of data mining techniques for malware detection using file features

paper_content:

This paper presents a survey of data mining techniques for malware detection using file features. The techniques are categorized based upon a three tier hierarchy that includes file features, analysis type and detection type. File features are the features extracted from binary programs, analysis type is either static or dynamic, and the detection type is borrowed from intrusion detection as either misuse or anomaly detection. It provides the reader with the major advancement in the malware research using data mining on file features and categorizes the surveyed work based upon the above stated hierarchy. This served as the major contribution of this paper.
---
paper_title: Robust signatures for kernel data structures

paper_content:

Kernel-mode rootkits hide objects such as processes and threads using a technique known as Direct Kernel Object Manipulation (DKOM). Many forensic analysis tools attempt to detect these hidden objects by scanning kernel memory with handmade signatures; however, such signatures are brittle and rely on non-essential features of these data structures, making them easy to evade. In this paper, we present an automated mechanism for generating signatures for kernel data structures and show that these signatures are robust: attempts to evade the signature by modifying the structure contents will cause the OS to consider the object invalid. Using dynamic analysis, we profile the target data structure to determine commonly used fields, and we then fuzz those fields to determine which are essential to the correct operation of the OS. These fields form the basis of a signature for the data structure. In our experiments, our new signature matched the accuracy of existing scanners for traditional malware and found processes hidden with our prototype rootkit that all current signatures missed. Our techniques significantly increase the difficulty of hiding objects from signature scanning.
---
paper_title: Operating System Interface Obfuscation and the Revealing of Hidden Operations

paper_content:

Many software security solutions--including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors--rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel's system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.
---
paper_title: Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing

paper_content:

Kernel rootkits pose a significant threat to computer systems as they run at the highest privilege level and have unrestricted access to the resources of their victims. Many current efforts in kernel rootkit defense focus on the detectionof kernel rootkits --- after a rootkit attack has taken place, while the smaller number of efforts in kernel rootkit preventionexhibit limitations in their capability or deployability. In this paper we present a kernel rootkit prevention system called NICKLE which addresses a common, fundamental characteristic of most kernel rootkits: the need for executing their own kernel code. NICKLE is a lightweight, virtual machine monitor (VMM) based system that transparently prevents unauthorized kernel code execution for unmodified commodity (guest) OSes. NICKLE is based on a new scheme called memory shadowing, wherein the trusted VMM maintains a shadow physical memory for a running VM and performs real-time kernel code authentication so that only authenticated kernel code will be stored in the shadow memory. Further, NICKLE transparently routes guest kernel instruction fetches to the shadow memory at runtime. By doing so, NICKLE guarantees that only the authenticated kernel code will be executed, foiling the kernel rootkit's attempt to strike in the first place. We have implemented NICKLE in three VMM platforms: QEMU+KQEMU, VirtualBox, and VMware Workstation. Our experiments with 23 real-world kernel rootkits targeting the Linux or Windows OSes demonstrate NICKLE's effectiveness. Furthermore, our performance evaluation shows that NICKLE introduces small overhead to the VMM platform.
---
paper_title: SecVisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity OSes

paper_content:

We propose SecVisor, a tiny hypervisor that ensures code integrity for commodity OS kernels. In particular, SecVisor ensures that only user-approved code can execute in kernel mode over the entire system lifetime. This protects the kernel against code injection attacks, such as kernel rootkits. SecVisor can achieve this propertyeven against an attacker who controls everything but the CPU, the memory controller, and system memory chips. Further, SecVisor can even defend against attackers with knowledge of zero-day kernel exploits. Our goal is to make SecVisor amenable to formal verificationand manual audit, thereby making it possible to rule out known classes of vulnerabilities. To this end, SecVisor offers small code size and small external interface. We rely on memory virtualization to build SecVisor and implement two versions, one using software memory virtualization and the other using CPU-supported memory virtualization. The code sizes of the runtime portions of these versions are 1739 and 1112 lines, respectively. The size of the external interface for both versions of SecVisor is 2 hypercalls. It is easy to port OS kernels to SecVisor. We port the Linux kernel version 2.6.20 by adding 12 lines and deleting 81 lines, out of a total of approximately 4.3 million lines of code in the kernel.
---
paper_title: Automatic placement of authorization hooks in the linux security modules framework

paper_content:

We present a technique for automatic placement of authorization hooks, and apply it to the Linux security modules (LSM) framework. LSM is a generic framework which allows diverse authorization policies to be enforced by the Linux kernel. It consists of a kernel module which encapsulates an authorization policy, and hooks into the kernel module placed at appropriate locations in the Linux kernel. The kernel enforces the authorization policy using hook calls. In current practice, hooks are placed manually in the kernel. This approach is tedious, and as prior work has shown, is prone to security holes.Our technique uses static analysis of the Linux kernel and the kernel module to automate hook placement. Given a non-hook-placed version of the Linux kernel, and a kernel module that implements an authorization policy, our technique infers the set of operations authorized by each hook, and the set of operations performed by each function in the kernel. It uses this information to infer the set of hooks that must guard each kernel function. We describe the design and implementation of a prototype tool called T AHOE (Tool for Authorization Hook Placement) that uses this technique. We demonstrate the effectiveness of T AHOE by using it with the LSM implementation of security-enhanced Linux (selinux). While our exposition in this paper focuses on hook placement for LSM, our technique can be used to place hooks in other LSM-like architectures as well.
---
paper_title: Multi-aspect profiling of kernel rootkit behavior

paper_content:

Kernel rootkits, malicious software designed to compromise a running operating system kernel, are difficult to analyze and profile due to their elusive nature, the variety and complexity of their behavior, and the privilege level at which they run. However, a comprehensive kernel rootkit profile that reveals key aspects of the rootkit's behavior is helpful in aiding a detailed manual analysis by a human expert. In this paper we present PoKeR, a kernel rootkit profiler capable of producing multi-aspect rootkit profiles which include the revelation of rootkit hooking behavior, the exposure of targeted kernel objects (both static and dynamic), assessment of user-level impacts, as well as the extraction of kernel rootkit code. The system is designed to be deployed in scenarios which can tolerate high overheads, such as honeypots. Our evaluation results with a number of real-world kernel rootkits show that PoKeR is able to accurately profile a variety of rootkits ranging from traditional ones with system call hooking to more advanced ones with direct kernel object manipulation. The obtained profiles lead to unique insights into the rootkits' characteristics and demonstrate PoKeR's usefulness as a tool for rootkit investigators.
---
paper_title: A Virtual Machine Introspection Based Architecture for Intrusion Detection

paper_content:

Today’s architectures for intrusion detection force the IDS designer to make a difficult choice. If the IDS resides on the host, it has an excellent view of what is happening in that host’s software, but is highly susceptible to attack. On the other hand, if the IDS resides in the network, it is more resistant to attack, but has a poor view of what is happening inside the host, making it more susceptible to evasion. In this paper we present an architecture that retains the visibility of a host-based IDS, but pulls the IDS outside of the host for greater attack resistance. We achieve this through the use of a virtual machine monitor. Using this approach allows us to isolate the IDS from the monitored host but still retain excellent visibility into the host’s state. The VMM also offers us the unique ability to completely mediate interactions between the host software and the underlying hardware. We present a detailed study of our architecture, including Livewire, a prototype implementation. We demonstrate Livewire by implementing a suite of simple intrusion detection policies and using them to detect real attacks.
---
paper_title: Valgrind: a framework for heavyweight dynamic binary instrumentation

paper_content:

Dynamic binary instrumentation (DBI) frameworks make it easy to build dynamic binary analysis (DBA) tools such as checkers and profilers. Much of the focus on DBI frameworks has been on performance; little attention has been paid to their capabilities. As a result, we believe the potential of DBI has not been fully exploited. In this paper we describe Valgrind, a DBI framework designed for building heavyweight DBA tools. We focus on its unique support for shadow values-a powerful but previously little-studied and difficult-to-implement DBA technique, which requires a tool to shadow every register and memory value with another value that describes it. This support accounts for several crucial design features that distinguish Valgrind from other DBI frameworks. Because of these features, lightweight tools built with Valgrind run comparatively slowly, but Valgrind can be used to build more interesting, heavyweight tools that are difficult or impossible to build with other DBI frameworks such as Pin and DynamoRIO.
---
paper_title: Operating System Interface Obfuscation and the Revealing of Hidden Operations

paper_content:

Many software security solutions--including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors--rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel's system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: DroidAnalyst: Synergic App Framework for Static and Dynamic App Analysis

paper_content:

Evolution of mobile devices, availability of additional resources coupled with enhanced functionality has leveraged smartphone to substitute the conventional computing devices. Mobile device users have adopted smartphones for online payments, sending emails, social networking, and stores the user sensitive information. The ever increasing mobile devices has attracted malware authors and cybercriminals to target mobile platforms. Android, the most popular open source mobile OS is being targeted by the malware writers. In particular, less monitored third party markets are being used as infection and propagation sources. Given the threats posed by the increasing number of malicious apps, security researchers must be able to analyze the malware quickly and efficiently; this may not be feasible with the manual analysis. Hence, automated analysis techniques for app vetting and malware detection are necessary. In this chapter, we present DroidAnalyst, a novel automated app vetting and malware analysis framework that integrates the synergy of static and dynamic analysis to improve accuracy and efficiency of analysis. DroidAnalyst generates a unified analysis model that combines the strengths of the complementary approaches with multiple detection methods, to increase the app code analysis. We have evaluated our proposed solution DroidAnalyst against a reasonable dataset consisting real-world benign and malware apps.
---
paper_title: Detecting and classifying method based on similarity matching of Android malware behavior with profile

paper_content:

AbstractMass-market mobile security threats have increased ::: recently due to the growth of mobile technologies and the popularity of mobile devices. Accordingly, techniques have been introduced for identifying, classifying, and defending against mobile threats utilizing static, dynamic, on-device, and off-device techniques. Static techniques are easy to evade, while dynamic techniques are expensive. On-device techniques are evasion, while off-device techniques need being always online. To address some of those shortcomings, we introduce Andro-profiler, a hybrid behavior based analysis and classification system for mobile malware. Andro-profiler main goals are efficiency, scalability, and accuracy. For that, Andro-profiler classifies malware by exploiting the behavior profiling extracted from the integrated system logs including system calls. Andro-profiler executes a malicious application on an emulator in order to generate the integrated system logs, and creates human-readable behavior profiles by analyzing the integrated system logs. By comparing the behavior profile of malicious application with representative behavior profile for each malware family using a weighted similarity matching technique, Andro-profiler detects and classifies it into malware families. The experiment results demonstrate that Andro-profiler is scalable, performs well in detecting and classifying malware with accuracy greater than 98 %, outperforms the existing state-of-the-art work, and is capable of identifying 0-day mobile malware samples.
---
paper_title: Searching for processes and threads in Microsoft Windows memory dumps

paper_content:

Current tools to analyze memory dumps of systems running Microsoft Windows usually build on the concept of enumerating lists maintained by the kernel to keep track of processes, threads and other objects. Therefore they will frequently fail to detect objects that are already terminated or which have been hidden by Direct Kernel Object Manipulation techniques. This article analyzes the in-memory structures which represent processes and threads. It develops search patterns which will then be used to scan the whole memory dump for traces of said objects, independent from the aforementioned lists. As demonstrated by a proof-of-concept implementation this approach could reveal hidden and terminated processes and threads, under some circumstances even after the system under examination has been rebooted.
---
paper_title: Detecting Kernel-Level Rootkits Using Data Structure Invariants

paper_content:

Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.
---
paper_title: Shredding your garbage: Reducing data lifetime through secure deallocation

paper_content:

Today's operating systems, word processors, web browsers, and other common software take no measures to promptly remove data from memory. Consequently, sensitive data, such as passwords, social security numbers, and confidential documents, often remains in memory indefinitely, significantly increasing the risk of exposure. ::: ::: We present a strategy for reducing the lifetime of data in memory called secure deallocation. With secure deal-location we zero data either at deallocation or within a short, predictable period afterward in general system allocators (e.g. user heap, user stack, kernel heap). This substantially reduces data lifetime with minimal implementation effort, negligible overhead, and without modifying existing applications. ::: ::: We demonstrate that secure deallocation generally clears data immediately after its last use, and that without such measures, data can remain in memory for days or weeks, even persisting across reboots. We further show that secure deallocation promptly eliminates sensitive data in a variety of important real world applications.
---
paper_title: The Art of Computer Virus Research and Defense

paper_content:

"Of all the computer-related books I've read recently, this one influenced my thoughts about security the most. There is very little trustworthy information about computer viruses. Peter Szor is one of the best virus analysts in the world and has the perfect credentials to write this book."-Halvar Flake, Reverse Engineer, SABRE Security GmbHSymantec's chief antivirus researcher has written the definitive guide to contemporary virus threats, defense techniques, and analysis tools. Unlike most books on computer viruses, The Art of Computer Virus Research and Defense is a reference written strictly for white hats: IT and security professionals responsible for protecting their organizations against malware. Peter Szor systematically covers everything you need to know, including virus behavior and classification, protection strategies, antivirus and worm-blocking techniques, and much more.Szor presents the state-of-the-art in both malware and protection, providing the full technical detail that professionals need to handle increasingly complex attacks. Along the way, he provides extensive information on code metamorphism and other emerging techniques, so you can anticipate and prepare for future threats.Szor also offers the most thorough and practical primer on virus analysis ever published-addressing everything from creating your own personal laboratory to automating the analysis process. This book's coverage includes Discovering how malicious code attacks on a variety of platforms Classifying malware strategies for infection, in-memory operation, self-protection, payload delivery, exploitation, and more Identifying and responding to code obfuscation threats: encrypted, polymorphic, and metamorphic Mastering empirical methods for analyzing malicious code-and what to do with what you learn Reverse-engineering malicious code with disassemblers, debuggers, emulators, and virtual machines Implementing technical defenses: scanning, code emulation, disinfection, inoculation, integrity checking, sandboxing, honeypots, behavior blocking, and much more Using worm blocking, host-based intrusion prevention, and network-level defense strategies © Copyright Pearson Education. All rights reserved.
---
paper_title: A survey of data mining techniques for malware detection using file features

paper_content:

This paper presents a survey of data mining techniques for malware detection using file features. The techniques are categorized based upon a three tier hierarchy that includes file features, analysis type and detection type. File features are the features extracted from binary programs, analysis type is either static or dynamic, and the detection type is borrowed from intrusion detection as either misuse or anomaly detection. It provides the reader with the major advancement in the malware research using data mining on file features and categorizes the surveyed work based upon the above stated hierarchy. This served as the major contribution of this paper.
---
paper_title: Robust signatures for kernel data structures

paper_content:

Kernel-mode rootkits hide objects such as processes and threads using a technique known as Direct Kernel Object Manipulation (DKOM). Many forensic analysis tools attempt to detect these hidden objects by scanning kernel memory with handmade signatures; however, such signatures are brittle and rely on non-essential features of these data structures, making them easy to evade. In this paper, we present an automated mechanism for generating signatures for kernel data structures and show that these signatures are robust: attempts to evade the signature by modifying the structure contents will cause the OS to consider the object invalid. Using dynamic analysis, we profile the target data structure to determine commonly used fields, and we then fuzz those fields to determine which are essential to the correct operation of the OS. These fields form the basis of a signature for the data structure. In our experiments, our new signature matched the accuracy of existing scanners for traditional malware and found processes hidden with our prototype rootkit that all current signatures missed. Our techniques significantly increase the difficulty of hiding objects from signature scanning.
---
paper_title: New malicious code detection using variable length n-grams

paper_content:

Most of the commercial antivirus software fail to detect unknown and new malicious code. In order to handle this problem generic virus detection is a viable option. Generic virus detector needs features that are common to viruses. Recently Kolter et al. [16] propose an efficient generic virus detector using n-grams as features. The fixed length n-grams used there suffer from the drawback that they cannot capture meaningful sequences of different lengths. In this paper we propose a new method of variable-length n-grams extraction based on the concept of episodes and demonstrate that they outperform fixed length n-grams in malicious code detection. The proposed algorithm requires only two scans over the whole data set whereas most of the classical algorithms require scans proportional to the maximum length of n-grams.
---
paper_title: Information flow control for standard OS abstractions

paper_content:

Decentralized Information Flow Control (DIFC) is an approach to security that allows application writers to control how data flows between the pieces of an application and the outside world. As applied to privacy, DIFC allows untrusted software to compute with private data while trusted security code controls the release of that data. As applied to integrity, DIFC allows trusted code to protect untrusted software from unexpected malicious inputs. In either case, only bugs in the trusted code, which tends to be small and isolated, can lead to security violations. We present Flume, a new DIFC model that applies at the granularity of operating system processes and standard OS abstractions (e.g., pipes and file descriptors). Flume was designed for simplicity of mechanism, to ease DIFC's use in existing applications, and to allow safe interaction between conventional and DIFC-aware processes. Flume runs as a user-level reference monitor onLinux. A process confined by Flume cannot perform most system calls directly; instead, an interposition layer replaces system calls with IPCto the reference monitor, which enforces data flowpolicies and performs safe operations on the process's behalf. We ported a complex web application (MoinMoin Wiki) to Flume, changingonly 2% of the original code. Performance measurements show a 43% slowdown on read workloadsand a 34% slowdown on write workloads, which aremostly due to Flume's user-level implementation.
---
paper_title: A sense of self for Unix processes

paper_content:

A method for anomaly detection is introduced in which ``normal'' is defined by short-range correlations in a process' system calls. Initial experiments suggest that the definition is stable during normal behavior for standard UNIX programs. Further, it is able to detect several common intrusions involving sendmail and lpr. This work is part of a research program aimed at building computer security systems that incorporate the mechanisms and algorithms used by natural immune systems.
---
paper_title: Modeling malicious activities in cyber space

paper_content:

Cyber attacks are an unfortunate part of society as an increasing amount of critical infrastructure is managed and controlled via the Internet. In order to protect legitimate users, it is critical for us to obtain an accurate and timely understanding of our cyber opponents. However, at the moment we lack effective tools to do this. In this article we summarize the work on modeling malicious activities from various perspectives, discuss the pros and cons of current models, and present promising directions for possible efforts in the near future.
---
paper_title: Anomaly detection using call stack information

paper_content:

The call stack of a program execution can be a very good information source for intrusion detection. There is no prior work on dynamically extracting information from the call stack and effectively using it to detect exploits. In this paper we propose a new method to do anomaly detection using call stack information. The basic idea is to extract return addresses from the call stack, and generate an abstract execution path between two program execution points. Experiments show that our method can detect some attacks that cannot be detected by other approaches, while its convergence and false positive performance is comparable to or better than the other approaches. We compare our method with other approaches by analyzing their underlying principles and thus achieve a better characterization of their performance, in particular on what and why attacks will be missed by the various approaches.
---
paper_title: Visual correlation of host processes and network traffic

paper_content:

Anomalous communication patterns are one of the leading indicators of computer system intrusions according to the system administrators we have interviewed. But a major problem is being able to correlate across the host/network boundary to see how network connections are related to running processes on a host. This paper introduces Portall, a visualization tool that gives system administrators a view of the communicating processes on the monitored machine correlated with the network activity in which the processes participate. Portall is a prototype of part of the Network Eye framework we have introduced in an earlier paper (Ball, et al., 2004). We discuss the Portall visualization, the supporting infrastructure it requires, and a formative usability study we conducted to obtain administrators' reactions to the tool.
---
paper_title: Exploiting Execution Context for the Detection of Anomalous System Calls

paper_content:

Attacks against privileged applications can be detected by analyzing the stream of system calls issued during process execution. In the last few years, several approaches have been proposed to detect anomalous system calls. These approaches are mostly based on modeling acceptable system call sequences. Unfortunately, the techniques proposed so far are either vulnerable to certain evasion attacks or are too expensive to be practical. This paper presents a novel approach to the analysis of system calls that uses a composition of dynamic analysis and learning techniques to characterize anomalous system call invocations in terms of both the invocation context and the parameters passed to the system calls. Our technique provides a more precise detection model with respect to solutions proposed previously, and, in addition, it is able to detect data modification attacks, which cannot be detected using only system call sequence analysis.
---
paper_title: Intrusion Detection Using Sequences of System Calls

paper_content:

A method is introduced for detecting intrusions at the level of privileged processes. Evidence is given that short sequences of system calls executed by running processes are a good discriminator between normal and abnormal operating characteristics of several common UNIX programs. Normal behavior is collected in two waysc Synthetically, by exercising as many normal modes of usage of a program as possible, and in a live user environment by tracing the actual execution of the program. In the former case several types of intrusive behavior were studieds in the latter case, results were analyzed for false positives.
---
paper_title: Fool Me If You Can: Mimicking Attacks and Anti-Attacks in Cyberspace

paper_content:

Botnets have become major engines for malicious activities in cyberspace nowadays. To sustain their botnets and disguise their malicious actions, botnet owners are mimicking legitimate cyber behavior to fly under the radar. This poses a critical challenge in anomaly detection. In this paper, we use web browsing on popular web sites as an example to tackle this problem. First of all, we establish a semi-Markov model for browsing behavior. Based on this model, we find that it is impossible to detect mimicking attacks based on statistics if the number of active bots of the attacking botnet is sufficiently large (no less than the number of active legitimate users). However, we also find it is hard for botnet owners to satisfy the condition to carry out a mimicking attack most of the time. With this new finding, we conclude that mimicking attacks can be discriminated from genuine flash crowds using second order statistical metrics. We define a new fine correntropy metrics and show its effectiveness compared to others. Our real world data set experiments and simulations confirm our theoretical claims. Furthermore, the findings can be widely applied to similar situations in other research fields.
---
paper_title: A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors

paper_content:

Anomaly detection on system call sequences has become perhaps the most successful approach for detecting novel intrusions. A natural way for learning sequences is to use a finite-state automaton (FSA). However previous research indicates that FSA-learning is computationally expensive, that it cannot be completely automated or that the space usage of the FSA may be excessive. We present a new approach that overcomes these difficulties. Our approach builds a compact FSA in a fully automatic and efficient manner, without requiring access to source code for programs. The space requirements for the FSA is low - of the order of a few kilobytes for typical programs. The FSA uses only a constant time per system call during the learning as well as the detection period. This factor leads to low overheads for intrusion detection. Unlike many of the previous techniques, our FSA-technique can capture both short term and long term temporal relationships among system calls, and thus perform more accurate detection. This enables our approach to generalize and predict future behaviors from past behaviors. As a result, the training periods needed for our FSA based approach are shorter. Moreover false positives are reduced without increasing the likelihood of missing attacks. This paper describes our FSA based technique and presents a comprehensive experimental evaluation of the technique.
---
paper_title: Metamorphic worm that carries its own morphing engine

paper_content:

Metamorphic malware changes its internal structure across generations, but its functionality remains unchanged. Well-designed metamorphic malware will evade signature detection. Recent research has revealed techniques based on hidden Markov models (HMMs) for detecting many types of metamorphic malware, as well as techniques for evading such detection. A worm is a type of malware that actively spreads across a network to other host systems. In this project we design and implement a prototype metamorphic worm that carries its own morphing engine. This is challenging, since the morphing engine itself must be morphed across replications, which imposes restrictions on the structure of the worm. Our design employs previously developed techniques to evade detection. We provide test results to confirm that this worm effectively evades signature and HMM-based detection, and we consider possible detection strategies. This worm provides a concrete example that should prove useful for additional metamorphic detection research.
---
paper_title: Opcode graph similarity and metamorphic detection

paper_content:

In this paper, we consider a method for computing the similarity of executable files, based on opcode graphs. We apply this technique to the challenging problem of metamorphic malware detection and compare the results to previous work based on hidden Markov models. In addition, we analyze the effect of various morphing techniques on the success of our proposed opcode graph-based detection scheme.
---
paper_title: Hunting for metamorphic engines

paper_content:

In this paper, we analyze several metamorphic virus generators. We define a similarity index and use it to precisely quantify the degree of metamorphism that each generator produces. Then we present a detector based on hidden Markov models and we consider a simpler detection method based on our similarity index. Both of these techniques detect all of the metamorphic viruses in our test set with extremely high accuracy. In addition, we show that popular commercial virus scanners do not detect the highly metamorphic virus variants in our test set.
---
paper_title: Profile hidden Markov models and metamorphic virus detection

paper_content:

Metamorphic computer viruses “mutate” by changing their internal structure and, consequently, different instances of the same virus may not exhibit a common signature. With the advent of construction kits, it is easy to generate metamorphic strains of a given virus. In contrast to standard hidden Markov models (HMMs), profile hidden Markov models (PHMMs) explicitly account for positional information. In principle, this positional information could yield stronger models for virus detection. However, there are many practical difficulties that arise when using PHMMs, as compared to standard HMMs. PHMMs are widely used in bioinformatics. For example, PHMMs are the most effective tool yet developed for finding family related DNA sequences. In this paper, we consider the utility of PHMMs for detecting metamorphic virus variants generated from virus construction kits. PHMMs are generated for each construction kit under consideration and the resulting models are used to score virus and non-virus files. Our results are encouraging, but several problems must be resolved for the technique to be truly practical.
---
paper_title: CODE OBFUSCATION AND VIRUS DETECTION

paper_content:

Typically, computer viruses and other malware are detected by searching for a string of bits which is found in the virus or malware. Such a string can be viewed as a “fingerprint” of the virus. These “fingerprints” are not generally unique; however they can be used to make rapid malware scanning feasible. This fingerprint is often called a signature and the technique of detecting viruses using signatures is known as signaturebased detection [8]. Today, virus writers often camouflage their viruses by using code obfuscation techniques in an effort to defeat signature-based detection schemes. So-called metamorphic viruses are viruses in which each instance has the same functionality but differs in its internal structure. Metamorphic viruses differ from polymorphic viruses in the method they use to hide their signature. While polymorphic viruses primarily rely on encryption for signature obfuscation, metamorphic viruses hide their signature via “mutating” their own code [3]. The paper [1] provides a rigorous proof that metamorphic viruses can bypass any signature-based detection, provided the code obfuscation has been done carefully based on a set of specified rules. Specifically, according to [1], if dead code is added and the control flow is changed sufficiently by inserting jump statements, the virus cannot be detected. In this project we first developed a code obfuscation engine conforming to the rules in [1]. We then used this engine to create metamorphic variants of a seed virus (created using the PS-MPK virus creation kit [15]) and demonstrated the validity of the assertion
---
paper_title: Unsupervised learning techniques for an intrusion detection system

paper_content:

With the continuous evolution of the types of attacks against computer networks, traditional intrusion detection systems, based on pattern matching and static signatures, are increasingly limited by their need of an up-to-date and comprehensive knowledge base. Data mining techniques have been successfully applied in host-based intrusion detection. Applying data mining techniques on raw network data, however, is made difficult by the sheer size of the input; this is usually avoided by discarding the network packet contents.In this paper, we introduce a two-tier architecture to overcome this problem: the first tier is an unsupervised clustering algorithm which reduces the network packets payload to a tractable size. The second tier is a traditional anomaly detection algorithm, whose efficiency is improved by the availability of data on the packet payload content.
---
paper_title: Toward Open Set Recognition

paper_content:

To date, almost all experimental evaluations of machine learning-based recognition algorithms in computer vision have taken the form of “closed set” recognition, whereby all testing classes are known at training time. A more realistic scenario for vision applications is “open set” recognition, where incomplete knowledge of the world is present at training time, and unknown classes can be submitted to an algorithm during testing. This paper explores the nature of open set recognition and formalizes its definition as a constrained minimization problem. The open set recognition problem is not well addressed by existing algorithms because it requires strong generalization. As a step toward a solution, we introduce a novel “1-vs-set machine,” which sculpts a decision space from the marginal distances of a 1-class or binary SVM with a linear kernel. This methodology applies to several different applications in computer vision where open set recognition is a challenging problem, including object recognition and face verification. We consider both in this work, with large scale cross-dataset experiments performed over the Caltech 256 and ImageNet sets, as well as face matching experiments performed over the Labeled Faces in the Wild set. The experiments highlight the effectiveness of machines adapted for open set evaluation compared to existing 1-class and binary SVMs for the same tasks.
---
paper_title: Classification and Novel Class Detection in Concept-Drifting Data Streams under Time Constraints

paper_content:

Most existing data stream classification techniques ignore one important aspect of stream data: arrival of a novel class. We address this issue and propose a data stream classification technique that integrates a novel class detection mechanism into traditional classifiers, enabling automatic detection of novel classes before the true labels of the novel class instances arrive. Novel class detection problem becomes more challenging in the presence of concept-drift, when the underlying data distributions evolve in streams. In order to determine whether an instance belongs to a novel class, the classification model sometimes needs to wait for more test instances to discover similarities among those instances. A maximum allowable wait time Tc is imposed as a time constraint to classify a test instance. Furthermore, most existing stream classification approaches assume that the true label of a data point can be accessed immediately after the data point is classified. In reality, a time delay Tl is involved in obtaining the true label of a data point since manual labeling is time consuming. We show how to make fast and correct classification decisions under these constraints and apply them to real benchmark data. Comparison with state-of-the-art stream classification techniques prove the superiority of our approach.
---
paper_title: Data Mining Approaches for Intrusion Detection

paper_content:

In this paper we discuss our research in developing general and systematic methods for intrusion detection. The key ideas are to use data mining techniques to discover consistent and useful patterns of system features that describe program and user behavior, and use the set of relevant system features to compute (inductively learned) classifiers that can recognize anomalies and known intrusions. Using experiments on the sendmail system call data and the network tcpdump data, we demonstrate that we can construct concise and accurate classifiers to detect anomalies. We provide an overview on two general data mining algorithms that we have implemented: the association rules algorithm and the frequent episodes algorithm. These algorithms can be used to compute the intra-and inter-audit record patterns, which are essential in describing program or user behavior. The discovered patterns can guide the audit data gathering process and facilitate feature selection. To meet the challenges of both efficient learning (mining) and real-time detection, we propose an agent-based architecture for intrusion detection systems where the learning agents continuously compute and provide the updated (detection) models to the detection agents.
---
paper_title: Decision tree classifier for network intrusion detection with GA-based feature selection

paper_content:

Machine Learning techniques such as Genetic Algorithms and Decision Trees have been applied to the field of intrusion detection for more than a decade. Machine Learning techniques can learn normal and anomalous patterns from training data and generate classifiers that then are used to detect attacks on computer systems. In general, the input data to classifiers is in a high dimension feature space, but not all of features are relevant to the classes to be classified. In this paper, we use a genetic algorithm to select a subset of input features for decision tree classifiers, with a goal of increasing the detection rate and decreasing the false alarm rate in network intrusion detection. We used the KDDCUP 99 data set to train and test the decision tree classifiers. The experiments show that the resulting decision trees can have better performance than those built with all available features.
---
paper_title: Data Streams: Models and Algorithms

paper_content:

This book primarily discusses issues related to the mining aspects of data streams and it is unique in its primary focus on the subject. This volume covers mining aspects of data streams comprehensively: each contributed chapter contains a survey on the topic, the key ideas in the field for that particular topic, and future research directions. The book is intended for a professional audience composed of researchers and practitioners in industry. This book is also appropriate for advanced-level students in computer science.
---
paper_title: Automatic network intrusion detection: Current techniques and open issues

paper_content:

Automatic network intrusion detection has been an important research topic for the last 20years. In that time, approaches based on signatures describing intrusive behavior have become the de-facto industry standard. Alternatively, other novel techniques have been used for improving automation of the intrusion detection process. In this regard, statistical methods, machine learning and data mining techniques have been proposed arguing higher automation capabilities than signature-based approaches. However, the majority of these novel techniques have never been deployed on real-life scenarios. The fact is that signature-based still is the most widely used strategy for automatic intrusion detection. In the present article we survey the most relevant works in the field of automatic network intrusion detection. In contrast to previous surveys, our analysis considers several features required for truly deploying each one of the reviewed approaches. This wider perspective can help us to identify the possible causes behind the lack of acceptance of novel techniques by network security experts.
---
paper_title: Feature engineering and classifier ensemble for KDD Cup 2010

paper_content:

KDD Cup 2010 is an educational data mining competition. Participants are asked to learn a model from students' past behavior and then predict their future performance. At National Taiwan University, we organized a course for this competition. Most student sub-teams expanded features by various binarization and discretization techniques. The resulting sparse feature sets were trained by logistic regression (using LIBLINEAR). One sub-team considered condensed features using simple statistical techniques and applied Random Forest (through Weka) for training. Initial development was conducted on an internal split of training data for training and validation. We identied some useful feature combinations to improve performance. For the nal submission, we combined results of student sub-teams by regularized linear regression. Our team is the rst prize winner of both tracks (all teams and student teams) of KDD Cup 2010.
---
paper_title: Improving Effectiveness of Intrusion Detection by Correlation Feature Selection

paper_content:

In this paper, the authors propose a new feature selection procedure for intrusion detection, which is based on filter method used in machine learning. They focus on Correlation Feature Selection CFS and transform the problem of feature selection by means of CFS measure into a mixed 0-1 linear programming problem with a number of constraints and variables that is linear in the number of full set features. The mixed 0-1 linear programming problem can then be solved by using branch-and-bound algorithm. This feature selection algorithm was compared experimentally with the best-first-CFS and the genetic-algorithm-CFS methods regarding the feature selection capabilities. Classification accuracies obtained after the feature selection by means of the C4.5 and the BayesNet over the KDD CUP'99 dataset were also tested. Experiments show that the authors' method outperforms the best-first-CFS and the genetic-algorithm-CFS methods by removing much more redundant features while keeping the classification accuracies or getting better performances.
---
paper_title: Adaptive Multiagent System for Network Traffic Monitoring

paper_content:

Individual anomaly-detection methods for monitoring computer network traffic have relatively high error rates. An agent-based trust-modeling system fuses anomaly data and progressively improves classification to achieve acceptable error rates.
---
paper_title: Intelligent agents for intrusion detection

paper_content:

The paper focuses on intrusion detection and countermeasures with respect to widely-used operating systems and networks. The design and architecture of an intrusion detection system built from distributed agents is proposed to implement an intelligent system on which data mining can be performed to provide global, temporal views of an entire networked system. A starting point for agent intelligence in the system is the research into the use of machine learning over system call traces from the privileged sendmail program on UNIX. The authors use a rule learning algorithm to classify the system call traces for intrusion detection purposes and show the results.
---
paper_title: Intrusion detection using neural networks and support vector machines

paper_content:

Information security is an issue of serious global concern. The complexity, accessibility, and openness of the Internet have served to increase the security risk of information systems tremendously. This paper concerns intrusion detection. We describe approaches to intrusion detection using neural networks and support vector machines. The key ideas are to discover useful patterns or features that describe user behavior on a system, and use the set of relevant features to build classifiers that can recognize anomalies and known intrusions, hopefully in real time. Using a set of benchmark data from a KDD (knowledge discovery and data mining) competition designed by DARPA, we demonstrate that efficient and accurate classifiers can be built to detect intrusions. We compare the performance of neural networks based, and support vector machine based, systems for intrusion detection.
---
paper_title: A comparative study of anomaly detection schemes in network intrusion detection

paper_content:

Intrusion detection corresponds to a suite of techniques that are used to identify attacks against computers and network infrastructures. Anomaly detection is a key element of intrusion detection in which perturbations of normal behavior suggest the presence of intentionally or unintentionally induced attacks, faults, defects, etc. This paper focuses on a detailed comparative study of several anomaly detection schemes for identifying different network intrusions. Several existing supervised and unsupervised anomaly detection schemes and their variations are evaluated on the DARPA 1998 data set of network connections [9] as well as on real network data using existing standard evaluation techniques as well as using several specific metrics that are appropriate when detecting attacks that involve a large number of connections. Our experimental results indicate that some anomaly detection schemes appear very promising when detecting novel intrusions in both DARPA’98 data and real network data.
---
paper_title: FEATURE SELECTION FOR INTRUSION DETECTION WITH NEURAL NETWORKS AND SUPPORT VECTOR MACHINES

paper_content:

Computational intelligence (CI) methods are increasingly being used for problem solving, and CI-type learning machines are being used for intrusion detection. Intrusion detection is a problem of general interest to transportation infrastructure protection, since one of its necessary tasks is to protect the computers responsible for the infrastructure's operational control, and an effective intrusion detection system (IDS) is essential for ensuring network security. Two classes of learning machines for IDSs are studied: artificial neural networks (ANNs) and support vector machines (SVMs). SVMs are shown to be superior to ANNs in three critical respects of IDSs: SVMs train and run an order of magnitude faster; they scale much better; and they give higher classification accuracy. A related issue is ranking the importance of input features, which is itself a problem of great interest. Since elimination of the insignificant (or useless) inputs leads to a simplified problem and possibly faster and more accurate...
---
paper_title: RT-UNNID: A practical solution to real-time network-based intrusion detection using unsupervised neural networks

paper_content:

With the growing rate of network attacks, intelligent methods for detecting new attacks have attracted increasing interest. The RT-UNNID system, introduced in this paper, is one such system, capable of intelligent real-time intrusion detection using unsupervised neural networks. Unsupervised neural nets can improve their analysis of new data over time without retraining. In previous work, we evaluated Adaptive Resonance Theory (ART) and Self-Organizing Map (SOM) neural networks using offline data. In this paper, we present a real-time solution using unsupervised neural nets to detect known and new attacks in network traffic. We evaluated our approach using 27 types of attack, and observed 97% precision using ART nets, and 95% precision using SOM nets.
---
paper_title: Explaining and Harnessing Adversarial Examples

paper_content:

Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Using this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.
---
paper_title: Network Intrusion Detection System Using Neural Networks

paper_content:

This paper presents a neural network-based intrusion detection method for the internet-based attacks on a computer network. Intrusion detection systems (IDS) have been created to predict and thwart current and future attacks. Neural networks are used to identify and predict unusual activities in the system. In particular, feedforward neural networks with the back propagation training algorithm were employed in this study. Training and testing data were obtained from the Defense Advanced Research Projects Agency (DARPA) intrusion detection evaluation data sets. The experimental results on real-data showed promising results on detection intrusion systems using neural networks.
---
paper_title: Identifying important features for intrusion detection using support vector machines and neural networks

paper_content:

Intrusion detection is a critical component of secure information systems. This paper addresses the issue of identifying important input features in building an intrusion detection system (IDS). Since elimination of the insignificant and/or useless inputs leads to a simplification of the problem, faster and more accurate detection may result. Feature ranking and selection, therefore, is an important issue in intrusion detection. We apply the technique of deleting one feature at a time to perform experiments on SVMs and neural networks to rank the importance of input features for the DARPA collected intrusion data. Important features for each of the 5 classes of intrusion patterns in the DARPA data are identified. It is shown that SVM-based and neural network based IDSs using a reduced number of features can deliver enhanced or comparable performance. An IDS for class-specific detection based on five SVMs is proposed.
---
paper_title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

paper_content:

Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study [30] revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call “fooling images” (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.
---
paper_title: A new approach to intrusion detection using Artificial Neural Networks and fuzzy clustering

paper_content:

Many researches have argued that Artificial Neural Networks (ANNs) can improve the performance of intrusion detection systems (IDS) when compared with traditional methods. However for ANN-based IDS, detection precision, especially for low-frequent attacks, and detection stability are still needed to be enhanced. In this paper, we propose a new approach, called FC-ANN, based on ANN and fuzzy clustering, to solve the problem and help IDS achieve higher detection rate, less false positive rate and stronger stability. The general procedure of FC-ANN is as follows: firstly fuzzy clustering technique is used to generate different training subsets. Subsequently, based on different training subsets, different ANN models are trained to formulate different base models. Finally, a meta-learner, fuzzy aggregation module, is employed to aggregate these results. Experimental results on the KDD CUP 1999 dataset show that our proposed new approach, FC-ANN, outperforms BPNN and other well-known methods such as decision tree, the naive Bayes in terms of detection precision and detection stability.
---
paper_title: A Self-organized Agent-based architecture for Power-aware Intrusion Detection in wireless ad-hoc networks

paper_content:

In this paper we propose SAPID-A Self-organized Agent-based architecture for Power-aware Intrusion Detection in wireless ad-hoc networks. We utilize an agent-based architecture that conserves the available bandwidth and segregates SAPID into two phases. Based on a power level metric and a hybrid metric that determine the duration and kinds of traffic that can be supported by a network-monitoring node, potential ad-hoc hosts are identified by repetitive training using an Adaptive Resonance Theory module. The agent architecture primarily consists of a Kohonen Self-Organizing Map to identify the appropriate patterns and recognize anomalies by way of unauthorized users in the network. A UNIX based session information file is utilized for testing and reporting possible intrusion attempts to the decision and action modules. Comprehensive experiments were carried out to clearly delineate and analyze the performance of the architecture.
---
paper_title: Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

paper_content:

In network intrusion detection research, one popular strategy for finding attacks is monitoring a network's activity for anomalies: deviations from profiles of normality previously learned from benign traffic, typically identified using tools borrowed from the machine learning community. However, despite extensive academic research one finds a striking gap in terms of actual deployments of such systems: compared with other intrusion detection approaches, machine learning is rarely employed in operational "real world" settings. We examine the differences between the network intrusion detection problem and other areas where machine learning regularly finds much more success. Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively. We support this claim by identifying challenges particular to network intrusion detection, and provide a set of guidelines meant to strengthen future research on anomaly detection.
---
paper_title: Bayesian Networks and Decision Graphs

paper_content:

Probabilistic graphical models and decision graphs are powerful modeling tools for reasoning and decision making under uncertainty. As modeling languages they allow a natural specification of problem domains with inherent uncertainty, and from a computational perspective they support efficient algorithms for automatic construction and query answering. This includes belief updating, finding the most probable explanation for the observed evidence, detecting conflicts in the evidence entered into the network, determining optimal strategies, analyzing for relevance, and performing sensitivity analysis. The book introduces probabilistic graphical models and decision graphs, including Bayesian networks and influence diagrams. The reader is introduced to the two types of frameworks through examples and exercises, which also instruct the reader on how to build these models. The book is a new edition of Bayesian Networks and Decision Graphs by Finn V. Jensen. The new edition is structured into two parts. The first part focuses on probabilistic graphical models. Compared with the previous book, the new edition also includes a thorough description of recent extensions to the Bayesian network modeling language, advances in exact and approximate belief updating algorithms, and methods for learning both the structure and the parameters of a Bayesian network. The second part deals with decision graphs, and in addition to the frameworks described in the previous edition, it also introduces Markov decision processes and partially ordered decision problems. The authors also provide a well-founded practical introduction to Bayesian networks, object-oriented Bayesian networks, decision trees, influence diagrams (and variants hereof), and Markov decision processes. give practical advice on the construction of Bayesian networks, decision trees, and influence diagrams from domain knowledge. give several examples and exercises exploiting computer systems for dealing with Bayesian networks and decision graphs. present a thorough introduction to state-of-the-art solution and analysis algorithms. The book is intended as a textbook, but it can also be used for self-study and as a reference book.
---
paper_title: Intrusion detection using neural networks and support vector machines

paper_content:

Information security is an issue of serious global concern. The complexity, accessibility, and openness of the Internet have served to increase the security risk of information systems tremendously. This paper concerns intrusion detection. We describe approaches to intrusion detection using neural networks and support vector machines. The key ideas are to discover useful patterns or features that describe user behavior on a system, and use the set of relevant features to build classifiers that can recognize anomalies and known intrusions, hopefully in real time. Using a set of benchmark data from a KDD (knowledge discovery and data mining) competition designed by DARPA, we demonstrate that efficient and accurate classifiers can be built to detect intrusions. We compare the performance of neural networks based, and support vector machine based, systems for intrusion detection.
---
paper_title: Letters: A hierarchical intrusion detection model based on the PCA neural networks

paper_content:

Most of existing intrusion detection (ID) models with a single-level structure can only detect either misuse or anomaly attacks. A hierarchical ID model using principal component analysis (PCA) neural networks is proposed to overcome such shortages. In the proposed model, PCA is applied for classification and neural networks are used for online computing. Experimental results and comparative studies based on the 1998 DARPA evaluation data sets are given, which show the proposed model can classify the network connections with satisfying performance.
---
paper_title: Extreme Value Distributions: Theory and Applications

paper_content:

Univariate extreme value distributions generalized extreme value distributions multivariate extreme value distributions.
---
paper_title: Improving one-class SVM for anomaly detection

paper_content:

With the tremendous growth of the Internet, information system security has become an issue of serious global concern due to the rapid connection and accessibility. Developing effective methods for intrusion detection, therefore, is an urgent task for assuring computer & information system security. Since most attacks and misuses can be recognized through the examination of system audit log files and pattern analysis therein, an approach for intrusion detection can be built on them. First we have made deep analysis on attacks and misuses patterns in log files; and then proposed an approach using support vector machines for anomaly detection. It is a one-class SVM based approach, trained with abstracted user audit logs data from 1999 DARPA.
---
paper_title: Anomaly Intrusion Detection Using Incremental Learning of an Infinite Mixture Model with Feature Selection

paper_content:

We propose an incremental nonparametric Bayesian approach for clustering. Our approach is based on a Dirichlet process mixture of generalized Dirichlet GD distributions. Unlike classic clustering approaches, our model does not require the number of clusters to be pre-defined. Moreover, an unsupervised feature selection scheme is integrated into the proposed nonparametric framework to improve clustering performance. By learning the proposed model using an incremental variational framework, the number of clusters as well as the features weights can be automatically and simultaneously computed. The effectiveness and merits of the proposed approach are investigated on a challenging application namely anomaly intrusion detection.
---
paper_title: Host-Based Intrusion Detection Using Dynamic and Static Behavioral Models

paper_content:

Intrusion detection has emerged as an important approach to network security. In this paper, we adopt an anomaly detection approach by detecting possible intrusions based on program or user profiles built from normal usage data. In particular, program profiles based on Unix system calls and user profiles based on Unix shell commands are modeled using two different types of behavioral models for data mining. The dynamic modeling approach is based on hidden Markov models (HMM) and the principle of maximum likelihood, while the static modeling approach is based on event occurrence frequency distributions and the principle of minimum cross entropy. The novelty detection approach is adopted to estimate the model parameters using normal training data only, as opposed to the classification approach which has to use both normal and intrusion data for training. To determine whether or not a certain behavior is similar enough to the normal model and hence should be classified as normal, we use a scheme that can be justified from the perspective of hypothesis testing. Our experimental results show that the dynamic modeling approach is better than the static modeling approach for the system call datasets, while the dynamic modeling approach is worse for the shell command datasets. Moreover, the static modeling approach is similar in performance to instance-based learning reported previously by others for the same shell command database but with much higher computational and storage requirements than our method.
---
paper_title: Anomaly intrusion detection using one class SVM

paper_content:

Kernel methods are widely used in statistical learning for many fields, such as protein classification and image processing. We recently extend kernel methods to intrusion detection domain by introducing a new family of kernels suitable for intrusion detection. These kernels, combined with an unsupervised learning method - one-class support vector machine, are used for anomaly detection. Our experiments show that the new anomaly detection methods are able to achieve better accuracy rates than the conventional anomaly detectors.
---
paper_title: Enhancing one-class support vector machines for unsupervised anomaly detection

paper_content:

Support Vector Machines (SVMs) have been one of the most successful machine learning techniques for the past decade. For anomaly detection, also a semi-supervised variant, the one-class SVM, exists. Here, only normal data is required for training before anomalies can be detected. In theory, the one-class SVM could also be used in an unsupervised anomaly detection setup, where no prior training is conducted. Unfortunately, it turns out that a one-class SVM is sensitive to outliers in the data. In this work, we apply two modifications in order to make one-class SVMs more suitable for unsupervised anomaly detection: Robust one-class SVMs and eta one-class SVMs. The key idea of both modifications is, that outliers should contribute less to the decision boundary as normal instances. Experiments performed on datasets from UCI machine learning repository show that our modifications are very promising: Comparing with other standard unsupervised anomaly detection algorithms, the enhanced one-class SVMs are superior on two out of four datasets. In particular, the proposed eta one-class SVM has shown the most promising results.
---
paper_title: Toward Open Set Recognition

paper_content:

To date, almost all experimental evaluations of machine learning-based recognition algorithms in computer vision have taken the form of “closed set” recognition, whereby all testing classes are known at training time. A more realistic scenario for vision applications is “open set” recognition, where incomplete knowledge of the world is present at training time, and unknown classes can be submitted to an algorithm during testing. This paper explores the nature of open set recognition and formalizes its definition as a constrained minimization problem. The open set recognition problem is not well addressed by existing algorithms because it requires strong generalization. As a step toward a solution, we introduce a novel “1-vs-set machine,” which sculpts a decision space from the marginal distances of a 1-class or binary SVM with a linear kernel. This methodology applies to several different applications in computer vision where open set recognition is a challenging problem, including object recognition and face verification. We consider both in this work, with large scale cross-dataset experiments performed over the Caltech 256 and ImageNet sets, as well as face matching experiments performed over the Labeled Faces in the Wild set. The experiments highlight the effectiveness of machines adapted for open set evaluation compared to existing 1-class and binary SVMs for the same tasks.
---
paper_title: Multi-class Open Set Recognition Using Probability of Inclusion

paper_content:

The perceived success of recent visual recognition approaches has largely been derived from their performance on classification tasks, where all possible classes are known at training time. But what about open set problems, where unknown classes appear at test time? Intuitively, if we could accurately model just the positive data for any known class without overfitting, we could reject the large set of unknown classes even under an assumption of incomplete class knowledge. In this paper, we formulate the problem as one of modeling positive training data at the decision boundary, where we can invoke the statistical extreme value theory. A new algorithm called the P I -SVM is introduced for estimating the unnormalized posterior probability of class inclusion.
---
paper_title: On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms

paper_content:

Outlier detection is a fundamental issue in data mining, specifically in fraud detection, network intrusion detection, network monitoring, etc. SmartSifter is an outlier detection engine addressing this problem from the viewpoint of statistical learning theory. This paper provides a theoretical basis for SmartSifter and empirically demonstrates its effectiveness. SmartSifter detects outliers in an on-line process through the on-line unsupervised learning of a probabilistic model (using a finite mixture model) of the information source. Each time a datum is input SmartSifter employs an on-line discounting learning algorithm to learn the probabilistic model. A score is given to the datum based on the learned model with a high score indicating a high possibility of being a statistical outlier. The novel features of SmartSifter are: (1) it is adaptive to non-stationary sources of datas (2) a score has a clear statistical/information-theoretic meanings (3) it is computationally inexpensives and (4) it can handle both categorical and continuous variables. An experimental application to network intrusion detection shows that SmartSifter was able to identify data with high scores that corresponded to attacks, with low computational costs. Further experimental application has identified a number of meaningful rare cases in actual health insurance pathology data from Australia's Health Insurance Commission.
---
paper_title: Traffic classification and verification using unsupervised learning of Gaussian Mixture Models

paper_content:

This paper presents the use of unsupervised Gaussian Mixture Models (GMMs) for the production of per-application models using their flows' statistics in order to be exploited in two different scenarios: (i) traffic classification, where the goal is to classify traffic flows by application (ii) traffic verification or traffic anomaly detection, where the aim is to confirm whether or not traffic flow generated by the claimed application conforms to its expected model. Unlike the first scenario, the second one is a new research path that has received less attention in the scope of Intrusion Detection System (IDS) research. The term “unsupervised” refers to the method ability to select the optimal number of components automatically without the need of careful initialization. Experiments are carried out using a public dataset collected from a real network. Favorable results indicate the effectiveness of unsupervised GMMs.
---
paper_title: Probability Models for Open Set Recognition

paper_content:

Real-world tasks in computer vision often touch upon open set recognition: multi-class recognition with incomplete knowledge of the world and many unknown inputs. Recent work on this problem has proposed a model incorporating an open space risk term to account for the space beyond the reasonable support of known classes. This paper extends the general idea of open space risk limiting classification to accommodate non-linear classifiers in a multi-class setting. We introduce a new open set recognition model called compact abating probability (CAP), where the probability of class membership decreases in value (abates) as points move from known data toward open space. We show that CAP models improve open set recognition for multiple algorithms. Leveraging the CAP formulation, we go on to describe the novel Weibull-calibrated SVM (W-SVM) algorithm, which combines the useful properties of statistical extreme value theory for score calibration with one-class and binary support vector machines. Our experiments show that the W-SVM is significantly better for open set object detection and OCR problems when compared to the state-of-the-art for the same tasks.
---
paper_title: Estimating the Support of a High-Dimensional Distribution

paper_content:

Suppose you are given some data set drawn from an underlying probability distribution P and you want to estimate a "simple" subset S of input space such that the probability that a test point drawn from P lies outside of S equals some a priori specified value between 0 and 1. We propose a method to approach this problem by trying to estimate a function f that is positive on S and negative on the complement. The functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data; it is regularized by controlling the length of the weight vector in an associated feature space. The expansion coefficients are found by solving a quadratic programming problem, which we do by carrying out sequential optimization over pairs of input patterns. We also provide a theoretical analysis of the statistical performance of our algorithm. The algorithm is a natural extension of the support vector algorithm to the case of unlabeled data.
---
paper_title: An Adaptive Weighted One-Class SVM for Robust Outlier Detection

paper_content:

This paper focuses on outlier detection from the perspective of classification. One-class support vector machine (OCSVM) is a widely applied and effective method of outlier detection. Unfortunately experiments show that the standard one-class SVM is easy to be influenced by the outliers contained in the training dataset. To cope with this problem, a robust OCSVM is presented in the paper. In consideration that the contribution yielded by the outlying instances and the normal data is different, a robust one-class SVM which assigns an adapting weight for every object in the training dataset was proposed in this paper. Experimental analysis shows the better performances of the proposed weighted method compared to the conventional one-class SVM on robustness.
---
paper_title: Towards Open World Recognition

paper_content:

With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimal downtime, even to learn. To handle these operational issues, we present the problem of Open World Recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance “open space risk” and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm that evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.
---
paper_title: A building block for awareness in technical systems: Online novelty detection and reaction with an application in intrusion detection

paper_content:

In this article we propose a new building block to realize awareness in technical systems, a two-stage algorithm for online novelty detection and reaction in a probabilistic framework. It uses a combination of parametric as well as nonparametric density modeling techniques. First, observed samples are categorized as potentially novel. Then, clusters of potentially novel samples are identified and finally probabilistic models of the observed environment are extended by adding new model components that describe the novel process. To demonstrate the applicability of the proposed algorithm in self-adapting technical systems, we investigate a case study in the field of intrusion detection, where new kinds of attacks have to be identified by an intrusion detection system. That is, the algorithm is used in this article to realize environment-awareness, but it could in principal be taken for self- or context-awareness mechanisms, too.
---
paper_title: Outlier Detection In Large-scale Traffic Data By Na\"ive Bayes Method and Gaussian Mixture Model Method

paper_content:

It is meaningful to detect outliers in traffic data for traffic management. However, this is a massive task for people from large-scale database to distinguish outliers. In this paper, we present two methods: Kernel Smoothing Na\"ive Bayes (NB) method and Gaussian Mixture Model (GMM) method to automatically detect any hardware errors as well as abnormal traffic events in traffic data collected at a four-arm junction in Hong Kong. Traffic data was recorded in a video format, and converted to spatial-temporal (ST) traffic signals by statistics. The ST signals are then projected to a two-dimensional (2D) (x,y)-coordinate plane by Principal Component Analysis (PCA) for dimension reduction. We assume that inlier data are normal distributed. As such, the NB and GMM methods are successfully applied in outlier detection (OD) for traffic data. The kernel smooth NB method assumes the existence of kernel distributions in traffic data and uses Bayes' Theorem to perform OD. In contrast, the GMM method believes the traffic data is formed by the mixture of Gaussian distributions and exploits confidence region for OD. This paper would address the modeling of each method and evaluate their respective performances. Experimental results show that the NB algorithm with Triangle kernel and GMM method achieve up to 93.78% and 94.50% accuracies, respectively.
---
paper_title: Novelty detection: a review - part 1: statistical approaches

paper_content:

Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.
---
paper_title: Online Adaboost-Based Parameterized Methods for Dynamic Distributed Network Intrusion Detection

paper_content:

Current network intrusion detection systems lack adaptability to the frequently changing network environments. Furthermore, intrusion detection in the new distributed architectures is now a major requirement. In this paper, we propose two online Adaboost-based intrusion detection algorithms. In the first algorithm, a traditional online Adaboost process is used where decision stumps are used as weak classifiers. In the second algorithm, an improved online Adaboost process is proposed, and online Gaussian mixture models (GMMs) are used as weak classifiers. We further propose a distributed intrusion detection framework, in which a local parameterized detection model is constructed in each node using the online Adaboost algorithm. A global detection model is constructed in each node by combining the local parametric models using a small number of samples in the node. This combination is achieved using an algorithm based on particle swarm optimization (PSO) and support vector machines. The global model in each node is used to detect intrusions. Experimental results show that the improved online Adaboost process with GMMs obtains a higher detection rate and a lower false alarm rate than the traditional online Adaboost process that uses decision stumps. Both the algorithms outperform existing intrusion detection algorithms. It is also shown that our PSO, and SVM-based algorithm effectively combines the local detection models into the global model in each node; the global model in a node can handle the intrusion types that are found in other nodes, without sharing the samples of these intrusion types.
---
paper_title: Robust Fusion: Extreme Value Theory for Recognition Score Normalization

paper_content:

Recognition problems in computer vision often benefit from a fusion of different algorithms and/or sensors, with score level fusion being among the most widely used fusion approaches. Choosing an appropriate score normalization technique before fusion is a fundamentally difficult problem because of the disparate nature of the underlying distributions of scores for different sources of data. Further complications are introduced when one or more fusion inputs outright fail or have adversarial inputs, which we find in the fields of biometrics and forgery detection. Ideally a score normalization should be robust to model assumptions, modeling errors, and parameter estimation errors, as well as robust to algorithm failure. In this paper, we introduce the w-score, a new technique for robust recognition score normalization. We do not assume a match or non-match distribution, but instead suggest that the top scores of a recognition system's non-match scores follow the statistical Extreme Value Theory, and show how to use that to provide consistent robust normalization with a strong statistical basis.
---
paper_title: Concept drift detection for online class imbalance learning

paper_content:

Concept drift detection methods are crucial components of many online learning approaches. Accurate drift detections allow prompt reaction to drifts and help to maintain high performance of online models over time. Although many methods have been proposed, no attention has been given to data streams with imbalanced class distributions, which commonly exist in real-world applications, such as fault diagnosis of control systems and intrusion detection in computer networks. This paper studies the concept drift problem for online class imbalance learning. We look into the impact of concept drift on single-class performance of online models based on three types of classifiers, under seven different scenarios with the presence of class imbalance. The analysis reveals that detecting drift in imbalanced data streams is a more difficult task than in balanced ones. Minority-class recall suffers from a significant drop after the drift involving the minority class. Overall accuracy is not suitable for drift detection. Based on the findings, we propose a new detection method DDM-OCI derived from the existing method DDM. DDM-OCI monitors minority-class recall online to capture the drift. The results show a quick response of the online model working with DDM-OCI to the new concept.
---
paper_title: Adaptive network intrusion detection system using a hybrid approach

paper_content:

Any activity aimed at disrupting a service or making a resource unavailable or gaining unauthorized access can be termed as an intrusion. Examples include buffer overflow attacks, flooding attacks, system break-ins, etc. Intrusion detection systems (IDSs) play a key role in detecting such malicious activities and enable administrators in securing network systems. Two key criteria should be met by an IDS for it to be effective: (i) ability to detect unknown attack types, (ii) having very less miss classification rate. In this paper we describe an adaptive network intrusion detection system, that uses a two stage architecture. In the first stage a probabilistic classifier is used to detect potential anomalies in the traffic. In the second stage a HMM based traffic model is used to narrow down the potential attack IP addresses. Various design choices that were made to make this system practical and difficulties faced in integrating with existing models are also described. We show that this system achieves good performance empirically.
---
paper_title: The Extreme Value Machine

paper_content:

It is often desirable to be able to recognize when inputs to a recognition function learned in a supervised manner correspond to classes unseen at training time. With this ability, new class labels could be assigned to these inputs by a human operator, allowing them to be incorporated into the recognition function—ideally under an efficient incremental update mechanism. While good algorithms that assume inputs from a fixed set of classes exist, e.g. , artificial neural networks and kernel machines, it is not immediately obvious how to extend them to perform incremental learning in the presence of unknown query classes. Existing algorithms take little to no distributional information into account when learning recognition functions and lack a strong theoretical foundation. We address this gap by formulating a novel, theoretically sound classifier—the Extreme Value Machine (EVM). The EVM has a well-grounded interpretation derived from statistical Extreme Value Theory (EVT), and is the first classifier to be able to perform nonlinear kernel-free variable bandwidth incremental learning. Compared to other classifiers in the same deep network derived feature space, the EVM is accurate and efficient on an established benchmark partition of the ImageNet dataset.
---
paper_title: Multi-class Open Set Recognition Using Probability of Inclusion

paper_content:

The perceived success of recent visual recognition approaches has largely been derived from their performance on classification tasks, where all possible classes are known at training time. But what about open set problems, where unknown classes appear at test time? Intuitively, if we could accurately model just the positive data for any known class without overfitting, we could reject the large set of unknown classes even under an assumption of incomplete class knowledge. In this paper, we formulate the problem as one of modeling positive training data at the decision boundary, where we can invoke the statistical extreme value theory. A new algorithm called the P I -SVM is introduced for estimating the unnormalized posterior probability of class inclusion.
---
paper_title: Probability Models for Open Set Recognition

paper_content:

Real-world tasks in computer vision often touch upon open set recognition: multi-class recognition with incomplete knowledge of the world and many unknown inputs. Recent work on this problem has proposed a model incorporating an open space risk term to account for the space beyond the reasonable support of known classes. This paper extends the general idea of open space risk limiting classification to accommodate non-linear classifiers in a multi-class setting. We introduce a new open set recognition model called compact abating probability (CAP), where the probability of class membership decreases in value (abates) as points move from known data toward open space. We show that CAP models improve open set recognition for multiple algorithms. Leveraging the CAP formulation, we go on to describe the novel Weibull-calibrated SVM (W-SVM) algorithm, which combines the useful properties of statistical extreme value theory for score calibration with one-class and binary support vector machines. Our experiments show that the W-SVM is significantly better for open set object detection and OCR problems when compared to the state-of-the-art for the same tasks.
---
paper_title: Applying CMAC-based online learning to intrusion detection

paper_content:

The timely and accurate detection of computer and network system intrusions has always been an elusive goal for system administrators and information security researchers. Existing intrusion detection approaches require either manual coding of new attacks in expert systems or the complete retraining of a neural network to improve analysis or learn new attacks. The paper presents an approach to applying adaptive neural networks to intrusion detection that is capable of autonomously learning new attacks rapidly by a modified reinforcement learning method that uses feedback from the protected system.
---
paper_title: Multi-attribute spaces: Calibration for attribute fusion and similarity search

paper_content:

Recent work has shown that visual attributes are a powerful approach for applications such as recognition, image description and retrieval. However, fusing multiple attribute scores — as required during multi-attribute queries or similarity searches — presents a significant challenge. Scores from different attribute classifiers cannot be combined in a simple way; the same score for different attributes can mean different things. In this work, we show how to construct normalized “multi-attribute spaces” from raw classifier outputs, using techniques based on the statistical Extreme Value Theory. Our method calibrates each raw score to a probability that the given attribute is present in the image. We describe how these probabilities can be fused in a simple way to perform more accurate multiattribute searches, as well as enable attribute-based similarity searches. A significant advantage of our approach is that the normalization is done after-the-fact, requiring neither modification to the attribute classification system nor ground truth attribute annotations. We demonstrate results on a large data set of nearly 2 million face images and show significant improvements over prior work. We also show that perceptual similarity of search results increases by using contextual attributes.
---
paper_title: Towards Open World Recognition

paper_content:

With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimal downtime, even to learn. To handle these operational issues, we present the problem of Open World Recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance “open space risk” and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm that evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.
---
paper_title: CLUSTERING-BASED NETWORK INTRUSION DETECTION

paper_content:

Recently data mining methods have gained importance in addressing network security issues, including network intrusion detection — a challenging task in network security. Intrusion detection systems aim to identify attacks with a high detection rate and a low false alarm rate. Classification-based data mining models for intrusion detection are often ineffective in dealing with dynamic changes in intrusion patterns and characteristics. Consequently, unsupervised learning methods have been given a closer look for network intrusion detection. We investigate multiple centroid-based unsupervised clustering algorithms for intrusion detection, and propose a simple yet effective self-labeling heuristic for detecting attack and normal clusters of network traffic audit data. The clustering algorithms investigated include, k-means, Mixture-Of-Spherical Gaussians, Self-Organizing Map, and Neural-Gas. The network traffic datasets provided by the DARPA 1998 offline intrusion detection project are used in our empirical investigation, which demonstrates the feasibility and promise of unsupervised learning methods for network intrusion detection. In addition, a comparative analysis shows the advantage of clustering-based methods over supervised classification techniques in identifying new or unseen attack types.
---
paper_title: Approaches to Online Learning and Concept Drift for User Identification in Computer Security

paper_content:

The task in the computer security domain of anomaly detection is to characterize the behaviors of a computer user (the 'valid' , or 'normal' user) so that unusual occurrences can be detected by comparison of the current input stream to the valid user's profile. This task requires an online learning system that can respond to concept drift and handle discrete non-metric time sequence data. We present an architecture for online learning in the anomaly detection domain and address the issues of incremental updating of system parameters and instance selection. We demonstrate a method for measuring direction and magnitude of concept drift in the classification space and present and evaluate approaches to the above stated issues which make use of the drift measurement.
---
paper_title: Anomalous Payload-Based Network Intrusion Detection

paper_content:

We present a payload-based anomaly detector, we call PAYL, for intrusion detection. PAYL models the normal application payload of network traffic in a fully automatic, unsupervised and very effecient fashion. We first compute during a training phase a profile byte frequency distribution and their standard deviation of the application payload flowing to a single host and port. We then use Mahalanobis distance during the detection phase to calculate the similarity of new data against the pre-computed profile. The detector compares this measure against a threshold and generates an alert when the distance of the new input exceeds this threshold. We demonstrate the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset we collected on the Columbia CS department network. In once case nearly 100% accuracy is achieved with 0.1% false positive rate for port 80 traffic.
---
paper_title: Generation of a new IDS test dataset: Time to retire the KDD collection

paper_content:

Intrusion detection systems are generally tested using datasets compiled at the end of last century, justified by the need for publicly available test data and the lack of any other alternative datasets. Prominent amongst this legacy group is the KDD project. Whilst a seminal contribution at the time of compilation, these datasets no longer represent relevant architecture or contemporary attack protocols, and are beset by data corruptions and inconsistencies. Hence, testing of new IDS approaches against these datasets does not provide an effective performance metric, and contributes to erroneous efficacy claims. This paper introduces a new publicly available dataset which is representative of modern attack structure and methodology. The new dataset is contrasted with the legacy datasets, and the performance difference of commonly used intrusion detection algorithms is highlighted.
---
paper_title: A Semantic Approach to Host-Based Intrusion Detection Systems Using Contiguousand Discontiguous System Call Patterns

paper_content:

Host-based anomaly intrusion detection system design is very challenging due to the notoriously high false alarm rate. This paper introduces a new host-based anomaly intrusion detection methodology using discontiguous system call patterns, in an attempt to increase detection rates whilst reducing false alarm rates. The key concept is to apply a semantic structure to kernel level system calls in order to reflect intrinsic activities hidden in high-level programming languages, which can help understand program anomaly behaviour. Excellent results were demonstrated using a variety of decision engines, evaluating the KDD98 and UNM data sets, and a new, modern data set. The ADFA Linux data set was created as part of this research using a modern operating system and contemporary hacking methods, and is now publicly available. Furthermore, the new semantic method possesses an inherent resilience to mimicry attacks, and demonstrated a high level of portability between different operating system versions.
---

Title: A Survey of Stealth Malware Attacks, Mitigation Measures, and Steps Toward Autonomous Open World Solutions

Section 1: A Survey of Existing Stealth Malware
Description 1: Discuss various types of stealth technology including rootkits, code mutation, anti-emulation, and targeting mechanisms, explaining each in detail.

Section 2: Component-Based Stealth Malware Countermeasures
Description 2: Explain anti-stealth malware techniques that protect system integrity, including hook detection, cross-view detection, invariant specification, and hardware and virtualization solutions.

Section 3: Pattern-Based Stealth Malware Countermeasures
Description 3: Describe generic recognition techniques for malware detection, focusing on static and dynamic analysis, signature analysis, and behavioral/heuristic analysis.

Section 4: Toward Adaptive Models for Stealth Malware Recognition
Description 4: Discuss the need for adaptive and interpretable recognition systems, detailing six flawed modeling assumptions and proposing an adaptive open world mathematical framework for improved stealth malware recognition.

Section 5: An Open Set Recognition Framework
Description 5: Explain the open set recognition problem, theorems proving open space risk management, and practical implementation to enhance existing intrusion recognition algorithms.

Section 6: Open World Archetypes for Stealth Malware Intrusion Recognition
Description 6: Discuss how open set recognition can be extended to open world recognition, supporting incremental model updates and prioritizing novel malware detection through malicious likelihood criteria.

Section 7: Conclusions and Open Issues
Description 7: Summarize findings, highlight the importance of combining countermeasures with machine learning solutions, and address open issues like bounding open space risk, misclassification costs, and the need for good datasets.