A Survey on Security Metrics
---
paper_title: Directions in Security Metrics Research

paper_content:

More than 100 years ago, Lord Kelvin insightfully observed that measurement is vital to deep knowledge and understanding in physical science. During the last few decades, researchers have made various attempts to develop measures and systems of measurement for computer security with varying degrees of success. This paper provides an overview of the security metrics area and looks at possible avenues of research that could be pursued to advance the state of the art.
---
paper_title: SoK: Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers

paper_content:

Run-time packers are often used by malware-writers to obfuscate their code and hinder static analysis. The packer problem has been widely studied, and several solutions have been proposed in order to generically unpack protected binaries. Nevertheless, these solutions commonly rely on a number of assumptions that may not necessarily reflect the reality of the packers used in the wild. Moreover, previous solutions fail to provide useful information about the structure of the packer or its complexity. In this paper, we describe a framework for packer analysis and we propose a taxonomy to measure the runtime complexity of packers. We evaluated our dynamic analysis system on two datasets, composed of both off-the-shelf packers and custom packed binaries. Based on the results of our experiments, we present several statistics about the packers complexity and their evolution over time.
---
paper_title: Model-based Security Metrics Using ADversary VIew Security Evaluation (ADVISE)

paper_content:

System architects need quantitative security metrics to make informed trade-off decisions involving system security. The security metrics need to provide insight on weak points in the system defense, considering characteristics of both the system and its adversaries. To provide such metrics, we formally define the ADversary View Security Evaluation (ADVISE) method. Our approach is to create an executable state-based security model of a system and an adversary that represents how the adversary is likely to attack the system and the results of such an attack. The attack decision function uses information about adversary attack preferences and possible attacks against the system to mimic how the adversary selects the most attractive next attack step. The adversary's decision involves looking ahead some number of attack steps. System architects can use ADVISE to compare the security strength of system architecture variants and analyze the threats posed by different adversaries. We demonstrate the feasibility and benefits of ADVISE using a case study. To produce quantitative model-based security metrics, we have implemented the ADVISE method in a tool that facilitates user input of system and adversary data and automatically generates executable models.
---
paper_title: Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices

paper_content:

The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this article, we survey and systematize common practices in the area of evaluation of such systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
---
paper_title: Binary-code obfuscations in prevalent packer tools

paper_content:

The first steps in analyzing defensive malware are understanding what obfuscations are present in real-world malware binaries, how these obfuscations hinder analysis, and how they can be overcome. While some obfuscations have been reported independently, this survey consolidates the discussion while adding substantial depth and breadth to it. This survey also quantifies the relative prevalence of these obfuscations by using the Dyninst binary analysis and instrumentation tool that was recently extended for defensive malware analysis. The goal of this survey is to encourage analysts to focus on resolving the obfuscations that are most prevalent in real-world malware.
---
paper_title: A taxonomy of computer program security flaws

paper_content:

An organized record of actual flaws can be useful to computer system designers, programmers, analysts, administrators, and users. This survey provides a taxonomy for computer program security flaws, with an Appendix that documents 50 actual security flaws. These flaws have all been described previously in the open literature, but in widely separated places. For those new to the field of computer security, they provide a good introduction to the characteristics of security flaws and how they can arise. Because these flaws were not randomly selected from a valid statistical sample of such flaws, we make no strong claims concerning the likely distribution of actual security flaws within the taxonomy. However, this method of organizing security flaw data can help those who have custody of more representative samples to organize them and to focus their efforts to remove and, eventually, to prevent the introduction of security flaws.
---
paper_title: Quantifying Information Flow for Dynamic Secrets

paper_content:

A metric is proposed for quantifying leakage of information about secrets and about how secrets change over time. The metric is used with a model of information flow for probabilistic, interactive systems with adaptive adversaries. The model and metric are implemented in a probabilistic programming language and used to analyze several examples. The analysis demonstrates that adaptivity increases information flow.
---
paper_title: Leakage Assessment Methodology - A Clear Roadmap for Side-Channel Evaluations.

paper_content:

Evoked by the increasing need to integrate side-channel countermeasures into security-enabled commercial devices, evaluation labs are seeking a standard approach that enables a fast, reliable and robust evaluation of the side-channel vulnerability of the given products. To this end, standardization bodies such as NIST intend to establish a leakage assessment methodology fulfilling these demands. One of such proposals is the Welch's t-test, which is being put forward by Cryptography Research Inc., and is able to relax the dependency between the evaluations and the device's underlying architecture. In this talk the theoretical background of the test's different flavors are reviewed, and a roadmap is presented that can be followed by the evaluation labs to efficiently and correctly conduct the tests. More precisely, a stable, robust and efficient way to perform the tests at higher orders is expressed. Further, the test is extended to multivariate settings, and details on how to efficiently and rapidly carry out such a multivariate higher-order test are provided.
---
paper_title: Model-based Security Metrics Using ADversary VIew Security Evaluation (ADVISE)

paper_content:

System architects need quantitative security metrics to make informed trade-off decisions involving system security. The security metrics need to provide insight on weak points in the system defense, considering characteristics of both the system and its adversaries. To provide such metrics, we formally define the ADversary View Security Evaluation (ADVISE) method. Our approach is to create an executable state-based security model of a system and an adversary that represents how the adversary is likely to attack the system and the results of such an attack. The attack decision function uses information about adversary attack preferences and possible attacks against the system to mimic how the adversary selects the most attractive next attack step. The adversary's decision involves looking ahead some number of attack steps. System architects can use ADVISE to compare the security strength of system architecture variants and analyze the threats posed by different adversaries. We demonstrate the feasibility and benefits of ADVISE using a case study. To produce quantitative model-based security metrics, we have implemented the ADVISE method in a tool that facilitates user input of system and adversary data and automatically generates executable models.
---
paper_title: A clinical study of risk factors related to malware infections

paper_content:

The success of malicious software (malware) depends upon both technical and human factors. The most security conscious users are vulnerable to zero-day exploits; the best security mechanisms can be circumvented by poor user choices. While there has been significant research addressing the technical aspects of malware attack and defense, there has been much less research reporting on how human behavior interacts with both malware and current malware defenses. In this paper we describe a proof-of-concept field study designed to examine the interactions between users, anti-virus (anti-malware) software, and malware as they occur on deployed systems. The 4-month study, conducted in a fashion similar to the clinical trials used to evaluate medical interventions, involved 50 subjects whose laptops were instrumented to monitor possible infections and gather data on user behavior. Although the population size was limited, this initial study produced some intriguing, non-intuitive insights into the efficacy of current defenses, particularly with regards to the technical sophistication of end users. We assert that this work shows the feasibility and utility of testing security software through long-term field studies with greater ecological validity than can be achieved through other means.
---
paper_title: Who falls for phish?: a demographic analysis of phishing susceptibility and effectiveness of interventions

paper_content:

In this paper we present the results of a roleplay survey instrument administered to 1001 online survey respondents to study both the relationship between demographics and phishing susceptibility and the effectiveness of several anti-phishing educational materials. Our results suggest that women are more susceptible than men to phishing and participants between the ages of 18 and 25 are more susceptible to phishing than other age groups. We explain these demographic factors through a mediation analysis. Educational materials reduced users' tendency to enter information into phishing webpages by 40% percent; however, some of the educational materials we tested also slightly decreased participants' tendency to click on legitimate links.
---
paper_title: The Psychology of Security for the Home Computer User

paper_content:

The home computer user is often said to be the weakest link in computer security. They do not always follow security advice, and they take actions, as in phishing, that compromise themselves. In general, we do not understand why users do not always behave safely, which would seem to be in their best interest. This paper reviews the literature of surveys and studies of factors that influence security decisions for home computer users. We organize the review in four sections: understanding of threats, perceptions of risky behavior, efforts to avoid security breaches and attitudes to security interventions. We find that these studies reveal a lot of reasons why current security measures may not match the needs or abilities of home computer users and suggest future work needed to inform how security is delivered to this user group.
---
paper_title: Statistical metrics for individual password strength

paper_content:

We propose several possible metrics for measuring the strength of an individual password or any other secret drawn from a known, skewed distribution. In contrast to previous ad hoc approaches which rely on textual properties of passwords, we consider the problem without any knowledge of password structure. This enables rating the strength of a password given a large sample distribution without assuming anything about password semantics. We compare the results of our generic metrics against those of the NIST metrics and other previous "entropy-based" metrics for a large password dataset, which suggest over-fitting in previous metrics.
---
paper_title: Testing metrics for password creation policies by attacking large sets of revealed passwords

paper_content:

In this paper we attempt to determine the effectiveness of using entropy, as defined in NIST SP800-63, as a measurement of the security provided by various password creation policies. This is accomplished by modeling the success rate of current password cracking techniques against real user passwords. These data sets were collected from several different websites, the largest one containing over 32 million passwords. This focus on actual attack methodologies and real user passwords quite possibly makes this one of the largest studies on password security to date. In addition we examine what these results mean for standard password creation policies, such as minimum password length, and character set requirements.
---
paper_title: The Science of Guessing: Analyzing an Anonymized Corpus of 70 Million Passwords

paper_content:

We report on the largest corpus of user-chosen passwords ever studied, consisting of anonymized password histograms representing almost 70 million Yahoo! users, mitigating privacy concerns while enabling analysis of dozens of subpopulations based on demographic factors and site usage characteristics. This large data set motivates a thorough statistical treatment of estimating guessing difficulty by sampling from a secret distribution. In place of previously used metrics such as Shannon entropy and guessing entropy, which cannot be estimated with any realistically sized sample, we develop partial guessing metrics including a new variant of guesswork parameterized by an attacker's desired success rate. Our new metric is comparatively easy to approximate and directly relevant for security engineering. By comparing password distributions with a uniform distribution which would provide equivalent security against different forms of guessing attack, we estimate that passwords provide fewer than 10 bits of security against an online, trawling attack, and only about 20 bits of security against an optimal offline dictionary attack. We find surprisingly little variation in guessing difficulty; every identifiable group of users generated a comparably weak password distribution. Security motivations such as the registration of a payment card have no greater impact than demographic factors such as age and nationality. Even proactive efforts to nudge users towards better password choices with graphical feedback make little difference. More surprisingly, even seemingly distant language communities choose the same weak passwords and an attacker never gains more than a factor of 2 efficiency gain by switching from the globally optimal dictionary to a population-specific lists.
---
paper_title: Guess Again (and Again and Again): Measuring Password Strength by Simulating Password-Cracking Algorithms

paper_content:

Text-based passwords remain the dominant authentication method in computer systems, despite significant advancement in attackers' capabilities to perform password cracking. In response to this threat, password composition policies have grown increasingly complex. However, there is insufficient research defining metrics to characterize password strength and using them to evaluate password-composition policies. In this paper, we analyze 12,000 passwords collected under seven composition policies via an online study. We develop an efficient distributed method for calculating how effectively several heuristic password-guessing algorithms guess passwords. Leveraging this method, we investigate (a) the resistance of passwords created under different conditions to guessing, (b) the performance of guessing algorithms under different training sets, (c) the relationship between passwords explicitly created under a given composition policy and other passwords that happen to meet the same requirements, and (d) the relationship between guess ability, as measured with password-cracking algorithms, and entropy estimates. Our findings advance understanding of both password-composition policies and metrics for quantifying password security.
---
paper_title: Measuring real-world accuracies and biases in modeling password guessability

paper_content:

Parameterized password guessability--how many guesses a particular cracking algorithm with particular training data would take to guess a password--has become a common metric of password security. Unlike statistical metrics, it aims to model real-world attackers and to provide per-password strength estimates. We investigate how cracking approaches often used by researchers compare to real-world cracking by professionals, as well as how the choice of approach biases research conclusions. ::: ::: We find that semi-automated cracking by professionals outperforms popular fully automated approaches, but can be approximated by combining multiple such approaches. These approaches are only effective, however, with careful configuration and tuning; in commonly used default configurations, they underestimate the real-world guessability of passwords. We find that analyses of large password sets are often robust to the algorithm used for guessing as long as it is configured effectively. However, cracking algorithms differ systematically in their effectiveness guessing passwords with certain common features (e.g., character substitutions). This has important implications for analyzing the security of specific password characteristics or of individual passwords (e.g., in a password meter or security audit). Our results highlight the danger of relying only on a single cracking algorithm as a measure of password strength and constitute the first scientific evidence that automated guessing can often approximate guessing by professionals.
---
paper_title: An Attack Surface Metric

paper_content:

Measurement of software security is a long-standing challenge to the research community. At the same time, practical security metrics and measurements are essential for secure software development. Hence, the need for metrics is more pressing now due to a growing demand for secure software. In this paper, we propose using a software system's attack surface measurement as an indicator of the system's security. We formalize the notion of a system's attack surface and introduce an attack surface metric to measure the attack surface in a systematic manner. Our measurement method is agnostic to a software system's implementation language and is applicable to systems of all sizes; we demonstrate our method by measuring the attack surfaces of small desktop applications and large enterprise systems implemented in C and Java. We conducted three exploratory empirical studies to validate our method. Software developers can mitigate their software's security risk by measuring and reducing their software's attack surfaces. Our attack surface reduction approach complements the software industry's traditional code quality improvement approach for security risk mitigation and is useful in multiple phases of the software development lifecycle. Our collaboration with SAP demonstrates the use of our metric in the software development process.
---
paper_title: After we knew it: empirical study and modeling of cost-effectiveness of exploiting prevalent known vulnerabilities across IaaS cloud

paper_content:

Infrastructure as a Service (IaaS) cloud has been attracting more and more customers as it provides the highest level of flexibility by offering configurable virtual machines (VMs) and computing infrastructures. Public VM images are usually available for customers to customize and launch. However, the 1 to N mapping between VM images and running instances in IaaS makes vulnerabilities propagate rapidly across the entire public cloud. Besides, IaaS cloud naturally comes with a larger and more stable attack surface and more concentrated target resources than traditional surroundings. In this paper, we first identify the threat of exploiting prevalent vulnerabilities over public IaaS cloud with an empirical study in Amazon EC2. We find that attackers can compromise a considerable number of VMs with trivial cost. We then do a qualitative cost-effectiveness analysis of this threat. Our main result is a two-fold observation: in IaaS cloud, exploiting prevalent vulnerabilities is much more cost-effective than traditional in-house computing environment, therefore attackers have stronger incentive; Fortunately, on the other hand, cloud defenders (cloud providers and customers) also have much lower cost-loss ratio than in traditional environment, therefore they can be more effective for defending attacks. We then build a game-theoretic model and conduct a risk-gain analysis to compare exploiting and patching strategies under cloud and traditional computing environments. Our modeling indicates that under cloud environment, both attack and defense become less cost-effective as time goes by, and the earlier actioner can be more rewarding. We propose countermeasures against such threat in order to bridge the gap between current security situation and defending mechanisms. To our best knowledge, we are the first to analyze and model the threat with prevalent known-vulnerabilities in public cloud.
---
paper_title: Vulnerability disclosure in the age of social media: exploiting twitter for predicting real-world exploits

paper_content:

In recent years, the number of software vulnerabilities discovered has grown significantly. This creates a need for prioritizing the response to new disclosures by assessing which vulnerabilities are likely to be exploited and by quickly ruling out the vulnerabilities that are not actually exploited in the real world. We conduct a quantitative and qualitative exploration of the vulnerability-related information disseminated on Twitter. We then describe the design of a Twitter-based exploit detector, and we introduce a threat model specific to our problem. In addition to response prioritization, our detection techniques have applications in risk modeling for cyber-insurance and they highlight the value of information provided by the victims of attacks.
---
paper_title: When governments hack opponents: a look at actors and technology

paper_content:

Repressive nation-states have long monitored telecommunications to keep tabs on political dissent. The Internet and online social networks, however, pose novel technical challenges to this practice, even as they open up new domains for surveillance. We analyze an extensive collection of suspicious files and links targeting activists, opposition members, and nongovernmental organizations in the Middle East over the past several years. We find that these artifacts reflect efforts to attack targets' devices for the purposes of eavesdropping, stealing information, and/or unmasking anonymous users. We describe attack campaigns we have observed in Bahrain, Syria, and the United Arab Emirates, investigating attackers, tools, and techniques. In addition to off-the-shelf remote access trojans and the use of third-party IP-tracking services, we identify commercial spyware marketed exclusively to governments, including Gamma's FinSpy and Hacking Team's Remote Control System (RCS). We describe their use in Bahrain and the UAE, and map out the potential broader scope of this activity by conducting global scans of the corresponding command-and-control (C&C) servers. Finally, we frame the real-world consequences of these campaigns via strong circumstantial evidence linking hacking to arrests, interrogations, and imprisonment.
---
paper_title: The Matter of Heartbleed

paper_content:

The Heartbleed vulnerability took the Internet by surprise in April 2014. The vulnerability, one of the most consequential since the advent of the commercial Internet, allowed attackers to remotely read protected memory from an estimated 24--55% of popular HTTPS sites. In this work, we perform a comprehensive, measurement-based analysis of the vulnerability's impact, including (1) tracking the vulnerable population, (2) monitoring patching behavior over time, (3) assessing the impact on the HTTPS certificate ecosystem, and (4) exposing real attacks that attempted to exploit the bug. Furthermore, we conduct a large-scale vulnerability notification experiment involving 150,000 hosts and observe a nearly 50% increase in patching by notified hosts. Drawing upon these analyses, we discuss what went well and what went poorly, in an effort to understand how the technical community can respond more effectively to such events in the future.
---
paper_title: After we knew it: empirical study and modeling of cost-effectiveness of exploiting prevalent known vulnerabilities across IaaS cloud

paper_content:

Infrastructure as a Service (IaaS) cloud has been attracting more and more customers as it provides the highest level of flexibility by offering configurable virtual machines (VMs) and computing infrastructures. Public VM images are usually available for customers to customize and launch. However, the 1 to N mapping between VM images and running instances in IaaS makes vulnerabilities propagate rapidly across the entire public cloud. Besides, IaaS cloud naturally comes with a larger and more stable attack surface and more concentrated target resources than traditional surroundings. In this paper, we first identify the threat of exploiting prevalent vulnerabilities over public IaaS cloud with an empirical study in Amazon EC2. We find that attackers can compromise a considerable number of VMs with trivial cost. We then do a qualitative cost-effectiveness analysis of this threat. Our main result is a two-fold observation: in IaaS cloud, exploiting prevalent vulnerabilities is much more cost-effective than traditional in-house computing environment, therefore attackers have stronger incentive; Fortunately, on the other hand, cloud defenders (cloud providers and customers) also have much lower cost-loss ratio than in traditional environment, therefore they can be more effective for defending attacks. We then build a game-theoretic model and conduct a risk-gain analysis to compare exploiting and patching strategies under cloud and traditional computing environments. Our modeling indicates that under cloud environment, both attack and defense become less cost-effective as time goes by, and the earlier actioner can be more rewarding. We propose countermeasures against such threat in order to bridge the gap between current security situation and defending mechanisms. To our best knowledge, we are the first to analyze and model the threat with prevalent known-vulnerabilities in public cloud.
---
paper_title: The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching

paper_content:

Vulnerability exploits remain an important mechanism for malware delivery, despite efforts to speed up the creation of patches and improvements in software updating mechanisms. Vulnerabilities in client applications (e.g., Browsers, multimedia players, document readers and editors) are often exploited in spear phishing attacks and are difficult to characterize using network vulnerability scanners. Analyzing their lifecycle requires observing the deployment of patches on hosts around the world. Using data collected over 5 years on 8.4 million hosts, available through Symantec's WINE platform, we present the first systematic study of patch deployment in client-side vulnerabilities. We analyze the patch deployment process of 1,593 vulnerabilities from 10 popular client applications, and we identify several new threats presented by multiple installations of the same program and by shared libraries distributed with several applications. For the 80 vulnerabilities in our dataset that affect code shared by two applications, the time between patch releases in the different applications is up to 118 days (with a median of 11 days). Furthermore, as the patching rates differ considerably among applications, many hosts patch the vulnerability in one application but not in the other one. We demonstrate two novel attacks that enable exploitation by invoking old versions of applications that are used infrequently, but remain installed. We also find that the median fraction of vulnerable hosts patched when exploits are released is at most 14%. Finally, we show that the patching rate is affected by user-specific and application-specific factors, for example, hosts belonging to security analysts and applications with an automated updating mechanism have significantly lower median times to patch.
---
paper_title: Targeted threat index: characterizing and quantifying politically-motivated targeted malware

paper_content:

Targeted attacks on civil society and nongovernmental organizations have gone underreported despite the fact that these organizations have been shown to be frequent targets of these attacks. In this paper, we shed light on targeted malware attacks faced by these organizations by studying malicious e-mails received by 10 civil society organizations (the majority of which are from groups related to China and Tibet issues) over a period of 4 years. ::: ::: Our study highlights important properties of malware threats faced by these organizations with implications on how these organizations defend themselves and how we quantify these threats. We find that the technical sophistication of malware we observe is fairly low, with more effort placed on socially engineering the e-mail content. Based on this observation, we develop the Targeted Threat Index (TTI), a metric which incorporates both social engineering and technical sophistication when assessing the risk of malware threats. We demonstrate that this metric is more effective than simple technical sophistication for identifying malware threats with the highest potential to successfully compromise victims. We also discuss how education efforts focused on changing user behaviour can help prevent compromise. For two of the three Tibetan groups in our study simple steps such as avoiding the use of email attachments could cut document-based malware threats delivered through e-mail that we observed by up to 95%.
---
paper_title: Comparing Vulnerability Severity and Exploits Using Case-Control Studies

paper_content:

(U.S.) Rule-based policies for mitigating software risk suggest using the CVSS score to measure the risk of an individual vulnerability and act accordingly. A key issue is whether the ‘danger’ score does actually match the risk of exploitation in the wild, and if and how such a score could be improved. To address this question, we propose using a case-control study methodology similar to the procedure used to link lung cancer and smoking in the 1950s. A case-control study allows the researcher to draw conclusions on the relation between some risk factor (e.g., smoking) and an effect (e.g., cancer) by looking backward at the cases (e.g., patients) and comparing them with controls (e.g., randomly selected patients with similar characteristics). The methodology allows us to quantify the risk reduction achievable by acting on the risk factor. We illustrate the methodology by using publicly available data on vulnerabilities, exploits, and exploits in the wild to (1) evaluate the performances of the current risk factor in the industry, the CVSS base score; (2) determine whether it can be improved by considering additional factors such the existence of a proof-of-concept exploit, or of an exploit in the black markets. Our analysis reveals that (a) fixing a vulnerability just because it was assigned a high CVSS score is equivalent to randomly picking vulnerabilities to fix; (b) the existence of proof-of-concept exploits is a significantly better risk factor; (c) fixing in response to exploit presence in black markets yields the largest risk reduction.
---
paper_title: Automatic Patch-Based Exploit Generation is Possible: Techniques and Implications

paper_content:

The automatic patch-based exploit generation problem is: given a program P and a patched version of the program P', automatically generate an exploit for the potentially unknown vulnerability present in P but fixed in P'. In this paper, we propose techniques for automatic patch-based exploit generation, and show that our techniques can automatically generate exploits for 5 Microsoft programs based upon patches provided via Windows Update. Although our techniques may not work in all cases, a fundamental tenant of security is to conservatively estimate the capabilities of attackers. Thus, our results indicate that automatic patch-based exploit generation should be considered practical. One important security implication of our results is that current patch distribution schemes which stagger patch distribution over long time periods, such as Windows Update, may allow attackers who receive the patch first to compromise the significant fraction of vulnerable hosts who have not yet received the patch.
---
paper_title: Directions in Security Metrics Research

paper_content:

More than 100 years ago, Lord Kelvin insightfully observed that measurement is vital to deep knowledge and understanding in physical science. During the last few decades, researchers have made various attempts to develop measures and systems of measurement for computer security with varying degrees of success. This paper provides an overview of the security metrics area and looks at possible avenues of research that could be pursued to advance the state of the art.
---
paper_title: Security impact ratings considered harmful

paper_content:

In this paper, we question the common practice of assigning security impact ratings to OS updates. Specifically, we present evidence that ranking updates by their perceived security importance, in order to defer applying some updates, exposes systems to significant risk. ::: ::: We argue that OS vendors and security groups should not focus on security updates to the detriment of other updates, but should instead seek update technologies that make it feasible to distribute updates for all disclosed OS bugs in a timely manner.
---
paper_title: J.A.: Mining your Ps and Qs: Detection of Widespread Weak Keys in Network Devices

paper_content:

RSA and DSA can fail catastrophically when used with malfunctioning random number generators, but the extent to which these problems arise in practice has never been comprehensively studied at Internet scale. We perform the largest ever network survey of TLS and SSH servers and present evidence that vulnerable keys are surprisingly widespread. We find that 0.75% of TLS certificates share keys due to insufficient entropy during key generation, and we suspect that another 1.70% come from the same faulty implementations and may be susceptible to compromise. Even more alarmingly, we are able to obtain RSA private keys for 0.50% of TLS hosts and 0.03% of SSH hosts, because their public keys shared nontrivial common factors due to entropy problems, and DSA private keys for 1.03% of SSH hosts, because of insufficient signature randomness. We cluster and investigate the vulnerable hosts, finding that the vast majority appear to be headless or embedded devices. In experiments with three software components commonly used by these devices, we are able to reproduce the vulnerabilities and identify specific software behaviors that induce them, including a boot-time entropy hole in the Linux random number generator. Finally, we suggest defenses and draw lessons for developers, users, and the security community.
---
paper_title: Evaluation of intrusion detectors: a decision theory approach

paper_content:

We present a method of analysis for evaluating intrusion detection systems. The method can be used to compare the performance of intrusion detectors, to evaluate performance goals for intrusion detectors, and to determine the best configuration of an intrusion detector for a given environment. The method uses a decision analysis that integrates and extends ROC (receiver operating characteristics) and cost analysis methods to provide an expected cost metric. We provide general results and illustrate the method in several numerical examples that cover a range of detectors that meet a performance goal and two actual detectors operating in a realistic environment. We demonstrate that, contrary to common advice, the value of an intrusion detection system and the optimal operation of that system depend not only on the system's ROC curve, but also on cost metrics and the hostility of the operating environment as summarized by the probability of intrusion. Extensions of the method are outlined, and conclusions are drawn.
---
paper_title: Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices

paper_content:

The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this article, we survey and systematize common practices in the area of evaluation of such systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
---
paper_title: ALDR: A New Metric for Measuring Effective Layering of Defenses ∗

paper_content:

Attackers continually innovate and craft attacks that penetrate existing defenses. New security product purchasing decisions are key in order to keep organizations as secure as possible. Current information available to inform these decisions is often limited to individual security product detection/blocking rates for some test set of attacks. Actual security performance, however, depends on how a security product performs in the context of an organization’s existing security products. Even a security product that tests well on its own may be completely redundant when deployed into an existing environment. We propose a new metric that measures the total security granted by a combination of security products. Also, this metric makes the computation of the added benefit of an additional security product easy. We take the results of each individual security product parsing a certain data set and then, take the union of the results of all security products deployed at that organization. Our metric is the attacks in this union divided by the total attacks in the data set or, in other words, the total detection rate achieved by the whole system. This metric can be computed using existing evaluation techniques and provides a more accurate overall picture of the security posture of an organization as well as a way to measure the real contribution of a specific security product in the context of other security layers. ∗This material is based on research partially sponsored by the National Science Foundation (NSF) under CCF grant 0950373. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF
---
paper_title: AV-Meter: An Evaluation of Antivirus Scans and Labels

paper_content:

Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection.
---
paper_title: ALDR: A New Metric for Measuring Effective Layering of Defenses ∗

paper_content:

Attackers continually innovate and craft attacks that penetrate existing defenses. New security product purchasing decisions are key in order to keep organizations as secure as possible. Current information available to inform these decisions is often limited to individual security product detection/blocking rates for some test set of attacks. Actual security performance, however, depends on how a security product performs in the context of an organization’s existing security products. Even a security product that tests well on its own may be completely redundant when deployed into an existing environment. We propose a new metric that measures the total security granted by a combination of security products. Also, this metric makes the computation of the added benefit of an additional security product easy. We take the results of each individual security product parsing a certain data set and then, take the union of the results of all security products deployed at that organization. Our metric is the attacks in this union divided by the total attacks in the data set or, in other words, the total detection rate achieved by the whole system. This metric can be computed using existing evaluation techniques and provides a more accurate overall picture of the security posture of an organization as well as a way to measure the real contribution of a specific security product in the context of other security layers. ∗This material is based on research partially sponsored by the National Science Foundation (NSF) under CCF grant 0950373. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF
---
paper_title: Heisenbyte: Thwarting Memory Disclosure Attacks using Destructive Code Reads

paper_content:

Vulnerabilities that disclose executable memory pages enable a new class of powerful code reuse attacks that build the attack payload at runtime. In this work, we present Heisenbyte, a system to protect against memory disclosure attacks. Central to Heisenbyte is the concept of destructive code reads -- code is garbled right after it is read. Garbling the code after reading it takes away from the attacker her ability to leverage memory disclosure bugs in both static code and dynamically generated just-in-time code. By leveraging existing virtualization support, Heisenbyte's novel use of destructive code reads sidesteps the problem of incomplete binary disassembly in binaries, and extends protection to close-sourced COTS binaries, which are two major limitations of prior solutions against memory disclosure vulnerabilities. Our experiments demonstrate that Heisenbyte can tolerate some degree of imperfect static analysis in disassembled binaries, while effectively thwarting dynamic code reuse exploits in both static and JIT code, at a modest 1.8% average runtime overhead due to virtualization and 16.5% average overhead due to the destructive code reads.
---
paper_title: Effective entropy: security-centric metric for memory randomization techniques

paper_content:

User space memory randomization techniques are an emerging field of cyber defensive technology which attempts to protect computing systems by randomizing the layout of memory. Quantitative metrics are needed to evaluate their effectiveness at securing systems against modern adversaries and to compare between randomization technologies. We introduce Effective Entropy, a measure of entropy in user space memory which quantitatively considers an adversary's ability to leverage low entropy regions of memory via absolute and dynamic intersection connections. Effective Entropy is indicative of adversary workload and enables comparison between different randomization techniques. Using Effective Entropy, we present a comparison of static Address Space Layout Randomization (ASLR), Position Independent Executable (PIE) ASLR, and a theoretical fine grain randomization technique.
---
paper_title: Oxymoron: making fine-grained memory randomization practical by allowing code sharing

paper_content:

The latest effective defense against code reuse attacks is fine-grained, per-process memory randomization. However, such process randomization prevents code sharing since there is no longer any identical code to share between processes. Without shared libraries, however, tremendous memory savings are forfeit. This drawback may hinder the adoption of fine-grained memory randomization. ::: ::: We present Oxymoron, a secure fine-grained memory randomization technique on a per-process level that does not interfere with code sharing. Executables and libraries built with Oxymoron feature 'memory-layout-agnostic code', which runs on a commodity Linux. Our theoretical and practical evaluations show that Oxymoron is the first solution to be secure against just-in-time code reuse attacks and demonstrate that fine-grained memory randomization is feasible without forfeiting the enormous memory savings of shared libraries.
---
paper_title: Just-In-Time Code Reuse: On the Effectiveness of Fine-Grained Address Space Layout Randomization

paper_content:

Fine-grained address space layout randomization (ASLR) has recently been proposed as a method of efficiently mitigating runtime attacks. In this paper, we introduce the design and implementation of a framework based on a novel attack strategy, dubbed just-in-time code reuse, that undermines the benefits of fine-grained ASLR. Specifically, we derail the assumptions embodied in fine-grained ASLR by exploiting the ability to repeatedly abuse a memory disclosure to map an application's memory layout on-the-fly, dynamically discover API functions and gadgets, and JIT-compile a target program using those gadgets -- all within a script environment at the time an exploit is launched. We demonstrate the power of our framework by using it in conjunction with a real-world exploit against Internet Explorer, and also provide extensive evaluations that demonstrate the practicality of just-in-time code reuse attacks. Our findings suggest that fine-grained ASLR may not be as promising as first thought.
---
paper_title: On the effectiveness of address-space randomization

paper_content:

Address-space randomization is a technique used to fortify systems against buffer overflow attacks. The idea is to introduce artificial diversity by randomizing the memory location of certain system components. This mechanism is available for both Linux (via PaX ASLR) and OpenBSD. We study the effectiveness of address-space randomization and find that its utility on 32-bit architectures is limited by the number of bits available for address randomization. In particular, we demonstrate a derandomization attack that will convert any standard buffer-overflow exploit into an exploit that works against systems protected by address-space randomization. The resulting exploit is as effective as the original exploit, although it takes a little longer to compromise a target machine: on average 216 seconds to compromise Apache running on a Linux PaX ASLR system. The attack does not require running code on the stack. We also explore various ways of strengthening address-space randomization and point out weaknesses in each. Surprisingly, increasing the frequency of re-randomizations adds at most 1 bit of security. Furthermore, compile-time randomization appears to be more effective than runtime randomization. We conclude that, on 32-bit architectures, the only benefit of PaX-like address-space randomization is a small slowdown in worm propagation speed. The cost of randomization is extra complexity in system support.
---
paper_title: SoK: Automated Software Diversity

paper_content:

The idea of automatic software diversity is at least two decades old. The deficiencies of currently deployed defenses and the transition to online software distribution (the "App store" model) for traditional and mobile computers has revived the interest in automatic software diversity. Consequently, the literature on diversity grew by more than two dozen papers since 2008. Diversity offers several unique properties. Unlike other defenses, it introduces uncertainty in the target. Precise knowledge of the target software provides the underpinning for a wide range of attacks. This makes diversity a broad rather than narrowly focused defense mechanism. Second, diversity offers probabilistic protection similar to cryptography-attacks may succeed by chance so implementations must offer high entropy. Finally, the design space of diversifying program transformations is large. As a result, researchers have proposed multiple approaches to software diversity that vary with respect to threat models, security, performance, and practicality. In this paper, we systematically study the state-of-the-art in software diversity and highlight fundamental trade-offs between fully automated approaches. We also point to open areas and unresolved challenges. These include "hybrid solutions", error reporting, patching, and implementation disclosure attacks on diversified software.
---
paper_title: Control Jujutsu: On the Weaknesses of Fine-Grained Control Flow Integrity

paper_content:

Control flow integrity (CFI) has been proposed as an approach to defend against control-hijacking memory corruption attacks. CFI works by assigning tags to indirect branch targets statically and checking them at runtime. Coarse-grained enforcements of CFI that use a small number of tags to improve the performance overhead have been shown to be ineffective. As a result, a number of recent efforts have focused on fine-grained enforcement of CFI as it was originally proposed. In this work, we show that even a fine-grained form of CFI with unlimited number of tags and a shadow stack (to check calls and returns) is ineffective in protecting against malicious attacks. We show that many popular code bases such as Apache and Nginx use coding practices that create flexibility in their intended control flow graph (CFG) even when a strong static analyzer is used to construct the CFG. These flexibilities allow an attacker to gain control of the execution while strictly adhering to a fine-grained CFI. We then construct two proof-of-concept exploits that attack an unlimited tag CFI system with a shadow stack. We also evaluate the difficulties of generating a precise CFG using scalable static analysis for real-world applications. Finally, we perform an analysis on a number of popular applications that highlights the availability of such attacks.
---
paper_title: Out of Control: Overcoming Control-Flow Integrity

paper_content:

As existing defenses like ASLR, DEP, and stack cookies are not sufficient to stop determined attackers from exploiting our software, interest in Control Flow Integrity (CFI) is growing. In its ideal form, CFI prevents flows of control that were not intended by the original program, effectively putting a stop to exploitation based on return oriented programming (and many other attacks besides). Two main problems have prevented CFI from being deployed in practice. First, many CFI implementations require source code or debug information that is typically not available for commercial software. Second, in its ideal form, the technique is very expensive. It is for this reason that current research efforts focus on making CFI fast and practical. Specifically, much of the work on practical CFI is applicable to binaries, and improves performance by enforcing a looser notion of control flow integrity. In this paper, we examine the security implications of such looser notions of CFI: are they still able to prevent code reuse attacks, and if not, how hard is it to bypass its protection? Specifically, we show that with two new types of gadgets, return oriented programming is still possible. We assess the availability of our gadget sets, and demonstrate the practicality of these results with a practical exploit against Internet Explorer that bypasses modern CFI implementations.
---
paper_title: CCFI: Cryptographically Enforced Control Flow Integrity

paper_content:

Control flow integrity (CFI) restricts jumps and branches within a program to prevent attackers from executing arbitrary code in vulnerable programs. However, traditional CFI still offers attackers too much freedom to chose between valid jump targets, as seen in recent attacks. We present a new approach to CFI based on cryptographic message authentication codes (MACs). Our approach, called cryptographic CFI (CCFI), uses MACs to protect control flow elements such as return addresses, function pointers, and vtable pointers. Through dynamic checks, CCFI enables much finer-grained classification of sensitive pointers than previous approaches, thwarting all known attacks and resisting even attackers with arbitrary access to program memory. We implemented CCFI in Clang/LLVM, taking advantage of recently available cryptographic CPU instructions (AES-NI). We evaluate our system on several large software packages (including nginx, Apache and memcache) as well as all their dependencies. The cost of protection ranges from a 3--18% decrease in server request rate. We also expect this overhead to shrink as Intel improves the performance AES-NI.
---
paper_title: ROP is still dangerous: breaking modern defenses

paper_content:

Return Oriented Programming (ROP) has become the exploitation technique of choice for modern memory-safety vulnerability attacks. Recently, there have been multiple attempts at defenses to prevent ROP attacks. In this paper, we introduce three new attack methods that break many existing ROP defenses. Then we show how to break kBouncer and ROPecker, two recent low-overhead defenses that can be applied to legacy software on existing hardware. We examine several recent ROP attacks seen in the wild and demonstrate that our techniques successfully cloak them so they are not detected by these defenses. Our attacks apply to many CFI-based defenses which we argue are weaker than previously thought. Future defenses will need to take our attacks into account.
---
paper_title: SoK: Eternal War in Memory

paper_content:

Memory corruption bugs in software written in low-level languages like C or C++ are one of the oldest problems in computer security. The lack of safety in these languages allows attackers to alter the program's behavior or take full control over it by hijacking its control flow. This problem has existed for more than 30 years and a vast number of potential solutions have been proposed, yet memory corruption attacks continue to pose a serious threat. Real world exploits show that all currently deployed protections can be defeated. This paper sheds light on the primary reasons for this by describing attacks that succeed on today's systems. We systematize the current knowledge about various protection techniques by setting up a general model for memory corruption attacks. Using this model we show what policies can stop which attacks. The model identifies weaknesses of currently deployed techniques, as well as other proposed protections enforcing stricter policies. We analyze the reasons why protection mechanisms implementing stricter polices are not deployed. To achieve wide adoption, protection mechanisms must support a multitude of features and must satisfy a host of requirements. Especially important is performance, as experience shows that only solutions whose overhead is in reasonable bounds get deployed. A comparison of different enforceable policies helps designers of new protection mechanisms in finding the balance between effectiveness (security) and efficiency. We identify some open research problems, and provide suggestions on improving the adoption of newer techniques.
---
paper_title: Per-Input Control-Flow Integrity

paper_content:

Control-Flow Integrity (CFI) is an effective approach to mitigating control-flow hijacking attacks. Conventional CFI techniques statically extract a control-flow graph (CFG) from a program and instrument the program to enforce that CFG. The statically generated CFG includes all edges for all possible inputs; however, for a concrete input, the CFG may include many unnecessary edges. We present Per-Input Control-Flow Integrity (PICFI), which is a new CFI technique that can enforce a CFG computed for each concrete input. PICFI starts executing a program with the empty CFG and lets the program itself lazily add edges to the enforced CFG if such edges are required for the concrete input. The edge addition is performed by PICFI-inserted instrumentation code. To prevent attackers from arbitrarily adding edges, PICFI uses a statically computed all-input CFG to constrain what edges can be added at runtime. To minimize performance overhead, operations for adding edges are designed to be idempotent, so they can be patched to no-ops after their first execution. As our evaluation shows, PICFI provides better security than conventional fine-grained CFI with comparable performance overhead.
---
paper_title: The undecidability of aliasing

paper_content:

Alias analysis is a prerequisite for performing most of the common program analyses such as reaching-definitions analysis or live-variables analysis. Landi [1992] recently established that it is impossible to compute statically precise alias information—either may-alias or must-alias—in languages with if statements, loops, dynamic storage, and recursive data structures: more precisely, he showed that the may-alias relation is not recursive, while the must-alias relation is not even recursively enumerable. This article presents simpler proofs of the same results.
---
paper_title: Stitching the gadgets: on the ineffectiveness of coarse-grained control-flow integrity protection

paper_content:

Return-oriented programming (ROP) offers a robust attack technique that has, not surprisingly, been extensively used to exploit bugs in modern software programs (e.g., web browsers and PDF readers). ROP attacks require no code injection, and have already been shown to be powerful enough to bypass fine-grained memory randomization (ASLR) defenses. To counter this ingenious attack strategy, several proposals for enforcement of (coarse-grained) control-flow integrity (CFI) have emerged. The key argument put forth by these works is that coarse-grained CFI policies are sufficient to prevent ROP attacks. As this reasoning has gained traction, ideas put forth in these proposals have even been incorporated into coarse-grained CFI defenses in widely adopted tools (e.g., Microsoft's EMET framework). ::: ::: In this paper, we provide the first comprehensive security analysis of various CFI solutions (covering kBouncer, ROPecker, CFI for COTS binaries, ROP-Guard, and Microsoft EMET 4.1). A key contribution is in demonstrating that these techniques can be effectively undermined, even under weak adversarial assumptions. More specifically, we show that with bare minimum assumptions, turing-complete and real-world ROP attacks can still be launched even when the strictest of enforcement policies is in use. To do so, we introduce several new ROP attack primitives, and demonstrate the practicality of our approach by transforming existing real-world exploits into more stealthy attacks that bypass coarse-grained CFI defenses.
---
paper_title: Control-flow bending: on the effectiveness of control-flow integrity

paper_content:

Control-Flow Integrity (CFI) is a defense which prevents control-flow hijacking attacks. While recent research has shown that coarse-grained CFI does not stop attacks, fine-grained CFI is believed to be secure. ::: ::: We argue that assessing the effectiveness of practical CFI implementations is non-trivial and that common evaluation metrics fail to do so. We then evaluate fullyprecise static CFI -- the most restrictive CFI policy that does not break functionality -- and reveal limitations in its security. Using a generalization of non-control-data attacks which we call Control-Flow Bending (CFB), we show how an attacker can leverage a memory corruption vulnerability to achieve Turing-complete computation on memory using just calls to the standard library. We use this attack technique to evaluate fully-precise static CFI on six real binaries and show that in five out of six cases, powerful attacks are still possible. Our results suggest that CFI may not be a reliable defense against memory corruption vulnerabilities. ::: ::: We further evaluate shadow stacks in combination with CFI and find that their presence for security is necessary: deploying shadow stacks removes arbitrary code execution capabilities of attackers in three of six cases.
---
paper_title: SoK: Automated Software Diversity

paper_content:

The idea of automatic software diversity is at least two decades old. The deficiencies of currently deployed defenses and the transition to online software distribution (the "App store" model) for traditional and mobile computers has revived the interest in automatic software diversity. Consequently, the literature on diversity grew by more than two dozen papers since 2008. Diversity offers several unique properties. Unlike other defenses, it introduces uncertainty in the target. Precise knowledge of the target software provides the underpinning for a wide range of attacks. This makes diversity a broad rather than narrowly focused defense mechanism. Second, diversity offers probabilistic protection similar to cryptography-attacks may succeed by chance so implementations must offer high entropy. Finally, the design space of diversifying program transformations is large. As a result, researchers have proposed multiple approaches to software diversity that vary with respect to threat models, security, performance, and practicality. In this paper, we systematically study the state-of-the-art in software diversity and highlight fundamental trade-offs between fully automated approaches. We also point to open areas and unresolved challenges. These include "hybrid solutions", error reporting, patching, and implementation disclosure attacks on diversified software.
---
paper_title: Enforcing forward-edge control-flow integrity in GCC & LLVM

paper_content:

Constraining dynamic control transfers is a common technique for mitigating software vulnerabilities. This defense has been widely and successfully used to protect return addresses and stack data; hence, current attacks instead typically corrupt vtable and function pointers to subvert a forward edge (an indirect jump or call) in the control-flow graph. Forward edges can be protected using Control-Flow Integrity (CFI) but, to date, CFI implementations have been research prototypes, based on impractical assumptions or ad hoc, heuristic techniques. To be widely adoptable, CFI mechanisms must be integrated into production compilers and be compatible with software-engineering aspects such as incremental compilation and dynamic libraries. ::: ::: This paper presents implementations of fine-grained, forward-edge CFI enforcement and analysis for GCC and LLVM that meet the above requirements. An analysis and evaluation of the security, performance, and resource consumption of these mechanisms applied to the SPEC CPU2006 benchmarks and common benchmarks for the Chromium web browser show the practicality of our approach: these fine-grained CFI mechanisms have significantly lower overhead than recent academic CFI prototypes. Implementing CFI in industrial compiler frameworks has also led to insights into design tradeoffs and practical challenges, such as dynamic loading.
---
paper_title: Lessons learned in using live red teams in IA experiments

paper_content:

The DARPA Information Assurance (IA) and Operational Partners in Experimentation (OPX) Programs have conducted over a dozen laboratory-based experiments involving live red teams since April 1999. This paper explores some of the lessons learned from the integrator's perspective that are common among those experiments.
---
paper_title: Characterizing Honeypot-Captured Cyber Attacks: Statistical Framework and Case Study

paper_content:

Rigorously characterizing the statistical properties of cyber attacks is an important problem. In this paper, we propose the first statistical framework for rigorously analyzing honeypot-captured cyber attack data. The framework is built on the novel concept of stochastic cyber attack process, a new kind of mathematical objects for describing cyber attacks. To demonstrate use of the framework, we apply it to analyze a low-interaction honeypot dataset, while noting that the framework can be equally applied to analyze high-interaction honeypot data that contains richer information about the attacks. The case study finds, for the first time, that long-range dependence (LRD) is exhibited by honeypot-captured cyber attacks. The case study confirms that by exploiting the statistical properties (LRD in this case), it is feasible to predict cyber attacks (at least in terms of attack rate) with good accuracy. This kind of prediction capability would provide sufficient early-warning time for defenders to adjust their defense configurations or resource allocations. The idea of “gray-box” (rather than “black-box”) prediction is central to the utility of the statistical framework, and represents a significant step towards ultimately understanding (the degree of) the predictability of cyber attacks.
---
paper_title: On the Mismanagement and Maliciousness of Networks

paper_content:

In this paper, we systematically explore the widely held, anecdotal belief that mismanaged networks are responsible for a wide range of security incidents. Utilizing Internet-scale measurements of DNS resolvers, BGP routers, and SMTP, HTTP, and DNS-name servers, we find there are thousands of networks where a large fraction of network services are misconfigured. Combining global feeds of malicious activities including spam, phishing, malware, and scanning, we find a statistically significant correlation between networks that are mismanaged and networks that are responsible for maliciousness.
---
paper_title: Markets for Cybercrime Tools and Stolen Data: Hackers' Bazaar

paper_content:

Criminal activities in cyberspace are increasingly facilitated by black markets. This report characterizes these markets and describes how they have evolved, to provide insight into how their existence can harm the information security environment.
---
paper_title: Predicting Cyber Attack Rates with Extreme Values

paper_content:

It is important to understand to what extent, and in what perspectives, cyber attacks can be predicted. Despite its evident importance, this problem was not investigated until very recently, when we proposed using the innovative methodology of gray-box prediction. This methodology advocates the use of gray-box models, which accommodate the statistical properties/phenomena exhibited by the data. Specifically, we showed that gray-box models that accommodate the long-range dependence phenomenon can predict the attack rate (i.e., the number of attacks per unit time) 1-h ahead-of-time with an accuracy of 70.2%–82.1%. To the best of our knowledge, this is the first result showing the feasibility of prediction in this domain. We observe that the prediction errors are partly caused by the models’ incapability in predicting the large attack rates, which are called extreme values in statistics. This motivates us to analyze the extreme-value phenomenon, using two complementary approaches: 1) the extreme value theory (EVT) and 2) the time series theory (TST). In this paper, we show that EVT can offer long-term predictions (e.g., 24-h ahead-of-time), while gray-box TST models can predict attack rates 1-h ahead-of-time with an accuracy of 86%–87.9%. We explore connections between the two approaches, and point out future research directions. Although our prediction study is based on specific cyber attack data, our methodology can be equally applied to analyze any cyber attack data of its kind.
---
paper_title: FIRE: FInding Rogue nEtworks

paper_content:

For many years, online criminals have been able to conduct their illicit activities by masquerading behind disreputable Internet Service Providers (ISPs). For example, organizations such as the Russian Business Network (RBN), Atrivo (a.k.a., Intercage), McColo, and most recently, the Triple Fiber Network (3FN) operated with impunity, providing a safe haven for Internet criminals for their own financial gain. What primarily sets these ISPs apart from others is the significant longevity of the malicious activities on their networks and the apparent lack of action taken in response to abuse reports. Interestingly, even though the Internet provides a certain degree of anonymity, such ISPs fear public attention. Once exposed, rogue networks often cease their malicious activities quickly, or are de-peered (disconnected) by their upstream providers. As a result, the Internet criminals are forced to relocate their operations. In this paper, we present FIRE, a novel system to identify and expose organizations and ISPs that demonstrate persistent, malicious behavior. The goal is to isolate the networks that are consistently implicated in malicious activity from those that are victims of compromise. To this end, FIRE actively monitors botnet communication channels, drive-by-download servers, and phishing web sites. This data is refined and correlated to quantify the degree of malicious activity for individual organizations. We present our results in real-time via the website maliciousnetworks.org. These results can be used to pinpoint and to track the activity of rogue organizations, preventing criminals from establishing strongholds on the Internet. Also, the information can be compiled into a null-routing blacklist to immediately halt traffic from malicious networks.
---
paper_title: Targeted threat index: characterizing and quantifying politically-motivated targeted malware

paper_content:

Targeted attacks on civil society and nongovernmental organizations have gone underreported despite the fact that these organizations have been shown to be frequent targets of these attacks. In this paper, we shed light on targeted malware attacks faced by these organizations by studying malicious e-mails received by 10 civil society organizations (the majority of which are from groups related to China and Tibet issues) over a period of 4 years. ::: ::: Our study highlights important properties of malware threats faced by these organizations with implications on how these organizations defend themselves and how we quantify these threats. We find that the technical sophistication of malware we observe is fairly low, with more effort placed on socially engineering the e-mail content. Based on this observation, we develop the Targeted Threat Index (TTI), a metric which incorporates both social engineering and technical sophistication when assessing the risk of malware threats. We demonstrate that this metric is more effective than simple technical sophistication for identifying malware threats with the highest potential to successfully compromise victims. We also discuss how education efforts focused on changing user behaviour can help prevent compromise. For two of the three Tibetan groups in our study simple steps such as avoiding the use of email attachments could cut document-based malware threats delivered through e-mail that we observed by up to 95%.
---
paper_title: Statistical mechanics of complex networks

paper_content:

The emergence of order in natural systems is a constant source of inspiration for both physical and biological sciences. While the spatial order characterizing for example the crystals has been the basis of many advances in contemporary physics, most complex systems in nature do not offer such high degree of order. Many of these systems form complex networks whose nodes are the elements of the system and edges represent the interactions between them. ::: Traditionally complex networks have been described by the random graph theory founded in 1959 by Paul Erdohs and Alfred Renyi. One of the defining features of random graphs is that they are statistically homogeneous, and their degree distribution (characterizing the spread in the number of edges starting from a node) is a Poisson distribution. In contrast, recent empirical studies, including the work of our group, indicate that the topology of real networks is much richer than that of random graphs. In particular, the degree distribution of real networks is a power-law, indicating a heterogeneous topology in which the majority of the nodes have a small degree, but there is a significant fraction of highly connected nodes that play an important role in the connectivity of the network. ::: The scale-free topology of real networks has very important consequences on their functioning. For example, we have discovered that scale-free networks are extremely resilient to the random disruption of their nodes. On the other hand, the selective removal of the nodes with highest degree induces a rapid breakdown of the network to isolated subparts that cannot communicate with each other. ::: The non-trivial scaling of the degree distribution of real networks is also an indication of their assembly and evolution. Indeed, our modeling studies have shown us that there are general principles governing the evolution of networks. Most networks start from a small seed and grow by the addition of new nodes which attach to the nodes already in the system. This process obeys preferential attachment: the new nodes are more likely to connect to nodes with already high degree. We have proposed a simple model based on these two principles wich was able to reproduce the power-law degree distribution of real networks. Perhaps even more importantly, this model paved the way to a new paradigm of network modeling, trying to capture the evolution of networks, not just their static topology.
---
paper_title: A Taxonomy of Botnet Structures

paper_content:

We propose a taxonomy of botnet structures, based on their utility to the botmaster. We propose key metrics to measure their utility for various activities (e.g., spam, ddos). Using these performance metrics, we consider the ability of different response techniques to degrade or disrupt botnets. In particular, our models show that targeted responses are particularly effective against scale free botnets and efforts to increase the robustness of scale free networks comes at a cost of diminished transitivity. Botmasters do not appear to have any structural solutions to this problem in scale free networks. We also show that random graph botnets (e.g., those using P2P formations) are highly resistant to both random and targeted responses. We evaluate the impact of responses on different topologies using simulation and demonstrate the utility of our proposed metrics by performing novel measurements of a P2P network. Our analysis shows how botnets may be classified according to structure and given rank or priority using our proposed metrics. This may help direct responses and suggests which general remediation strategies are more likely to succeed.
---
paper_title: A graph-based system for network-vulnerability analysis

paper_content:

This paper presents a graph-based approach to network vulnerability analysis. The method is flexible, allowing analysis of attacks from both outside and inside the network. It can analyze risks to a specific network asset, or examine the universe of possible consequences following a successful attack. The graph-based tool can identify the set of attack paths that have a high probability of success (or a low effort cost) for the attacker. The system could be used to test the effectiveness of making configuration changes, implementing an intrusion detection system, etc. The analysis system requires as input a database of common attacks, broken into atomic steps, specific network configuration and topology information, and an attacker profile. The attack information is matched with the network configuration information and an attacker profile to create a superset attack graph. Nodes identify a stage of attack, for example the class of machines the attacker has accessed and the user privilege level he or she has compromised. The arcs in the attack graph represent attacks or stages of attacks. By assigning probabilities of success on the arcs or costs representing level-of-effort for the attacker, various graph algorithms such as shortest-path algorithms can identify the attack paths with the highest probability of success.
---
paper_title: Aggregating Vulnerability Metrics in Enterprise Networks using Attack Graphs

paper_content:

Quantifying security risk is an important and yet difficult task in enterprise network security management. While metrics exist for individual software vulnerabilities, there is currently no standard way of aggregating such metrics. We present a model that can be used to aggregate vulnerability metrics in an enterprise network, producing quantitative metrics that measure the likelihood breaches can occur within a given network configuration. A clear semantic model for this aggregation is an important first step toward a comprehensive network security metric model. We utilize existing work in attack graphs and apply probabilistic reasoning to produce an aggregation that has clear semantics and sound computation. We ensure that shared dependencies between attack paths have a proportional effect on the final calculation. We correctly reason over cycles, ensuring that privileges are evaluated without any self-referencing effect. We introduce additional modeling artifacts in our probabilistic graphical model to capture and account for hidden correlations among exploit steps. The paper shows that a clear semantic model for aggregation is critical in interpreting the results, calibrating the metric model, and explaining insights gained from empirical evaluation. Our approach has been rigorously evaluated using a number of network models, as well as data from production systems.
---
paper_title: Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security

paper_content:

This paper presents the results of an experiment in security evaluation. The system is modeled as a privilege graph that exhibits its security vulnerabilities. Quantitative measures that estimate the effort an attacker might expend to exploit these vulnerabilities to defeat the system security objectives are proposed. A set of tools has been developed to compute such measures and has been used in an experiment to monitor a large real system for nearly two years. The experimental results are presented and the validity of the measures is discussed. Finally, the practical usefulness of such tools for operational security monitoring is shown and a comparison with other existing approaches is given.
---
paper_title: Time-efficient and cost-effective network hardening using attack graphs

paper_content:

Attack graph analysis has been established as a powerful tool for analyzing network vulnerability. However, previous approaches to network hardening look for exact solutions and thus do not scale. Further, hardening elements have been treated independently, which is inappropriate for real environments. For example, the cost for patching many systems may be nearly the same as for patching a single one. Or patching a vulnerability may have the same effect as blocking traffic with a firewall, while blocking a port may deny legitimate service. By failing to account for such hardening interdependencies, the resulting recommendations can be unrealistic and far from optimal. Instead, we formalize the notion of hardening strategy in terms of allowable actions, and define a cost model that takes into account the impact of interdependent hardening actions. We also introduce a near-optimal approximation algorithm that scales linearly with the size of the graphs, which we validate experimentally.
---
paper_title: Using model checking to analyze network vulnerabilities

paper_content:

Even well administered networks are vulnerable to attacks due to the security ramifications of offering a variety of combined services. That is, services that are secure when offered in isolation nonetheless provide an attacker with a vulnerability to exploit when offered simultaneously. Many current tools address vulnerabilities in the context of a single host. We address vulnerabilities due to the configuration of various hosts in a network. In a different line of research, formal methods are often useful for generating test cases, and model checkers are particularly adept at this task due to their ability to generate counterexamples. We address the network vulnerabilities problem with test cases, which amount to attack scenarios, generated by a model checker. We encode the vulnerabilities in a state machine description suitable for a model checker and then assert that an attacker cannot acquire a given privilege on a given host. The model checker either offers assurance that the assertion is true on the actual network or provides a counterexample detailing each step of a successful attack.
---
paper_title: Metrics of Security

paper_content:

Discussion of challenges and ways of improving Cyber Situational Awareness dominated our previous chapters. However, we have not yet touched on how to quantify any improvement we might achieve. Indeed, to get an accurate assessment of network security and provide sufficient Cyber Situational Awareness (CSA), simple but meaningful metrics—the focus of the Metrics of Security chapter—are necessary. The adage, “what can’t be measured can’t be effectively managed,” applies here. Without good metrics and the corresponding evaluation methods, security analysts and network operators cannot accurately evaluate and measure the security status of their networks and the success of their operations. In particular, this chapter explores two distinct issues: (i) how to define and use metrics as quantitative characteristics to represent the security state of a network, and (ii) how to define and use metrics to measure CSA from a defender’s point of view.
---
paper_title: Adversarial machine learning

paper_content:

In this paper (expanded from an invited talk at AISEC 2010), we discuss an emerging field of study: adversarial machine learning---the study of effective machine learning techniques against an adversarial opponent. In this paper, we: give a taxonomy for classifying attacks against online machine learning algorithms; discuss application-specific factors that limit an adversary's capabilities; introduce two models for modeling an adversary's capabilities; explore the limits of an adversary's knowledge about the algorithm, feature space, training, and input data; explore vulnerabilities in machine learning algorithms; discuss countermeasures against attacks; introduce the evasion challenge; and discuss privacy-preserving learning techniques.
---
paper_title: Adversarial learning

paper_content:

Many classification tasks, such as spam filtering, intrusion detection, and terrorism detection, are complicated by an adversary who wishes to avoid detection. Previous work on adversarial classification has made the unrealistic assumption that the attacker has perfect knowledge of the classifier [2]. In this paper, we introduce the adversarial classifier reverse engineering (ACRE) learning problem, the task of learning sufficient information about a classifier to construct adversarial attacks. We present efficient algorithms for reverse engineering linear classifiers with either continuous or Boolean features and demonstrate their effectiveness using real data from the domain of spam filtering.
---
paper_title: SoK: Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers

paper_content:

Run-time packers are often used by malware-writers to obfuscate their code and hinder static analysis. The packer problem has been widely studied, and several solutions have been proposed in order to generically unpack protected binaries. Nevertheless, these solutions commonly rely on a number of assumptions that may not necessarily reflect the reality of the packers used in the wild. Moreover, previous solutions fail to provide useful information about the structure of the packer or its complexity. In this paper, we describe a framework for packer analysis and we propose a taxonomy to measure the runtime complexity of packers. We evaluated our dynamic analysis system on two datasets, composed of both off-the-shelf packers and custom packed binaries. Based on the results of our experiments, we present several statistics about the packers complexity and their evolution over time.
---
paper_title: Binary-code obfuscations in prevalent packer tools

paper_content:

The first steps in analyzing defensive malware are understanding what obfuscations are present in real-world malware binaries, how these obfuscations hinder analysis, and how they can be overcome. While some obfuscations have been reported independently, this survey consolidates the discussion while adding substantial depth and breadth to it. This survey also quantifies the relative prevalence of these obfuscations by using the Dyninst binary analysis and instrumentation tool that was recently extended for defensive malware analysis. The goal of this survey is to encourage analysts to focus on resolving the obfuscations that are most prevalent in real-world malware.
---
paper_title: Characterizing the power of moving target defense via cyber epidemic dynamics

paper_content:

Moving Target Defense (MTD) can enhance the resilience of cyber systems against attacks. Although there have been many MTD techniques, there is no systematic understanding and {\em quantitative} characterization of the power of MTD. In this paper, we propose to use a cyber epidemic dynamics approach to characterize the power of MTD. We define and investigate two complementary measures that are applicable when the defender aims to deploy MTD to achieve a certain security goal. One measure emphasizes the maximum portion of time during which the system can afford to stay in an undesired configuration (or posture), without considering the cost of deploying MTD. The other measure emphasizes the minimum cost of deploying MTD, while accommodating that the system has to stay in an undesired configuration (or posture) for a given portion of time. Our analytic studies lead to algorithms for optimally deploying MTD.
---
paper_title: Emergent behavior in cybersecurity

paper_content:

We argue that <i>emergent behavior</i> is inherent to cybersecurity.
---
paper_title: A Stochastic Model of Multivirus Dynamics

paper_content:

Understanding the spreading dynamics of computer viruses (worms, attacks) is an important research problem, and has received much attention from the communities of both computer security and statistical physics. However, previous studies have mainly focused on single-virus spreading dynamics. In this paper, we study multivirus spreading dynamics, where multiple viruses attempt to infect computers while possibly combating against each other because, for example, they are controlled by multiple botmasters. Specifically, we propose and analyze a general model (and its two special cases) of multivirus spreading dynamics in arbitrary networks (i.e., we do not make any restriction on network topologies), where the viruses may or may not coreside on computers. Our model offers analytical results for addressing questions such as: What are the sufficient conditions (also known as epidemic thresholds) under which the multiple viruses will die out? What if some viruses can "rob” others? What characteristics does the multivirus epidemic dynamics exhibit when the viruses are (approximately) equally powerful? The analytical results make a fundamental connection between two types of factors: defense capability and network connectivity. This allows us to draw various insights that can be used to guide security defense.
---
paper_title: Cyber Epidemic Models with Dependences

paper_content:

Studying models of cyber epidemics over arbitrary complex networks can deepen our understanding of cyber security from a whole-system perspective. In this work, we initiate the investigation of cyber epidemic models that accommodate the dependences between the cyber attack events. Due to the notorious difficulty in dealing with such dependences, essentially all existing cyber epidemic models have disregarded them. Specifically, we introduce the idea of copulas into cyber epidemic models for accommodating the dependences between the cyber attack events. We investigate the epidemic equilibrium thresholds as well as the bounds for both equilibrium and nonequilibrium infection probabilities. We further characterize the side effects of disregarding the due dependences between the cyber attack events by showing that the results thereof are unnecessarily restrictive or even incorrect.
---
paper_title: Active cyber defense dynamics exhibiting rich phenomena

paper_content:

The Internet is a man-made complex system under constant attacks (e.g., Advanced Persistent Threats and malwares). It is therefore important to understand the phenomena that can be induced by the interaction between cyber attacks and cyber defenses. In this paper, we explore the rich phenomena that can be exhibited when the defender employs active defense to combat cyber attacks. To the best of our knowledge, this is the first study that shows that {\em active cyber defense dynamics} (or more generally, {\em cybersecurity dynamics}) can exhibit the bifurcation and chaos phenomena. This has profound implications for cyber security measurement and prediction: (i) it is infeasible (or even impossible) to accurately measure and predict cyber security under certain circumstances; (ii) the defender must manipulate the dynamics to avoid such {\em unmanageable situations} in real-life defense operations.
---
paper_title: Model-based Security Metrics Using ADversary VIew Security Evaluation (ADVISE)

paper_content:

System architects need quantitative security metrics to make informed trade-off decisions involving system security. The security metrics need to provide insight on weak points in the system defense, considering characteristics of both the system and its adversaries. To provide such metrics, we formally define the ADversary View Security Evaluation (ADVISE) method. Our approach is to create an executable state-based security model of a system and an adversary that represents how the adversary is likely to attack the system and the results of such an attack. The attack decision function uses information about adversary attack preferences and possible attacks against the system to mimic how the adversary selects the most attractive next attack step. The adversary's decision involves looking ahead some number of attack steps. System architects can use ADVISE to compare the security strength of system architecture variants and analyze the threats posed by different adversaries. We demonstrate the feasibility and benefits of ADVISE using a case study. To produce quantitative model-based security metrics, we have implemented the ADVISE method in a tool that facilitates user input of system and adversary data and automatically generates executable models.
---
paper_title: A Stochastic Model of Active Cyber Defense Dynamics

paper_content:

The concept of active cyber defense has been proposed for years. However, there are no mathematical models for characterizing the effectiveness of active cyber defense. In this paper, we fill the void by proposing a novel Markov process model that is native to the interaction between cyber attack and active cyber defense. Unfortunately, the native Markov process model cannot be tackled by the techniques we are aware of. We therefore simplify, via mean-field approximation, the Markov process model as a Dynamic System model that is amenable to analysis. This allows us to derive a set of valuable analytical results that characterize the effectiveness of four types of active cyber defense dynamics. Simulations show that the analytical results are inherent to the native Markov process model, and therefore justify the validity of the Dynamic System model. We also discuss the side-effect of the mean-field approximation and its implications.
---
paper_title: A quantitative model of the security intrusion process based on attacker behavior

paper_content:

The paper is based on a conceptual framework in which security can be split into two generic types of characteristics, behavioral and preventive. Here, preventive security denotes the system's ability to protect itself from external attacks. One way to describe the preventive security of a system is in terms of its interaction with the alleged attacker, i.e., by describing the intrusion process. To our knowledge, very little is done to model this process in quantitative terms. Therefore, based on empirical data collected from intrusion experiments, we have worked out a hypothesis on typical attacker behavior. The hypothesis suggests that the attacking process can be split into three phases: the learning phase, the standard attack phase, and the innovative attack phase. The probability for successful attacks during the learning and innovative phases is expected to be small, although for different reasons. During the standard attack phase it is expected to be considerably higher. The collected data indicates that the breaches during the standard attack phase are statistically equivalent and that the times between breaches are exponentially distributed. This would actually imply that traditional methods for reliability modeling could be applicable.
---
paper_title: Modeling and quantification of security attributes of software systems

paper_content:

Quite often failures in network based services and server systems may not be accidental, but rather caused by deliberate security intrusions. We would like such systems to either completely preclude the possibility of a security intrusion or design them to be robust enough to continue functioning despite security attacks. Not only is it important to prevent or tolerate security intrusions, it is equally important to treat security as a QoS attribute at par with, if not more important than other QoS attributes such as availability and performability. This paper deals with various issues related to quantifying the security attribute of an intrusion tolerant system, such as the SITAR system. A security intrusion and the response of an intrusion tolerant system to the attack is modeled as a random process. This facilitates the use of stochastic modeling techniques to capture the attacker behavior as well as the system's response to a security intrusion. This model is used to analyze and quantify the security attributes of the system. The security quantification analysis is first carried out for steady-state behavior leading to measures like steady-state availability. By transforming this model to a model with absorbing states, we compute a security measure called the "mean time (or effort) to security failure" and also compute probabilities of security failure due to violations of different security attributes.
---
paper_title: A behavioral theory of insider-threat risks: A system dynamics approach

paper_content:

The authors describe a behavioral theory of the dynamics of insider-threat risks. Drawing on data related to information technology security violations and on a case study created to explain the dynamics observed in that data, the authors constructed a system dynamics model of a theory of the development of insider-threat risks and conducted numerical simulations to explore the parameter and response spaces of the model. By examining several scenarios in which attention to events, increased judging capabilities, better information, and training activities are simulated, the authors theorize about why information technology security effectiveness changes over time. The simulation results argue against the common presumption that increased security comes at the cost of reduced production.
---
paper_title: Vulnerability disclosure in the age of social media: exploiting twitter for predicting real-world exploits

paper_content:

In recent years, the number of software vulnerabilities discovered has grown significantly. This creates a need for prioritizing the response to new disclosures by assessing which vulnerabilities are likely to be exploited and by quickly ruling out the vulnerabilities that are not actually exploited in the real world. We conduct a quantitative and qualitative exploration of the vulnerability-related information disseminated on Twitter. We then describe the design of a Twitter-based exploit detector, and we introduce a threat model specific to our problem. In addition to response prioritization, our detection techniques have applications in risk modeling for cyber-insurance and they highlight the value of information provided by the victims of attacks.
---
paper_title: Characterizing Honeypot-Captured Cyber Attacks: Statistical Framework and Case Study

paper_content:

Rigorously characterizing the statistical properties of cyber attacks is an important problem. In this paper, we propose the first statistical framework for rigorously analyzing honeypot-captured cyber attack data. The framework is built on the novel concept of stochastic cyber attack process, a new kind of mathematical objects for describing cyber attacks. To demonstrate use of the framework, we apply it to analyze a low-interaction honeypot dataset, while noting that the framework can be equally applied to analyze high-interaction honeypot data that contains richer information about the attacks. The case study finds, for the first time, that long-range dependence (LRD) is exhibited by honeypot-captured cyber attacks. The case study confirms that by exploiting the statistical properties (LRD in this case), it is feasible to predict cyber attacks (at least in terms of attack rate) with good accuracy. This kind of prediction capability would provide sufficient early-warning time for defenders to adjust their defense configurations or resource allocations. The idea of “gray-box” (rather than “black-box”) prediction is central to the utility of the statistical framework, and represents a significant step towards ultimately understanding (the degree of) the predictability of cyber attacks.
---
paper_title: Statistical metrics for individual password strength

paper_content:

We propose several possible metrics for measuring the strength of an individual password or any other secret drawn from a known, skewed distribution. In contrast to previous ad hoc approaches which rely on textual properties of passwords, we consider the problem without any knowledge of password structure. This enables rating the strength of a password given a large sample distribution without assuming anything about password semantics. We compare the results of our generic metrics against those of the NIST metrics and other previous "entropy-based" metrics for a large password dataset, which suggest over-fitting in previous metrics.
---
paper_title: Testing metrics for password creation policies by attacking large sets of revealed passwords

paper_content:

In this paper we attempt to determine the effectiveness of using entropy, as defined in NIST SP800-63, as a measurement of the security provided by various password creation policies. This is accomplished by modeling the success rate of current password cracking techniques against real user passwords. These data sets were collected from several different websites, the largest one containing over 32 million passwords. This focus on actual attack methodologies and real user passwords quite possibly makes this one of the largest studies on password security to date. In addition we examine what these results mean for standard password creation policies, such as minimum password length, and character set requirements.
---
paper_title: Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security

paper_content:

This paper presents the results of an experiment in security evaluation. The system is modeled as a privilege graph that exhibits its security vulnerabilities. Quantitative measures that estimate the effort an attacker might expend to exploit these vulnerabilities to defeat the system security objectives are proposed. A set of tools has been developed to compute such measures and has been used in an experiment to monitor a large real system for nearly two years. The experimental results are presented and the validity of the measures is discussed. Finally, the practical usefulness of such tools for operational security monitoring is shown and a comparison with other existing approaches is given.
---
paper_title: Who falls for phish?: a demographic analysis of phishing susceptibility and effectiveness of interventions

paper_content:

In this paper we present the results of a roleplay survey instrument administered to 1001 online survey respondents to study both the relationship between demographics and phishing susceptibility and the effectiveness of several anti-phishing educational materials. Our results suggest that women are more susceptible than men to phishing and participants between the ages of 18 and 25 are more susceptible to phishing than other age groups. We explain these demographic factors through a mediation analysis. Educational materials reduced users' tendency to enter information into phishing webpages by 40% percent; however, some of the educational materials we tested also slightly decreased participants' tendency to click on legitimate links.
---
paper_title: The Matter of Heartbleed

paper_content:

The Heartbleed vulnerability took the Internet by surprise in April 2014. The vulnerability, one of the most consequential since the advent of the commercial Internet, allowed attackers to remotely read protected memory from an estimated 24--55% of popular HTTPS sites. In this work, we perform a comprehensive, measurement-based analysis of the vulnerability's impact, including (1) tracking the vulnerable population, (2) monitoring patching behavior over time, (3) assessing the impact on the HTTPS certificate ecosystem, and (4) exposing real attacks that attempted to exploit the bug. Furthermore, we conduct a large-scale vulnerability notification experiment involving 150,000 hosts and observe a nearly 50% increase in patching by notified hosts. Drawing upon these analyses, we discuss what went well and what went poorly, in an effort to understand how the technical community can respond more effectively to such events in the future.
---
paper_title: After we knew it: empirical study and modeling of cost-effectiveness of exploiting prevalent known vulnerabilities across IaaS cloud

paper_content:

Infrastructure as a Service (IaaS) cloud has been attracting more and more customers as it provides the highest level of flexibility by offering configurable virtual machines (VMs) and computing infrastructures. Public VM images are usually available for customers to customize and launch. However, the 1 to N mapping between VM images and running instances in IaaS makes vulnerabilities propagate rapidly across the entire public cloud. Besides, IaaS cloud naturally comes with a larger and more stable attack surface and more concentrated target resources than traditional surroundings. In this paper, we first identify the threat of exploiting prevalent vulnerabilities over public IaaS cloud with an empirical study in Amazon EC2. We find that attackers can compromise a considerable number of VMs with trivial cost. We then do a qualitative cost-effectiveness analysis of this threat. Our main result is a two-fold observation: in IaaS cloud, exploiting prevalent vulnerabilities is much more cost-effective than traditional in-house computing environment, therefore attackers have stronger incentive; Fortunately, on the other hand, cloud defenders (cloud providers and customers) also have much lower cost-loss ratio than in traditional environment, therefore they can be more effective for defending attacks. We then build a game-theoretic model and conduct a risk-gain analysis to compare exploiting and patching strategies under cloud and traditional computing environments. Our modeling indicates that under cloud environment, both attack and defense become less cost-effective as time goes by, and the earlier actioner can be more rewarding. We propose countermeasures against such threat in order to bridge the gap between current security situation and defending mechanisms. To our best knowledge, we are the first to analyze and model the threat with prevalent known-vulnerabilities in public cloud.
---
paper_title: Behavioral Analysis of Insider Threat: A Survey and Bootstrapped Prediction in Imbalanced Data

paper_content:

The problem of insider threat is receiving increasing attention both within the computer science community as well as government and industry. This paper starts by presenting a broad, multidisciplinary survey of insider threat capturing contributions from computer scientists, psychologists, criminologists, and security practitioners. Subsequently, we present the behavioral analysis of insider threat ( $ \textsf{BAIT}$ ) framework, in which we conduct a detailed experiment involving 795 subjects on Amazon Mechanical Turk (AMT) in order to gauge the behaviors that real human subjects follow when attempting to exfiltrate data from within an organization. In the real world, the number of actual insiders found is very small, so supervised machine-learning methods encounter a challenge. Unlike past works, we develop bootstrapping algorithms that learn from highly imbalanced data, mostly unlabeled, and almost no history of user behavior from an insider threat perspective. We develop and evaluate seven algorithms using $ \textsf{BAIT}$ and show that they can produce a realistic (and acceptable) balance of precision and recall.
---
paper_title: The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching

paper_content:

Vulnerability exploits remain an important mechanism for malware delivery, despite efforts to speed up the creation of patches and improvements in software updating mechanisms. Vulnerabilities in client applications (e.g., Browsers, multimedia players, document readers and editors) are often exploited in spear phishing attacks and are difficult to characterize using network vulnerability scanners. Analyzing their lifecycle requires observing the deployment of patches on hosts around the world. Using data collected over 5 years on 8.4 million hosts, available through Symantec's WINE platform, we present the first systematic study of patch deployment in client-side vulnerabilities. We analyze the patch deployment process of 1,593 vulnerabilities from 10 popular client applications, and we identify several new threats presented by multiple installations of the same program and by shared libraries distributed with several applications. For the 80 vulnerabilities in our dataset that affect code shared by two applications, the time between patch releases in the different applications is up to 118 days (with a median of 11 days). Furthermore, as the patching rates differ considerably among applications, many hosts patch the vulnerability in one application but not in the other one. We demonstrate two novel attacks that enable exploitation by invoking old versions of applications that are used infrequently, but remain installed. We also find that the median fraction of vulnerable hosts patched when exploits are released is at most 14%. Finally, we show that the patching rate is affected by user-specific and application-specific factors, for example, hosts belonging to security analysts and applications with an automated updating mechanism have significantly lower median times to patch.
---
paper_title: Targeted threat index: characterizing and quantifying politically-motivated targeted malware

paper_content:

Targeted attacks on civil society and nongovernmental organizations have gone underreported despite the fact that these organizations have been shown to be frequent targets of these attacks. In this paper, we shed light on targeted malware attacks faced by these organizations by studying malicious e-mails received by 10 civil society organizations (the majority of which are from groups related to China and Tibet issues) over a period of 4 years. ::: ::: Our study highlights important properties of malware threats faced by these organizations with implications on how these organizations defend themselves and how we quantify these threats. We find that the technical sophistication of malware we observe is fairly low, with more effort placed on socially engineering the e-mail content. Based on this observation, we develop the Targeted Threat Index (TTI), a metric which incorporates both social engineering and technical sophistication when assessing the risk of malware threats. We demonstrate that this metric is more effective than simple technical sophistication for identifying malware threats with the highest potential to successfully compromise victims. We also discuss how education efforts focused on changing user behaviour can help prevent compromise. For two of the three Tibetan groups in our study simple steps such as avoiding the use of email attachments could cut document-based malware threats delivered through e-mail that we observed by up to 95%.
---
paper_title: An Attack Surface Metric

paper_content:

Measurement of software security is a long-standing challenge to the research community. At the same time, practical security metrics and measurements are essential for secure software development. Hence, the need for metrics is more pressing now due to a growing demand for secure software. In this paper, we propose using a software system's attack surface measurement as an indicator of the system's security. We formalize the notion of a system's attack surface and introduce an attack surface metric to measure the attack surface in a systematic manner. Our measurement method is agnostic to a software system's implementation language and is applicable to systems of all sizes; we demonstrate our method by measuring the attack surfaces of small desktop applications and large enterprise systems implemented in C and Java. We conducted three exploratory empirical studies to validate our method. Software developers can mitigate their software's security risk by measuring and reducing their software's attack surfaces. Our attack surface reduction approach complements the software industry's traditional code quality improvement approach for security risk mitigation and is useful in multiple phases of the software development lifecycle. Our collaboration with SAP demonstrates the use of our metric in the software development process.
---
paper_title: Predicting Cyber Attack Rates with Extreme Values

paper_content:

It is important to understand to what extent, and in what perspectives, cyber attacks can be predicted. Despite its evident importance, this problem was not investigated until very recently, when we proposed using the innovative methodology of gray-box prediction. This methodology advocates the use of gray-box models, which accommodate the statistical properties/phenomena exhibited by the data. Specifically, we showed that gray-box models that accommodate the long-range dependence phenomenon can predict the attack rate (i.e., the number of attacks per unit time) 1-h ahead-of-time with an accuracy of 70.2%–82.1%. To the best of our knowledge, this is the first result showing the feasibility of prediction in this domain. We observe that the prediction errors are partly caused by the models’ incapability in predicting the large attack rates, which are called extreme values in statistics. This motivates us to analyze the extreme-value phenomenon, using two complementary approaches: 1) the extreme value theory (EVT) and 2) the time series theory (TST). In this paper, we show that EVT can offer long-term predictions (e.g., 24-h ahead-of-time), while gray-box TST models can predict attack rates 1-h ahead-of-time with an accuracy of 86%–87.9%. We explore connections between the two approaches, and point out future research directions. Although our prediction study is based on specific cyber attack data, our methodology can be equally applied to analyze any cyber attack data of its kind.
---
paper_title: When private keys are public: results from the 2008 Debian OpenSSL vulnerability

paper_content:

We report on the aftermath of the discovery of a severe vulnerability in the Debian Linux version of OpenSSL. Systems affected by the bug generated predictable random numbers, most importantly public/private keypairs. To study user response to this vulnerability, we collected a novel dataset of daily remote scans of over 50,000 SSL/TLS-enabled Web servers, of which 751 displayed vulnerable certificates. We report three primary results. First, as expected from previous work, we find an extremely slow rate of fixing, with 30% of the hosts vulnerable when we began our survey on day 4 after disclosure still vulnerable almost six months later. However, unlike conventional vulnerabilities, which typically show a short, fast fixing phase, we observe a much flatter curve with fixing extending six months after the announcement. Second, we identify some predictive factors for the rate of upgrading. Third, we find that certificate authorities continued to issue certificates to servers with weak keys long after the vulnerability was disclosed.
---
paper_title: The Science of Guessing: Analyzing an Anonymized Corpus of 70 Million Passwords

paper_content:

We report on the largest corpus of user-chosen passwords ever studied, consisting of anonymized password histograms representing almost 70 million Yahoo! users, mitigating privacy concerns while enabling analysis of dozens of subpopulations based on demographic factors and site usage characteristics. This large data set motivates a thorough statistical treatment of estimating guessing difficulty by sampling from a secret distribution. In place of previously used metrics such as Shannon entropy and guessing entropy, which cannot be estimated with any realistically sized sample, we develop partial guessing metrics including a new variant of guesswork parameterized by an attacker's desired success rate. Our new metric is comparatively easy to approximate and directly relevant for security engineering. By comparing password distributions with a uniform distribution which would provide equivalent security against different forms of guessing attack, we estimate that passwords provide fewer than 10 bits of security against an online, trawling attack, and only about 20 bits of security against an optimal offline dictionary attack. We find surprisingly little variation in guessing difficulty; every identifiable group of users generated a comparably weak password distribution. Security motivations such as the registration of a payment card have no greater impact than demographic factors such as age and nationality. Even proactive efforts to nudge users towards better password choices with graphical feedback make little difference. More surprisingly, even seemingly distant language communities choose the same weak passwords and an attacker never gains more than a factor of 2 efficiency gain by switching from the globally optimal dictionary to a population-specific lists.
---
paper_title: Guess Again (and Again and Again): Measuring Password Strength by Simulating Password-Cracking Algorithms

paper_content:

Text-based passwords remain the dominant authentication method in computer systems, despite significant advancement in attackers' capabilities to perform password cracking. In response to this threat, password composition policies have grown increasingly complex. However, there is insufficient research defining metrics to characterize password strength and using them to evaluate password-composition policies. In this paper, we analyze 12,000 passwords collected under seven composition policies via an online study. We develop an efficient distributed method for calculating how effectively several heuristic password-guessing algorithms guess passwords. Leveraging this method, we investigate (a) the resistance of passwords created under different conditions to guessing, (b) the performance of guessing algorithms under different training sets, (c) the relationship between passwords explicitly created under a given composition policy and other passwords that happen to meet the same requirements, and (d) the relationship between guess ability, as measured with password-cracking algorithms, and entropy estimates. Our findings advance understanding of both password-composition policies and metrics for quantifying password security.
---
paper_title: A Taxonomy of Botnet Structures

paper_content:

We propose a taxonomy of botnet structures, based on their utility to the botmaster. We propose key metrics to measure their utility for various activities (e.g., spam, ddos). Using these performance metrics, we consider the ability of different response techniques to degrade or disrupt botnets. In particular, our models show that targeted responses are particularly effective against scale free botnets and efforts to increase the robustness of scale free networks comes at a cost of diminished transitivity. Botmasters do not appear to have any structural solutions to this problem in scale free networks. We also show that random graph botnets (e.g., those using P2P formations) are highly resistant to both random and targeted responses. We evaluate the impact of responses on different topologies using simulation and demonstrate the utility of our proposed metrics by performing novel measurements of a P2P network. Our analysis shows how botnets may be classified according to structure and given rank or priority using our proposed metrics. This may help direct responses and suggests which general remediation strategies are more likely to succeed.
---
paper_title: Using model checking to analyze network vulnerabilities

paper_content:

Even well administered networks are vulnerable to attacks due to the security ramifications of offering a variety of combined services. That is, services that are secure when offered in isolation nonetheless provide an attacker with a vulnerability to exploit when offered simultaneously. Many current tools address vulnerabilities in the context of a single host. We address vulnerabilities due to the configuration of various hosts in a network. In a different line of research, formal methods are often useful for generating test cases, and model checkers are particularly adept at this task due to their ability to generate counterexamples. We address the network vulnerabilities problem with test cases, which amount to attack scenarios, generated by a model checker. We encode the vulnerabilities in a state machine description suitable for a model checker and then assert that an attacker cannot acquire a given privilege on a given host. The model checker either offers assurance that the assertion is true on the actual network or provides a counterexample detailing each step of a successful attack.
---
paper_title: Metrics of Security

paper_content:

Discussion of challenges and ways of improving Cyber Situational Awareness dominated our previous chapters. However, we have not yet touched on how to quantify any improvement we might achieve. Indeed, to get an accurate assessment of network security and provide sufficient Cyber Situational Awareness (CSA), simple but meaningful metrics—the focus of the Metrics of Security chapter—are necessary. The adage, “what can’t be measured can’t be effectively managed,” applies here. Without good metrics and the corresponding evaluation methods, security analysts and network operators cannot accurately evaluate and measure the security status of their networks and the success of their operations. In particular, this chapter explores two distinct issues: (i) how to define and use metrics as quantitative characteristics to represent the security state of a network, and (ii) how to define and use metrics to measure CSA from a defender’s point of view.
---
paper_title: Measuring real-world accuracies and biases in modeling password guessability

paper_content:

Parameterized password guessability--how many guesses a particular cracking algorithm with particular training data would take to guess a password--has become a common metric of password security. Unlike statistical metrics, it aims to model real-world attackers and to provide per-password strength estimates. We investigate how cracking approaches often used by researchers compare to real-world cracking by professionals, as well as how the choice of approach biases research conclusions. ::: ::: We find that semi-automated cracking by professionals outperforms popular fully automated approaches, but can be approximated by combining multiple such approaches. These approaches are only effective, however, with careful configuration and tuning; in commonly used default configurations, they underestimate the real-world guessability of passwords. We find that analyses of large password sets are often robust to the algorithm used for guessing as long as it is configured effectively. However, cracking algorithms differ systematically in their effectiveness guessing passwords with certain common features (e.g., character substitutions). This has important implications for analyzing the security of specific password characteristics or of individual passwords (e.g., in a password meter or security audit). Our results highlight the danger of relying only on a single cracking algorithm as a measure of password strength and constitute the first scientific evidence that automated guessing can often approximate guessing by professionals.
---
paper_title: Control-flow bending: on the effectiveness of control-flow integrity

paper_content:

Control-Flow Integrity (CFI) is a defense which prevents control-flow hijacking attacks. While recent research has shown that coarse-grained CFI does not stop attacks, fine-grained CFI is believed to be secure. ::: ::: We argue that assessing the effectiveness of practical CFI implementations is non-trivial and that common evaluation metrics fail to do so. We then evaluate fullyprecise static CFI -- the most restrictive CFI policy that does not break functionality -- and reveal limitations in its security. Using a generalization of non-control-data attacks which we call Control-Flow Bending (CFB), we show how an attacker can leverage a memory corruption vulnerability to achieve Turing-complete computation on memory using just calls to the standard library. We use this attack technique to evaluate fully-precise static CFI on six real binaries and show that in five out of six cases, powerful attacks are still possible. Our results suggest that CFI may not be a reliable defense against memory corruption vulnerabilities. ::: ::: We further evaluate shadow stacks in combination with CFI and find that their presence for security is necessary: deploying shadow stacks removes arbitrary code execution capabilities of attackers in three of six cases.
---
paper_title: Continuous Security Metrics for Prevalent Network Threats: Introduction and First Four Metrics

paper_content:

Abstract : The goal of this work is to introduce meaningful security metrics that motivate effective improvements in network security. We present a methodology for directly deriving security metrics from realistic mathematical models of adversarial behaviors and systems and also a maturity model to guide the adoption and use of these metrics. Four security metrics are described that assess the risk from prevalent network threats. These can be computed automatically and continuously on a network to assess the effectiveness of controls. Each new metric directly assesses the effect of controls that mitigate vulnerabilities, continuously estimates the risk from one adversary, and provides direct insight into what changes must be made to improve security. Details of an explicit maturity model are provided for each metric that guide security practitioners through three stages where they (1) develop foundational understanding, tools and procedures, (2) make accurate and timely measurements that cover all relevant network components and specify security conditions to test, and (3) perform continuous risk assessments and network improvements. Metrics are designed to address specific threats, maintain practicality and simplicity, and motivate risk reduction. These initial four metrics and additional ones we are developing should be added incrementally to a network to gradually improve overall security as scores drop to acceptable levels and the risks from associated cyber threats are mitigated.
---
paper_title: Emergent behavior in cybersecurity

paper_content:

We argue that <i>emergent behavior</i> is inherent to cybersecurity.
---
paper_title: Security Metrics: Replacing Fear, Uncertainty, and Doubt

paper_content:

The Definitive Guide to Quantifying, Classifying, and Measuring Enterprise IT Security OperationsSecurity Metrics is the first comprehensive best-practice guide to defining, creating, and utilizing security metrics in the enterprise.Using sample charts, graphics, case studies, and war stories, Yankee Group Security Expert Andrew Jaquith demonstrates exactly how to establish effective metrics based on your organization's unique requirements. You'll discover how to quantify hard-to-measure security activities, compile and analyze all relevant data, identify strengths and weaknesses, set cost-effective priorities for improvement, and craft compelling messages for senior management.Security Metrics successfully bridges management's quantitative viewpoint with the nuts-and-bolts approach typically taken by security professionals. It brings together expert solutions drawn from Jaquith's extensive consulting work in the software, aerospace, and financial services industries, including new metrics presented nowhere else. You'll learn how to:· Replace nonstop crisis response with a systematic approach to security improvement· Understand the differences between “good” and “bad” metrics· Measure coverage and control, vulnerability management, password quality, patch latency, benchmark scoring, and business-adjusted risk· Quantify the effectiveness of security acquisition, implementation, and other program activities· Organize, aggregate, and analyze your data to bring out key insights· Use visualization to understand and communicate security issues more clearly· Capture valuable data from firewalls and antivirus logs, third-party auditor reports, and other resources· Implement balanced scorecards that present compact, holistic views of organizational security effectivenessWhether you're an engineer or consultant responsible for security and reporting to managementi??or an executive who needs better information for decision-makingi??Security Metrics is the resource you have been searching for.Andrew Jaquith, program manager for Yankee Group's Security Solutions and Services Decision Service, advises enterprise clients on prioritizing and managing security resources. He also helps security vendors develop product, service, and go-to-market strategies for reaching enterprise customers. He co-founded @stake, Inc., a security consulting pioneer acquired by Symantec Corporation in 2004. His application security and metrics research has been featured in CIO, CSO, InformationWeek, IEEE Security and Privacy, and The Economist. Forewordi? i? i? i? i? i? i? i? i? Prefacei? i? i? i? i? i? i? i? i? i? i? i? Acknowledgmentsi? i? i? i? i? i? i? i? i? About the Authori? i? i? i? i? i? i? i? i? i? i? Chapter 1 i? i? i? i? i? i? i? i? Introduction: Escaping the Hamster Wheel of Paini? i? i? i? i? i? i? i? i? i? Chapter 2 i? i? i? i? i? i? i? i? Defining Security Metricsi? i? i? i? i? i? i? i? i? i? i? Chapter 3 i? i? i? i? i? i? i? i? Diagnosing Problems and Measuring Technical Securityi? i? Chapter 4 i? i? i? i? i? i? i? i? Measuring Program Effectivenessi? i? i? i? i? i? i? i? i? i? i? Chapter 5 i? i? i? i? i? i? i? i? Analysis Techniquesi? i? i? i? i? Chapter 6 i? i? i? i? i? i? i? i? Visualizationi? i? i? i? i? Chapter 7 i? i? i? i? i? i? i? i? Automating Metrics CalculationsChapter 8 i? i? i? i? i? i? i? i? Designing Security Scorecardsi? i? Indexi? i? i?
---

Title: A Survey on Security Metrics

Section 1: Introduction
Description 1: Introduce the security metrics and the existing challenges and motivations for conducting the survey.

Section 2: Scope
Description 2: Discuss the scope of the literature that is surveyed and limitations of the current focus.

Section 3: Survey Methodology
Description 3: Outline the methodology used to conduct the survey, including the perspective of cyber attack-defense interactions.

Section 4: Metrics for Measuring Vulnerabilities
Description 4: Discuss metrics that measure system vulnerabilities, including user vulnerabilities, password vulnerabilities, interface-induced vulnerabilities, software vulnerabilities, and cryptographic key vulnerabilities.

Section 5: Metrics for Measuring Defenses
Description 5: Review metrics that measure the effectiveness of various defenses such as blacklisting, attack detection, ASLR, CFI, and overall defense power.

Section 6: Metrics for Measuring Threats
Description 6: Examine metrics related to threats, including the threat landscape, zero-day attacks, the power of individual attacks, and obfuscation sophistication.

Section 7: Metrics for Measuring Situations
Description 7: Describe metrics that measure outcomes of attack-defense interactions, including security states, security incidents, and security investments.

Section 8: Fundamental Questions and Challenges
Description 8: Discuss fundamental questions and challenges that must be addressed to improve security metrics and their application, including what needs to be measured and the properties of good metrics.

Section 9: Conclusion
Description 9: Summarize the survey findings, the suggested desirable metrics, and the partnership needed between academia, industry, and government to advance the research in security metrics.