Approaches for the combined use of risk analysis and testing: a systematic literature review
---
paper_title: Testing Long-Lived Web Services Transactions Using a Risk-Based Approach

paper_content:

Transactions are crucial to ensuring the quality (such as recovery and reliability) of web services applications by constraining them to a mutually agreed outcome. This paper addresses the issue of testing the long-lived web services transactions which has been given little attention by the current research. It proposes a risk-based approach and also defines a set of properties for web services transactions. The proposed approach identifies for each property a set of potential situations that must be tested. We present an analysis for the Recovery property using a Fault Tree diagram where the leaf nodes represent potentially dangerous scenarios that must be checked. Finally we show with a case study how this Fault Tree can be used to derive test cases for web services transactions.
---
paper_title: Risk-based Testing Techniques: A Perspective Study

paper_content:

Doing more with less has become a mantra for IT organization in today’s business environment. Nowadays, there are more projects, more competitive pressures and greater failure risk which needs to be managed with fewer resources with tighter timelines. But with all these constraints, there's simply no room for compromise on quality and stability in today’s competitive world especially in case of important business critical applications. So, instead of doing more with less and risking late projects, increased costs or low quality, we need to find ways to achieve better with less. The focus of testing has to be placed on aspects of software that matter most with an aim of reducing the risk of failure as well as ensuring the quality and stability of the business applications. This can be achieved by applying the principle of Risk Based Prioritization of tests, known as Risk-based testing (RBT). The aim of Risk Based testing approach is to ensure that appropriate testing activities are identified and prioritized based on risk. The primary role of risk-based testing is to optimize available resources and time without affecting the quality of the product. RBT approach reduces the risk of failure to the business and increase customer satisfaction. In this light, this paper presents the progress different risk-based testing metrics to measure and control test cases and test activities progress, efforts and costs. IT organizations must adopt a focused approach and a comprehensive methodology for end-to-end testing. Riskbased testing helps quantify and mitigate risks in the lifecycle of applications, and prioritize tests more effectively. Under RBT, we create Optimized Regression Test Suite based on Business Severity and Priority. The Success of testing team will be the ability to identify high risk defects in software and ensure they are fixed.
---
paper_title: A review of research on risk analysis methods for IT systems

paper_content:

Context: At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. This means that there is a need for risk analysis in the development of this kind of systems. Risk analysis of technical systems has a long history in mechanical and electrical engineering. Objective: Even if a number of methods for risk analysis of technical systems exist, the failure behavior of information systems is typically very different from mechanical systems. Therefore, risk analysis of IT systems requires different risk analysis techniques, or at least adaptations of traditional approaches. This means that there is a need to understand what types of methods are available for IT systems and what research that has been conducted on these methods. Method: In this paper we present a systematic mapping study on risk analysis for IT systems. 1086 unique papers were identified in a database search and 57 papers were identified as relevant for this study. These papers were classified based on 5 different criteria. Results: This classification, for example, shows that most of the discussed risk analysis methods are qualitative and not quantitative and that most of the risk analysis methods that are presented in these papers are developed for IT systems in general and not for specific types of IT system. Conclusions: The results show that many new risk analysis methods have been proposed in the last decade but even more that there is a need for more empirical evaluations of the different risk analysis methods. Many papers were identified that propose new risk analysis methods, but few papers discuss a systematic evaluation of these methods or a comparison of different methods based on empirical data.
---
paper_title: A Framework to Test Advanced Web Services Transactions

paper_content:

Transactions are a key issue in the reliability of distributed applications because they ensure all the participants achieve a mutually agreed outcome. However, current research has given little attention to testing transactions in web services. This paper presents a conceptual framework, inspired in risk-based methodologies, to address this gap. It also reports on preliminary results and identifies future work.
---
paper_title: Theory and practice of risk-based testing 1

paper_content:

This paper extends a theory of risk-based test planning that was outlined in a previous paper, introduces techniques to facilitate it, explains their practical application, and highlights opportunities for research to enquire into their feasibility and efficacy. Copyright © 2004 John Wiley & Sons, Ltd.
---
paper_title: Integrating Manual and Automatic Risk Assessment for Risk-Based Testing

paper_content:

In this paper we define a model-based risk assessment procedure that integrates automatic risk assessment by static analysis, semi-automatic risk assessment and guided manual risk assessment. In this process probability and impact criteria are determined by metrics which are combined to estimate the risk of specific system development artifacts. The risk values are propagated to the assigned test cases providing a prioritization of test cases. This supports to optimize the allocation of limited testing time and budget in a risk-based testing methodology. Therefore, we embed our risk assessment process into a generic risk-based testing methodology. The calculation of probability and impact metrics is based on system and requirements artifacts which are formalized as model elements. Additional time metrics consider the temporal development of the system under test and take for instance the bug and version history of the system into account. The risk assessment procedure integrates several stakeholders and is explained by a running example.
---
paper_title: A TEST CASE PRIORITIZATION BASED ON DEGREE OF RISK EXPOSURE AND ITS EMPIRICAL STUDY

paper_content:

We propose a test case prioritization strategy for risk based testing, in which the risk exposure is employed as the key criterion of evaluation. Existing approaches to risk based testing typically employ risk exposure values as assessed by the tester. In contrast, we employ exposure values that have been determined by experts during the risk assessment stage of the risk management process. If a given method produces greater accuracy in fault detection, that approach is considered more valuable for software testing. We demonstrate the value of our proposed risk based testing method in this sense through its application.
---
paper_title: Exploring Risk-based Testing and Its Implications 1

paper_content:

If software cannot be tested exhaustively, it must be tested selectively. But, on what should selection be based in order to maximize test effectiveness? It seems sensible to concentrate on the parts of the software where the risks are greatest, but what risks should be sought and how can they be identified and analysed? ‘Risk-based testing’ is a term in current use, for example in an accredited test-practitioner's course syllabus, but there is no broadly accepted definition of the phrase and no literature or body of knowledge to underpin the subject implied by it. Moreover, there has so far been no suggestion that it requires an understanding of the subject of risk. This paper examines what is implied by risk-based testing, shows that its practice requires an understanding of risk, and points to the need for research into the topic and the development of a body of knowledge to underpin it. Copyright © 2004 John Wiley & Sons, Ltd.
---
paper_title: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study

paper_content:

Abstract The idea of risk-based testing is to focus testing and spend more time on critical functions. By combining the focused process with metrics it is possible to manage the test process by intelligent assessment and to communicate the expected consequences of decisions taken. This paper discusses an approach to risk-based testing and how risk-based testing was carried out in a large project in a financial institution. The paper concludes with how practical risk-based testing experience should inform theory and provide advice on organizational requirements that are necessary to achieve success.
---
paper_title: Experiences and Challenges of Introducing Risk-Based Testing in an Industrial Project

paper_content:

Risk-based testing has a high potential to improve the software test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations the integration of risk-based testing into an existing test process is a challenging task. In this paper we present a generic risk-based testing methodology and a procedure how it can be introduced in a test process. Based on this procedure we derive four stages of risk-based test integration, i.e., initial risk-based testing, risk-based test reporting, risk-based test planning, and optimization of risk-based testing. We then discuss how this procedure could be applied based on an industrial project and identify several challenges and lessons learned in introducing risk-based testing.
---
paper_title: Hazard-based selection of test cases

paper_content:

It is a challenge to engineer and select admissable test cases. This work approaches selection based on hazardous states through integrated modelling of the environment and the system, and by integration of requirements engineering activities into the tasks of system test engineering. The elaboration of test models and test case specifications focuses automation in system verification in order to detect hazard-oriented failures as well as in requirements validation in order to disclose potential defects in specification artefacts.
---
paper_title: Risk analysis: a guiding force in the improvement of testing

paper_content:

The authors propose a state-based risk assessment methodology at the analysis and design stage of Software Development Life Cycle. First, a method is proposed to estimate the risk for various states of a component within a scenario and then, the risk for the whole scenario is estimated. The key data needed for risk assessment are complexity and severity. An Inter-Component State-Dependence Graph is introduced to estimate the complexity for a state of a component within a system. The severity for a component within a scenario is decided based on three hazard techniques: Functional Failure Analysis, Software Failure Mode and Effect Analysis and Software Fault Tree Analysis. The risk for a scenario is estimated based on the risk of interacting components in various states within the scenario and State COllaboration TEst Model of the scenario. Finally, the system risk is estimated based on two inputs: scenarios risks and Interaction Overview Diagram of the system. The methodology is applied on a Library Management System case study. An experimental comparative analysis is performed and observed that the testing team guided by our state-based risk assessment approach achieves high test efficiency compared with it with an existing component-based risk assessment approach.
---
paper_title: Automated Risk-Based Testing by Integrating Safety Analysis Information into System Behavior Models

paper_content:

The development of safety-critical software-intensive systems requires systematic quality assurance on all stages of the development process. Executable development artifacts are validated against the system specifications. Risk-based test approaches enable the distribution of test effort in a specific way to cover critical system parts, functions, and requirements. The development process of safety-critical systems usually implies analysis activities for determining and understanding hazards and risks. Moreover, it requires a systematic design of the system structure and behavior based on the specification. For achieving a high degree of automation of test case derivation, existing formal models from the risk analysis and system design phases are combined. The approach presented here focuses on integration of fault trees into state-based behavior models. Therefore, fault trees are analyzed and their elements are assessed for their validity and significance for the test modeling. The approach systematically transforms the relevant fault tree elements like single critical basic events, system states, or sequences of events into elements of the state-based behavior model. The resulting model enables the automated generation of test cases considering risk-based test purposes such as the coverage of critical states, transitions, or sub-models. The feasibility of the approach is shown in a small case study.
---
paper_title: Risk-Based Testing of Safety-Critical Embedded Systems Driven by Fault Tree Analysis

paper_content:

One important aspect of the quality assurance process of safety-critical embedded systems is verifying the appropriateness, correctness of the implementation and effectiveness of safety functions. Due to the rapid growth in complexity, manual verification activities are no longer feasible. This holds especially for testing. A popular method for testing such complex systems is model-based testing. Recent techniques for model-based testing do not sufficiently take into consideration the information derived from the safety analyses like Failure Mode and Effect Analysis and Fault Tree Analyses (FTA). In this paper, we describe an approach to use the results of FTA during the construction of test models, such that test cases can be derived, selected and prioritized according to the severity of the identified risks and the number of basic events that cause it. This approach is demonstrated on an example from the automation domain, namely a modular production system. We find that the method provides a significant increase in coverage of safety functions, compared to regular model based testing.
---
paper_title: Automated Security Test Generation with Formal Threat Models

paper_content:

Security attacks typically result from unintended behaviors or invalid inputs. Security testing is labor intensive because a real-world program usually has too many invalid inputs. It is highly desirable to automate or partially automate security-testing process. This paper presents an approach to automated generation of security tests by using formal threat models represented as Predicate/Transition nets. It generates all attack paths, i.e., security tests, from a threat model and converts them into executable test code according to the given Model-Implementation Mapping (MIM) specification. We have applied this approach to two real-world systems, Magento (a web-based shopping system being used by many online stores) and FileZilla Server (a popular FTP server implementation in C++). Threat models are built systematically by examining all potential STRIDE (spoofing identity, tampering with data, repudiation, information disclosure, denial of service, and elevation of privilege) threats to system functions. The security tests generated from these models have found multiple security risks in each system. The test code for most of the security tests can be generated and executed automatically. To further evaluate the vulnerability detection capability of the testing approach, the security tests have been applied to a number of security mutants where vulnerabilities are injected deliberately. The mutants are created according to the common vulnerabilities in C++ and web applications. Our experiments show that the security tests have killed the majority of the mutants.
---
paper_title: A Systematic Approach to Risk-Based Testing Using Risk-annotated Requirements Models

paper_content:

Nowadays, software-intensive systems continuously pervade several areas of daily life, even critical ones, and replace established mechanical or manual solutions. Development and quality assurance methods have to ensure that these software-intensive systems are delivered both with adequate quality, optimized resources and within the scheduled time frame. The idea of risk-based testing is to prioritize testing activities to what is deemed critical for the software-intensive system. Although there is a common agreement that risk-based testing techniques ought to be rigorously applied, especially for safety- and security-critical systems, there is actually little knowledge available on how to systematically come to risk-optimized test suites. This paper presents a novel approach to risk-based testing that deals with the transition from risk management and requirements engineering to test design activities and test case generation by using models. The main contribution of the paper is the description of a methodology that allows an easy combination of test generation directives and risk level in order to generate risk-optimized test suites.
---
paper_title: Introducing model-based testing in an industrial scrum project

paper_content:

Various approaches for the automated test case generation in the area of graphical user interface (GUI) testing have emerged in recent years. A notable trend is model-based testing (MBT). In this experience report we shed light on the challenges faced during the introduction and every day use of a concrete technique which leverages MBT in a Scrum project along with practical solutions found. Such topics as process of model definition and maintenance for the purposes of regression and risk-based testing of GUIs, suitable test case derivation algorithms, human factors as well as choice of appropriate architecture are discussed.
---
paper_title: An automated technique for risk-based test case generation and prioritization

paper_content:

In practice, available testing budgets limit the number of test cases that can be executed. Thus, a representative subset of all possible test cases must be chosen to guarantee adequate coverage of a test object. In risk-based testing, the probability of a fault and the damage that this fault can cause when leading to a failure is considered for test case prioritization. Existing approaches for risk-based testing provide guidelines for deriving test cases. However, those guidelines lack the level of detail and precision needed for automation. In this contribution, we introduce the risk-based testing technique RiteDAP, which automatically generates system test cases from activity diagrams and prioritizes those test cases based on risk. The results of applying the technique to a practical example are presented and the ability of different prioritization strategies to uncover faults is evaluated.
---
paper_title: A source-based risk analysis approach for software test optimization

paper_content:

In this paper we introduce our proposed technique for software component test prioritization and optimization which is based on a source-code based risk analysis. Software test is one of the most critical steps in the software development. Considering that the time and human resources of a software project are limited, software test should be scheduled and planned very carefully. In this paper we introduce a classification approach that provides the developers with a risk model of the application which is specifically designed to assist the testing process by identifying the most important components and their corresponding test effort estimation. We designed an analyser tool to apply our technique to a test software project and we presented the results in this paper.
---
paper_title: Source code-based software risk assessing

paper_content:

The more complex a software system is, the more likely it is that programmers will make mistakes that introduce faults which can lead to execution failures. A risk in a software system can be viewed as a potential problem, and a problem is a risk that has manifested. In order to reduce the risk of software operations, code which has the potential to cause problems has to be identified so that necessary actions (e.g., performing a more thorough testing on such code) can be taken to prevent any such problems from occurring. Consequently, this can help programmers detect faults in the software before it is deployed and reduce the overall maintenance code. In this paper, we propose a static and a dynamic risk model using metrics collected based on the source code; more specifically, metrics which are either related to the static structure of the source code or the dynamic test coverage of the code. The computation of the risk of code is automated at different granularity levels ranging from basic blocks to functions. An experiment to demonstrate the feasibility of using our method is reported. High risk code, so identified by our method, can be integrated with information collected from other software quality assurance practices to further ensure the safe operation of software applications.
---
paper_title: Enabling the adoption of aspects - testing aspects: a risk model, fault model and patterns

paper_content:

Aspect oriented programming (AOP) has started to achieve industry adoption for custom programs and some adoption in frameworks such as the Spring framework. Aspect oriented programming provides many benefits -- it can increase the scope of concerns that can be captured cleanly, it has explicit language support, and the separation provided by AOP provides an elegant mechanism for custom solutions. In this paper we present a model for AOP testing. This includes a model for risk assessment, an associated fault model and AOP testing patterns. We also propose further opportunities for research in the area for automated AOP risk assessment and testing. At ApTSi™ (Applied Technology Solutions, Inc.) we have been applying AOP in the creation of our SOASense™ framework, and in our consulting engagements. We are seeing adoption typically in classical AOP areas such as Logging, Error Handling, Audit events, etc. In these scenarios, having a reliable AOP implementation is critical. For example, having an Audit event not occur for a service call due to a faulty join-point definition can have severe legal implications. We need a solution that provides reliability, is repeatable and enables us to assess risk.
---
paper_title: Testing Long-Lived Web Services Transactions Using a Risk-Based Approach

paper_content:

Transactions are crucial to ensuring the quality (such as recovery and reliability) of web services applications by constraining them to a mutually agreed outcome. This paper addresses the issue of testing the long-lived web services transactions which has been given little attention by the current research. It proposes a risk-based approach and also defines a set of properties for web services transactions. The proposed approach identifies for each property a set of potential situations that must be tested. We present an analysis for the Recovery property using a Fault Tree diagram where the leaf nodes represent potentially dangerous scenarios that must be checked. Finally we show with a case study how this Fault Tree can be used to derive test cases for web services transactions.
---
paper_title: Risk-Based Adaptive Group Testing of Semantic Web Services

paper_content:

Comprehensive testing is necessary to ensure the quality of complex web services that are loosely coupled, dynamic bound and integrated through standard protocols. Testing of such web services can be however very expensive due to the diversified user requirements and the large numbers of service combinations delivered by the open platform. Group testing was introduced in our previous research as a selective testing technique to reduce test cost and improve test efficiencies. It applies test cases efficiently so that the largest percent of problematic web service is detected as early as possible. The paper proposes a risk-based approach to group test selection. With this approach, test cases are categorized and scheduled with respect to the risks of their target service features. The approach is based on the assumption that for a service-based system, the tolerance to a feature’s failure is an inverse ratio to its risk. The risky features should be tested earlier and with more tests. We specially address the problem in the context of semantic Web Services and report a first attempt for an ontology-based quantitative risk assessment. The paper also discusses risk-based group testing process and strategies for ranking and ruling-out services of the test groups, at each risk level. Runtime monitoring mechanism is incorporated to detect the dynamic changes in service configuration and composition so that the risks can be continuously adjusted online.
---
paper_title: RISK ASSESSMENT AND ADAPTIVE GROUP TESTING OF SEMANTIC WEB SERVICES

paper_content:

Testing is necessary to ensure the quality of web services that are loosely coupled, dynamic bound and integrated through standard protocols. Exhaustive testing of web services is usually impossible due to unavailable source code, diversified user requirements and large number of possible service combinations delivered by the open platform. This paper proposes a risk-based approach for selecting and prioritizing test cases for testing service-based systems. We specially address the problem in the context of semantic web services. Semantic web services introduce semantics to service integration and interoperation using ontology models and specifications. Semantic errors are considered more difficult to detect than syntactic errors. Due to the complexity of conceptual uniformity, it is hard to ensure the completeness, consistency and unified quality of ontology model. A failure of the semantic service-based software may result from many factors such as misused data, unsuccessful service binding, and unexpected usage scenarios. This work analyzes the two factors of risk estimation: failure probability and importance, from three aspects: ontology data, service and composite service. With this approach, test cases are associated to semantic features, and are scheduled based on the risks of their target features. Risk assessment is used to control the process of Web Services progressive group testing, including test case ranking, test case selection and service ruling out. This paper discusses the control architecture and adaptive measurement mechanism for adaptive group testing. As a statistical testing technique, the proposed approach aims to detect, as early as possible, the problems with highest impact on the users.
---
paper_title: Risk-Based Security Testing in Cloud Computing Environments

paper_content:

Assuring the security of a software system in terms of testing nowadays still is a quite tricky task to conduct. Security requirements are taken as a foundation to derive tests to be executed against a system under test. Yet, these positive requirements by far do not cover all the relevant security aspects to be considered. Hence, especially in the event of security testing, negative requirements, derived from risk analysis, are vital to be incorporated. If considering today's emerging trend in the adoption of cloud computing, security testing even has a more important significance. Due to a cloud's openness, in theory there exists an infinite number of tests. Hence, a concise technique to incorporate the results of risk analysis in security testing is inevitable. We therefore propose a new model-driven methodology for the security testing of cloud environments, ingesting misuse cases, defined by negative requirements derived from risk analysis.
---
paper_title: Towards a Model Based Security Testing Approach of Cloud Computing Environments

paper_content:

In recent years Cloud computing became one of the most aggressively emerging computer paradigms resulting in a growing rate of application in the area of IT outsourcing. However, as recent studies have shown, security most of the time is the one requirement, neglected at all. Yet, especially because of the nature of usage of Cloud computing, security is inevitable. Unfortunately, assuring the security of a Cloud computing environment is not a one time task, it is a task to be performed during the complete lifespan of the Cloud. This is motivated by the fact that Clouds undergo daily changes in terms of newly deployed applications and offered services. Based on this assumption, in this paper, we propose a novel model -- based, change -- driven approach, employing risk analysis, to test the security of a Cloud computing environment among all layers. As a main intrusion point, our approach exploits the public service interfaces, as they are a major source of newly introduced vulnerabilities, possibly leading to severe security incidents.
---
paper_title: A Framework to Test Advanced Web Services Transactions

paper_content:

Transactions are a key issue in the reliability of distributed applications because they ensure all the participants achieve a mutually agreed outcome. However, current research has given little attention to testing transactions in web services. This paper presents a conceptual framework, inspired in risk-based methodologies, to address this gap. It also reports on preliminary results and identifies future work.
---
paper_title: Leveraging Risk Based Testing in Enterprise Systems Security Validation

paper_content:

As the security breaches continue to grow, and as project teams become larger and more distributed the cost of testing the security of the software systems will increase rapidly. Companies that take the time to test the security of their software systems before releasing it to production are at a disadvantage because of higher costs and potential late arrival to the market. On the other side of the fence additional financial incentives may come from customers demanding a certain level of security testing from a software supplier, before agreeing to sign a contract to purchase their products. This document discusses and provides information on proven Risk Based Techniques for Software Security Testing that provides the test coverage needed while maintaining the time to market under competitive cost pressures without compromising security.
---
paper_title: Risk-Based Testing: A Case Study

paper_content:

This paper describes the application of risk-based testing for a software product evaluation in a real case study. Risk-based testing consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to their likelihood and impact and test cases are designed based on the strategies for treatment of the identified risk factors. Thus, test efforts are continuously adjusted according to risk monitoring. The paper also briefly reviews available risk-based approaches, describes the benefits that are likely to accrue from the growing body of work in this area and provides a set of problems, challenges and future work.
---
paper_title: Hazard-based selection of test cases

paper_content:

It is a challenge to engineer and select admissable test cases. This work approaches selection based on hazardous states through integrated modelling of the environment and the system, and by integration of requirements engineering activities into the tasks of system test engineering. The elaboration of test models and test case specifications focuses automation in system verification in order to detect hazard-oriented failures as well as in requirements validation in order to disclose potential defects in specification artefacts.
---
paper_title: Risk analysis: a guiding force in the improvement of testing

paper_content:

The authors propose a state-based risk assessment methodology at the analysis and design stage of Software Development Life Cycle. First, a method is proposed to estimate the risk for various states of a component within a scenario and then, the risk for the whole scenario is estimated. The key data needed for risk assessment are complexity and severity. An Inter-Component State-Dependence Graph is introduced to estimate the complexity for a state of a component within a system. The severity for a component within a scenario is decided based on three hazard techniques: Functional Failure Analysis, Software Failure Mode and Effect Analysis and Software Fault Tree Analysis. The risk for a scenario is estimated based on the risk of interacting components in various states within the scenario and State COllaboration TEst Model of the scenario. Finally, the system risk is estimated based on two inputs: scenarios risks and Interaction Overview Diagram of the system. The methodology is applied on a Library Management System case study. An experimental comparative analysis is performed and observed that the testing team guided by our state-based risk assessment approach achieves high test efficiency compared with it with an existing component-based risk assessment approach.
---
paper_title: Measurement and control for risk-based test cases and activities

paper_content:

Risk-based testing is an approach that consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to its likelihood and impact, and the test cases are projected based on the strategies for treatment of the identified risk factors. Then, test efforts are continuously adjusted according the risk monitoring. Most risk-based testing approaches focuses on activities related to risk identification, analysis and prioritizing. However, metrics are fundamental as they quantify characteristics of a process or product and support software project management activities. In this light, this paper proposes and discusses risk-based testing metrics to measure and control test cases and test activities progress, efforts and costs.
---
paper_title: RISK-DRIVEN SOFTWARE TESTING AND RELIABILITY

paper_content:

We develop a risk-driven reliability model and testing process that borrows concepts from classical sequential testing methodology that is used for hardware, with adaptation to software. Both consumer and producer risk are considered, reflecting the fact the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Using the consumer-producer framework, we develop a model and process for executing sequential tests, based on software risk and reliability and model risk and reliability prediction accuracy. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. The test rules are integrated with several levels of criticality of software (i.e., the higher the criticality, the more stringent the tests). Based on our experience, we provide lessons learned with the objective of improving the model and process for future applications. We start the analysis by developing a model and process template based on the Poisson distribution of failures. This hypothetical example allows us to debug the model and process before it is applied to a real application involving the NASA Space Shuttle flight software.
---
paper_title: Risk-Based Testing: A Case Study

paper_content:

This paper describes the application of risk-based testing for a software product evaluation in a real case study. Risk-based testing consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to their likelihood and impact and test cases are designed based on the strategies for treatment of the identified risk factors. Thus, test efforts are continuously adjusted according to risk monitoring. The paper also briefly reviews available risk-based approaches, describes the benefits that are likely to accrue from the growing body of work in this area and provides a set of problems, challenges and future work.
---
paper_title: Risk-Based Adaptive Group Testing of Semantic Web Services

paper_content:

Comprehensive testing is necessary to ensure the quality of complex web services that are loosely coupled, dynamic bound and integrated through standard protocols. Testing of such web services can be however very expensive due to the diversified user requirements and the large numbers of service combinations delivered by the open platform. Group testing was introduced in our previous research as a selective testing technique to reduce test cost and improve test efficiencies. It applies test cases efficiently so that the largest percent of problematic web service is detected as early as possible. The paper proposes a risk-based approach to group test selection. With this approach, test cases are categorized and scheduled with respect to the risks of their target service features. The approach is based on the assumption that for a service-based system, the tolerance to a feature’s failure is an inverse ratio to its risk. The risky features should be tested earlier and with more tests. We specially address the problem in the context of semantic Web Services and report a first attempt for an ontology-based quantitative risk assessment. The paper also discusses risk-based group testing process and strategies for ranking and ruling-out services of the test groups, at each risk level. Runtime monitoring mechanism is incorporated to detect the dynamic changes in service configuration and composition so that the risks can be continuously adjusted online.
---
paper_title: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study

paper_content:

Abstract The idea of risk-based testing is to focus testing and spend more time on critical functions. By combining the focused process with metrics it is possible to manage the test process by intelligent assessment and to communicate the expected consequences of decisions taken. This paper discusses an approach to risk-based testing and how risk-based testing was carried out in a large project in a financial institution. The paper concludes with how practical risk-based testing experience should inform theory and provide advice on organizational requirements that are necessary to achieve success.
---
paper_title: An automated technique for risk-based test case generation and prioritization

paper_content:

In practice, available testing budgets limit the number of test cases that can be executed. Thus, a representative subset of all possible test cases must be chosen to guarantee adequate coverage of a test object. In risk-based testing, the probability of a fault and the damage that this fault can cause when leading to a failure is considered for test case prioritization. Existing approaches for risk-based testing provide guidelines for deriving test cases. However, those guidelines lack the level of detail and precision needed for automation. In this contribution, we introduce the risk-based testing technique RiteDAP, which automatically generates system test cases from activity diagrams and prioritizes those test cases based on risk. The results of applying the technique to a practical example are presented and the ability of different prioritization strategies to uncover faults is evaluated.
---
paper_title: RISK ASSESSMENT AND ADAPTIVE GROUP TESTING OF SEMANTIC WEB SERVICES

paper_content:

Testing is necessary to ensure the quality of web services that are loosely coupled, dynamic bound and integrated through standard protocols. Exhaustive testing of web services is usually impossible due to unavailable source code, diversified user requirements and large number of possible service combinations delivered by the open platform. This paper proposes a risk-based approach for selecting and prioritizing test cases for testing service-based systems. We specially address the problem in the context of semantic web services. Semantic web services introduce semantics to service integration and interoperation using ontology models and specifications. Semantic errors are considered more difficult to detect than syntactic errors. Due to the complexity of conceptual uniformity, it is hard to ensure the completeness, consistency and unified quality of ontology model. A failure of the semantic service-based software may result from many factors such as misused data, unsuccessful service binding, and unexpected usage scenarios. This work analyzes the two factors of risk estimation: failure probability and importance, from three aspects: ontology data, service and composite service. With this approach, test cases are associated to semantic features, and are scheduled based on the risks of their target features. Risk assessment is used to control the process of Web Services progressive group testing, including test case ranking, test case selection and service ruling out. This paper discusses the control architecture and adaptive measurement mechanism for adaptive group testing. As a statistical testing technique, the proposed approach aims to detect, as early as possible, the problems with highest impact on the users.
---
paper_title: Risk-Based Security Testing in Cloud Computing Environments

paper_content:

Assuring the security of a software system in terms of testing nowadays still is a quite tricky task to conduct. Security requirements are taken as a foundation to derive tests to be executed against a system under test. Yet, these positive requirements by far do not cover all the relevant security aspects to be considered. Hence, especially in the event of security testing, negative requirements, derived from risk analysis, are vital to be incorporated. If considering today's emerging trend in the adoption of cloud computing, security testing even has a more important significance. Due to a cloud's openness, in theory there exists an infinite number of tests. Hence, a concise technique to incorporate the results of risk analysis in security testing is inevitable. We therefore propose a new model-driven methodology for the security testing of cloud environments, ingesting misuse cases, defined by negative requirements derived from risk analysis.
---
paper_title: Risk analysis: a guiding force in the improvement of testing

paper_content:

The authors propose a state-based risk assessment methodology at the analysis and design stage of Software Development Life Cycle. First, a method is proposed to estimate the risk for various states of a component within a scenario and then, the risk for the whole scenario is estimated. The key data needed for risk assessment are complexity and severity. An Inter-Component State-Dependence Graph is introduced to estimate the complexity for a state of a component within a system. The severity for a component within a scenario is decided based on three hazard techniques: Functional Failure Analysis, Software Failure Mode and Effect Analysis and Software Fault Tree Analysis. The risk for a scenario is estimated based on the risk of interacting components in various states within the scenario and State COllaboration TEst Model of the scenario. Finally, the system risk is estimated based on two inputs: scenarios risks and Interaction Overview Diagram of the system. The methodology is applied on a Library Management System case study. An experimental comparative analysis is performed and observed that the testing team guided by our state-based risk assessment approach achieves high test efficiency compared with it with an existing component-based risk assessment approach.
---
paper_title: Towards a Model Based Security Testing Approach of Cloud Computing Environments

paper_content:

In recent years Cloud computing became one of the most aggressively emerging computer paradigms resulting in a growing rate of application in the area of IT outsourcing. However, as recent studies have shown, security most of the time is the one requirement, neglected at all. Yet, especially because of the nature of usage of Cloud computing, security is inevitable. Unfortunately, assuring the security of a Cloud computing environment is not a one time task, it is a task to be performed during the complete lifespan of the Cloud. This is motivated by the fact that Clouds undergo daily changes in terms of newly deployed applications and offered services. Based on this assumption, in this paper, we propose a novel model -- based, change -- driven approach, employing risk analysis, to test the security of a Cloud computing environment among all layers. As a main intrusion point, our approach exploits the public service interfaces, as they are a major source of newly introduced vulnerabilities, possibly leading to severe security incidents.
---
paper_title: Measurement and control for risk-based test cases and activities

paper_content:

Risk-based testing is an approach that consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to its likelihood and impact, and the test cases are projected based on the strategies for treatment of the identified risk factors. Then, test efforts are continuously adjusted according the risk monitoring. Most risk-based testing approaches focuses on activities related to risk identification, analysis and prioritizing. However, metrics are fundamental as they quantify characteristics of a process or product and support software project management activities. In this light, this paper proposes and discusses risk-based testing metrics to measure and control test cases and test activities progress, efforts and costs.
---
paper_title: Integrating Manual and Automatic Risk Assessment for Risk-Based Testing

paper_content:

In this paper we define a model-based risk assessment procedure that integrates automatic risk assessment by static analysis, semi-automatic risk assessment and guided manual risk assessment. In this process probability and impact criteria are determined by metrics which are combined to estimate the risk of specific system development artifacts. The risk values are propagated to the assigned test cases providing a prioritization of test cases. This supports to optimize the allocation of limited testing time and budget in a risk-based testing methodology. Therefore, we embed our risk assessment process into a generic risk-based testing methodology. The calculation of probability and impact metrics is based on system and requirements artifacts which are formalized as model elements. Additional time metrics consider the temporal development of the system under test and take for instance the bug and version history of the system into account. The risk assessment procedure integrates several stakeholders and is explained by a running example.
---
paper_title: Automated Security Test Generation with Formal Threat Models

paper_content:

Security attacks typically result from unintended behaviors or invalid inputs. Security testing is labor intensive because a real-world program usually has too many invalid inputs. It is highly desirable to automate or partially automate security-testing process. This paper presents an approach to automated generation of security tests by using formal threat models represented as Predicate/Transition nets. It generates all attack paths, i.e., security tests, from a threat model and converts them into executable test code according to the given Model-Implementation Mapping (MIM) specification. We have applied this approach to two real-world systems, Magento (a web-based shopping system being used by many online stores) and FileZilla Server (a popular FTP server implementation in C++). Threat models are built systematically by examining all potential STRIDE (spoofing identity, tampering with data, repudiation, information disclosure, denial of service, and elevation of privilege) threats to system functions. The security tests generated from these models have found multiple security risks in each system. The test code for most of the security tests can be generated and executed automatically. To further evaluate the vulnerability detection capability of the testing approach, the security tests have been applied to a number of security mutants where vulnerabilities are injected deliberately. The mutants are created according to the common vulnerabilities in C++ and web applications. Our experiments show that the security tests have killed the majority of the mutants.
---
paper_title: A Systematic Approach to Risk-Based Testing Using Risk-annotated Requirements Models

paper_content:

Nowadays, software-intensive systems continuously pervade several areas of daily life, even critical ones, and replace established mechanical or manual solutions. Development and quality assurance methods have to ensure that these software-intensive systems are delivered both with adequate quality, optimized resources and within the scheduled time frame. The idea of risk-based testing is to prioritize testing activities to what is deemed critical for the software-intensive system. Although there is a common agreement that risk-based testing techniques ought to be rigorously applied, especially for safety- and security-critical systems, there is actually little knowledge available on how to systematically come to risk-optimized test suites. This paper presents a novel approach to risk-based testing that deals with the transition from risk management and requirements engineering to test design activities and test case generation by using models. The main contribution of the paper is the description of a methodology that allows an easy combination of test generation directives and risk level in order to generate risk-optimized test suites.
---
paper_title: RISK-DRIVEN SOFTWARE TESTING AND RELIABILITY

paper_content:

We develop a risk-driven reliability model and testing process that borrows concepts from classical sequential testing methodology that is used for hardware, with adaptation to software. Both consumer and producer risk are considered, reflecting the fact the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Using the consumer-producer framework, we develop a model and process for executing sequential tests, based on software risk and reliability and model risk and reliability prediction accuracy. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. The test rules are integrated with several levels of criticality of software (i.e., the higher the criticality, the more stringent the tests). Based on our experience, we provide lessons learned with the objective of improving the model and process for future applications. We start the analysis by developing a model and process template based on the Poisson distribution of failures. This hypothetical example allows us to debug the model and process before it is applied to a real application involving the NASA Space Shuttle flight software.
---
paper_title: Source code-based software risk assessing

paper_content:

The more complex a software system is, the more likely it is that programmers will make mistakes that introduce faults which can lead to execution failures. A risk in a software system can be viewed as a potential problem, and a problem is a risk that has manifested. In order to reduce the risk of software operations, code which has the potential to cause problems has to be identified so that necessary actions (e.g., performing a more thorough testing on such code) can be taken to prevent any such problems from occurring. Consequently, this can help programmers detect faults in the software before it is deployed and reduce the overall maintenance code. In this paper, we propose a static and a dynamic risk model using metrics collected based on the source code; more specifically, metrics which are either related to the static structure of the source code or the dynamic test coverage of the code. The computation of the risk of code is automated at different granularity levels ranging from basic blocks to functions. An experiment to demonstrate the feasibility of using our method is reported. High risk code, so identified by our method, can be integrated with information collected from other software quality assurance practices to further ensure the safe operation of software applications.
---
paper_title: Experiences and Challenges of Introducing Risk-Based Testing in an Industrial Project

paper_content:

Risk-based testing has a high potential to improve the software test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations the integration of risk-based testing into an existing test process is a challenging task. In this paper we present a generic risk-based testing methodology and a procedure how it can be introduced in a test process. Based on this procedure we derive four stages of risk-based test integration, i.e., initial risk-based testing, risk-based test reporting, risk-based test planning, and optimization of risk-based testing. We then discuss how this procedure could be applied based on an industrial project and identify several challenges and lessons learned in introducing risk-based testing.
---
paper_title: Theory and practice of risk-based testing 1

paper_content:

This paper extends a theory of risk-based test planning that was outlined in a previous paper, introduces techniques to facilitate it, explains their practical application, and highlights opportunities for research to enquire into their feasibility and efficacy. Copyright © 2004 John Wiley & Sons, Ltd.
---
paper_title: Integrating Manual and Automatic Risk Assessment for Risk-Based Testing

paper_content:

In this paper we define a model-based risk assessment procedure that integrates automatic risk assessment by static analysis, semi-automatic risk assessment and guided manual risk assessment. In this process probability and impact criteria are determined by metrics which are combined to estimate the risk of specific system development artifacts. The risk values are propagated to the assigned test cases providing a prioritization of test cases. This supports to optimize the allocation of limited testing time and budget in a risk-based testing methodology. Therefore, we embed our risk assessment process into a generic risk-based testing methodology. The calculation of probability and impact metrics is based on system and requirements artifacts which are formalized as model elements. Additional time metrics consider the temporal development of the system under test and take for instance the bug and version history of the system into account. The risk assessment procedure integrates several stakeholders and is explained by a running example.
---
paper_title: Automated Risk-Based Testing by Integrating Safety Analysis Information into System Behavior Models

paper_content:

The development of safety-critical software-intensive systems requires systematic quality assurance on all stages of the development process. Executable development artifacts are validated against the system specifications. Risk-based test approaches enable the distribution of test effort in a specific way to cover critical system parts, functions, and requirements. The development process of safety-critical systems usually implies analysis activities for determining and understanding hazards and risks. Moreover, it requires a systematic design of the system structure and behavior based on the specification. For achieving a high degree of automation of test case derivation, existing formal models from the risk analysis and system design phases are combined. The approach presented here focuses on integration of fault trees into state-based behavior models. Therefore, fault trees are analyzed and their elements are assessed for their validity and significance for the test modeling. The approach systematically transforms the relevant fault tree elements like single critical basic events, system states, or sequences of events into elements of the state-based behavior model. The resulting model enables the automated generation of test cases considering risk-based test purposes such as the coverage of critical states, transitions, or sub-models. The feasibility of the approach is shown in a small case study.
---
paper_title: Risk-Based Testing: A Case Study

paper_content:

This paper describes the application of risk-based testing for a software product evaluation in a real case study. Risk-based testing consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to their likelihood and impact and test cases are designed based on the strategies for treatment of the identified risk factors. Thus, test efforts are continuously adjusted according to risk monitoring. The paper also briefly reviews available risk-based approaches, describes the benefits that are likely to accrue from the growing body of work in this area and provides a set of problems, challenges and future work.
---
paper_title: Enabling the adoption of aspects - testing aspects: a risk model, fault model and patterns

paper_content:

Aspect oriented programming (AOP) has started to achieve industry adoption for custom programs and some adoption in frameworks such as the Spring framework. Aspect oriented programming provides many benefits -- it can increase the scope of concerns that can be captured cleanly, it has explicit language support, and the separation provided by AOP provides an elegant mechanism for custom solutions. In this paper we present a model for AOP testing. This includes a model for risk assessment, an associated fault model and AOP testing patterns. We also propose further opportunities for research in the area for automated AOP risk assessment and testing. At ApTSi™ (Applied Technology Solutions, Inc.) we have been applying AOP in the creation of our SOASense™ framework, and in our consulting engagements. We are seeing adoption typically in classical AOP areas such as Logging, Error Handling, Audit events, etc. In these scenarios, having a reliable AOP implementation is critical. For example, having an Audit event not occur for a service call due to a faulty join-point definition can have severe legal implications. We need a solution that provides reliability, is repeatable and enables us to assess risk.
---
paper_title: Introducing model-based testing in an industrial scrum project

paper_content:

Various approaches for the automated test case generation in the area of graphical user interface (GUI) testing have emerged in recent years. A notable trend is model-based testing (MBT). In this experience report we shed light on the challenges faced during the introduction and every day use of a concrete technique which leverages MBT in a Scrum project along with practical solutions found. Such topics as process of model definition and maintenance for the purposes of regression and risk-based testing of GUIs, suitable test case derivation algorithms, human factors as well as choice of appropriate architecture are discussed.
---
paper_title: Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study

paper_content:

Abstract The idea of risk-based testing is to focus testing and spend more time on critical functions. By combining the focused process with metrics it is possible to manage the test process by intelligent assessment and to communicate the expected consequences of decisions taken. This paper discusses an approach to risk-based testing and how risk-based testing was carried out in a large project in a financial institution. The paper concludes with how practical risk-based testing experience should inform theory and provide advice on organizational requirements that are necessary to achieve success.
---
paper_title: A source-based risk analysis approach for software test optimization

paper_content:

In this paper we introduce our proposed technique for software component test prioritization and optimization which is based on a source-code based risk analysis. Software test is one of the most critical steps in the software development. Considering that the time and human resources of a software project are limited, software test should be scheduled and planned very carefully. In this paper we introduce a classification approach that provides the developers with a risk model of the application which is specifically designed to assist the testing process by identifying the most important components and their corresponding test effort estimation. We designed an analyser tool to apply our technique to a test software project and we presented the results in this paper.
---
paper_title: An automated technique for risk-based test case generation and prioritization

paper_content:

In practice, available testing budgets limit the number of test cases that can be executed. Thus, a representative subset of all possible test cases must be chosen to guarantee adequate coverage of a test object. In risk-based testing, the probability of a fault and the damage that this fault can cause when leading to a failure is considered for test case prioritization. Existing approaches for risk-based testing provide guidelines for deriving test cases. However, those guidelines lack the level of detail and precision needed for automation. In this contribution, we introduce the risk-based testing technique RiteDAP, which automatically generates system test cases from activity diagrams and prioritizes those test cases based on risk. The results of applying the technique to a practical example are presented and the ability of different prioritization strategies to uncover faults is evaluated.
---
paper_title: Risk-Based Security Testing in Cloud Computing Environments

paper_content:

Assuring the security of a software system in terms of testing nowadays still is a quite tricky task to conduct. Security requirements are taken as a foundation to derive tests to be executed against a system under test. Yet, these positive requirements by far do not cover all the relevant security aspects to be considered. Hence, especially in the event of security testing, negative requirements, derived from risk analysis, are vital to be incorporated. If considering today's emerging trend in the adoption of cloud computing, security testing even has a more important significance. Due to a cloud's openness, in theory there exists an infinite number of tests. Hence, a concise technique to incorporate the results of risk analysis in security testing is inevitable. We therefore propose a new model-driven methodology for the security testing of cloud environments, ingesting misuse cases, defined by negative requirements derived from risk analysis.
---
paper_title: Towards a Model Based Security Testing Approach of Cloud Computing Environments

paper_content:

In recent years Cloud computing became one of the most aggressively emerging computer paradigms resulting in a growing rate of application in the area of IT outsourcing. However, as recent studies have shown, security most of the time is the one requirement, neglected at all. Yet, especially because of the nature of usage of Cloud computing, security is inevitable. Unfortunately, assuring the security of a Cloud computing environment is not a one time task, it is a task to be performed during the complete lifespan of the Cloud. This is motivated by the fact that Clouds undergo daily changes in terms of newly deployed applications and offered services. Based on this assumption, in this paper, we propose a novel model -- based, change -- driven approach, employing risk analysis, to test the security of a Cloud computing environment among all layers. As a main intrusion point, our approach exploits the public service interfaces, as they are a major source of newly introduced vulnerabilities, possibly leading to severe security incidents.
---
paper_title: RISK-DRIVEN SOFTWARE TESTING AND RELIABILITY

paper_content:

We develop a risk-driven reliability model and testing process that borrows concepts from classical sequential testing methodology that is used for hardware, with adaptation to software. Both consumer and producer risk are considered, reflecting the fact the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Using the consumer-producer framework, we develop a model and process for executing sequential tests, based on software risk and reliability and model risk and reliability prediction accuracy. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. The test rules are integrated with several levels of criticality of software (i.e., the higher the criticality, the more stringent the tests). Based on our experience, we provide lessons learned with the objective of improving the model and process for future applications. We start the analysis by developing a model and process template based on the Poisson distribution of failures. This hypothetical example allows us to debug the model and process before it is applied to a real application involving the NASA Space Shuttle flight software.
---
paper_title: Source code-based software risk assessing

paper_content:

The more complex a software system is, the more likely it is that programmers will make mistakes that introduce faults which can lead to execution failures. A risk in a software system can be viewed as a potential problem, and a problem is a risk that has manifested. In order to reduce the risk of software operations, code which has the potential to cause problems has to be identified so that necessary actions (e.g., performing a more thorough testing on such code) can be taken to prevent any such problems from occurring. Consequently, this can help programmers detect faults in the software before it is deployed and reduce the overall maintenance code. In this paper, we propose a static and a dynamic risk model using metrics collected based on the source code; more specifically, metrics which are either related to the static structure of the source code or the dynamic test coverage of the code. The computation of the risk of code is automated at different granularity levels ranging from basic blocks to functions. An experiment to demonstrate the feasibility of using our method is reported. High risk code, so identified by our method, can be integrated with information collected from other software quality assurance practices to further ensure the safe operation of software applications.
---
paper_title: Experiences and Challenges of Introducing Risk-Based Testing in an Industrial Project

paper_content:

Risk-based testing has a high potential to improve the software test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations the integration of risk-based testing into an existing test process is a challenging task. In this paper we present a generic risk-based testing methodology and a procedure how it can be introduced in a test process. Based on this procedure we derive four stages of risk-based test integration, i.e., initial risk-based testing, risk-based test reporting, risk-based test planning, and optimization of risk-based testing. We then discuss how this procedure could be applied based on an industrial project and identify several challenges and lessons learned in introducing risk-based testing.
---

Title: Approaches for the combined use of risk analysis and testing: a systematic literature review

Section 1: Introduction
Description 1: Introduce the topic, the importance of combining risk analysis and testing, and the overarching goal of the paper.

Section 2: Terminology
Description 2: Define the technical terms within risk analysis and testing used in the literature survey.

Section 3: Research Process
Description 3: Describe the systematic literature review process conducted, including objectives, research questions, inclusion and exclusion criteria, and the search process.

Section 4: Overview and Classification of the Approaches
Description 4: Provide a summary and classification of the identified approaches, categorized into different groups based on their main focus.

Section 5: Approaches Addressing Combination at a General Level
Description 5: Discuss approaches that address the combination of risk analysis and testing at a general level with specific examples.

Section 6: Approaches with Focus on Model-Based Risk Estimation
Description 6: Describe approaches that focus primarily on model-based risk estimation.

Section 7: Approaches with Focus on Test-Case Generation
Description 7: Focus on approaches that emphasize the automatic generation of test cases from models with risk annotations.

Section 8: Approaches with Focus on Test-Case Analysis
Description 8: Explain approaches that prioritize and select the most critical test cases based on risk analysis.

Section 9: Approaches Based on Automatic Source Code Analysis
Description 9: Discuss approaches that use automatic source code analysis for risk-based testing and test-based risk analysis.

Section 10: Approaches Targeting Specific Programming Paradigms
Description 10: Examine approaches that target specific programming paradigms, such as aspect-oriented programming and object-oriented programming.

Section 11: Approaches Targeting Specific Applications
Description 11: Describe approaches that are designed for specific applications like semantic web services and cloud computing environments.

Section 12: Approaches Aiming at Measurement
Description 12: Focus on approaches that aim at measuring risk-based testing and test-based risk analysis, including the metrics used for evaluation.

Section 13: Relationships Between Publications
Description 13: Analyze the inter-relationships between the approaches based on citations and references among the publications included in the survey.

Section 14: Discussion
Description 14: Discuss the findings with respect to the research questions, including common strategies, level of formality, and tool support.

Section 15: Conclusions
Description 15: Summarize the key findings of the survey, identify gaps in the current research, and suggest areas for future research.

Section 16: Appendix: Search Strings
Description 16: Provide the detailed search strings used during the literature search process.