Variability in quality attributes of service-based software systems: A systematic literature review
---
paper_title: Variability management in software product lines: an investigation of contemporary industrial challenges

paper_content:

Variability management is critical for achieving the large scale reuse promised by the software product line paradigm. It has been studied for almost 20 years. We assert that it is important to explore how well the body of knowledge of variability management solves the challenges faced by industrial practitioners, and what are the remaining and (or) emerging challenges. To gain such understanding of the challenges of variability management faced by practitioners, we have conducted an empirical study using focus group as data collection method. The results of the study highlight several technical challenges that are often faced by practitioners in their daily practices. Different from previous studies, the results also reveal and shed light on several non-technical challenges that were almost neglected by existing research.
---
paper_title: Systematic literature reviews in software engineering - A tertiary study

paper_content:

Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to provide an annotated catalogue of SLRs available to software engineering researchers and practitioners. This study updates our previous study using a broad automated search. Method: We performed a broad automated search to find SLRs published in the time period 1st January 2004 to 30th June 2008. We contrast the number, quality and source of these SLRs with SLRs found in the original study. Results: Our broad search found an additional 35 SLRs corresponding to 33 unique studies. Of these papers, 17 appeared relevant to the undergraduate educational curriculum and 12 appeared of possible interest to practitioners. The number of SLRs being published is increasing. The quality of papers in conferences and workshops has improved as more researchers use SLR guidelines. Conclusion: SLRs appear to have gone past the stage of being used solely by innovators but cannot yet be considered a main stream software engineering research methodology. They are addressing a wide range of topics but still have limitations, such as often failing to assess primary study quality.
---
paper_title: Systematic Review in Software Engineering

paper_content:

A kit of assemblable components for implantation into the bone of a mammal for use in distraction osteogenesis. The kit comprises a fixture, a footing and a distracter, the fixture including a longitudinally extending body portion with a proximal end and a distal end, the body portion having an exterior surface adapted for contact and integration with bone tissue, the body portion having a generally longitudinally extending bore extending from a proximal opening adjacent the proximal end to a distal opening adjacent the distal end. The footing includes a proximal surface and a distal surface. The distracter comprises a generally rod-shaped body including a distal end and a proximal end, and the proximal end of the distracter is adapted to bear against the footing. There are first and second engaging means on the fixture and the distracter respectively for adjustably locating the fixture relative to the distracter.
---
paper_title: An Overview of Software Engineering Approaches to Service Oriented Architectures in Various Fields

paper_content:

For the last few years, a rise has been observed in research activity in Service Oriented Architectures, with applications in different sectors. Several new technologies have been introduced and even more are being currently researched and aimed to the future. In this paper we present and analyze some of the most influential approaches from a software engineer’s point of view that belong either to the academic or to the industrial field. Despite their differences though, all of these approaches share a service oriented mentality, with the purpose of lessening the issues of clients and companies, students and teachers, citizens and government employees alike. Lastly, we discuss our findings from the comparison and present possible new research opportunities for the immediate future.
---
paper_title: Exploring service-oriented system engineering challenges: a systematic literature review

paper_content:

Service-oriented system engineering (SOSE) has drawn increasing attention since service-oriented computing was introduced in the beginning of this decade. A large number of SOSE challenges that call for special software engineering efforts have been proposed in the research community. Our goal is to gain insight into the current status of SOSE research issues as published to date. To this end, we conducted a systematic literature review exploring SOSE challenges that have been claimed between January 2000 and July 2008. This paper presents the results of the systematic review as well as the empirical research method we followed. In this review, of the 729 publications that have been examined, 51 were selected as primary studies, from which more than 400 SOSE challenges were elicited. By applying qualitative data analysis methods to the extracted data from the review, we proved our hypotheses about the classification scheme. We are able to conclude that the SOSE challenges can be classified along two dimensions: (a) based on themes (or topics) that they cover and (b) based on characteristics (or types) that they reveal. By analyzing the distribution of the SOSE challenges on the topics and types in the years 2000–2008, we are able to point out the trend in SOSE research activities. The findings of this review further provide empirical evidence for establishing future SOSE research agendas.
---
paper_title: Systematic literature reviews in software engineering – A systematic literature review

paper_content:

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.
---
paper_title: Experiences conducting systematic reviews from novices' perspective

paper_content:

A systematic review (SR) is a sound methodology for collecting evidence on a research topic of interest and establishing the context of future research. Unlike ordinary or even expert literature reviews, SRs are systematic thus increasing the confidence in the findings from the previous published literature. SRs can be carried out by both experienced and novice researchers; however, while expert researchers? experiences with conducting SRs are important for improving the SR body of knowledge, we believe that novice researchers? experiences are equally important to establish what distinct problems they face while carrying out SRs. With a prior knowledge of these issues, novice researchers can better plan their SRs and seek guidance from expert researchers.
---
paper_title: Variability management in software product lines: a systematic review

paper_content:

Variability Management (VM) in Software Product Line (SPL) is a key activity that usually affects the degree to which a SPL is successful. SPL community has spent huge amount of resources on developing various approaches to dealing with variability related challenges over the last decade. To provide an overview of different aspects of the proposed VM approaches, we carried out a systematic literature review of the papers reporting VM in SPL. This paper presents and discusses the findings from this systematic literature review. The results reveal the chronological backgrounds of various approaches over the history of VM research, and summarize the key issues that drove the evolution of different approaches. This study has also identified several gaps that need to be filled by future efforts in this line of research.
---
paper_title: Variability management in software product lines: an investigation of contemporary industrial challenges

paper_content:

Variability management is critical for achieving the large scale reuse promised by the software product line paradigm. It has been studied for almost 20 years. We assert that it is important to explore how well the body of knowledge of variability management solves the challenges faced by industrial practitioners, and what are the remaining and (or) emerging challenges. To gain such understanding of the challenges of variability management faced by practitioners, we have conducted an empirical study using focus group as data collection method. The results of the study highlight several technical challenges that are often faced by practitioners in their daily practices. Different from previous studies, the results also reveal and shed light on several non-technical challenges that were almost neglected by existing research.
---
paper_title: An Overview of Software Engineering Approaches to Service Oriented Architectures in Various Fields

paper_content:

For the last few years, a rise has been observed in research activity in Service Oriented Architectures, with applications in different sectors. Several new technologies have been introduced and even more are being currently researched and aimed to the future. In this paper we present and analyze some of the most influential approaches from a software engineer’s point of view that belong either to the academic or to the industrial field. Despite their differences though, all of these approaches share a service oriented mentality, with the purpose of lessening the issues of clients and companies, students and teachers, citizens and government employees alike. Lastly, we discuss our findings from the comparison and present possible new research opportunities for the immediate future.
---
paper_title: Exploring service-oriented system engineering challenges: a systematic literature review

paper_content:

Service-oriented system engineering (SOSE) has drawn increasing attention since service-oriented computing was introduced in the beginning of this decade. A large number of SOSE challenges that call for special software engineering efforts have been proposed in the research community. Our goal is to gain insight into the current status of SOSE research issues as published to date. To this end, we conducted a systematic literature review exploring SOSE challenges that have been claimed between January 2000 and July 2008. This paper presents the results of the systematic review as well as the empirical research method we followed. In this review, of the 729 publications that have been examined, 51 were selected as primary studies, from which more than 400 SOSE challenges were elicited. By applying qualitative data analysis methods to the extracted data from the review, we proved our hypotheses about the classification scheme. We are able to conclude that the SOSE challenges can be classified along two dimensions: (a) based on themes (or topics) that they cover and (b) based on characteristics (or types) that they reveal. By analyzing the distribution of the SOSE challenges on the topics and types in the years 2000–2008, we are able to point out the trend in SOSE research activities. The findings of this review further provide empirical evidence for establishing future SOSE research agendas.
---
paper_title: Software Architecture in Practice

paper_content:

The award-winning and highly influential Software Architecture in Practice, Third Edition, has been substantially revised to reflect the latest developments in the field. In a real-world setting, the book once again introduces the concepts and best practices of software architecturehow a software system is structured and how that systems elements are meant to interact. Distinct from the details of implementation, algorithm, and data representation, an architecture holds the key to achieving system quality, is a reusable asset that can be applied to subsequent systems, and is crucial to a software organizations business strategy. The authors have structured this edition around the concept of architecture influence cycles. Each cycle shows how architecture influences, and is influenced by, a particular context in which architecture plays a critical role. Contexts include technical environment, the life cycle of a project, an organizations business profile, and the architects professional practices. The authors also have greatly expanded their treatment of quality attributes, which remain central to their architecture philosophywith an entire chapter devoted to each attributeand broadened their treatment of architectural patterns. If you design, develop, or manage large software systems (or plan to do so), you will find this book to be a valuable resource for getting up to speed on the state of the art. Totally new material covers Contexts of software architecture: technical, project, business, and professional Architecture competence: what this means both for individuals and organizations The origins of business goals and how this affects architecture Architecturally significant requirements, and how to determine them Architecture in the life cycle, including generate-and-test as a design philosophy; architecture conformance during implementation; architecture and testing; and architecture and agile development Architecture and current technologies, such as the cloud, social networks, and end-user devices
---
paper_title: Quality Attributes for Service-Oriented Architectures

paper_content:

The SOA approach is a very popular choice today for the implementation of distributed systems. The use of SOA or more specifically the Web services technology is an important architecture decision. An architect should understand how different quality attributes for a system are impacted by that decision. While there are significant benefits with respect to interoperability and modifiability, other qualities such as performance, security and testability are concerns. This paper discusses how the different quality attributes of a system can be positively or negatively affected by the use of such technology. It describes the factors related to each attribute, as well as possible tradeoffs and existing efforts to achieve that quality. The paper also discusses open issues in service level agreements that are used to contract the level of service quality between service providers and users.
---
paper_title: Managing Variation in Services in a Software Product Line Context

paper_content:

Abstract : Software product line (SPL) and service-oriented architecture (SOA) approaches both enable an organization to reuse existing assets and capabilities rather than repeatedly redeveloping them for new systems. Organizations can capitalize on such reuse in software-reliant systems to achieve business goals such as productivity gains, decreased development costs, improved time to market, increased reliability, increased agility, and competitive advantage. Both approaches accommodate variation in the software that is being reused or the way in which it is employed. Meeting business goals through a product line or a set of service-oriented systems requires managing the variation of assets, including services. This report examines combining existing SOA and software product line approaches for variation management. This examination has two objectives: (1) for service-oriented systems development, to present an approach for managing variation by identifying and designing services explicitly targeted to multiple service-oriented systems, and (2) for SPL systems, to present an approach for managing variation where services are a mechanism for variation within a product line or for expanding the product line scope.
---
paper_title: Quality Attributes and Service-Oriented Architectures

paper_content:

Abstract : This report examines the relationship between service-oriented architectures (SOAs) and quality attributes. Because software architecture is the bridge between mission/business goals and a software-intensive system, and quality attribute requirements drive software architecture design, it is important to understand how SOAs support these requirements. This report gives a short introduction to SOAs and outlines some of the main business goals that may lead an organization to choose an SOA to design and implement a system. The report outlines a set of quality attributes that may be derived from an organization's business goals and examines how those attributes relate to an SOA. In addition, the report describes how the SOA impacts those attributes and how choosing an SOA can help an organization achieve its business goals.
---
paper_title: Requirements and Tools for Variability Management

paper_content:

Explicit and software-supported Business Process Management has become the core infrastructure of any medium and large organization that has a need to be efficient and effective. The number of processes of a single organization can be very high, furthermore, they might be very similar, be in need of momentary change, or evolve frequently. If the adhoc adaptation and customization of processes is currently the dominant way, it clearly is not the best. In fact, providing tools for supporting the explicit management of variation in processes (due to customization or evolution needs) has a profound impact on the overall life-cycle of processes in organizations. Additionally, with the increasing adoption of Service-Oriented Architectures, the infrastructure to support automatic reconfiguration and adaptation of business process is solid. In this paper, after defining variability in business process management, we consider the requirements for explicit variation handling for (service based) business process systems. eGovernment serves as an illustrative example of reuse. In this case study, all local municipalities need to implement the same general legal process while adapting it to the local business practices and IT infrastructure needs. Finally, an evaluation of existing tools for explicit variability management is provided with respect to the requirements identified.
---
paper_title: The Landscape of Service-Oriented Systems: A Research Perspective

paper_content:

Service orientation has been touted as one of the most important technologies for designing, implementing and deploying large scale service provision software systems. In this position paper we attempt to investigate an initial classification of challenge areas related to service orientation and service-oriented systems. We start by organizing the research issues related to service orientation in three general categories- business, engineering and operations, plus a set of cross-cutting concerns across domain. We further propose the notion of Service Strategy as a binding model for these three categories. Finally, concluding this position paper, we outline a set of emerging opportunities to be used for further discussion.
---
paper_title: A systematic review of quality attributes and measures for software product lines

paper_content:

It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e.g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines.
---
paper_title: Issues in the Design of Flexible and Dynamic Service-Oriented Systems

paper_content:

Due to the use of the concepts embedded in Service- Oriented Architecture (SOA), software design, now more than ever, involves the use of incomplete information. Applications that utilizeWeb Services are also highly impacted by the problem of deployment and subsequent undeployment of services. Specifically, there is a level of uncertainty caused by the potential for services to become unavailable (either temporarily or permanently). In a scenario where an application must switch from one service to another due to the undeployment problem, the client application may require that new or different handlers be used to cope with the properties of the alternative service. In this current development climate, the design issues become clear: there is a need to reason about how a design is impacted by discovered services, design analysis must consider the transaction and event properties of discovered services, and design of systems must incorporate fault tolerance and high-integrity issues to cope with the dynamic landscape caused by the uncertainty associated with using services.
---
paper_title: Testing in Service Oriented Architectures with Dynamic Binding: A Mapping Study

paper_content:

Context: Service Oriented Architectures (SOA) have emerged as a new paradigm to develop interoperable and highly dynamic applications. Objective: This paper aims to identify the state of the art in the research on testing in Service Oriented Architectures with dynamic binding. Method: A mapping study has been performed employing both manual and automatic search in journals, conference/workshop proceedings and electronic databases. Results: A total of 33 studies have been reviewed in order to extract relevant information regarding a previously defined set of research questions. The detection of faults and the decision making based on the information gathered from the tests have been identified as the main objectives of these studies. To achieve these goals, monitoring and test case generation are the most proposed techniques testing both functional and non-functional properties. Furthermore, different stakeholders have been identified as participants in the tests, which are performed in specific points in time during the life cycle of the services. Finally, it has been observed that a relevant group of studies have not validated their approach yet. Conclusions: Although we have only found 33 studies that address the testing of SOA where the discovery and binding of the services are performed at runtime, this number can be considered significant due to the specific nature of the reviewed topic. The results of this study have contributed to provide a body of knowledge that allows identifying current gaps in improving the quality of the dynamic binding in SOA using testing approaches.
---
paper_title: Adaptation of service-based systems

paper_content:

The advances in modern technology development and future technology changes dictate new challenges and requirements to the engineering and provision of services and service-based systems (SBS). These services and systems should become drastically more flexible; they should be able to operate and evolve in highly dynamic environments and to adequately react to various changes in these environments. In these settings, adaptability becomes a key feature of services as it provides a way for an application to continuously change itself in order to satisfy new contextual requirements.
---
paper_title: Exploring service-oriented system engineering challenges: a systematic literature review

paper_content:

Service-oriented system engineering (SOSE) has drawn increasing attention since service-oriented computing was introduced in the beginning of this decade. A large number of SOSE challenges that call for special software engineering efforts have been proposed in the research community. Our goal is to gain insight into the current status of SOSE research issues as published to date. To this end, we conducted a systematic literature review exploring SOSE challenges that have been claimed between January 2000 and July 2008. This paper presents the results of the systematic review as well as the empirical research method we followed. In this review, of the 729 publications that have been examined, 51 were selected as primary studies, from which more than 400 SOSE challenges were elicited. By applying qualitative data analysis methods to the extracted data from the review, we proved our hypotheses about the classification scheme. We are able to conclude that the SOSE challenges can be classified along two dimensions: (a) based on themes (or topics) that they cover and (b) based on characteristics (or types) that they reveal. By analyzing the distribution of the SOSE challenges on the topics and types in the years 2000–2008, we are able to point out the trend in SOSE research activities. The findings of this review further provide empirical evidence for establishing future SOSE research agendas.
---
paper_title: Systematic literature reviews in software engineering – A systematic literature review

paper_content:

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.
---
paper_title: The Landscape of Service-Oriented Systems: A Research Perspective

paper_content:

Service orientation has been touted as one of the most important technologies for designing, implementing and deploying large scale service provision software systems. In this position paper we attempt to investigate an initial classification of challenge areas related to service orientation and service-oriented systems. We start by organizing the research issues related to service orientation in three general categories- business, engineering and operations, plus a set of cross-cutting concerns across domain. We further propose the notion of Service Strategy as a binding model for these three categories. Finally, concluding this position paper, we outline a set of emerging opportunities to be used for further discussion.
---
paper_title: Variability management in software product lines: a systematic review

paper_content:

Variability Management (VM) in Software Product Line (SPL) is a key activity that usually affects the degree to which a SPL is successful. SPL community has spent huge amount of resources on developing various approaches to dealing with variability related challenges over the last decade. To provide an overview of different aspects of the proposed VM approaches, we carried out a systematic literature review of the papers reporting VM in SPL. This paper presents and discusses the findings from this systematic literature review. The results reveal the chronological backgrounds of various approaches over the history of VM research, and summarize the key issues that drove the evolution of different approaches. This study has also identified several gaps that need to be filled by future efforts in this line of research.
---
paper_title: The Goal Question Metric Approach

paper_content:

As with any engineering discipline, software development requires a measurement mechanism for feedback and evaluation. Measurement is a mechanism for creating a corporate memory and an aid in answering a variety of questions associated with the enactment of any software process. It helps support project planning (e.g., How much will a new project cost?); it allows us to determine the strengths and weaknesses of the current processes and products (e.g., What is the frequency of certain types of errors?); it provides a rationale for adopting/refining techniques (e.g., What is the impact of the technique XX on the productivity of the projects?); it allows us to evaluate the quality of specific processes and products (e.g., What is the defect density in a specific system after deployment?). Measurement also helps, during the course of a project, to assess its progress, to take corrective action based on this assessment, and to evaluate the impact of such action.
---
paper_title: On searching relevant studies in software engineering

paper_content:

BACKGROUND: Systematic Literature Review (SLR) has become an important research methodology in software engineering since 2004. One critical step in applying this methodology is to design and execute appropriate and effective search strategy. This is quite time consuming and error-prone step, which needs to be carefully planned and implemented. There is an apparent need of a systematic approach to designing, executing, and evaluating a suitable search strategy for optimally retrieving the target literature from digital libraries. ::: ::: OBJECTIVE: The main objective of the research reported in this paper is to improve the search step of doing SLRs in SE by devising and evaluating systematic and practical approaches to identifying relevant studies in SE. ::: ::: OUTCOMES: We have systematically selected and analytically studied a large number of papers to understand the state-of-the-practice of search strategies in EBSE. Having identified the limitations of the current ad-hoc nature of search strategies used by SE researchers for SLR, we have devised a systematic approach to developing and executing optimal search strategies in SLRs. The proposed approach incorporates the concept of 'quasi-gold standard', which consists of collection of known studies and corresponding 'quasi-sensitivity' into the search process for evaluating search performance. We report the case study and its finding to demonstrate that the approach is able to improve the rigor of search process in an SLR, and can serves as the supplements to the guidelines for SLRs in EBSE. We plan to further evaluate the proposed approach using several case studies with varying topics in software engineering.
---
paper_title: Exploring service-oriented system engineering challenges: a systematic literature review

paper_content:

Service-oriented system engineering (SOSE) has drawn increasing attention since service-oriented computing was introduced in the beginning of this decade. A large number of SOSE challenges that call for special software engineering efforts have been proposed in the research community. Our goal is to gain insight into the current status of SOSE research issues as published to date. To this end, we conducted a systematic literature review exploring SOSE challenges that have been claimed between January 2000 and July 2008. This paper presents the results of the systematic review as well as the empirical research method we followed. In this review, of the 729 publications that have been examined, 51 were selected as primary studies, from which more than 400 SOSE challenges were elicited. By applying qualitative data analysis methods to the extracted data from the review, we proved our hypotheses about the classification scheme. We are able to conclude that the SOSE challenges can be classified along two dimensions: (a) based on themes (or topics) that they cover and (b) based on characteristics (or types) that they reveal. By analyzing the distribution of the SOSE challenges on the topics and types in the years 2000–2008, we are able to point out the trend in SOSE research activities. The findings of this review further provide empirical evidence for establishing future SOSE research agendas.
---
paper_title: Variability management in software product lines: a systematic review

paper_content:

Variability Management (VM) in Software Product Line (SPL) is a key activity that usually affects the degree to which a SPL is successful. SPL community has spent huge amount of resources on developing various approaches to dealing with variability related challenges over the last decade. To provide an overview of different aspects of the proposed VM approaches, we carried out a systematic literature review of the papers reporting VM in SPL. This paper presents and discusses the findings from this systematic literature review. The results reveal the chronological backgrounds of various approaches over the history of VM research, and summarize the key issues that drove the evolution of different approaches. This study has also identified several gaps that need to be filled by future efforts in this line of research.
---
paper_title: A systematic review of comparative evidence of aspect-oriented programming

paper_content:

Context: Aspect-oriented programming (AOP) promises to improve many facets of software quality by providing better modularization and separation of concerns, which may have system wide affect. There have been numerous claims in favor and against AOP compared with traditional programming languages such as Objective Oriented and Structured Programming Languages. However, there has been no attempt to systematically review and report the available evidence in the literature to support the claims made in favor or against AOP compared with non-AOP approaches. Objective: This research aimed to systematically identify, analyze, and report the evidence published in the literature to support the claims made in favor or against AOP compared with non-AOP approaches. Method: We performed a systematic literature review of empirical studies of AOP based development, published in major software engineering journals and conference proceedings. Results: Our search strategy identified 3307 papers, of which 22 were identified as reporting empirical studies comparing AOP with non-AOP approaches. Based on the analysis of the data extracted from those 22 papers, our findings show that for performance, code size, modularity, and evolution related characteristics, a majority of the studies reported positive effects, a few studies reported insignificant effects, and no study reported negative effects; however, for cognition and language mechanism, negative effects were reported. Conclusion: AOP is likely to have positive effect on performance, code size, modularity, and evolution. However its effect on cognition and language mechanism is less likely to be positive. Care should be taken using AOP outside the context in which it has been validated.
---
paper_title: On searching relevant studies in software engineering

paper_content:

BACKGROUND: Systematic Literature Review (SLR) has become an important research methodology in software engineering since 2004. One critical step in applying this methodology is to design and execute appropriate and effective search strategy. This is quite time consuming and error-prone step, which needs to be carefully planned and implemented. There is an apparent need of a systematic approach to designing, executing, and evaluating a suitable search strategy for optimally retrieving the target literature from digital libraries. ::: ::: OBJECTIVE: The main objective of the research reported in this paper is to improve the search step of doing SLRs in SE by devising and evaluating systematic and practical approaches to identifying relevant studies in SE. ::: ::: OUTCOMES: We have systematically selected and analytically studied a large number of papers to understand the state-of-the-practice of search strategies in EBSE. Having identified the limitations of the current ad-hoc nature of search strategies used by SE researchers for SLR, we have devised a systematic approach to developing and executing optimal search strategies in SLRs. The proposed approach incorporates the concept of 'quasi-gold standard', which consists of collection of known studies and corresponding 'quasi-sensitivity' into the search process for evaluating search performance. We report the case study and its finding to demonstrate that the approach is able to improve the rigor of search process in an SLR, and can serves as the supplements to the guidelines for SLRs in EBSE. We plan to further evaluate the proposed approach using several case studies with varying topics in software engineering.
---
paper_title: Exploring service-oriented system engineering challenges: a systematic literature review

paper_content:

Service-oriented system engineering (SOSE) has drawn increasing attention since service-oriented computing was introduced in the beginning of this decade. A large number of SOSE challenges that call for special software engineering efforts have been proposed in the research community. Our goal is to gain insight into the current status of SOSE research issues as published to date. To this end, we conducted a systematic literature review exploring SOSE challenges that have been claimed between January 2000 and July 2008. This paper presents the results of the systematic review as well as the empirical research method we followed. In this review, of the 729 publications that have been examined, 51 were selected as primary studies, from which more than 400 SOSE challenges were elicited. By applying qualitative data analysis methods to the extracted data from the review, we proved our hypotheses about the classification scheme. We are able to conclude that the SOSE challenges can be classified along two dimensions: (a) based on themes (or topics) that they cover and (b) based on characteristics (or types) that they reveal. By analyzing the distribution of the SOSE challenges on the topics and types in the years 2000–2008, we are able to point out the trend in SOSE research activities. The findings of this review further provide empirical evidence for establishing future SOSE research agendas.
---
paper_title: Refining the systematic literature review process—two participant-observer case studies

paper_content:

Systematic literature reviews (SLRs) are a major tool for supporting evidence-based software engineering. Adapting the procedures involved in such a review to meet the needs of software engineering and its literature remains an ongoing process. As part of this process of refinement, we undertook two case studies which aimed 1) to compare the use of targeted manual searches with broad automated searches and 2) to compare different methods of reaching a consensus on quality. For Case 1, we compared a tertiary study of systematic literature reviews published between January 1, 2004 and June 30, 2007 which used a manual search of selected journals and conferences and a replication of that study based on a broad automated search. We found that broad automated searches find more studies than manual restricted searches, but they may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers, or they are assessing research trends in research methodologies. For Case 2, we analyzed the process used to evaluate the quality of SLRs. We conclude that if quality evaluation of primary studies is a critical component of a specific SLR, assessments should be based on three independent evaluators incorporating at least two rounds of discussion.
---
paper_title: A systematic review of comparative evidence of aspect-oriented programming

paper_content:

Context: Aspect-oriented programming (AOP) promises to improve many facets of software quality by providing better modularization and separation of concerns, which may have system wide affect. There have been numerous claims in favor and against AOP compared with traditional programming languages such as Objective Oriented and Structured Programming Languages. However, there has been no attempt to systematically review and report the available evidence in the literature to support the claims made in favor or against AOP compared with non-AOP approaches. Objective: This research aimed to systematically identify, analyze, and report the evidence published in the literature to support the claims made in favor or against AOP compared with non-AOP approaches. Method: We performed a systematic literature review of empirical studies of AOP based development, published in major software engineering journals and conference proceedings. Results: Our search strategy identified 3307 papers, of which 22 were identified as reporting empirical studies comparing AOP with non-AOP approaches. Based on the analysis of the data extracted from those 22 papers, our findings show that for performance, code size, modularity, and evolution related characteristics, a majority of the studies reported positive effects, a few studies reported insignificant effects, and no study reported negative effects; however, for cognition and language mechanism, negative effects were reported. Conclusion: AOP is likely to have positive effect on performance, code size, modularity, and evolution. However its effect on cognition and language mechanism is less likely to be positive. Care should be taken using AOP outside the context in which it has been validated.
---
paper_title: Generalizing a Model of Software Architecture Design from Five Industrial Approaches

paper_content:

We compare five industrial software architecture design methods and we extract from their commonalities a general software architecture design approach. Using this general approach, we compare across the five methods the artifacts and activities they use or recommend, and we pinpoint similarities and differences. Once we get beyond the great variance in terminology and description, we find that the 5 approaches have a lot in common and match more or less the "ideal" pattern we introduced.
---
paper_title: Requirements Engineering for Software Product Lines: A Systematic Literature Review

paper_content:

Context: Software product line engineering (SPLE) is a growing area showing promising results in research and practice. In order to foster its further development and acceptance in industry, it is necessary to assess the quality of the research so that proper evidence for adoption and validity are ensured. This holds in particular for requirements engineering (RE) within SPLE, where a growing number of approaches have been proposed. Objective: This paper focuses on RE within SPLE and has the following goals: assess research quality, synthesize evidence to suggest important implications for practice, and identify research trends, open problems, and areas for improvement. Method: A systematic literature review was conducted with three research questions and assessed 49 studies, dated from 1990 to 2009. Results: The evidence for adoption of the methods is not mature, given the primary focus on toy examples. The proposed approaches still have serious limitations in terms of rigor, credibility, and validity of their findings. Additionally, most approaches still lack tool support addressing the heterogeneity and mostly textual nature of requirements formats as well as address only the proactive SPLE adoption strategy. Conclusions: Further empirical studies should be performed with sufficient rigor to enhance the body of evidence in RE within SPLE. In this context, there is a clear need for conducting studies comparing alternative methods. In order to address scalability and popularization of the approaches, future research should be invested in tool support and in addressing combined SPLE adoption strategies.
---
paper_title: Applying Systematic Reviews to Diverse Study Types: An Experience Report

paper_content:

Systematic reviews are one of the key building blocks of evidence-based software engineering. Current guidelines for such reviews are, for a large part, based on standard meta-analytic techniques. However, such quantitative techniques have only limited applicability to software engineering research. In this paper, therefore, we describe our experience with an approach to combine diverse study types in a systematic review of empirical research of agile software development.
---
paper_title: A systematic review of evaluation of variability management approaches in software product lines

paper_content:

ContextVariability management (VM) is one of the most important activities of software product-line engineering (SPLE), which intends to develop software-intensive systems using platforms and mass customization. VM encompasses the activities of eliciting and representing variability in software artefacts, establishing and managing dependencies among different variabilities, and supporting the exploitation of the variabilities for building and evolving a family of software systems. Software product line (SPL) community has allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, there has been no systematic effort to study how the reported VM approaches have been evaluated. ObjectiveThe objectives of this research are to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches. MethodWe carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007. ResultsWe selected 97 papers according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues. We found that only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. A detailed investigation of the reviewed studies employing empirical research methods revealed significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects. ConclusionThe findings from this systematic review show that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. However, the reported evidence is quite consistent across different studies. That means the proposed approaches may be very beneficial when they are applied properly in appropriate situations. Hence, it can be concluded that further investigations need to pay more attention to the contexts under which different approaches can be more beneficial.
---
paper_title: Increasing business flexibility and SOA adoption through effective SOA governance

paper_content:

Most organizations understand the need to address service-oriented architecture (SOA) governance during SOA adoption. An abundance of information is available defining SOA governance: what it is and what it is not, why it is important, and why organizational change must be addressed. Increasingly business and information technology (IT) stakeholders, executive and technical, acknowledge that SOA governance is essential for realizing the benefits of SOA adoption: building more-flexible IT architectures, improving the fusion between business and IT models, and making business processes more flexible and reusable. However, what is not clear is how an organization gets started. What works and what does not work? More importantly, what is required in SOA governance for organizations to see sustained and realized benefits? This paper describes a framework, the SOA governance model, that can be used to scope and identify what is required for effective SOA governance. Based on client experiences, we describe four approaches to getting started with SOA governance, and we describe how to use these four approaches to make shared services (services used by two or more consumers), reuse, and flexibility a reality. We also discuss lessons learned in using these four approaches.
---
paper_title: Variability management in software product lines: a systematic review

paper_content:

Variability Management (VM) in Software Product Line (SPL) is a key activity that usually affects the degree to which a SPL is successful. SPL community has spent huge amount of resources on developing various approaches to dealing with variability related challenges over the last decade. To provide an overview of different aspects of the proposed VM approaches, we carried out a systematic literature review of the papers reporting VM in SPL. This paper presents and discusses the findings from this systematic literature review. The results reveal the chronological backgrounds of various approaches over the history of VM research, and summarize the key issues that drove the evolution of different approaches. This study has also identified several gaps that need to be filled by future efforts in this line of research.
---
paper_title: A method for evaluating rigor and industrial relevance of technology evaluations

paper_content:

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.
---
paper_title: Variability management in software product lines: a systematic review

paper_content:

Variability Management (VM) in Software Product Line (SPL) is a key activity that usually affects the degree to which a SPL is successful. SPL community has spent huge amount of resources on developing various approaches to dealing with variability related challenges over the last decade. To provide an overview of different aspects of the proposed VM approaches, we carried out a systematic literature review of the papers reporting VM in SPL. This paper presents and discusses the findings from this systematic literature review. The results reveal the chronological backgrounds of various approaches over the history of VM research, and summarize the key issues that drove the evolution of different approaches. This study has also identified several gaps that need to be filled by future efforts in this line of research.
---
paper_title: A systematic review of evaluation of variability management approaches in software product lines

paper_content:

ContextVariability management (VM) is one of the most important activities of software product-line engineering (SPLE), which intends to develop software-intensive systems using platforms and mass customization. VM encompasses the activities of eliciting and representing variability in software artefacts, establishing and managing dependencies among different variabilities, and supporting the exploitation of the variabilities for building and evolving a family of software systems. Software product line (SPL) community has allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, there has been no systematic effort to study how the reported VM approaches have been evaluated. ObjectiveThe objectives of this research are to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches. MethodWe carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007. ResultsWe selected 97 papers according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues. We found that only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. A detailed investigation of the reviewed studies employing empirical research methods revealed significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects. ConclusionThe findings from this systematic review show that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. However, the reported evidence is quite consistent across different studies. That means the proposed approaches may be very beneficial when they are applied properly in appropriate situations. Hence, it can be concluded that further investigations need to pay more attention to the contexts under which different approaches can be more beneficial.
---
paper_title: A method for evaluating rigor and industrial relevance of technology evaluations

paper_content:

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.
---
paper_title: Mathematical notation in formal specification: too difficult for the masses?

paper_content:

The phrase "not much mathematics required" can imply a variety of skill levels. When this phrase is applied to computer scientists, software engineers, and clients in the area of formal specification, the word "much" can be widely misinterpreted with disastrous consequences. A small experiment in reading specifications revealed that students already trained in discrete mathematics and the specification notation performed very poorly; much worse than could reasonably be expected if formal methods proponents are to be believed.
---
paper_title: A method for evaluating rigor and industrial relevance of technology evaluations

paper_content:

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.
---
paper_title: A method for evaluating rigor and industrial relevance of technology evaluations

paper_content:

One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor. In addition, the research field does not show any improvements in terms of industrial relevance over time.
---
paper_title: On searching relevant studies in software engineering

paper_content:

BACKGROUND: Systematic Literature Review (SLR) has become an important research methodology in software engineering since 2004. One critical step in applying this methodology is to design and execute appropriate and effective search strategy. This is quite time consuming and error-prone step, which needs to be carefully planned and implemented. There is an apparent need of a systematic approach to designing, executing, and evaluating a suitable search strategy for optimally retrieving the target literature from digital libraries. ::: ::: OBJECTIVE: The main objective of the research reported in this paper is to improve the search step of doing SLRs in SE by devising and evaluating systematic and practical approaches to identifying relevant studies in SE. ::: ::: OUTCOMES: We have systematically selected and analytically studied a large number of papers to understand the state-of-the-practice of search strategies in EBSE. Having identified the limitations of the current ad-hoc nature of search strategies used by SE researchers for SLR, we have devised a systematic approach to developing and executing optimal search strategies in SLRs. The proposed approach incorporates the concept of 'quasi-gold standard', which consists of collection of known studies and corresponding 'quasi-sensitivity' into the search process for evaluating search performance. We report the case study and its finding to demonstrate that the approach is able to improve the rigor of search process in an SLR, and can serves as the supplements to the guidelines for SLRs in EBSE. We plan to further evaluate the proposed approach using several case studies with varying topics in software engineering.
---
paper_title: Systematic literature reviews in software engineering - A tertiary study

paper_content:

Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to provide an annotated catalogue of SLRs available to software engineering researchers and practitioners. This study updates our previous study using a broad automated search. Method: We performed a broad automated search to find SLRs published in the time period 1st January 2004 to 30th June 2008. We contrast the number, quality and source of these SLRs with SLRs found in the original study. Results: Our broad search found an additional 35 SLRs corresponding to 33 unique studies. Of these papers, 17 appeared relevant to the undergraduate educational curriculum and 12 appeared of possible interest to practitioners. The number of SLRs being published is increasing. The quality of papers in conferences and workshops has improved as more researchers use SLR guidelines. Conclusion: SLRs appear to have gone past the stage of being used solely by innovators but cannot yet be considered a main stream software engineering research methodology. They are addressing a wide range of topics but still have limitations, such as often failing to assess primary study quality.
---
paper_title: A systematic review of evaluation of variability management approaches in software product lines

paper_content:

ContextVariability management (VM) is one of the most important activities of software product-line engineering (SPLE), which intends to develop software-intensive systems using platforms and mass customization. VM encompasses the activities of eliciting and representing variability in software artefacts, establishing and managing dependencies among different variabilities, and supporting the exploitation of the variabilities for building and evolving a family of software systems. Software product line (SPL) community has allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, there has been no systematic effort to study how the reported VM approaches have been evaluated. ObjectiveThe objectives of this research are to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches. MethodWe carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007. ResultsWe selected 97 papers according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues. We found that only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. A detailed investigation of the reviewed studies employing empirical research methods revealed significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects. ConclusionThe findings from this systematic review show that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. However, the reported evidence is quite consistent across different studies. That means the proposed approaches may be very beneficial when they are applied properly in appropriate situations. Hence, it can be concluded that further investigations need to pay more attention to the contexts under which different approaches can be more beneficial.
---
paper_title: Requirements Engineering for Software Product Lines: A Systematic Literature Review

paper_content:

Context: Software product line engineering (SPLE) is a growing area showing promising results in research and practice. In order to foster its further development and acceptance in industry, it is necessary to assess the quality of the research so that proper evidence for adoption and validity are ensured. This holds in particular for requirements engineering (RE) within SPLE, where a growing number of approaches have been proposed. Objective: This paper focuses on RE within SPLE and has the following goals: assess research quality, synthesize evidence to suggest important implications for practice, and identify research trends, open problems, and areas for improvement. Method: A systematic literature review was conducted with three research questions and assessed 49 studies, dated from 1990 to 2009. Results: The evidence for adoption of the methods is not mature, given the primary focus on toy examples. The proposed approaches still have serious limitations in terms of rigor, credibility, and validity of their findings. Additionally, most approaches still lack tool support addressing the heterogeneity and mostly textual nature of requirements formats as well as address only the proactive SPLE adoption strategy. Conclusions: Further empirical studies should be performed with sufficient rigor to enhance the body of evidence in RE within SPLE. In this context, there is a clear need for conducting studies comparing alternative methods. In order to address scalability and popularization of the approaches, future research should be invested in tool support and in addressing combined SPLE adoption strategies.
---
paper_title: Systematic literature reviews in software engineering – A systematic literature review

paper_content:

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.
---

Title: Variability in Quality Attributes of Service-Based Software Systems: A Systematic Literature Review

Section 1: Introduction
Description 1: Introduce the concept of variability and its significance, especially in the context of service-based software systems. State the objective of the paper.

Section 2: Background
Description 2: Provide definitions and descriptions of service-based systems, quality attributes, and variability as used in the paper.

Section 2.1: Service-based systems and quality attributes
Description 2.1: Describe the principles of service-orientation and the adopted definitions of quality attributes.

Section 2.2: Variability
Description 2.2: Explain the concept of variability, its introduction through variation points, and differentiate between design time and runtime variability.

Section 2.3: Lack of existing reviews
Description 2.3: Highlight the existing gap in systematic reviews specifically focusing on variability in quality attributes of service-based systems.

Section 3: Paper goal and contributions
Description 3: Articulate the specific goals of the paper following the Goal-Question-Metric (GQM) perspectives and identify the target audience.

Section 4: Paper structure
Description 4: Provide an overview of the paper's organization and structure.

Section 5: Research method
Description 5: Describe the systematic literature review method used, its significance, and the protocol followed.

Section 5.1: Research questions
Description 5.1: List and explain the research questions that the review seeks to answer.

Section 5.2: Search strategy
Description 5.2: Detail the search strategy used to identify relevant studies, including preliminary searches, reviews, trial searches, and expert consultation.

Section 5.3: Search method
Description 5.3: Explain the method of conducting an automatic search using search strings on electronic data sources and manual searches to form a "quasi-gold" standard.

Section 5.4: Search terms for automatic search
Description 5.4: Outline the steps and terms used to generate search strings for automatic searches.

Section 5.5: Search scope and sources to be searched
Description 5.5: Define the scope of the search in terms of publication period and source.

Section 5.6: Search process
Description 5.6: Describe the stages of the study selection process, from searching databases to final inclusion/exclusion decisions.

Section 5.7: Inclusion and exclusion criteria
Description 5.7: List the inclusion and exclusion criteria used to filter relevant studies.

Section 5.8: Quality criteria
Description 5.8: Explain the quality criteria used to evaluate the papers, relating to rationale, context, research design, data analysis, findings, bias, and credibility.

Section 5.9: Data collection
Description 5.9: Describe the data extraction process and the use of data extraction forms for analysis.

Section 5.10: Data analysis
Description 5.10: Explain how data were summarized, synthetized, and analyzed using descriptive statistics and frequency analysis.

Section 6: Results and analysis
Description 6: Present the findings of the systematic review in response to the research questions.

Section 6.1: Results overview and demographics
Description 6.1: Provide an overview of the identified studies and highlight trends in published papers.

Section 6.2: RQ1: What quality attributes do existing methods for variability in quality attributes of service-based systems handle?
Description 6.2: Analyze the quality attributes addressed by the identified studies, identifying common sets and specific domains.

Section 6.3: RQ2: What software development activities are addressed by existing methods for handling variability in quality attributes of service-based systems?
Description 6.3: Evaluate and categorize the software development activities considered by the studies.

Section 6.4: RQ3: What solution types are used by methods to handle variability in quality attributes of service-based systems?
Description 6.4: Identify and classify the types of solutions used in the proposed methods.

Section 6.5: RQ4: What evidence is available to adopt proposed methods for handling variability in quality attributes of service-based systems?
Description 6.5: Assess the credibility of the studies based on citation counts, quality scores, evidence levels, and evaluation approaches.

Section 6.6: RQ5: Are methods only applicable to variability of design-time or run-time quality attributes?
Description 6.6: Determine the focus of the methods on design-time or runtime quality attributes.

Section 6.7: RQ6: Is there support for practitioners concerning how to use current methods?
Description 6.7: Evaluate the support available for practitioners in terms of practical implementations and tool support.

Section 7: Discussion of results
Description 7: Summarize the main findings, discuss limitations, and address threats to validity.

Section 7.1: Focus on certain quality attributes
Description 7.1: Discuss the emphasis on specific quality attributes in current research.

Section 7.2: Impact of product line engineering
Description 7.2: Reflect on the influence of product line engineering on current variability methods.

Section 7.3: Poor evidence of proposed methods
Description 7.3: Evaluate the evidence supporting proposed methods, highlighting gaps in industrial evidence.

Section 7.4: Implications for practitioners and researchers
Description 7.4: Discuss the relevance of the findings for both academic and industrial audiences.

Section 7.5: Research direction for future work
Description 7.5: Suggest future research directions based on the identified gaps and limitations.

Section 7.6: Inaccuracy and bias in selected papers for review
Description 7.6: Address potential inaccuracies and biases during the paper selection process.

Section 7.7: Inaccuracy and bias in data extraction
Description 7.7: Discuss measures taken to mitigate bias and inaccuracy in data extraction.

Section 7.8: Deviations from the procedures for systematic reviews
Description 7.8: Note deviations from standard systematic review procedures and their implications.

Section 7.9: Evaluation of review
Description 7.9: Evaluate the systematic review based on predefined quality questions.

Section 8: Conclusions
Description 8: Summarize the conclusions of the systematic review, highlighting key findings and recommendations for future research.