A systematic literature review of open source software quality assessment models
---
paper_title: The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation

paper_content:

Software quality evaluation has always been an important part of software business. The quality evaluation process is usually based on hierarchical quality models that measure various aspects of software quality and deduce a characterization of the product quality being evaluated. The particular nature of open source software has rendered existing models inappropriate for detailed quality evaluations. In this paper, we present a hierarchical quality model that evaluates source code and community processes, based on automatic calculation of metric values and their correlation to a set of predefined quality profiles.1
---
paper_title: The QualOSS open source assessment model measuring the performance of open source communities

paper_content:

Open Source Software (OSS) has an increasing importance for the software industry. Similar to traditional (closed) software acquisition, OSS acquisition requires an assessment of whether its quality is sufficient for the intended purpose, and whether the chances of being maintained and supported in the future, as well as of keeping certain quality standards over time, are sufficiently high. In this paper, we present an approach toward a comprehensive measurement framework for OSS projects, developed in the EU project QualOSS. This approach takes into account product quality as well as process maturity and sustainability of the underlying OSS community.
---
paper_title: A Review of Software Quality Models for the Evaluation of Software Products

paper_content:

Actually, software products are increasing in a fast way and are used in almost all activities of human life. Consequently measuring and evaluating the quality of a software product has become a critical task for many companies. Several models have been proposed to help diverse types of users with quality issues. The development of techniques for building software has influenced the creation of models to assess the quality. Since 2000 the construction of software started to depend on generated or manufactured components and gave rise to new challenges for assessing quality. These components introduce new concepts such as configurability, reusability, availability, better quality and lower cost. Consequently the models are classified in basic models which were developed until 2000, and those based on components called tailored quality models. The purpose of this article is to describe the main models with their strengths and point out some deficiencies. In this work, we conclude that in the present age, aspects of communications play an important factor in the quality of software
---
paper_title: Quantitative Quality Model for Evaluating Open Source Web Applications: Case Study of Repository Software

paper_content:

Many open source web applications exist today and universities also find them useful. For instance, universities now manage most of their research output by storing them in their respective institutional repositories. These repositories are often built as open source web applications and known as repository software. Several of these exist but three popular ones include: DSpace, EPrints and Greenstone (DEG). These three are open source and built by different institutions. Considering their increasing adoption and usage by universities today, it would be useful to have a model that can compare between the quality of two or more web applications and suggest the better option to an institution intending to adopt one. This paper therefore proposes a model for measuring quality in open source web applications (focusing on repository software) by adapting existing quality models. The proposed model is used to measure quality in DEG. The proposed model is validated through real data and the results presented and discussed. Overall, the model rated DSpace as the better option.
---
paper_title: Predicting Software Product Quality: A Systematic Mapping Study

paper_content:

Predicting software product quality (SPQ)is becoming a permanent concern during software lifecycle phases. In this paper, a systematic mapping studywas performed to summarize the existing SPQ prediction(SPQP) approaches in literature and to organizethe selected studies according to seven classificationcriteria: SPQP approaches, research types, empiricaltypes, data sets used in the empirical evaluation of thesestudies, artifacts, SQ models, and SQ characteristics.Publication channels and trends were also identified.After identifying 182 documents in ACM Digital Library,IEEE Xplore, ScienceDirect, SpringerLink, and Googlescholar, 69 papers were selected. The results showthat the main publication source of the papers identifiedwas conference. Data mining techniques are themost frequently SPQP approaches reported in literature.Solution proposal was the main research type identified.The majority of the papers selected were history-basedevaluations using existing data which were mainly obtainedfrom open source software projects and domainspecific projects. Source code was the main artifactconcerned with SPQP approaches. Well-known SQmodels were hardly mentioned and reliability is the SQcharacteristic through which SPQP was mainly achieved.SPQP-related subject seems to need more investigationfrom researchers and practitioners. Moreover, SQ modelsand standards need to be considered more in futureSPQP research.
---
paper_title: Software quality models: Purposes, usage scenarios and requirements

paper_content:

Software quality models are a well-accepted means to support quality management of software systems. Over the last 30 years, a multitude of quality models have been proposed and applied with varying degrees of success. Despite successes and standardisation efforts, quality models are still being criticised, as their application in practice exhibits various problems. To some extent, this criticism is caused by an unclear definition of what quality models are and which purposes they serve. Beyond this, there is a lack of explicitly stated requirements for quality models with respect to their intended mode of application. To remedy this, this paper describes purposes and usage scenarios of quality models and, based on the literature and experiences from the authors, collects critique of existing models. From this, general requirements for quality models are derived. The requirements can be used to support the evaluation of existing quality models for a given context or to guide further quality model development.
---
paper_title: A Study on Maturity Model of Open Source Software Community to Estimate the Quality of Products

paper_content:

Abstract Open Source Software (OSS) is a new paradigm to develop software by community, in which groups of developer collaborating each other. The method becomes very popular because the products of OSS projects, such as source code, documents, results of tests, are published by open-license. One of noticeable features of OSS is openness of project. Anyone can access the products of OSS projects. It is expected that the quality of products could be higher than those developed by conventional methods, as more people access OSS products and the chance to find defect could be larger. However, there are no established methods for the evaluation of OSS, neither actual terms of the evaluation of OSS. We propose an evaluation method, which is based on the maturity model of OSS development community.
---
paper_title: Evaluating Software Product Quality: A Systematic Mapping Study

paper_content:

Evaluating software product quality (SPQ) is an important task to ensure the quality of software products. In this paper a systematic mapping study was performed to summarize the existing SPQ evaluation (SPQE) approaches in literature and to classify the selected studies according to seven classification criteria: SPQE approaches, research types, empirical types, data sets used in the empirical evaluation of these studies, artifacts, SQ models, and SQ characteristics. Publication channels and trends were also identified. 57 papers were selected. The results show that the main publication sources of the papers identified were journals. Data mining techniques are the most frequently approaches reported in literature. Solution proposals were the main research type identified. The majority of the selected papers were history-based evaluations using existing data, which were mainly obtained from open source software projects and domain specific projects. Source code was the main artifacts used by SPQE approaches. Well-known SQ models were mentioned by half of the selected papers and reliability is the SQ characteristic through which SPQE was mainly achieved. SPQE-related subjects seem to attract more interest from researchers since the past years.
---
paper_title: Challenges in using open source software in product development: a review of the literature

paper_content:

Component-Based Software Development has become a popular approach to building software intensive systems. Besides using Commercial Off-The-Shelf components, an organization may choose to use Open Source Software components. Using OSS has been reported to have many benefits, but there are also challenges involved. Understanding the potential challenges of using OSS in developing products is important for practitioners, so they become aware of them and can anticipate them and take appropriate measures to address these challenges. We have performed a thorough review of the literature to identify challenges that may arise, as reported in the literature. This paper presents and discusses these findings. Researchers can discuss potential causes and solutions of our synthesized findings as well as benefit from provided references to literature on OSS challenges as input for future research.
---
paper_title: Quantitative Quality Model for Evaluating Open Source Web Applications: Case Study of Repository Software

paper_content:

Many open source web applications exist today and universities also find them useful. For instance, universities now manage most of their research output by storing them in their respective institutional repositories. These repositories are often built as open source web applications and known as repository software. Several of these exist but three popular ones include: DSpace, EPrints and Greenstone (DEG). These three are open source and built by different institutions. Considering their increasing adoption and usage by universities today, it would be useful to have a model that can compare between the quality of two or more web applications and suggest the better option to an institution intending to adopt one. This paper therefore proposes a model for measuring quality in open source web applications (focusing on repository software) by adapting existing quality models. The proposed model is used to measure quality in DEG. The proposed model is validated through real data and the results presented and discussed. Overall, the model rated DSpace as the better option.
---
paper_title: Quantitative Quality Model for Evaluating Open Source Web Applications: Case Study of Repository Software

paper_content:

Many open source web applications exist today and universities also find them useful. For instance, universities now manage most of their research output by storing them in their respective institutional repositories. These repositories are often built as open source web applications and known as repository software. Several of these exist but three popular ones include: DSpace, EPrints and Greenstone (DEG). These three are open source and built by different institutions. Considering their increasing adoption and usage by universities today, it would be useful to have a model that can compare between the quality of two or more web applications and suggest the better option to an institution intending to adopt one. This paper therefore proposes a model for measuring quality in open source web applications (focusing on repository software) by adapting existing quality models. The proposed model is used to measure quality in DEG. The proposed model is validated through real data and the results presented and discussed. Overall, the model rated DSpace as the better option.
---
paper_title: Systematic mapping studies in software engineering

paper_content:

BACKGROUND: A software engineering systematic map is a defined method to build a classification scheme and structure a software engineering field of interest. The analysis of results focuses on frequencies of publications for categories within the scheme. Thereby, the coverage of the research field can be determined. Different facets of the scheme can also be combined to answer more specific research questions. ::: ::: OBJECTIVE: We describe how to conduct a systematic mapping study in software engineering and provide guidelines. We also compare systematic maps and systematic reviews to clarify how to chose between them. This comparison leads to a set of guidelines for systematic maps. ::: ::: METHOD: We have defined a systematic mapping process and applied it to complete a systematic mapping study. Furthermore, we compare systematic maps with systematic reviews by systematically analyzing existing systematic reviews. ::: ::: RESULTS: We describe a process for software engineering systematic mapping studies and compare it to systematic reviews. Based on this, guidelines for conducting systematic maps are defined. ::: ::: CONCLUSIONS: Systematic maps and reviews are different in terms of goals, breadth, validity issues and implications. Thus, they should be used complementarily and require different methods (e.g., for analysis).
---
paper_title: Systematic literature review of machine learning based software development effort estimation models

paper_content:

Context: Software development effort estimation (SDEE) is the process of predicting the effort required to develop a software system. In order to improve estimation accuracy, many researchers have proposed machine learning (ML) based SDEE models (ML models) since 1990s. However, there has been no attempt to analyze the empirical evidence on ML models in a systematic way. Objective: This research aims to systematically analyze ML models from four aspects: type of ML technique, estimation accuracy, model comparison, and estimation context. Method: We performed a systematic literature review of empirical studies on ML model published in the last two decades (1991-2010). Results: We have identified 84 primary studies relevant to the objective of this research. After investigating these studies, we found that eight types of ML techniques have been employed in SDEE models. Overall speaking, the estimation accuracy of these ML models is close to the acceptable level and is better than that of non-ML models. Furthermore, different ML models have different strengths and weaknesses and thus favor different estimation contexts. Conclusion: ML models are promising in the field of SDEE. However, the application of ML models in industry is still limited, so that more effort and incentives are needed to facilitate the application of ML models. To this end, based on the findings of this review, we provide recommendations for researchers as well as guidelines for practitioners.
---
paper_title: An open source usability maturity model (OS-UMM)

paper_content:

User satisfaction has always been a major factor in the success of software, regardless of whether it is closed proprietary or open source software (OSS). In open source projects, usability aspects cannot be improved unless there are ways to test and measure them. Hence, the increasing popularity of open source projects among novice and non-technical users necessitates a usability evaluation methodology. Consequently, this paper presents a usability maturity model specifically aimed at usability-related issues for open source projects. In particular, the model examines the degree of coordination between open source projects and their usability aspects. The measuring instrument of the model contains factors that have been selected from four of our empirical studies, which examine the perspectives of OSS users, developers, contributors and the industry. In addition to presenting the usability maturity model, this paper discusses assessment questionnaires, a rating methodology and two case studies.
---
paper_title: Challenges in using open source software in product development: a review of the literature

paper_content:

Component-Based Software Development has become a popular approach to building software intensive systems. Besides using Commercial Off-The-Shelf components, an organization may choose to use Open Source Software components. Using OSS has been reported to have many benefits, but there are also challenges involved. Understanding the potential challenges of using OSS in developing products is important for practitioners, so they become aware of them and can anticipate them and take appropriate measures to address these challenges. We have performed a thorough review of the literature to identify challenges that may arise, as reported in the literature. This paper presents and discusses these findings. Researchers can discuss potential causes and solutions of our synthesized findings as well as benefit from provided references to literature on OSS challenges as input for future research.
---
paper_title: The Application of DEA to Measure the Efficiency of Open Source Security Tool Production

paper_content:

There are a wide variety of open source security tools available for deployment within the enterprise. Despite the success of many security-based open source software (OSS) projects, large numbers of these projects become inactive and are eventually abandoned. The purpose of this research is to develop an empirical study to determine the relative efficiency of security-based OSS projects. A security-based OSS projects utilize two key resources in their development, the number of software developers and the number of users submitting bugs found when using OSS code. This research develops a model to measure the relative efficiency of each project and determines the number of inefficient projects benchmarking each efficient project. The result of this research is a model that can be used by project developers to evaluate the relative efficiency of their projects. Our empirical study will be based on the analysis of publicly available data from a repository of OSS project at sourceforge.net.
---
paper_title: Applying EFFORT for evaluating CRM open source systems

paper_content:

Free Open Source Software (FIOSS) for Customer relationship management (CRM) represents a great opportunity for small and medium enterprises as they can significantly impact their competitiveness. These software solutions can be adopted with success, whatever the size, and offer customized solutions for clients, even with few people working of it. This paper presents an analysis of the quality of CRM FIOSS systems using a predefined quality model. The analysis focuses on the main relevant aspect of both open source domain and application domain proposing a trade off assessment among these.
---
paper_title: An open source usability maturity model (OS-UMM)

paper_content:

User satisfaction has always been a major factor in the success of software, regardless of whether it is closed proprietary or open source software (OSS). In open source projects, usability aspects cannot be improved unless there are ways to test and measure them. Hence, the increasing popularity of open source projects among novice and non-technical users necessitates a usability evaluation methodology. Consequently, this paper presents a usability maturity model specifically aimed at usability-related issues for open source projects. In particular, the model examines the degree of coordination between open source projects and their usability aspects. The measuring instrument of the model contains factors that have been selected from four of our empirical studies, which examine the perspectives of OSS users, developers, contributors and the industry. In addition to presenting the usability maturity model, this paper discusses assessment questionnaires, a rating methodology and two case studies.
---
paper_title: Quality evaluation of floss projects: Application to ERP systems

paper_content:

Abstract Context The selection and adoption of open source software can significantly influence the competitiveness of organisations. Open source software solutions offer great opportunities for cost reduction and quality improvement, especially for small and medium enterprises that typically have to address major difficulties due to the limited resources available for selecting and adopting a new software system. Objective This paper aims to provide support for selecting the open source software that is most suitable to the specific needs of an enterprise from among the options offering equivalent or overlapping functionality. Method This paper proposes a framework for evaluating the quality and functionality of open source software systems. The name of the framework is EFFORT (Evaluation Framework for Free/Open souRce projecTs). It supports the evaluation of product quality, community trustworthiness and product attractiveness. The framework needs to be customised to the analysis of software systems for a specific context. Results The paper presents the customisation of EFFORT for evaluating Enterprise Resource Planning (ERP) open source software systems. The customised framework was applied to the evaluation and comparison of five ERP open source software systems. The results obtained permitted both the refinement of the measurement framework and the identification of the ERP open source software system that achieved the highest score for each chosen characteristic. Conclusion EFFORT is a useful tool for evaluating and selecting an open source software system. It may significantly reduce the amount of negotiation conducted among an enterprise’s members and reduce the time and cost required for gathering and interpreting data. The EFFORT framework also considers the users’ opinions by introducing relevance markers associated with the metrics and questions in the data aggregation process.
---
paper_title: Operationalised product quality models and assessment: The Quamoco approach

paper_content:

ContextSoftware quality models provide either abstract quality characteristics or concrete quality measurements; there is no seamless integration of these two aspects. Quality assessment approaches are, hence, also very specific or remain abstract. Reasons for this include the complexity of quality and the various quality profiles in different domains which make it difficult to build operationalised quality models. ObjectiveIn the project Quamoco, we developed a comprehensive approach aimed at closing this gap. MethodThe project combined constructive research, which involved a broad range of quality experts from academia and industry in workshops, sprint work and reviews, with empirical studies. All deliverables within the project were peer-reviewed by two project members from a different area. Most deliverables were developed in two or three iterations and underwent an evaluation. ResultsWe contribute a comprehensive quality modelling and assessment approach: (1) A meta quality model defines the structure of operationalised quality models. It includes the concept of a product factor, which bridges the gap between concrete measurements and abstract quality aspects, and allows modularisation to create modules for specific domains. (2) A largely technology-independent base quality model reduces the effort and complexity of building quality models for specific domains. For Java and C# systems, we refined it with about 300 concrete product factors and 500 measures. (3) A concrete and comprehensive quality assessment approach makes use of the concepts in the meta-model. (4) An empirical evaluation of the above results using real-world software systems showed: (a) The assessment results using the base model largely match the expectations of experts for the corresponding systems. (b) The approach and models are well understood by practitioners and considered to be both consistent and well suited for getting an overall view on the quality of a software product. The validity of the base quality model could not be shown conclusively, however. (5) The extensive, open-source tool support is in a mature state. (6) The model for embedded software systems is a proof-of-concept for domain-specific quality models. ConclusionWe provide a broad basis for the development and application of quality models in industrial practice as well as a basis for further extension, validation and comparison with other approaches in research.
---
paper_title: State of art surveys of overviews on MCDM/MADM methods

paper_content:

AbstractDecision-making is primarily a process that involves different actors: people, groups of people, institutions and the state. As a discipline, multi-criteria decision-making has a relatively short history. Since 1950s and 1960s, when foundations of modern multi-criteria decision-making methods have been laid, many researches devoted their time to development of new multi-criteria decision-making models and techniques. In the past decades, researches and development in the field have accelerated and seem to continue growing exponentially. Despite the intensive development worldwide, few attempts have been made to systematically present the theoretical bases and developments of multi-criteria decision-making methods. However, the methodological choices and framework for assessment of decisions are still under discussion. The article describes the situation with reviews of MCDM/MADM methods. Furthermore, there is a need for research to study the strengths and weaknesses of different decision-making me...
---
paper_title: Evaluating the Quality of Software in e-Book Using the ISO 9126 Model

paper_content:

Electronic Book or e-Book has become a new medium in education. An important consideration of e-Book in the education system is the quality of software that are shipped with it. The term ‘e-Book’ is ambiguous at best, and has been used interchangeably with netbook, mini laptop and e-reader to name a few. This paper is interested in determining an appropriate model for evaluating the quality of software in e-Book. It is argued that in order to reap the best possible outcome from e-Book (and on-line learning), quality of the software used should be of ‘acceptable’ standard. Past research efforts are reviewed and a comparison is made to identify best quality ‘characteristics’ that should be used in the evaluation of e-Book. An extension of the ISO 9126 model for assessment is proposed, categorizing five quality characteristics for e-Book.
---
paper_title: Challenges in using open source software in product development: a review of the literature

paper_content:

Component-Based Software Development has become a popular approach to building software intensive systems. Besides using Commercial Off-The-Shelf components, an organization may choose to use Open Source Software components. Using OSS has been reported to have many benefits, but there are also challenges involved. Understanding the potential challenges of using OSS in developing products is important for practitioners, so they become aware of them and can anticipate them and take appropriate measures to address these challenges. We have performed a thorough review of the literature to identify challenges that may arise, as reported in the literature. This paper presents and discusses these findings. Researchers can discuss potential causes and solutions of our synthesized findings as well as benefit from provided references to literature on OSS challenges as input for future research.
---
paper_title: Evaluating software quality in use using user reviews mining

paper_content:

Reviews of software from experienced users play an important role for software acquisition decision. In order to share their experiences, an online software recommendation system has been developed. This information is not only useful for users or customers, but it is also be used for evaluating the software. Since there are many of reviews are accumulated and expressed in both formal and informal written languages, it takes time for concluding the evaluation. Therefore, we are interested in an automatically process to extract software information attributes from the reviews in order to provide software review representation. One essential problem is the different sentiment of the same sentence in different environment. To solve this problem, rule-based classification is used as our machine learning model. In this research, software quality extracted from user perspective with respect to ISO 9126 is selected to be the characteristic model. We also propose a methodology for a software product reviews mining based on software quality ontology and a product software quality in use scores for software review representation. Our classification approach is applied from two lists of sentiment words (positive and negative words) combining with rule-based classification method. Our result yields four percent of the accuracy improvement from simple classification applied only two lists of sentiment words.
---
paper_title: The evaluation of Learning Management Systems using an artificial intelligence fuzzy logic algorithm

paper_content:

There are many open source and commercially available Learning Management System (LMS) on the Internet and one of the important problems in this field is how to choose an LMS that will be the most effective one and that will satisfy the requirements. In order to help in the solution of this problem, the author has developed a computer program to aid in the selection of an LMS. The developed system is web-based and can easily be used over the Internet any where over the world at any time. The developed system is basically a web-based decision support system used to evaluate LMSs by using a flexible and smart algorithm derived from artificial intelligent concepts with fuzzy logic values. The paper describes the development of the LMS evaluation system. The individuals who are most likely to be interested in the LMS evaluation process are teachers, students, and any educational organizations such as: universities, schools, institutes, and anyone else who seeks to have a LMS.
---
paper_title: The Application of a Multi-Attribute Decision-Making Algorithm to Learning Management Systems Evaluation.

paper_content:

Recently, because of the rapid increase in the popularity of the Internet, the delivery of learning programmes has gradually shifted from local desktop to online-based applications. As more and more technological tools become available for online education, there is an increasing interest among educators and other professionals in the application of these tools with regard to online courses. In large-scale operations, online learning management systems (LMSs) can save cost and time. LMSs are software applications that comprise a suite of tools for learning and teaching online. In conventional educational settings, online LMSs can help to improve the speed and effectiveness of the educational process, communication among learners, and also between staff and students. There are a large number of open-source and commercially available LMSs on the marketplace, and one of the problems facing an administrator or instructor is how to choose the LMS that can meet his/her needs. This paper describes a software system developed by the author to aid in the selection of a suitable LMS. The software is a web-based decision support system, titled Easy Way to Evaluate LMSs, and is based on using a multi-attribute decision-making algorithm for the effective selection of an LMS.
---

Title: A Systematic Literature Review of Open Source Software Quality Assessment Models

Section 1: Background
Description 1: Discuss the evolution of open source software quality models, their distinguishing features, and the existing gaps in current models.

Section 2: Methods
Description 2: Describe the research questions, search strategy, inclusion criteria, quality assessment, and data extraction strategy used in this study.

Section 3: Research Questions
Description 3: Present the three research questions this study aims to answer and the rationale behind each.

Section 4: Search Strategy
Description 4: Outline the search string and the digital libraries used to retrieve relevant papers, along with the inclusion criteria for selecting papers.

Section 5: Quality Assessment
Description 5: Explain the criteria and process used to evaluate the quality of the selected primary studies.

Section 6: Data Extraction Strategy
Description 6: Describe how data was extracted from the selected studies and the process used to ensure accuracy and consistency.

Section 7: RQ1: Key Quality Characteristics
Description 7: Analyze the key quality characteristics possessed by the existing OSS quality assessment models.

Section 8: RQ2: Methods for Selection Decisions
Description 8: Evaluate the different methods used by OSS quality assessment models for making selection decisions.

Section 9: RQ3: Domain of Application
Description 9: Examine the various domains of application for the existing OSS quality assessment models.

Section 10: Principal Findings
Description 10: Summarize the principal findings from the study, highlighting significant trends and common patterns in existing models.

Section 11: Implications of the Results
Description 11: Discuss the implications of the study’s findings for practitioners and researchers developing new OSS quality assessment models.

Section 12: Threats to Validity
Description 12: Analyze the potential threats to the validity of the study and the measures taken to mitigate them.

Section 13: Conclusion and Future Work
Description 13: Conclude with the overall findings of the study and suggest directions for future research in OSS quality assessment models.