A Survey of Software Testing in the Cloud
---
paper_title: Adoption issues for cloud computing

paper_content:

Cloud computing allows users to use only a Web browser to receive computing services via the Internet. Users only need to pay for the services they actually use. It appears that a wide adoption of cloud computing in the foreseeable future is inevitable, and its adoption will bring about a sea change in the pricing and distribution practices for both software and hardware. There are, however, various issues that will impede adoption of cloud computing. Most of them can be solved. We discuss the status of cloud computing today and various adoption issues. We also provide a market prognosis.
---
paper_title: A model-driven approach for automating mobile applications testing

paper_content:

Software testing faces up several challenges. One out of these is the opposition between time-to-market software delivery and the excessive length of testing activities. The latter results from the growth of the application complexity along with the diversity of handheld devices. The economical competition, branding impose zero-defect products, putting forward testing as an even more crucial activity. In this paper, we describe a Domain-Specific Modeling Language (DSML) built upon an industrial platform (a test bed) which aims to automate mobile application checking. A key characteristic of this DSML is its ability to cope with variability in the spirit of software product line engineering. We discuss this DSML as part of a tool suite enabling the test of remote devices having variable features.
---
paper_title: Cloud Computing – Issues, Research and Implementations

paper_content:

"Cloud" computing – a relatively recent term, builds on decades of research in virtualization, distributed computing, utility computing, and more recently networking, web and software services. It implies a service oriented architecture, reduced information technology overhead for the end-user, great flexibility, reduced total cost of ownership, on-demand services and many other things. This paper discusses the concept of “cloud” computing, some of the issues it tries to address, related research topics, and a “cloud” implementation available today.
---
paper_title: Software Testing as an Online Service: Observations from Practice

paper_content:

The objective of this qualitative study was to explore and understand the conditions that influence software testing as an online service and elicit important research issues. Interviews were conducted with managers from eleven organizations. The study used qualitative grounded theory as its research method. The results indicate that the demand for software testing as an online service is on the rise and is influenced by conditions such as the level of domain knowledge needed to effectively test an application, flexibility and cost effectiveness as benefits, security and pricing as top requirements, cloud computing as the delivery mode and the need for software testers to hone their skills. Potential research areas suggested include application areas best suited for online software testing, pricing and handling of test data among others.
---
paper_title: Testing as a Service over Cloud

paper_content:

Testing-as-a-service (TaaS) is a new model to provide testing capabilities to end users. Users save the cost of complicated maintenance and upgrade effort, and service providers can upgrade their services without impact on the end-users. Due to uneven volumes of concurrent requests, it is important to address the elasticity of TaaS platform in a cloud environment. Scheduling and dispatching algorithms are developed to improve the utilization of computing resources. We develop a prototype of TaaS over cloud, and evaluate the scalability of the platform by increasing the test task load, analyze the distribution of computing time on test task scheduling and test task processing over the cloud, and examine the performance of proposed algorithms by comparing others
---
paper_title: Customizing Virtual Machine with Fault Injector by Integrating with SpecC Device Model for a Software Testing Environment D-Cloud

paper_content:

D-Cloud is a software testing environment for dependable parallel and distributed systems using cloud computing technology. We use Eucalyptus as cloud management software to manage virtual machines designed based on QEMU, called FaultVM, which have a fault injection mechanism. D-Cloud enables the test procedures to be automated using a large amount of computing resources in the cloud by interpreting the system configuration and the test scenario written in XML in D-Cloud front end and enables tests including hardware faults by emulating hardware faults by FaultVM flexibly. In the present paper, we describe the customization facility of FaultVM used to add new device models. We use SpecC, which is a system description language, to describe the behavior of devices, and a simulator generated from the description by SpecC is linked and integrated into FaultVM. This also makes the definition and injection of faults flexible without the modification of the original QEMU source codes. This facility allows D-Cloud to be used to test distributed systems with customized devices.
---
paper_title: Splitter: a proxy-based approach for post-migration testing of web applications

paper_content:

The benefits of virtualized IT environments, such as compute clouds, have drawn interested enterprises to migrate their applications onto new platforms to gain the advantages of reduced hardware and energy costs, increased flexibility and deployment speed, and reduced management complexity. However, the process of migrating a complex application takes a considerable amount of effort, particularly when performing post-migration testing to verify that the application still functions correctly in the target environment. The traditional approach of test case generation and execution can take weeks and synthetic test cases may not adequately reflect actual application usage. In this paper, we propose and evaluate a black-box approach for post-migration testing of Web applications without manually creating test cases. A Web proxy is put in front of the production application to intercept all requests from real users, and these requests are simultaneously sent to the production and migrated applications. Results generated by both applications are then compared, and mismatches due to migration problems can be easily detected and presented to testing teams for resolution. We implement this approach in Splitter, a software module that is deployed as a reverse Web proxy. Through our evaluation using a number of real applications, we show that it Splitter can effectively automate post-migration testing while also reduce the number of mismatches that must be manually inspected. Equally important, it imposes a relatively small performance overhead on the production environment.
---
paper_title: Testing as a Service over Cloud

paper_content:

Testing-as-a-service (TaaS) is a new model to provide testing capabilities to end users. Users save the cost of complicated maintenance and upgrade effort, and service providers can upgrade their services without impact on the end-users. Due to uneven volumes of concurrent requests, it is important to address the elasticity of TaaS platform in a cloud environment. Scheduling and dispatching algorithms are developed to improve the utilization of computing resources. We develop a prototype of TaaS over cloud, and evaluate the scalability of the platform by increasing the test task load, analyze the distribution of computing time on test task scheduling and test task processing over the cloud, and examine the performance of proposed algorithms by comparing others
---
paper_title: Open Cirrus: A Global Cloud Computing Testbed

paper_content:

Open Cirrus is a cloud computing testbed that, unlike existing alternatives, federates distributed data centers. It aims to spur innovation in systems and applications research and catalyze development of an open source service stack for the cloud.
---
paper_title: Testing Tasks Management in Testing Cloud Environment

paper_content:

In testing Cloud environment testing tasks requested by different tenants have many uncertainties. The arriving time, deadline and the number of tasks are unknown in advance. Especially, the relationships between testing tasks and testing environments are very complex. How to efficiently manage these tasks is really a challenging problem. This paper studies the special features of testing tasks and presents a task management framework. We analyze the dependencies and conflicts associated with testing tasks and their related runtime environments, using rule matching mechanism to derive the relationships supported by domain knowledge. Based on these analyses, improved algorithms are introduced to cluster and dynamically schedule testing tasks to minimize the make-span or meet deadlines with the consideration of testing task resource requirements and Cloud resource utilization balance at the same time. A fault tolerance mechanism is built to cope with testing errors, whose results are studied to ameliorate clustering and scheduling algorithms. A suite of experiments compares the effectiveness of the proposed approach with other algorithms.
---
paper_title: Modeling and testing of cloud applications

paper_content:

What is a cloud application precisely? In this paper, we formulate a computing cloud as a kind of graph, a computing resource such as services or intellectual property access rights as an attribute of a graph node, and the use of a resource as a predicate on an edge of the graph. We also propose to model cloud computation semantically as a set of paths in a subgraph of the cloud such that every edge contains a predicate that is evaluated to be true. Finally, we present algorithms to compose cloud computations and a family of model-based testing criteria to support the testing of cloud applications.
---
paper_title: D-Cloud: Design of a Software Testing Environment for Reliable Distributed Systems Using Cloud Computing Technology

paper_content:

In this paper, we propose a software testing environment, called D-Cloud, using cloud computing technology and virtual machines with fault injection facility. Nevertheless, the importance of high dependability in a software system has recently increased, and exhaustive testing of software systems is becoming expensive and time-consuming, and, in many cases, sufficient software testing is not possible. In particular, it is often difficult to test parallel and distributed systems in the real world after deployment, although reliable systems, such as high-availability servers, are parallel and distributed systems. D-Cloud is a cloud system which manages virtual machines with fault injection facility. D-Cloud sets up a test environment on the cloud resources using a given system configuration file and executes several tests automatically according to a given scenario. In this scenario, D-Cloud enables fault tolerance testing by causing device faults by virtual machine. We have designed the D-Cloud system using Eucalyptus software and a description language for system configuration and the scenario of fault injection written in XML. We found that the D-Cloud system, which allows a user to easily set up and test a distributed system on the cloud and effectively reduces the cost and time of testing.
---
paper_title: Research in concurrent software testing: a systematic review

paper_content:

The current increased demand for distributed applications in domains such as web services and cloud computing has significantly increased interest in concurrent programming. This demand in turn has resulted in new testing methodologies for such systems, which take account of the challenges necessary to test these applications. This paper presents a systematic review of the published research related to concurrent testing approaches, bug classification and testing tools. A systematic review is a process of collection, assessment and interpretation of the published papers related to a specific search question, designed to provide a background for further research. The results include information about the research relationships and research teams that are working in the different areas of concurrent programs testing.
---
paper_title: Modeling and testing of cloud applications

paper_content:

What is a cloud application precisely? In this paper, we formulate a computing cloud as a kind of graph, a computing resource such as services or intellectual property access rights as an attribute of a graph node, and the use of a resource as a predicate on an edge of the graph. We also propose to model cloud computation semantically as a set of paths in a subgraph of the cloud such that every edge contains a predicate that is evaluated to be true. Finally, we present algorithms to compose cloud computations and a family of model-based testing criteria to support the testing of cloud applications.
---
paper_title: Software Engineering Challenges for Migration to the Service Cloud Paradigm: Ongoing Work in the REMICS Project

paper_content:

This paper presents on-going work in a research project on defining methodology and tools for model-driven migration of legacy applications to a service-oriented architecture with deployment in the cloud, i.e. the Service Cloud paradigm. We have performed a comprehensive state of the art analysis and present some findings here. In parallel, the two industrial participants in the project have specified their requirements and expectations regarding modernization of their applications. The SOA paradigm implies the breakdown of architecture into high-grain components providing business services. For taking advantage of the services of cloud computing technologies, the clients' architecture should be decomposed, decoupled and be made scalable. Also requirements regarding servers, data storage and security, networking and response time, business models and pricing should be projected. We present software engineering challenges related to these aspects and examples of these in the context of one of the industrial cases in the project.
---
paper_title: Research Issues for Software Testing in the Cloud

paper_content:

Cloud computing is causing a paradigm shift in the provision and use of computing services, away from the traditional desktop form to online services. This implies that the manner in which these computing services are tested should also change. This paper discusses the research issues that cloud computing imposes on software testing. These issues were gathered during interviews with industry practitioners from eleven software organizations. The interviews were analyzed using qualitative grounded theory method. Findings of the study were compared with existing literature. The research issues were categorized according to application, management, legal and financial issues. By addressing these issues, researchers can offer reliable recommendation for practitioners in the industry.
---
paper_title: Engineering the cloud from software modules

paper_content:

Cloud computing faces many of the challenges and difficulties of distributed and parallel software. While the service interface hides the actual application from the remote user, the application developer still needs to come to terms with distributed software that needs to run on dynamic clusters and operate under a wide range of configurations. In this paper, we outline our vision of a model and runtime platform for the development, deployment, and management of software applications on the cloud. Our basic idea is to turn the notion of software module into a first class entity used for management and distribution that can be autonomously managed by the underlying software fabric of the cloud. In the paper we present our model, outline an initial implementation, and describe a first application developed using the ideas presented in the paper.
---
paper_title: Parallel symbolic execution for automated real-world software testing

paper_content:

This paper introduces Cloud9, a platform for automated testing of real-world software. Our main contribution is the scalable parallelization of symbolic execution on clusters of commodity hardware, to help cope with path explosion. Cloud9 provides a systematic interface for writing "symbolic tests" that concisely specify entire families of inputs and behaviors to be tested, thus improving testing productivity. Cloud9 can handle not only single-threaded programs but also multi-threaded and distributed systems. It includes a new symbolic environment model that is the first to support all major aspects of the POSIX interface, such as processes, threads, synchronization, networking, IPC, and file I/O. We show that Cloud9 can automatically test real systems, like memcached, Apache httpd, lighttpd, the Python interpreter, rsync, and curl. We show how Cloud9 can use existing test suites to generate new test cases that capture untested corner cases (e.g., network stream fragmentation). Cloud9 can also diagnose incomplete bug fixes by analyzing the difference between buggy paths before and after a patch.
---
paper_title: Customizing Virtual Machine with Fault Injector by Integrating with SpecC Device Model for a Software Testing Environment D-Cloud

paper_content:

D-Cloud is a software testing environment for dependable parallel and distributed systems using cloud computing technology. We use Eucalyptus as cloud management software to manage virtual machines designed based on QEMU, called FaultVM, which have a fault injection mechanism. D-Cloud enables the test procedures to be automated using a large amount of computing resources in the cloud by interpreting the system configuration and the test scenario written in XML in D-Cloud front end and enables tests including hardware faults by emulating hardware faults by FaultVM flexibly. In the present paper, we describe the customization facility of FaultVM used to add new device models. We use SpecC, which is a system description language, to describe the behavior of devices, and a simulator generated from the description by SpecC is linked and integrated into FaultVM. This also makes the definition and injection of faults flexible without the modification of the original QEMU source codes. This facility allows D-Cloud to be used to test distributed systems with customized devices.
---
paper_title: Testing in the Cloud: Exploring the Practice

paper_content:

As applications and services migrate to the cloud, testing will follow the same trend. Therefore, organizations must understand the dynamics of cloud-based testing. This article presents interviews with eight organizations that use cloud computing. The results suggest that cloud computing can make testing faster and enhance the delivery of testing services. Cloud computing also highlights important aspects of testing that require attention, such as integration and interoperability. This article includes a Web extra that provides additional references for further study.
---
paper_title: Application migration to cloud: a taxonomy of critical factors

paper_content:

Cloud computing has attracted attention as an important platform for software deployment, with perceived benefits such as elasticity to fluctuating load, and reduced operational costs compared to running in enterprise data centers. While some software is written from scratch specially for the cloud, many organizations also wish to migrate existing applications to a cloud platform. Such a migration exercise to a cloud platform is not easy: some changes need to be made to deal with differences in software environment, such as programming model and data storage APIs, as well as varying performance qualities. We report here on experiences in doing a number of sample migrations. We propose a taxonomy of the migration tasks involved, and we show the breakdown of costs among categories of task, for a case-study which migrated a .NET n-tier application to run on Windows Azure. We also indicate important factors that impact on the cost of various migration tasks. This work contributes towards our future direction of building a framework for cost-benefit tradeoff analysis that would apply to migrating applications to cloud platforms, and could help decision-makers evaluate proposals for using cloud computing.
---
paper_title: Using realistic simulation for performance analysis of mapreduce setups

paper_content:

Recently, there has been a huge growth in the amount of data processed by enterprises and the scientific computing community. Two promising trends ensure that applications will be able to deal with ever increasing data volumes: First, the emergence of cloud computing, which provides transparent access to a large number of compute, storage and networking resources; and second, the development of the MapReduce programming model, which provides a high-level abstraction for data-intensive computing. However, the design space of these systems has not been explored in detail. Specifically, the impact of various design choices and run-time parameters of a MapReduce system on application performance remains an open question. To this end, we embarked on systematically understanding the performance of MapReduce systems, but soon realized that understanding effects of parameter tweaking in a large-scale setup with many variables was impractical. Consequently, in this paper, we present the design of an accurate MapReduce simulator, MRPerf, for facilitating exploration of MapReduce design space. MRPerf captures various aspects of a MapReduce setup, and uses this information to predict expected application performance. In essence, MRPerf can serve as a design tool for MapReduce infrastructure, and as a planning tool for making MapReduce deployment far easier via reduction in the number of parameters that currently have to be hand-tuned using rules of thumb. Our validation of MRPerf using data from medium-scale production clusters shows that it is able to predict application performance accurately, and thus can be a useful tool in enabling cloud computing. Moreover, an initial application of MRPerf to our test clusters running Hadoop, revealed a performance bottleneck, fixing which resulted in up to 28.05% performance improvement.
---
paper_title: Splitter: a proxy-based approach for post-migration testing of web applications

paper_content:

The benefits of virtualized IT environments, such as compute clouds, have drawn interested enterprises to migrate their applications onto new platforms to gain the advantages of reduced hardware and energy costs, increased flexibility and deployment speed, and reduced management complexity. However, the process of migrating a complex application takes a considerable amount of effort, particularly when performing post-migration testing to verify that the application still functions correctly in the target environment. The traditional approach of test case generation and execution can take weeks and synthetic test cases may not adequately reflect actual application usage. In this paper, we propose and evaluate a black-box approach for post-migration testing of Web applications without manually creating test cases. A Web proxy is put in front of the production application to intercept all requests from real users, and these requests are simultaneously sent to the production and migrated applications. Results generated by both applications are then compared, and mismatches due to migration problems can be easily detected and presented to testing teams for resolution. We implement this approach in Splitter, a software module that is deployed as a reverse Web proxy. Through our evaluation using a number of real applications, we show that it Splitter can effectively automate post-migration testing while also reduce the number of mismatches that must be manually inspected. Equally important, it imposes a relatively small performance overhead on the production environment.
---
paper_title: Software Testing as an Online Service: Observations from Practice

paper_content:

The objective of this qualitative study was to explore and understand the conditions that influence software testing as an online service and elicit important research issues. Interviews were conducted with managers from eleven organizations. The study used qualitative grounded theory as its research method. The results indicate that the demand for software testing as an online service is on the rise and is influenced by conditions such as the level of domain knowledge needed to effectively test an application, flexibility and cost effectiveness as benefits, security and pricing as top requirements, cloud computing as the delivery mode and the need for software testers to hone their skills. Potential research areas suggested include application areas best suited for online software testing, pricing and handling of test data among others.
---
paper_title: VIAF: Verification-Based Integrity Assurance Framework for MapReduce

paper_content:

MapReduce, a cloud computing paradigm, is gaining popularity. However, like all open distributed computing frameworks, MapReduce suffers from the integrity assurance vulnerability: it takes merely one malicious worker to render the overall computation result useless. Existing solutions are effective in defeating the malicious behavior of non-collusive workers, but are futile in detecting collusive workers. In this paper, we focus on the mappers, which typically constitute the majority of workers, and propose the Verification-based Integrity Assurance Framework (VIAF) to detect both non-collusive and collusive mappers. The basic idea of VIAF is to combine task replication with non-deterministic verification, in which consistent but malicious results from collusive mappers can be detected by a trusted verifier. We have implemented VIAF in Hadoop, an open source MapReduce implementation. Our theoretical analysis and experimental result show that VIAF can achieve high task accuracy while imposing acceptable overhead.
---
paper_title: When to Migrate Software Testing to the Cloud?

paper_content:

Testing is a challenging activity for many software engineering projects, especially for large-scale systems. The amount of tests cases can range from a few hundred to several thousands, requiring significant computing resources and lengthy execution times. Cloud computing offers the potential to address both of these issues: it offers resources such as virtualized hardware, effectively unlimited storage, and software services that can aid in reducing the execution time of large test suites in a cost-effective manner. However, migrating to the cloud is not without cost, nor is it necessarily the best solution to all testing problems. This paper discusses when to migrate software testing to the cloud from two perspectives: the characteristics of an application under test, and the types of testing performed on the application.
---
paper_title: On the Standardization of a Testing Framework for Application Deployment on Grid and Cloud Infrastructures

paper_content:

An important requirement in the successful deployment of grid and cloud computing technology in industry or governmental institutions is the ability to compose their infrastructures using equipment from different vendors. These equipments have to be engineered for and assessed to assure a problem-free interoperation. Focusing on software interoperability, we present a testing framework for the assessment of interoperability of grid and cloud computing infrastructures. This testing framework is part of an initiative for standardizing the use of grid and cloud technology in the context of telecommunication at the European Telecommunications Standards Institute. Following the test development process developed and used at the European Telecommunications Standards Institute, its application is exemplified by the assessment of for resource reservation and application deployment onto grid and cloud infrastructures based on standardized Grid Component Model descriptors. The presented testing framework has been applied successfully in an interoperability event.
---
paper_title: Open Cirrus: A Global Cloud Computing Testbed

paper_content:

Open Cirrus is a cloud computing testbed that, unlike existing alternatives, federates distributed data centers. It aims to spur innovation in systems and applications research and catalyze development of an open source service stack for the cloud.
---
paper_title: Towards a scalable and robust multi-tenancy SaaS

paper_content:

Software-as-as-Service (SaaS) is a new approach for developing software, and it is characterized by its multi-tenancy architecture and its ability to provide flexible customization to individual tenant. However, the multi-tenancy architecture and customization requirements have brought up new issues in software, such as database design, database partition, scalability, recovery, and continuous testing. This paper proposes a hybrid test database design to support SaaS customization with two-layer database partitioning. The database is further extended with a new built-in redundancy with ontology so that the SaaS can recover from ontology, data or meta-data failures. Furthermore, constraints in metadata can be used either as test cases or policies to support SaaS continuous testing and policy enforcement.
---
paper_title: Evaluating Cloud Platform Architecture with the CARE Framework

paper_content:

There is an emergence of Cloud application platforms such as Microsoft’s Azure, Google’s App Engine and Amazon’s EC2/SimpleDB/S3. Startups and Enterprise alike, lured by the promise of ‘infinite scalability’, ‘ease of development’, ‘low infrastructure setup cost’ are increasingly using these Cloud service building blocks to develop and deploy their web based applications. However, the precise nature of these Cloud platforms and the resultant Cloud application runtime behavior is still largely an unknown. Given the black box nature of these platforms, and the novel programming and data models of Cloud, there is a dearth of tools and techniques for enabling the rigorously evaluation of Cloud platforms at runtime. This paper introduces the CARE (Cloud Architecture Runtime Evaluation) approach, a framework for evaluating Cloud application development and runtime platforms. CARE implements a unified interface with WSDL and REST in order to evaluate different Cloud platforms for Cloud application hosting servers and Cloud databases. With the unified interface, we are able to perform selective high stress and low stress evaluations corresponding to desired test scenarios. Result shows the effectiveness of CARE in the evaluation of Cloud variations in terms of scalability, availability and responsiveness, across both compute and storage capabilities. Thus placing CARE as an important tool in the path of Cloud computing research.
---
paper_title: Risk-Based Security Testing in Cloud Computing Environments

paper_content:

Assuring the security of a software system in terms of testing nowadays still is a quite tricky task to conduct. Security requirements are taken as a foundation to derive tests to be executed against a system under test. Yet, these positive requirements by far do not cover all the relevant security aspects to be considered. Hence, especially in the event of security testing, negative requirements, derived from risk analysis, are vital to be incorporated. If considering today's emerging trend in the adoption of cloud computing, security testing even has a more important significance. Due to a cloud's openness, in theory there exists an infinite number of tests. Hence, a concise technique to incorporate the results of risk analysis in security testing is inevitable. We therefore propose a new model-driven methodology for the security testing of cloud environments, ingesting misuse cases, defined by negative requirements derived from risk analysis.
---
paper_title: Cloud9: a software testing service

paper_content:

Cloud9 aims to reduce the resource-intensive and laborintensive nature of high-quality software testing. First, Cloud9 parallelizes symbolic execution (an effective, but still poorly scalable test automation technique) to large shared-nothing clusters. To our knowledge, Cloud9 is the first symbolic execution engine that scales to large clusters of machines, thus enabling thorough automated testing of real software in conveniently short amounts of time. Preliminary results indicate one to two orders of magnitude speedup over a state-of-the-art symbolic execution engine. Second, Cloud9 is an on-demand software testing service: it runs on compute clouds, like Amazon EC2, and scales its use of resources over a wide dynamic range, proportionally with the testing task at hand.
---
paper_title: An environment for evaluation and testing of service workflow schedulers in clouds

paper_content:

Workflows built through service composition bring new challenges, making the scheduling task even more complex. Besides that, scheduling researchers need a wide range of workflows and their respective services to validate new achievements. In this paper we present a service oriented testbed where experiments can be conducted to develop and evaluate algorithms, heuristics, and scheduling policies for workflows. Our testbed offers an emulator service which allows the workflow characterization through the description of its services. With this, researchers can build workflows which have similar behavior to the real workflow applications, emulating them without the need of implementing all applications and services involved in a real application execution. To demonstrate the utilization of both the testbed and emulator, we conducted workflow executions emulating service workflow applications, such as Montage and LIGO.
---
paper_title: An Approach for Service Composition and Testing for Cloud Computing

paper_content:

As cloud services proliferate, it becomes difficult to facilitate service composition and testing in clouds. In traditional service-oriented computing, service composition and testing are carried out independently. This paper proposes a new approach to manage services on the cloud so that it can facilitate service composition and testing. The paper uses service implementation selection to facilitate service composition similar to Google’s Guice and Spring tools, and apply the group testing technique to identify the oracle, and use the established oracle to perform continuous testing for new services or compositions. The paper extends the existing concept of template based service composition and focus on testing the same workflow of service composition. In addition, all these testing processes can be executed in parallel, and the paper illustrates how to apply service-level MapReduce technique to accelerate the testing process.
---
paper_title: Application-Oriented Remote Verification Trust Model in Cloud Computing

paper_content:

The emergence and application of cloud computing can help users access to various computing resources and services more conveniently. However, it also brings forth many security challenges. This paper proposes the application oriented remote verification trust model, which is capable of adjusting the useri¯s trust authorization verification contents according to the specific security requirements of different applications. The model also dynamically adjusts the usersi¯trust value with the trust feedback mechanism to determine whether or not the requested resource or service should be provided, so as to guarantee the security of information resources. This paper provides a formal description of the basic components and trust properties of the model with a belief formula, and describes the framework for the implementation of the model.
---
paper_title: Large-Scale Software Testing Environment Using Cloud Computing Technology for Dependable Parallel and Distributed Systems

paper_content:

Various information systems are widely used in information society era, and the demand for highly dependable system is increasing year after year. However, software testing for such a system becomes more difficult due to the enlargement and the complexity of the system. In particular, it is too difficult to test parallel and distributed systems sufficiently although dependable systems such as high-availability servers usually form parallel and distributed systems. To solve these problems, we proposed a software testing environment for dependable parallel and distributed system using the cloud computing technology, named D-Cloud. D-Cloud includes Eucalyptus as the cloud management software, and FaultVM based on QEMU as the virtualization software, and D-Cloud frontend for interpreting test scenario. D-Cloud enables not only to automate the system configuration and the test procedure but also to perform a number of test cases simultaneously, and to emulate hardware faults flexibly. In this paper, we present the concept and design of D-Cloud, and describe how to specify the system configuration and the test scenario. Furthermore, the preliminary test example as the software testing using D-Cloud was presented. Its result shows that D-Cloud allows to set up the environment easily, and to test the software testing for the distributed system.
---
paper_title: Cloud Services Testing: An Understanding

paper_content:

There is a mammoth quantity of Cloud Services being deployed on the Cloud, nowadays. Everyday users and customers use these services to fulfil their needs. The use of Cloud Services is getting more and more complex with the pace of time. It is a possibility that several recurrent service requests cannot be fulfilled using just one Cloud service. Cloud services are usually composed manually, which is a time consuming and monotonous task. We can find several numbers of successful methods for automatic Cloud service composition, the main issue with that is the lack of test environment with some standards to compare and evaluate these methods. This research work is about a short survey to explore Cloud Services testing methods. This study compares several software testing researches and pose questions for further research work to find Cloud suited testing techniques for the software testers. This survey paper poses few questions to the Cloud computing research community to concentrate and find suited answers for software testing community.
---
paper_title: Parallel symbolic execution for structural test generation

paper_content:

Symbolic execution is a popular technique for automatically generating test cases achieving high structural coverage. Symbolic execution suffers from scalability issues since the number of symbolic paths that need to be explored is very large (or even infinite) for most realistic programs. To address this problem, we propose a technique, Simple Static Partitioning, for parallelizing symbolic execution. The technique uses a set of pre-conditions to partition the symbolic execution tree, allowing us to effectively distribute symbolic execution and decrease the time needed to explore the symbolic execution tree. The proposed technique requires little communication between parallel instances and is designed to work with a variety of architectures, ranging from fast multi-core machines to cloud or grid computing environments. We implement our technique in the Java PathFinder verification tool-set and evaluate it on six case studies with respect to the performance improvement when exploring a finite symbolic execution tree and performing automatic test generation. We demonstrate speedup in both the analysis time over finite symbolic execution trees and in the time required to generate tests relative to sequential execution, with a maximum analysis time speedup of 90x observed using 128 workers and a maximum test generation speedup of 70x observed using 64 workers.
---
paper_title: VATS: Virtualized-Aware Automated Test Service

paper_content:

It is anticipated that by 2015 more than 75% of Information Technology infrastructure will be purchased as a service from service providers. Services will be hosted in virtualized shared resource pools referred to as Clouds. Service providers will need to ensure that customer performance requirements are satisfied while consuming an acceptable quantity of resources. This paper describes a Virtualization-aware Automated Testing Service (VATS). VATS is a framework for automated test execution in Cloud computing environments. It executes tests, manipulates virtualized infrastructure, and collects performance information. VATS uses HP LoadRunner as a load generator and provides the foundation for an automatic performance evaluator for Cloud environments. A case study describes our use of VATS with an SAP R/3 system running in a Xen-based virtualized resource pool. The results from VATS are used to determine the impact of virtual machine configuration parameters on a SAP system.
---
paper_title: Leveraging Cloud Platform for Custom Application Development

paper_content:

Compared with packaged application, custom application developments (CAD) experience the frustration of higher project overhead and less certainty. The typical time spent on building the infrastructure for a CAD project is, on average, several weeks. Project uncertainty comes from unique customer requirements and lack of standardized methods and toolsets to follow. Therefore, a CAD project is more difficult to achieve cost reduction and asset reuse. In this paper, we present a cloud platform to alleviate this problem through an integration of a) standard methods, b) standardized toolsets aligned with those methods, c) project management environments with pre-defined work breakdown structure (WBS) aligned with those methods and toolsets, and d) infrastructure support from the cloud technology. We believe that such a cloud platform will become a fundamental approach for large enterprises to develop CAD or other solutions for their clients.
---
paper_title: Environment Modeling for Automated Testing of Cloud Applications

paper_content:

Recently, cloud computing platforms, such as Microsoft Azure, are available to provide convenient infrastructures such that cloud applications could conduct cloud and data-intensive computing. To ensure high quality of cloud applications under development, developer testing (also referred to as unit testing) could be used. The behavior of a unit in a cloud application is dependent on the test inputs as well as the state of the cloud environment. Generally, manually providing various test inputs and cloud states for conducting developer testing is time-consuming and labor-intensive. To reduce the manual effort, developers could employ automated test generation tools. However, applying an automated test generation tool faces the challenge of generating various cloud states for achieving effective testing, such as achieving high structural coverage of the cloud application since these tools cannot control the cloud environment. To address this challenge, we propose an approach to (1) model the cloud environment for simulating the behavior of the real environment and, (2) apply Dynamic Symbolic Execution (DSE), to both generate test inputs and cloud states to achieve high structural coverage. We apply our approach on some open source Azure cloud applications. The result shows that our approach automatically generates test inputs and cloud states to achieve high structural coverage of the cloud application.
---
paper_title: Automated software testing as a service

paper_content:

This paper makes the case for TaaS--automated software testing as a cloud-based service. We present three kinds of TaaS: a "programmer's sidekick" enabling developers to thoroughly and promptly test their code with minimal upfront resource investment; a "home edition" on-demand testing service for consumers to verify the software they are about to install on their PC or mobile device; and a public "certification service," akin to Underwriters Labs, that independently assesses the reliability, safety, and security of software. TaaS automatically tests software, without human involvement from the service user's or provider's side. This is unlike today's "testing as a service" businesses, which employ humans to write tests. Our goal is to take recently proposed techniques for automated testing--even if usable only on to y programs--and make them practical by modifying them to harness the resources of compute clouds. Preliminary work suggests it is technically feasible to do so, and we find that TaaS is also compelling from a social and business point of view.
---
paper_title: Research Issues for Software Testing in the Cloud

paper_content:

Cloud computing is causing a paradigm shift in the provision and use of computing services, away from the traditional desktop form to online services. This implies that the manner in which these computing services are tested should also change. This paper discusses the research issues that cloud computing imposes on software testing. These issues were gathered during interviews with industry practitioners from eleven software organizations. The interviews were analyzed using qualitative grounded theory method. Findings of the study were compared with existing literature. The research issues were categorized according to application, management, legal and financial issues. By addressing these issues, researchers can offer reliable recommendation for practitioners in the industry.
---
paper_title: Software Testing as an Online Service: Observations from Practice

paper_content:

The objective of this qualitative study was to explore and understand the conditions that influence software testing as an online service and elicit important research issues. Interviews were conducted with managers from eleven organizations. The study used qualitative grounded theory as its research method. The results indicate that the demand for software testing as an online service is on the rise and is influenced by conditions such as the level of domain knowledge needed to effectively test an application, flexibility and cost effectiveness as benefits, security and pricing as top requirements, cloud computing as the delivery mode and the need for software testers to hone their skills. Potential research areas suggested include application areas best suited for online software testing, pricing and handling of test data among others.
---
paper_title: Software Engineering Challenges for Migration to the Service Cloud Paradigm: Ongoing Work in the REMICS Project

paper_content:

This paper presents on-going work in a research project on defining methodology and tools for model-driven migration of legacy applications to a service-oriented architecture with deployment in the cloud, i.e. the Service Cloud paradigm. We have performed a comprehensive state of the art analysis and present some findings here. In parallel, the two industrial participants in the project have specified their requirements and expectations regarding modernization of their applications. The SOA paradigm implies the breakdown of architecture into high-grain components providing business services. For taking advantage of the services of cloud computing technologies, the clients' architecture should be decomposed, decoupled and be made scalable. Also requirements regarding servers, data storage and security, networking and response time, business models and pricing should be projected. We present software engineering challenges related to these aspects and examples of these in the context of one of the industrial cases in the project.
---
paper_title: Application migration to cloud: a taxonomy of critical factors

paper_content:

Cloud computing has attracted attention as an important platform for software deployment, with perceived benefits such as elasticity to fluctuating load, and reduced operational costs compared to running in enterprise data centers. While some software is written from scratch specially for the cloud, many organizations also wish to migrate existing applications to a cloud platform. Such a migration exercise to a cloud platform is not easy: some changes need to be made to deal with differences in software environment, such as programming model and data storage APIs, as well as varying performance qualities. We report here on experiences in doing a number of sample migrations. We propose a taxonomy of the migration tasks involved, and we show the breakdown of costs among categories of task, for a case-study which migrated a .NET n-tier application to run on Windows Azure. We also indicate important factors that impact on the cost of various migration tasks. This work contributes towards our future direction of building a framework for cost-benefit tradeoff analysis that would apply to migrating applications to cloud platforms, and could help decision-makers evaluate proposals for using cloud computing.
---
paper_title: Using realistic simulation for performance analysis of mapreduce setups

paper_content:

Recently, there has been a huge growth in the amount of data processed by enterprises and the scientific computing community. Two promising trends ensure that applications will be able to deal with ever increasing data volumes: First, the emergence of cloud computing, which provides transparent access to a large number of compute, storage and networking resources; and second, the development of the MapReduce programming model, which provides a high-level abstraction for data-intensive computing. However, the design space of these systems has not been explored in detail. Specifically, the impact of various design choices and run-time parameters of a MapReduce system on application performance remains an open question. To this end, we embarked on systematically understanding the performance of MapReduce systems, but soon realized that understanding effects of parameter tweaking in a large-scale setup with many variables was impractical. Consequently, in this paper, we present the design of an accurate MapReduce simulator, MRPerf, for facilitating exploration of MapReduce design space. MRPerf captures various aspects of a MapReduce setup, and uses this information to predict expected application performance. In essence, MRPerf can serve as a design tool for MapReduce infrastructure, and as a planning tool for making MapReduce deployment far easier via reduction in the number of parameters that currently have to be hand-tuned using rules of thumb. Our validation of MRPerf using data from medium-scale production clusters shows that it is able to predict application performance accurately, and thus can be a useful tool in enabling cloud computing. Moreover, an initial application of MRPerf to our test clusters running Hadoop, revealed a performance bottleneck, fixing which resulted in up to 28.05% performance improvement.
---
paper_title: Splitter: a proxy-based approach for post-migration testing of web applications

paper_content:

The benefits of virtualized IT environments, such as compute clouds, have drawn interested enterprises to migrate their applications onto new platforms to gain the advantages of reduced hardware and energy costs, increased flexibility and deployment speed, and reduced management complexity. However, the process of migrating a complex application takes a considerable amount of effort, particularly when performing post-migration testing to verify that the application still functions correctly in the target environment. The traditional approach of test case generation and execution can take weeks and synthetic test cases may not adequately reflect actual application usage. In this paper, we propose and evaluate a black-box approach for post-migration testing of Web applications without manually creating test cases. A Web proxy is put in front of the production application to intercept all requests from real users, and these requests are simultaneously sent to the production and migrated applications. Results generated by both applications are then compared, and mismatches due to migration problems can be easily detected and presented to testing teams for resolution. We implement this approach in Splitter, a software module that is deployed as a reverse Web proxy. Through our evaluation using a number of real applications, we show that it Splitter can effectively automate post-migration testing while also reduce the number of mismatches that must be manually inspected. Equally important, it imposes a relatively small performance overhead on the production environment.
---
paper_title: Software Testing as an Online Service: Observations from Practice

paper_content:

The objective of this qualitative study was to explore and understand the conditions that influence software testing as an online service and elicit important research issues. Interviews were conducted with managers from eleven organizations. The study used qualitative grounded theory as its research method. The results indicate that the demand for software testing as an online service is on the rise and is influenced by conditions such as the level of domain knowledge needed to effectively test an application, flexibility and cost effectiveness as benefits, security and pricing as top requirements, cloud computing as the delivery mode and the need for software testers to hone their skills. Potential research areas suggested include application areas best suited for online software testing, pricing and handling of test data among others.
---
paper_title: When to Migrate Software Testing to the Cloud?

paper_content:

Testing is a challenging activity for many software engineering projects, especially for large-scale systems. The amount of tests cases can range from a few hundred to several thousands, requiring significant computing resources and lengthy execution times. Cloud computing offers the potential to address both of these issues: it offers resources such as virtualized hardware, effectively unlimited storage, and software services that can aid in reducing the execution time of large test suites in a cost-effective manner. However, migrating to the cloud is not without cost, nor is it necessarily the best solution to all testing problems. This paper discusses when to migrate software testing to the cloud from two perspectives: the characteristics of an application under test, and the types of testing performed on the application.
---
paper_title: Towards a scalable and robust multi-tenancy SaaS

paper_content:

Software-as-as-Service (SaaS) is a new approach for developing software, and it is characterized by its multi-tenancy architecture and its ability to provide flexible customization to individual tenant. However, the multi-tenancy architecture and customization requirements have brought up new issues in software, such as database design, database partition, scalability, recovery, and continuous testing. This paper proposes a hybrid test database design to support SaaS customization with two-layer database partitioning. The database is further extended with a new built-in redundancy with ontology so that the SaaS can recover from ontology, data or meta-data failures. Furthermore, constraints in metadata can be used either as test cases or policies to support SaaS continuous testing and policy enforcement.
---
paper_title: Evaluating Cloud Platform Architecture with the CARE Framework

paper_content:

There is an emergence of Cloud application platforms such as Microsoft’s Azure, Google’s App Engine and Amazon’s EC2/SimpleDB/S3. Startups and Enterprise alike, lured by the promise of ‘infinite scalability’, ‘ease of development’, ‘low infrastructure setup cost’ are increasingly using these Cloud service building blocks to develop and deploy their web based applications. However, the precise nature of these Cloud platforms and the resultant Cloud application runtime behavior is still largely an unknown. Given the black box nature of these platforms, and the novel programming and data models of Cloud, there is a dearth of tools and techniques for enabling the rigorously evaluation of Cloud platforms at runtime. This paper introduces the CARE (Cloud Architecture Runtime Evaluation) approach, a framework for evaluating Cloud application development and runtime platforms. CARE implements a unified interface with WSDL and REST in order to evaluate different Cloud platforms for Cloud application hosting servers and Cloud databases. With the unified interface, we are able to perform selective high stress and low stress evaluations corresponding to desired test scenarios. Result shows the effectiveness of CARE in the evaluation of Cloud variations in terms of scalability, availability and responsiveness, across both compute and storage capabilities. Thus placing CARE as an important tool in the path of Cloud computing research.
---
paper_title: An environment for evaluation and testing of service workflow schedulers in clouds

paper_content:

Workflows built through service composition bring new challenges, making the scheduling task even more complex. Besides that, scheduling researchers need a wide range of workflows and their respective services to validate new achievements. In this paper we present a service oriented testbed where experiments can be conducted to develop and evaluate algorithms, heuristics, and scheduling policies for workflows. Our testbed offers an emulator service which allows the workflow characterization through the description of its services. With this, researchers can build workflows which have similar behavior to the real workflow applications, emulating them without the need of implementing all applications and services involved in a real application execution. To demonstrate the utilization of both the testbed and emulator, we conducted workflow executions emulating service workflow applications, such as Montage and LIGO.
---
paper_title: An Approach for Service Composition and Testing for Cloud Computing

paper_content:

As cloud services proliferate, it becomes difficult to facilitate service composition and testing in clouds. In traditional service-oriented computing, service composition and testing are carried out independently. This paper proposes a new approach to manage services on the cloud so that it can facilitate service composition and testing. The paper uses service implementation selection to facilitate service composition similar to Google’s Guice and Spring tools, and apply the group testing technique to identify the oracle, and use the established oracle to perform continuous testing for new services or compositions. The paper extends the existing concept of template based service composition and focus on testing the same workflow of service composition. In addition, all these testing processes can be executed in parallel, and the paper illustrates how to apply service-level MapReduce technique to accelerate the testing process.
---
paper_title: YETI on the Cloud

paper_content:

The York Extensible Testing Infrastructure (YETI) is an automated random testing tool that allows to test programs written in various programming languages. While YETI is one of the fastest random testing tools with over a million method calls per minute on fast code, testing large programs or slow code -- such as libraries using intensively the memory -- might benefit from parallel executions of testing sessions. This paper presents the cloud-enabled version of YETI. It relies on the Hadoop package and its map/reduce implementation to distribute tasks over potentially many computers. This would allow to distribute the cloud version of YETI over Amazon's Elastic Compute Cloud (EC2).
---
paper_title: Parallel symbolic execution for structural test generation

paper_content:

Symbolic execution is a popular technique for automatically generating test cases achieving high structural coverage. Symbolic execution suffers from scalability issues since the number of symbolic paths that need to be explored is very large (or even infinite) for most realistic programs. To address this problem, we propose a technique, Simple Static Partitioning, for parallelizing symbolic execution. The technique uses a set of pre-conditions to partition the symbolic execution tree, allowing us to effectively distribute symbolic execution and decrease the time needed to explore the symbolic execution tree. The proposed technique requires little communication between parallel instances and is designed to work with a variety of architectures, ranging from fast multi-core machines to cloud or grid computing environments. We implement our technique in the Java PathFinder verification tool-set and evaluate it on six case studies with respect to the performance improvement when exploring a finite symbolic execution tree and performing automatic test generation. We demonstrate speedup in both the analysis time over finite symbolic execution trees and in the time required to generate tests relative to sequential execution, with a maximum analysis time speedup of 90x observed using 128 workers and a maximum test generation speedup of 70x observed using 64 workers.
---
paper_title: VATS: Virtualized-Aware Automated Test Service

paper_content:

It is anticipated that by 2015 more than 75% of Information Technology infrastructure will be purchased as a service from service providers. Services will be hosted in virtualized shared resource pools referred to as Clouds. Service providers will need to ensure that customer performance requirements are satisfied while consuming an acceptable quantity of resources. This paper describes a Virtualization-aware Automated Testing Service (VATS). VATS is a framework for automated test execution in Cloud computing environments. It executes tests, manipulates virtualized infrastructure, and collects performance information. VATS uses HP LoadRunner as a load generator and provides the foundation for an automatic performance evaluator for Cloud environments. A case study describes our use of VATS with an SAP R/3 system running in a Xen-based virtualized resource pool. The results from VATS are used to determine the impact of virtual machine configuration parameters on a SAP system.
---
paper_title: Leveraging Cloud Platform for Custom Application Development

paper_content:

Compared with packaged application, custom application developments (CAD) experience the frustration of higher project overhead and less certainty. The typical time spent on building the infrastructure for a CAD project is, on average, several weeks. Project uncertainty comes from unique customer requirements and lack of standardized methods and toolsets to follow. Therefore, a CAD project is more difficult to achieve cost reduction and asset reuse. In this paper, we present a cloud platform to alleviate this problem through an integration of a) standard methods, b) standardized toolsets aligned with those methods, c) project management environments with pre-defined work breakdown structure (WBS) aligned with those methods and toolsets, and d) infrastructure support from the cloud technology. We believe that such a cloud platform will become a fundamental approach for large enterprises to develop CAD or other solutions for their clients.
---
paper_title: Environment Modeling for Automated Testing of Cloud Applications

paper_content:

Recently, cloud computing platforms, such as Microsoft Azure, are available to provide convenient infrastructures such that cloud applications could conduct cloud and data-intensive computing. To ensure high quality of cloud applications under development, developer testing (also referred to as unit testing) could be used. The behavior of a unit in a cloud application is dependent on the test inputs as well as the state of the cloud environment. Generally, manually providing various test inputs and cloud states for conducting developer testing is time-consuming and labor-intensive. To reduce the manual effort, developers could employ automated test generation tools. However, applying an automated test generation tool faces the challenge of generating various cloud states for achieving effective testing, such as achieving high structural coverage of the cloud application since these tools cannot control the cloud environment. To address this challenge, we propose an approach to (1) model the cloud environment for simulating the behavior of the real environment and, (2) apply Dynamic Symbolic Execution (DSE), to both generate test inputs and cloud states to achieve high structural coverage. We apply our approach on some open source Azure cloud applications. The result shows that our approach automatically generates test inputs and cloud states to achieve high structural coverage of the cloud application.
---

Title: A Survey of Software Testing in the Cloud

Section 1: INTRODUCTION
Description 1: Provide an introduction to cloud computing and its relevance to software testing, discussing the motivations for moving software testing to the cloud and the benefits and challenges associated with it.

Section 2: CLOUD COMPUTING
Description 2: Describe cloud computing paradigms, essential characteristics, service delivery models, deployment models, and how virtualization plays a role in software testing within the cloud.

Section 3: RESEARCH METHODOLOGY
Description 3: Explain the methodology used in the research, including the aim to classify cloud-based testing activities, clarify terminology, identify gaps, and address issues in the literature.

Section 4: Categorization
Description 4: Categorize the papers reviewed based on different criteria such as test levels, test types, contributions, and associated delivery models in cloud computing.

Section 5: Gaps
Description 5: Identify gaps in existing research, particularly the lack of studies on the effects of different cloud deployment models on software testing as a service, and opportunities for further research.

Section 6: Testing for the Cloud
Description 6: Define and discuss testing activities specifically for applications developed to run on cloud platforms.

Section 7: Testing on the Cloud
Description 7: Explore testing activities for on-premise applications that utilize cloud infrastructure for testing purposes.

Section 8: RELATED WORK
Description 8: Review related research on software testing in the cloud, highlighting the absence of comprehensive literature reviews that categorize existing research by problem and solution domains.

Section 9: CONCLUSION
Description 9: Summarize the findings, emphasize open research areas, and suggest future research directions to address gaps identified in the survey.