Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming
---
paper_title: The Mythical Man-Month

paper_content:

The book, The Mythical Man-Month, Addison-Wesley, 1975 (excerpted in Datamation, December 1974), gathers some of the published data about software engineering and mixes it with the assertion of a lot of personal opinions. In this presentation, the author will list some of the assertions and invite dispute or support from the audience. This is intended as a public discussion of the published book, not a regular paper.
---
paper_title: Engineering the Software for Understanding Climate Change

paper_content:

Climate scientists build large, complex simulations with little or no software engineering training—and don't readily adopt the latest software engineering tools and techniques. This ethnographic study of climate scientists shows that their culture and practices share many features of agile and open source projects, but with highly customized software validation and verification techniques.
---
paper_title: The Chimera of Software Quality

paper_content:

Despite years of computing progress, today's systems experience spectacular and all-too-frequent crashes, while many enormously expensive projects fail to produce anything useful. Of equal importance, and potentially more damaging, are the misleading smaller defects we tend to miss. From time to time, we must remind ourselves that the underlying quality of the software that our results and progress increasingly depend on will likely be flawed and even more dependent on independent corroboration than the science itself. Many scientific results are corrupted, perhaps fatally so, by undiscovered mistakes in the software used to calculate and present those results.
---
paper_title: The T-experiments: errors in scientific software

paper_content:

Extensive tests showed that many software codes widely used in science and engineering are not as accurate as we would like to think. It is argued that better software engineering practices would help solve this problem, but realizing that the problem exists is an important first step.
---
paper_title: Uncertainty, Complexity and Concepts of Good Science in Climate Change Modelling: Are GCMs the Best Tools?

paper_content:

In this paper we explore the dominant position of a particular style of scientific modelling in the provision of policy-relevant scientific knowledge on future climate change. We describe how the apical position of General Circulation Models (GCMs) appears to follow ‘logically’ both from conventional understandings of scientific representation and the use of knowledge, so acquired, in decision-making. We argue, however, that both of these particular understandings are contestable. In addition to questioning their current policy-usefulness, we draw upon existing analyses of GCMs which discuss model trade-offs, errors, and the effects of parameterisations, to raise questions about the validity of the conception of complexity in conventional accounts. An alternative approach to modelling, incorporating concepts of uncertainty, is discussed, and an illustrative example given for the case of the global carbon cycle. In then addressing the question of how GCMs have come to occupy their dominant position, we argue that the development of global climate change science and global environmental ‘management’ frameworks occurs concurrently and in a mutually supportive fashion, so uniting GCMs and environmental policy developments in certain industrialised nations and international organisations. The more basic questions about what kinds of commitments to theories of knowledge underpin different models of ‘complexity’ as a normative principle of ‘good science’ are concealed in this mutual reinforcement. Additionally, a rather technocratic policy orientation to climate change may be supported by such science, even though it involves political choices which deserve to be more widely debated.
---
paper_title: Mechanizing Proof: Computing, Risk, and Trust

paper_content:

Most aspects of our private and social lives---our safety, the integrity of the financial system, the functioning of utilities and other services, and national security---now depend on computing. But how can we know that this computing is trustworthy? In Mechanizing Proof, Donald MacKenzie addresses this key issue by investigating the interrelations of computing, risk, and mathematical proof over the last half century from the perspectives of history and sociology. His discussion draws on the technical literature of computer science and artificial intelligence and on extensive interviews with scientists and engineers.
---
paper_title: Software release build process and components in ATLAS offline

paper_content:

ATLAS is one of the largest collaborations in the physical sciences and involves 3000 scientists and engineers from 174 institutions in 38 countries. The geographically dispersed developer community has produced a large amount of software which is organized in 10 projects. In this presentation we discuss how the software is built on a variety of compiler and operating system combinations every night. File level and package level parallelism together with multi-core build servers are used to perform fast builds of the different platforms in several branches. We discuss the different tools involved during the software release build process and also the various mechanisms implemented to provide performance gains and error detection and retry mechanisms in order to counteract network and other instabilities that would otherwise degrade the robustness of the system. The goal is to provide high quality software built as fast as possible ready for final validation and deployment.
---
paper_title: Lessons from the JMCB Archive

paper_content:

We examine the online archive of the Journal of Money, Credit, and Banking, in which an author is required to deposit the data and code that replicate the results of his paper. We find that most authors do not fulfill this requirement. Of more than 150 empirical articles, fewer than 15 could be replicated. Despite all this, there is no doubt that a data/code archive is more conducive to replicable research than the alternatives. We make recommendations to improve the functioning of the archive.
---
paper_title: The Role of Data and Program Code Archives in the Future of Economic Research

paper_content:

This essay examines the role of data and program-code archives in making economic research “replicable.” Replication of published results is recognized as an essential part of the scientific method. Yet, historically, both the “demand for” and “supply of” replicable results in economics has been minimal. “Respect for the scientific method” is not sufficient to motivate either economists or editors of professional journals to ensure the replicability of published results. We enumerate the costs and benefits of mandatory data and code archives, and argue that the benefits far exceed the costs. Progress has been made since the gloomy assessment of Dewald, Thursby and Anderson some twenty years ago in the American Economic Review, but much remains to be done before empirical economics ceases to be a “dismal science” when judged by the replicability of its published results. JEL Classification: B4, C8 (This is Federal Reserve Bank of St. Louis Research Division working paper 2005-14.) Revised version of paper prepared for the American Economic Association meeting, Philadelphia PA, January 9, 2005. Views expressed herein are solely those of the authors and not necessarily those of the Federal Reserve Bank of St. Louis or the Federal Reserve System, New York University, Drexel University, Fordham University, or their staffs. Correspondence: (1) anderson@stls.frb.org; (2) wgreene@stern.nyu.edu; (3) bdmccullough@drexel.edu; (4) vinod@fordham.edu.
---
paper_title: A longitudinal study of development and maintenance in Norway: Report from the 2003 investigation

paper_content:

Abstract The amount of work on application systems being taken up by maintenance activities (work done on an IT-system after being put in production) has been one of the arguments of those speaking about a ‘software crisis’. We have earlier investigated the applicability of this notion, and propose to rather look at the percentage of work being done on application portfolio upkeep (work made to keep up the functional coverage of the application system portfolio of the organization. This also includes the development of replacement systems), to assess the efficiency of the application systems support in an organisation. This paper presents the main results of a survey investigation performed in 2003 in 54 Norwegian organisations within this area. The amount of application portfolio upkeep is significantly higher than in a similar investigation conducted in 1993. The level of maintenance is smaller (although not significantly) than in another similar investigation conducted in 1998. There was a significant increase in both maintenance and application portfolio upkeep from 1993 to 1998, which could partly be attributed to be the extra maintenance and replacement-oriented work necessary to deal with the ‘year 2000 problem.’ As for the 2003 investigation, the slow IT-market in general seems to have influenced the results negatively seen from the point of view of application systems support efficiency in organization.
---
paper_title: Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

paper_content:

Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.
---
paper_title: Measuring Reproducibility in Computer Systems Research

paper_content:

We describe a study into the willingness of Computer Systems researchers to share their code and data. We nd that . . . . We also propose a novel sharing specication scheme that will require researchers to specify the level of reproducibility that reviewers and readers can assume from a paper either submitted for publication, or published.
---
paper_title: Uncertainty, Complexity and Concepts of Good Science in Climate Change Modelling: Are GCMs the Best Tools?

paper_content:

In this paper we explore the dominant position of a particular style of scientific modelling in the provision of policy-relevant scientific knowledge on future climate change. We describe how the apical position of General Circulation Models (GCMs) appears to follow ‘logically’ both from conventional understandings of scientific representation and the use of knowledge, so acquired, in decision-making. We argue, however, that both of these particular understandings are contestable. In addition to questioning their current policy-usefulness, we draw upon existing analyses of GCMs which discuss model trade-offs, errors, and the effects of parameterisations, to raise questions about the validity of the conception of complexity in conventional accounts. An alternative approach to modelling, incorporating concepts of uncertainty, is discussed, and an illustrative example given for the case of the global carbon cycle. In then addressing the question of how GCMs have come to occupy their dominant position, we argue that the development of global climate change science and global environmental ‘management’ frameworks occurs concurrently and in a mutually supportive fashion, so uniting GCMs and environmental policy developments in certain industrialised nations and international organisations. The more basic questions about what kinds of commitments to theories of knowledge underpin different models of ‘complexity’ as a normative principle of ‘good science’ are concealed in this mutual reinforcement. Additionally, a rather technocratic policy orientation to climate change may be supported by such science, even though it involves political choices which deserve to be more widely debated.
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: Growth in a Time of Debt

paper_content:

We study economic growth and inflation at different levels of government and external debt. Our analysis is based on new data on forty-four countries spanning about two hundred years. The dataset incorporates over 3,700 annual observations covering a wide range of political systems, institutions, exchange rate arrangements, and historic circumstances. Our main findings are: First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 percent of GDP. Above 90 percent, median growth rates fall by one percent, and average growth falls considerably more. We find that the threshold for public debt is similar in advanced and emerging economies. Second, emerging markets face lower thresholds for external debt (public and private)--which is usually denominated in a foreign currency. When external debt reaches 60 percent of GDP, annual growth declines by about two percent; for higher levels, growth rates are roughly cut in half. Third, there is no apparent contemporaneous link between inflation and public debt levels for the advanced countries as a group (some countries, such as the United States, have experienced higher inflation when debt/GDP is high.) The story is entirely different for emerging markets, where inflation rises sharply as debt increases.
---
paper_title: Lessons from the JMCB Archive

paper_content:

We examine the online archive of the Journal of Money, Credit, and Banking, in which an author is required to deposit the data and code that replicate the results of his paper. We find that most authors do not fulfill this requirement. Of more than 150 empirical articles, fewer than 15 could be replicated. Despite all this, there is no doubt that a data/code archive is more conducive to replicable research than the alternatives. We make recommendations to improve the functioning of the archive.
---
paper_title: When Software Engineers Met Research Scientists: A Case Study

paper_content:

This paper describes a case study of software engineers developing a library of software components for a group of research scientists, using a traditional, staged, document-led methodology. The case study reveals two problems with the use of the methodology. The first is that it demands an upfront articulation of requirements, whereas the scientists had experience, and hence expectations, of emergent requirements; the second is that the project documentation does not suffice to construct a shared understanding. Reflecting on our case study, we discuss whether combining agile elements with a traditional methodology might have alleviated these problems. We then argue that the rich picture painted by the case study, and the reflections on methodology that it inspires, has a relevance that reaches beyond the original context of the study.
---
paper_title: An overview of the Trilinos project

paper_content:

The Trilinos Project is an effort to facilitate the design, development, integration, and ongoing support of mathematical software libraries within an object-oriented framework for the solution of large-scale, complex multiphysics engineering and scientific problems. Trilinos addresses two fundamental issues of developing software for these problems: (i) providing a streamlined process and set of tools for development of new algorithmic implementations and (ii) promoting interoperability of independently developed software.Trilinos uses a two-level software structure designed around collections of packages. A Trilinos package is an integral unit usually developed by a small team of experts in a particular algorithms area such as algebraic preconditioners, nonlinear solvers, etc. Packages exist underneath the Trilinos top level, which provides a common look-and-feel, including configuration, documentation, licensing, and bug-tracking.Here we present the overall Trilinos design, describing our use of abstract interfaces and default concrete implementations. We discuss the services that Trilinos provides to a prospective package and how these services are used by various packages. We also illustrate how packages can be combined to rapidly develop new algorithms. Finally, we discuss how Trilinos facilitates high-quality software engineering practices that are increasingly required from simulation software.
---
paper_title: Configuration Management for Large-Scale Scientific Computing at the UK Met Office

paper_content:

The UK Met Office's flexible configuration management (FCM) system uses existing open source tools, adapted for use with high-performance scientific Fortran code, to help manage evolving code in its large-scale climate simulation and weather forecasting models. FCM has simplified the development process, improved team coordination, and reduced release cycles.
---
paper_title: A survey of scientific software development

paper_content:

Software for scientific research purposes has received increased attention in recent years. Case studies have noted development practices, limitations, and problems in the development of scientific software. However, applicability of the results of these studies to improving the wider scientific software development practices is not known. This paper presents a survey of 60 scientific software developers. The survey was conducted online from August--September 2009, and aims to identify where improvements to scientific software practices can be made. While our results generally confirm previous work, we have found some notable differences. The use of IDEs and version control tools among the surveyed scientific software developers has increased, and trace-ability of scientific software is not as important to scientific software developers as it is to scientific software users. Documentation also appears to be more widely produced than previous studies indicate. However, there remains room for improvement in tool use, documentation, testing, and verification activities for scientific software development.
---
paper_title: When Software Engineers Met Research Scientists: A Case Study

paper_content:

This paper describes a case study of software engineers developing a library of software components for a group of research scientists, using a traditional, staged, document-led methodology. The case study reveals two problems with the use of the methodology. The first is that it demands an upfront articulation of requirements, whereas the scientists had experience, and hence expectations, of emergent requirements; the second is that the project documentation does not suffice to construct a shared understanding. Reflecting on our case study, we discuss whether combining agile elements with a traditional methodology might have alleviated these problems. We then argue that the rich picture painted by the case study, and the reflections on methodology that it inspires, has a relevance that reaches beyond the original context of the study.
---
paper_title: An overview of the Trilinos project

paper_content:

The Trilinos Project is an effort to facilitate the design, development, integration, and ongoing support of mathematical software libraries within an object-oriented framework for the solution of large-scale, complex multiphysics engineering and scientific problems. Trilinos addresses two fundamental issues of developing software for these problems: (i) providing a streamlined process and set of tools for development of new algorithmic implementations and (ii) promoting interoperability of independently developed software.Trilinos uses a two-level software structure designed around collections of packages. A Trilinos package is an integral unit usually developed by a small team of experts in a particular algorithms area such as algebraic preconditioners, nonlinear solvers, etc. Packages exist underneath the Trilinos top level, which provides a common look-and-feel, including configuration, documentation, licensing, and bug-tracking.Here we present the overall Trilinos design, describing our use of abstract interfaces and default concrete implementations. We discuss the services that Trilinos provides to a prospective package and how these services are used by various packages. We also illustrate how packages can be combined to rapidly develop new algorithms. Finally, we discuss how Trilinos facilitates high-quality software engineering practices that are increasingly required from simulation software.
---
paper_title: Software Development Environments for Scientific and Engineering Software: A Series of Case Studies

paper_content:

The need for high performance computing applications for computational science and engineering projects is growing rapidly, yet there have been few detailed studies of the software engineering process used for these applications. The DARPA High Productivity Computing Systems Program has sponsored a series of case studies of representative computational science and engineering projects to identify the steps involved in developing such applications (i.e. the life cycle, the workflows, technical challenges, and organizational challenges). Secondary goals were to characterize tool usage and identify enhancements that would increase the programmers' productivity. Finally, these studies were designed to develop a set of lessons learned that can be transferred to the general computational science and engineering community to improve the software engineering process used for their applications. Nine lessons learned from five representative projects are presented, along with their software engineering implications, to provide insight into the software development environments in this domain.
---
paper_title: Software Development Cultures and Cooperation Problems: A Field Study of the Early Stages of Development of Software for a Scientific Community

paper_content:

In earlier work, I identified a particular class of end-user developers, who include scientists and whom I term `professional end-user developers', as being of especial interest. Here, I extend this work by articulating a culture of professional end-user development, and illustrating by means of a field-study how the influence of this culture causes cooperation problems in an inter-disciplinary team developing a software system for a scientific community. My analysis of the field study data is informed by some recent literature on multi-national work cultures. Whilst acknowledging that viewing a scientific development through a lens of software development culture does not give a full picture, I argue that it nonetheless provides deep insights.
---
paper_title: Configuration Management for Large-Scale Scientific Computing at the UK Met Office

paper_content:

The UK Met Office's flexible configuration management (FCM) system uses existing open source tools, adapted for use with high-performance scientific Fortran code, to help manage evolving code in its large-scale climate simulation and weather forecasting models. FCM has simplified the development process, improved team coordination, and reduced release cycles.
---
paper_title: The ASC-Alliance Projects: A Case Study of Large-Scale Parallel Scientific Code Development

paper_content:

Computational scientists face many challenges when developing software that runs on large-scale parallel machines. However, software-engineering researchers haven't studied their software development processes in much detail. To better understand the nature of software development in this context, the authors examined five large-scale computational science software projects operated at the five ASC-Alliance centers.
---
paper_title: Supporting scientific SE process improvement

paper_content:

The increasing complexity of scientific software can result in significant impacts on the research itself. In traditional software development projects, teams adopt historical best practices into their development processes to mitigate the risk of such problems. In contrast, the gap that has formed between the traditional and scientific software communities leaves scientists to rely on only their own experience when facing software process improvement (SPI) decisions. Rather than expect scientists to become software engineering (SE) experts or the SE community to learn all of the intricacies involved in scientific software development projects, we seek a middle ground. The Scientific Software Process Improvement Framework (SciSPIF) will allow scientists to self-drive their own SPI efforts while leveraging the collective experiences of their peers and linking their concerns to established SE best practices. This proposal outlines the known challenges of scientific software development, relevant concepts from traditional SE research, and our planned methodology for collecting the data required to construct SciSPIF while staying grounded in the actual goals and concerns of the scientists.
---
paper_title: Spatial Data e-Infrastructure

paper_content:

This paper examines overlap between e-Science and geospatial communities using work undertaken as part of the JISC funded Secure Access to Geospatial Services (SEE-GEO) project. Working from a position that open standards provide the best means of achieving interoperability, the case studies demonstrate the use of Grid technology and open geospatial standard interfaces to realise classic spatial data infrastructure scenarios. These examples are used to illustrate just how e-infrastructures enable seamless, security driven spatial data access at the national, regional and global scale.
---
paper_title: Exploring XP for scientific research

paper_content:

Can we successfully apply XP (Extreme Programming) in a scientific research context? A pilot project at the NASA Langley Research Center tested XPs applicability in this context. Since the cultural environment at a government research center differs from the customer-centric business view, eight of XPs 12 practices seemed incompatible with the existing research culture. Despite initial awkwardness, the authors determined that XP can function in situations for which it appears to be ill suited.
---
paper_title: Engineering the Software for Understanding Climate Change

paper_content:

Climate scientists build large, complex simulations with little or no software engineering training—and don't readily adopt the latest software engineering tools and techniques. This ethnographic study of climate scientists shows that their culture and practices share many features of agile and open source projects, but with highly customized software validation and verification techniques.
---
paper_title: Assuring the Future? A Look at Validating Climate Model Software

paper_content:

The scientific community studying climate change uses a variety of strategies to assess the correctness of their models. These software systems represent large, sophisticated, fine-grained scientific tools. The validation practices described are thus tailored to a domain in which software and software engineering practices are useful but cannot be allowed to get in the way of the science. In audio interviews, two scientists--Robert Jacob, a computational climate scientist at Argonne National Laboratory, and Gavin Schmidt, a climatologist and climate modeler at the NASA Goddard Institute for Space Studies--discuss what it means to develop and communicate ground-breaking results.
---
paper_title: Scientific Software Development at a Research Facility

paper_content:

Software engineers at Daresbury Laboratory develop experiment control and data acquisition software to support scientific research. Here, they review their experiences and learning over the years.
---
paper_title: Agile methods in biomedical software development: a multi-site experience report

paper_content:

BackgroundAgile is an iterative approach to software development that relies on strong collaboration and automation to keep pace with dynamic environments. We have successfully used agile development approaches to create and maintain biomedical software, including software for bioinformatics. This paper reports on a qualitative study of our experiences using these methods.ResultsWe have found that agile methods are well suited to the exploratory and iterative nature of scientific inquiry. They provide a robust framework for reproducing scientific results and for developing clinical support systems. The agile development approach also provides a model for collaboration between software engineers and researchers. We present our experience using agile methodologies in projects at six different biomedical software development organizations. The organizations include academic, commercial and government development teams, and included both bioinformatics and clinical support applications. We found that agile practices were a match for the needs of our biomedical projects and contributed to the success of our organizations.ConclusionWe found that the agile development approach was a good fit for our organizations, and that these practices should be applicable and valuable to other biomedical software development efforts. Although we found differences in how agile methods were used, we were also able to identify a set of core practices that were common to all of the groups, and that could be a focus for others seeking to adopt these methods.
---
paper_title: What Do We Know about Scientific Software Development's Agile Practices?

paper_content:

The development of scientific software has similarities with processes that follow the software engineering "agile manifesto": responsiveness to change and collaboration are of utmost importance. But how well do current scientific software-development processes match the practices found in agile development methods, and what are the effects of using agile practices in such processes?
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: Chaste: using agile programming techniques to develop computational biology software

paper_content:

Cardiac modelling is the area of physiome modelling where the available simulation software is perhaps most mature, and it therefore provides an excellent starting point for considering the software requirements for the wider physiome community. In this paper, we will begin by introducing some of the most advanced existing software packages for simulating cardiac electrical activity. We consider the software development methods used in producing codes of this type, and discuss their use of numerical algorithms, relative computational efficiency, usability, robustness and extensibility. We then go on to describe a class of software development methodologies known as test-driven agile methods and argue that such methods are more suitable for scientific software development than the traditional academic approaches. A case study is a project of our own, Cancer, Heart and Soft Tissue Environment, which is a library of computational biology software that began as an experiment in the use of agile programming methods. We present our experiences with a review of our progress thus far, focusing on the advantages and disadvantages of this new approach compared with the development methods used in some existing packages. We conclude by considering whether the likely wider needs of the cardiac modelling community are currently being met and suggest that, in order to respond effectively to changing requirements, it is essential that these codes should be more malleable. Such codes will allow for reliable extensions to include both detailed mathematical models--of the heart and other organs--and more efficient numerical techniques that are currently being developed by many research groups worldwide.
---
paper_title: Introducing agile development into bioinformatics: an experience report

paper_content:

This experience report describes our efforts to introduce agile development techniques incrementally into our customer's organization in the National Cancer Institute and develop a partnering relationship in the process. The report addresses the steps we have taken not only to deploy the practices, but also to gain customer support for them. It addresses variations we have used to adapt to our customer's environment, including our approach to involving customer personnel at remote locations. We also address challenges we still must face, including how best to manage a product-line with agile development techniques.
---
paper_title: Domain-specific languages: an annotated bibliography

paper_content:

We survey the literature available on the topic of domain-specific languages as used for the construction and maintenance of software systems. We list a selection of 75 key publications in the area, and provide a summary for each of the papers. Moreover, we discuss terminology, risks and benefits, example domain-specific languages, design methodologies, and implementation techniques.
---
paper_title: Braincurry: A Domain- Specific Language for Integrative Neuroscience

paper_content:

This paper describes Braincurry, a domain-specific, declarative language for describing and analysing experiments in neuroscience. Braincurry has three goals: to allow experiments and data analysis to be described in a way that is sufficiently abstract to serve as a definition; to facilitate carrying out experiments by executing such descriptions; and to be directly usable by end users: neuroscientists. We adopted an experimental and incremental approach to the design and implementation of Braincurry, focusing on the neurophysiological response to visual stimuli in locusts as a test case. Braincurry is currently implemented as an embedding in Haskell, which is a highly effective tool for this kind of exploratory language design. The declarative nature of Haskell and its flexible syntax fitted with our goals. We discuss the requirements for a realistic language meeting the above goals, describe the current Braincurry design and how it may be generalised, and explain how some particularly challenging hard real-time requirements were met.
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: Chaste: using agile programming techniques to develop computational biology software

paper_content:

Cardiac modelling is the area of physiome modelling where the available simulation software is perhaps most mature, and it therefore provides an excellent starting point for considering the software requirements for the wider physiome community. In this paper, we will begin by introducing some of the most advanced existing software packages for simulating cardiac electrical activity. We consider the software development methods used in producing codes of this type, and discuss their use of numerical algorithms, relative computational efficiency, usability, robustness and extensibility. We then go on to describe a class of software development methodologies known as test-driven agile methods and argue that such methods are more suitable for scientific software development than the traditional academic approaches. A case study is a project of our own, Cancer, Heart and Soft Tissue Environment, which is a library of computational biology software that began as an experiment in the use of agile programming methods. We present our experiences with a review of our progress thus far, focusing on the advantages and disadvantages of this new approach compared with the development methods used in some existing packages. We conclude by considering whether the likely wider needs of the cardiac modelling community are currently being met and suggest that, in order to respond effectively to changing requirements, it is essential that these codes should be more malleable. Such codes will allow for reliable extensions to include both detailed mathematical models--of the heart and other organs--and more efficient numerical techniques that are currently being developed by many research groups worldwide.
---
paper_title: A Software Chasm: Software Engineering and Scientific Computing

paper_content:

Some time ago, a chasm opened between the scientific-computing community and the software engineering community. Originally, computing meant scientific computing. Today, science and engineering applications are at the heart of software systems such as environmental monitoring systems, rocket guidance systems, safety studies for nuclear stations, and fuel injection systems. Failures of such health-, mission-, or safety-related systems have served as examples to promote the use of software engineering best practices. Yet, the bulk of the software engineering community's research is on anything but scientific-application software. This chasm has many possible causes. In this article, we look at the impact of one particular contributor in industry.
---
paper_title: Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research

paper_content:

Scholarly dissemination and communication standards are changing to reflect the increasingly computational nature of scholarly research, primarily to include the sharing of the data and code associated with published results. This paper presents a formalized set of best practice recommendations for computational scientists wishing to disseminate reproducible research, facilitate innovation by enabling data and code re-use, and enable broader communication of the output of digital scientific research. We distinguish two forms of collaboration to motivate choices of software environment for computational scientific research. We also present these Best Practices as a living, evolving, and changing document on wiki.
---
paper_title: A Risk-Based, Practice-Centered Approach to Project Management for HPCMP CREATE

paper_content:

The Department of Defense's High Performance Computing Modernization Program Computational Research and Engineering Acquisition Tools and Environments Computational Research and Engineering Acquisition Tools and Environments (CREATE) program is developing and deploying multiphysics high-performance computing software applications for engineers to design and make accurate performance predictions for military aircraft, ships, ground vehicles, and radio frequency antennas. When CREATE started (2007), no commonly recognized set of successful software project management practices existed for developing multiphysics, HPC engineering software. Based on lessons learned from the HPC and computational engineering communities, CREATE leadership developed and implemented a risk-based, practice-centered strategy to organize and manage the highly distributed program. This approach led to a good balance between ensuring a sufficiently structured workflow and accountability and providing the flexibility and agility necessary to create new sets of engineering tools with the capabilities needed to design next-generation weapon systems.
---
paper_title: A new DoD initiative: the Computational Research and Engineering Acquisition Tools and Environments (CREATE) program

paper_content:

In FY2008, the U.S. Department of Defense (DoD) initiated the Computational Research and Engineering Acquisition Tools and Environments (CREATE) program, a $360M program with a two-year planning phase and a ten-year execution phase. CREATE will develop and deploy three computational engineering tool sets for DoD acquisition programs to use to design aircraft, ships and radio-frequency antennas. The planning and execution of CREATE are based on the 'lessons learned' from case studies of large-scale computational science and engineering projects. The case studies stress the importance of a stable, close-knit development team; a focus on customer needs and requirements; verification and validation; flexible and agile planning, management, and development processes; risk management; realistic schedules and resource levels; balanced short- and long-term goals and deliverables; and stable, long-term support by the program sponsor. Since it began in FY2008, the CREATE program has built a team and project structure, developed requirements and begun validating them, identified candidate products, established initial connections with the acquisition programs, begun detailed project planning and development, and generated the initial collaboration infrastructure necessary for success by its multi-institutional, multidisciplinary teams.
---
paper_title: MAINTAINING CORRECTNESS IN SCIENTIFIC PROGRAMS

paper_content:

Combine a high rate of change (which makes correctness hard to maintain) with an increased sensitivity to failure to maintain correctness and you have a big problem. Solving this problem must be the focus of our methodology. In this paper, the author describes the layered approach that he found to be the most successful in maintaining correctness in the face of rapid change.
---
paper_title: Five Recommended Practices for Computational Scientists Who Write Software

paper_content:

Few software engineering techniques and approaches are specifically useful for computational scientists, and despite recent efforts, it could be many years before a consolidated handbook is available. Meanwhile, computational scientists can look to the practices of other scientists who write successful software.
---
paper_title: Software Carpentry: Getting Scientists to Write Better Code by Making Them More Productive

paper_content:

For the past years, my colleagues and I have developed a one-semester course that teaches scientists and engineers the "common core" of modern software development. Our experience shows that an investment of 150 hours-25 of lectures and the rest of practical work-can improve productivity by roughly 20 percent. That's one day a week, one less semester in a master's degree, or one less year for a typical PhD. The course is called software carpentry, rather than software engineering, to emphasize the fact that it focuses on small-scale and immediately practical issues. All of the material is freely available under an open-source license at www.swc.scipy.org and can be used both for self-study and in the classroom. This article describes what the course contains, and why
---
paper_title: Verification of Computer Simulation Models

paper_content:

The problem of validating computer simulation models of industrial systems has received only limited attention in the management science literature. The purpose of this paper is to consider the problem of validating computer models in the light of contemporary thought in the fields of philosophy of science, economic theory, and statistics. In order to achieve this goal we have attempted to gather together and present some of the ideas of scientific philosophers, economists, statisticians, and practitioners in the field of simulation which are relevant to the problem of verifying simulation models. We have paid particular attention to the writings of economists who have been concerned with testing the validity of economic models. Among the questions which we shall consider are included: What does it mean to verify a computer model of an industrial system? Are there any differences between the verification of computer models and the verification of other types of models? If so, what are some of these differences? Also considered are a number of measures and techniques for testing the "goodness of fit" of time series generated by computer models to observed historical series.
---
paper_title: 'Integronsters', integral and integrated modeling

paper_content:

In many cases model integration treats models as software components only, ignoring the fluid relationship between models and reality, the evolving nature of models and their constant modification and recalibration. As a result, with integrated models we find increased complexity, where changes that used to impact only relatively contained models of subsystems, now propagate throughout the whole integrated system. This makes it harder to keep the overall complexity under control and, in a way, defeats the purpose of modularity, when efficiency is supposed to be gained from independent development of modules. Treating models only as software in solving the integration challenge may give birth to 'integronsters' - constructs that are perfectly valid as software products but ugly or even useless as models. We argue that one possible remedy is to learn to use data sets as modules and integrate them into the models. Then the data that are available for module calibration can serve as an intermediate linkage tool, sitting between modules and providing a module-independent baseline dynamics, which is then incremented when scenarios are to be run. In this case it is not the model output that is directed into the next model input, but model output is presented as a variation around the baseline trajectory, and it is this variation that is then fed into the next module down the chain. However still with growing overall complexity, calibration can become an important limiting factor, giving more promise to the integral approach, when the system is modeled and simplified as a whole.
---
paper_title: Computational Science Demands a New Paradigm

paper_content:

The field has reached a threshold at which better organization becomes crucial. New methods of verifying and validating complex codes are mandatory if computational science is to fulfill its promise for science and society.
---
paper_title: Lessons from Space

paper_content:

Given the parallels between the complexity of human spaceflight and large software systems, there are many things we developers can learn from successful space programs, such as the Soyuz. First, limiting a project's scope and complexity early on can have a dramatic payoff in its success and longevity. In addition, adding generous margins to early estimates (and any subsequent revisions) will ease the pain of development and deployment. Furthermore, gradual evolution with a working program at each step, rather than massive rewrites, benefits from successful architectures and teams, while also retaining the software's customer base and third-party contributors. Finally, a well-defined modular structure can increase the software's versatility yielding economies of scope and scale over its lifetime.
---
paper_title: Engineering the Software for Understanding Climate Change

paper_content:

Climate scientists build large, complex simulations with little or no software engineering training—and don't readily adopt the latest software engineering tools and techniques. This ethnographic study of climate scientists shows that their culture and practices share many features of agile and open source projects, but with highly customized software validation and verification techniques.
---
paper_title: Predictive Capability Maturity Model for computational modeling and simulation.

paper_content:

The Predictive Capability Maturity Model (PCMM) is a new model that can be used to assess the level of maturity of computational modeling and simulation (M&S) efforts. The development of the model is based on both the authors experience and their analysis of similar investigations in the past. The perspective taken in this report is one of judging the usefulness of a predictive capability that relies on the numerical solution to partial differential equations to better inform and improve decision making. The review of past investigations, such as the Software Engineering Institute's Capability Maturity Model Integration and the National Aeronautics and Space Administration and Department of Defense Technology Readiness Levels, indicates that a more restricted, more interpretable method is needed to assess the maturity of an M&S effort. The PCMM addresses six contributing elements to M&S: (1) representation and geometric fidelity, (2) physics and material model fidelity, (3) code verification, (4) solution verification, (5) model validation, and (6) uncertainty quantification and sensitivity analysis. For each of these elements, attributes are identified that characterize four increasing levels of maturity. Importantly, the PCMM is a structured method for assessing the maturity of an M&S effort that is directed toward an engineering applicationmore » of interest. The PCMM does not assess whether the M&S effort, the accuracy of the predictions, or the performance of the engineering system satisfies or does not satisfy specified application requirements.« less
---
paper_title: Self-Perceptions about Software Engineering: A Survey of Scientists and Engineers

paper_content:

Scientists and engineers devote considerable effort to developing large, complex codes to solve important problems. However, while they often develop useful code, many scientists and engineers are frequently unaware of how various software engineering practices can help them write better code. This article presents the results of a survey on this topic.
---
paper_title: Understanding the High-Performance-Computing Community: A Software Engineer's Perspective

paper_content:

Computational scientists developing software for HPC systems face unique software engineering issues. Attempts to transfer SE technologies to this domain must take these issues into account.
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: Managing Chaos: Lessons Learned Developing Software in the Life Sciences

paper_content:

In the life sciences, the need to balance the costs and benefits of introducing software processes into a research environment presents a distinct set of challenges due to the cultural disconnect between life sciences research and software engineering. The Institute for Systems Biology's research informatics team has studied these challenges and developed a software process to address them.
---
paper_title: Hints on Test Data Selection: Help for the Practicing Programmer

paper_content:

In many cases tests of a program that uncover simple errors are also effective in uncovering much more complex errors. This so-called coupling effect can be used to save work during the testing process.
---
paper_title: A Scientific Function Test Framework for Modular Environmental Model Development: Application to the Community Land Model

paper_content:

As environmental models have become more complicated, we need new tools to analyze and validate these models and to facilitate collaboration among field scientists, observation dataset providers, environmental system modelers, and computer scientists. Modular design and function test of environmental models have gained attention recently within the Biological and Environmental Research Program of the U.S. Department of Energy. In this paper, we will present our methods and software tools 1) to analyze environmental software and 2) to generate modules for scientific function testing of environmental models. We have applied these methods to the Community Land Model with three typical scenarios: 1) benchmark case function validation, 2) observation-constraint function validation, and 3) a virtual root module generation for root function investigation and evaluation. We believe that our strategies and experience in scientific function test framework can be beneficial to many other research programs that adapt integrated environmental modeling methodology.
---
paper_title: Testing Scientific Software: A Systematic Literature Review

paper_content:

Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. Results: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. Conclusions: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
---
paper_title: Examining random and designed tests to detect code mistakes in scientific software

paper_content:

Abstract Successfully testing computational software to detect code mistakes is impacted by multiple factors. One factor is the tolerance accepted in test output. Other factors are the nature of the code mistake, the characteristics of the code structure, and the choice of test input. We have found that randomly generated test input is a viable approach to testing for code mistakes and that simple structural metrics have little predictive power in the type of testing required. We provide further evidence that reduction of tolerance in expected test output has a much larger impact than running many more tests to discover code mistakes.
---
paper_title: Lessons from Space

paper_content:

Given the parallels between the complexity of human spaceflight and large software systems, there are many things we developers can learn from successful space programs, such as the Soyuz. First, limiting a project's scope and complexity early on can have a dramatic payoff in its success and longevity. In addition, adding generous margins to early estimates (and any subsequent revisions) will ease the pain of development and deployment. Furthermore, gradual evolution with a working program at each step, rather than massive rewrites, benefits from successful architectures and teams, while also retaining the software's customer base and third-party contributors. Finally, a well-defined modular structure can increase the software's versatility yielding economies of scope and scale over its lifetime.
---
paper_title: Building PDE codes to be verifiable and validatable

paper_content:

For codes that solve nonlinear partial differential equations (PDEs), powerful methodologies already exist for verification of codes, verification of calculations, and validation (V2V). If computational scientists and engineers are serious about these issues, they will take the responsibility and the relatively little extra effort to design (or modify) their codes so that independent users can confirm V2V.
---
paper_title: Engineering the Software for Understanding Climate Change

paper_content:

Climate scientists build large, complex simulations with little or no software engineering training—and don't readily adopt the latest software engineering tools and techniques. This ethnographic study of climate scientists shows that their culture and practices share many features of agile and open source projects, but with highly customized software validation and verification techniques.
---
paper_title: Assuring the Future? A Look at Validating Climate Model Software

paper_content:

The scientific community studying climate change uses a variety of strategies to assess the correctness of their models. These software systems represent large, sophisticated, fine-grained scientific tools. The validation practices described are thus tailored to a domain in which software and software engineering practices are useful but cannot be allowed to get in the way of the science. In audio interviews, two scientists--Robert Jacob, a computational climate scientist at Argonne National Laboratory, and Gavin Schmidt, a climatologist and climate modeler at the NASA Goddard Institute for Space Studies--discuss what it means to develop and communicate ground-breaking results.
---
paper_title: Limited discrepancy search revisited

paper_content:

Harvey and Ginsberg's limited discrepancy search (LDS) is based on the assumption that costly heuristic mistakes are made early in the search process. Consequently, LDS repeatedly probes the state space, going against the heuristic (i.e., taking discrepancies) a specified number of times in all possible ways and attempts to take those discrepancies as early as possible. LDS was improved by Richard Korf, to become improved LDS (ILDS), but in doing so, discrepancies were taken as late as possible, going against the original assumption. Many subsequent algorithms have faithfully inherited Korf's interpretation of LDS, and take discrepancies late. This then raises the question: Should we take our discrepancies late or early? We repeat the original experiments performed by Harvey and Ginsberg and those by Korf in an attempt to answer this question. We also investigate the early stopping condition of the YIELDS algorithm, demonstrating that it is simple, elegant and efficient.
---
paper_title: On Testing Non-testable Programs

paper_content:

It is widely accepted that the fundamental limitation of using program testing techniques to determine the correctness of a program is the inability to extrapolate from the correctness of results for a proper subset of the input domain to the program's correctness for all elements of the domain. In particular, for any proper subset of the domain there are infinitely many programs which produce the correct output on those elements, but produce an incorrect output for some other domain element. None the less we routinely test programs to increase our confidence in their correctness, and a great deal of research is currently being devoted to improving the effectiveness of program testing. These efforts fall into three primary categories: (1) the development of a sound theoretical basis for testing; (2) devising and improving testing methodologies, particularly mechanizable ones; (3) the definition of accurate measures of and criteria for test data adequacy. Almost all of the research on software testing therefore focuses on the development and analysis of input data. In particular there is an underlying assumption that once this phase is complete, the remaining tasks are straightforward. These consist of running the program on the selected data, producing output which is then examined to determine the program's correctness on the test data. The mechanism which checks this correctness is known as an oracle, and the belief that the tester is routinely able to determine whether or not the test output is correct is the oracle assumption.' • 2
---
paper_title: Software Engineering for Scientists

paper_content:

At two recent workshops, participants discussed the juxtaposition of software engineering with the development of scientific computational software.
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: The ASC-Alliance Projects: A Case Study of Large-Scale Parallel Scientific Code Development

paper_content:

Computational scientists face many challenges when developing software that runs on large-scale parallel machines. However, software-engineering researchers haven't studied their software development processes in much detail. To better understand the nature of software development in this context, the authors examined five large-scale computational science software projects operated at the five ASC-Alliance centers.
---
paper_title: Assessing climate model software quality : a defect density analysis of three models

paper_content:

Abstract. A climate model is an executable theory of the climate; the model encapsulates climatological theories in software so that they can be simulated and their implications investigated. Thus, in order to trust a climate model, one must trust that the software it is built from is built correctly. Our study explores the nature of software quality in the context of climate modelling. We performed an analysis of defect reports and defect fixes in several versions of leading global climate models by collecting defect data from bug tracking systems and version control repository comments. We found that the climate models all have very low defect densities compared to well-known, similarly sized open-source projects. We discuss the implications of our findings for the assessment of climate model software trustworthiness.
---
paper_title: How Well Do Coupled Models Simulate Today's Climate?

paper_content:

Coupled climate models are sophisticated tools designed to simulate the Earth climate system and the complex interactions between its components. Currently, more than a dozen centers around the world develop climate models to enhance our understanding of climate and climate change and to support the activities of the Intergovernmental Panel on Climate Change (IPCC). However, climate models are not perfect. Our theoretical understanding of climate is still incomplete, and certain simplifying assumptions are unavoidable when building these models. This introduces biases into their simulations, which sometimes are surprisingly difficult to correct. Model imperfections have attracted criticism, with some arguing that model-based projections of climate
---
paper_title: Software Testing and Verification in Climate Model Development

paper_content:

Over the past 30 years, most climate models have grown from relatively simple representations of a few atmospheric processes to complex multidisciplinary systems. Computer infrastructure over that period has gone from punchcard mainframes to modern parallel clusters. Model implementations have become complex, brittle, and increasingly difficult to extend and maintain. Verification processes for model implementations rely almost exclusively on some combination of detailed analyses of output from full climate simulations and system-level regression tests. Besides being costly in terms of developer time and computing resources, these testing methodologies are limited in the types of defects they can detect, isolate, and diagnose. Mitigating these weaknesses of coarse-grained testing with finer-grained unit tests has been perceived as cumbersome and counterproductive. Recent advances in commercial software tools and methodologies have led to a renaissance of systematic fine-grained testing. This opens new possibilities for testing climate-modeling-software methodologies.
---
paper_title: The ASC-Alliance Projects: A Case Study of Large-Scale Parallel Scientific Code Development

paper_content:

Computational scientists face many challenges when developing software that runs on large-scale parallel machines. However, software-engineering researchers haven't studied their software development processes in much detail. To better understand the nature of software development in this context, the authors examined five large-scale computational science software projects operated at the five ASC-Alliance centers.
---
paper_title: Task-directed software inspection

paper_content:

Software inspection is recognized as an effective verification technique. Despite this fact, the use of inspection is surprisingly low. This paper describes a new inspection technique, called task-directed inspection (TDI), and a light-weight process, that were used to introduce inspection in a particular industrial environment. This environment had no history of inspections, was resistant to the idea of inspection, but had a situation where confidence in a safety-related legacy suite of software had to be increased. The characteristics of TDI are explored. They give rise to a variety of approaches that may encourage more widespread use of inspections. This paper examines the industrial exercise as a case study, with the intent that it be useful in other situations that share characteristics with the situation described.
---
paper_title: Scientific Software Testing: Analysis with Four Dimensions

paper_content:

By analyzing our testing exercise through the four dimensions of context, goals, techniques, and adequacy, we developed a better understanding of how to effectively test a piece of scientific software. Once we considered the scientist-tester as part of the testing system, the exercise evolved in a way that made use of and increased his knowledge of the software. One result was an approach to software assessment that combines inspection with code execution. An other result was the suppression of process-driven testing in favor of goal centric approaches. The combination of software engineer working with scientist was success ful in this case. The software engineer brings a toolkit of ideas, and the scientist chooses and fashions the tools into some thing that works for a specific situation. Unlike many other types of software systems, scientific software includes the scientist as an integral part of the system. The tools that support the scientist must include the scientist's knowledge and goals in their design. This represents a different way of considering the juxtaposition of software engineering with scientific software development.
---
paper_title: The Chimera of Software Quality

paper_content:

Despite years of computing progress, today's systems experience spectacular and all-too-frequent crashes, while many enormously expensive projects fail to produce anything useful. Of equal importance, and potentially more damaging, are the misleading smaller defects we tend to miss. From time to time, we must remind ourselves that the underlying quality of the software that our results and progress increasingly depend on will likely be flawed and even more dependent on independent corroboration than the science itself. Many scientific results are corrupted, perhaps fatally so, by undiscovered mistakes in the software used to calculate and present those results.
---
paper_title: The T-experiments: errors in scientific software

paper_content:

Extensive tests showed that many software codes widely used in science and engineering are not as accurate as we would like to think. It is argued that better software engineering practices would help solve this problem, but realizing that the problem exists is an important first step.
---
paper_title: What every computer scientist should know about floating-point arithmetic

paper_content:

Floating-point arithmetic is considered as esoteric subject by many people. This is rather surprising, because floating-point is ubiquitous in computer systems: Almost every language has a floating-point datatype; computers from PCs to supercomputers have floating-point accelerators; most compilers will be called upon to compile floating-point algorithms from time to time; and virtually every operating system must respond to floating-point exceptions such as overflow. This paper presents a tutorial on the aspects of floating-point that have a direct impact on designers of computer systems. It begins with background on floating-point representation and rounding error, continues with a discussion of the IEEE floating point standard, and concludes with examples of how computer system builders can better support floating point.
---
paper_title: Dealing with Risk in Scientific Software Development

paper_content:

The development of scientific software involves risk in the underlying theory, its implementation, and its use. Through a series of interviews, the authors explored how research scientists at two Canadian universities developed their software. These interviews indicated that the scientists used a set of strategies to address risk. They also suggested where the software engineering community could perform research focused on specific problems faced by scientific software developers.
---
paper_title: A Component Architecture for High-Performance Scientific Computing

paper_content:

The Common Component Architecture (CCA) provides a means for software developers to manage the complexity of large-scale scientific simulations and to move toward a plug-and-play environment for high-performance coputing. In the scientific computing context, component models also promote collaboration using independently developed software, thereby allowing particular individals or groups to focus on the aspects of greatest interest to them. The CCA supports parallel and distributed coputing as well as local high-performance connections between components in a language-independent manner. The design places minimal requirements on components and thus facilitates the integration of existing code into the CCA environment. The CCA model imposes minimal ovehead to minimize the impact on application performance. The focus on high performance distinguishes the CCA from most other component models. The CCA is being applied within an increasing range of disciplines, including cobustion research, global climate simulation, and computtional chemistry.
---
paper_title: A Component Architecture for High-Performance Scientific Computing

paper_content:

The Common Component Architecture (CCA) provides a means for software developers to manage the complexity of large-scale scientific simulations and to move toward a plug-and-play environment for high-performance coputing. In the scientific computing context, component models also promote collaboration using independently developed software, thereby allowing particular individals or groups to focus on the aspects of greatest interest to them. The CCA supports parallel and distributed coputing as well as local high-performance connections between components in a language-independent manner. The design places minimal requirements on components and thus facilitates the integration of existing code into the CCA environment. The CCA model imposes minimal ovehead to minimize the impact on application performance. The focus on high performance distinguishes the CCA from most other component models. The CCA is being applied within an increasing range of disciplines, including cobustion research, global climate simulation, and computtional chemistry.
---
paper_title: Object construction and destruction design patterns in Fortran 2003

paper_content:

Abstract This paper presents object-oriented design patterns in the context of object construction and destruction. The examples leverage the newly supported object-oriented features of Fortran 2003. We describe from the client perspective two patterns articulated by Gamma et al. [1]: abstract factory and factory method. We also describe from the implementation perspective one new pattern: the object pattern. We apply the Gamma et al. patterns to solve a partial differential equation, and we discuss applying the new pattern to a quantum vortex dynamics code. Finally, we address consequences and describe the use of the patterns in two open-source software projects: ForTrilinos and Morfeus.
---
paper_title: Patterns in scientific software: an introduction

paper_content:

Patterns are well understood methodology for object-oriented software architecture, especially for business applications. Scientific programmers have generally avoided object-oriented approaches because of their heavy computational overhead, but the benefits of using patterns for scientific problems can outweigh their costs. This article introduces the concept of object oriented software patterns and discusses how they can be applied to scientific software problems. After a brief explanation of what patterns are and why they can be relevant to scientific software, the author explores the application of patterns to dynamic-systems simulation, such as molecular dynamics, and identifies four design patterns that emerge in modeling such systems. To illustrate how to reuse a general pattern for a specific problem, he applies one of the dynamic simulation patterns to the different problem of hydrodynamic chemistry tracers.
---
paper_title: Object-oriented design patterns in Fortran 90/95: mazev1, mazev2 and mazev3

paper_content:

This paper discusses the concept, application, and usefulness of software design patterns for scientific programming in Fortran 90/95. An example from the discipline of object-oriented design patterns, that of a game based on navigation through a maze, is used to describe how some important patterns can be implemented in Fortran 90/95 and how the progressive introduction of design patterns can usefully restructure Fortran software as it evolves. This example is complemented by a discussion of how design patterns have been used in a real-life simulation of Particle-in-Cell plasma physics. The following patterns are mentioned in this paper: Factory, Strategy, Template, Abstract Factory and Facade.
---
paper_title: Design Patterns for e-Science

paper_content:

This is a book about a code and about coding. The code is a case study which has been used to teachcourses in e-Science atthe Australian NationalUniv- sity since 2001. Students learn advanced programming skills and techniques TM in the Java language. Above all, they learn to apply useful object-oriented design patterns as they progressively refactor and enhance the software. We think our case study,EScope, is as close to real life as you can get! It is a smaller version of a networked, graphical, waveform browser which is used in the control rooms of fusion energy experiments around the world. It is quintessential e-Science in the sense of e-Science being computer science and information technology in the service of science. It is not, speci?cally, Grid-enabled, but we develop it in a way that will facilitate its deployment onto the Grid. The standard version ofEScope interfaces with a specialised database for waveforms, and related data, known asMDSplus. On the acc- panying CD, we have provided you with software which will enable you to installMDSplus,EScope and sample data ?les onto Windows or Linux c- puters. There is much additional software including many versions of the case study as it gets built up and progressively refactored using design patterns. There will be a home web-site for this book which will contain up-to-date information about the software and other aspects of the case study.
---
paper_title: Design patterns and Fortran 90/95

paper_content:

In the literature on object oriented programming (OO), design patterns are a very popular subject. Apart from any hype that may be connected to the concept, they are supposed to help you look at a programming problem and come up with a robust design for its solution. The reason design patterns work is not that they are something new, but instead that they are time-honoured, well-developed solutions.I will not repeat the story about architectural design patterns and Christopher Alexander who recognised their potential. Instead I will try to explain how these (software) design patterns can be used in setting up Fortran 90/95 programs, despite the "fact" that Fortran 90/95 lacks certain OO features, such as inheritance and polymorphism. It may not be stressed in all OO literature, but design patterns help you find solutions that do not necessarily involve inheritance or polymorphism (cf. Shalloway and Trott, 2002).Design patterns come by fancy names such as the Adapter pattern or the Decorations pattern and explaining what they are and how to use them is best done via a few examples.
---
paper_title: Software Engineering for Scientists

paper_content:

At two recent workshops, participants discussed the juxtaposition of software engineering with the development of scientific computational software.
---
paper_title: Refactoring to Patterns

paper_content:

In 1994, Design Patterns changed the landscape of object-oriented development by introducing classic solutions to recurring design problems. In 1999, Refactoring revolutionized design by introducing an effective process for improving code. With the highly anticipated Refactoring to Patterns, Joshua Kerievsky has changed our approach to design by forever uniting patterns with the evolutionary process of refactoring.This book introduces the theory and practice of pattern-directed refactorings: sequences of low-level refactorings that allow designers to safely move designs to, towards, or away from pattern implementations. Using code from real-world projects, Kerievsky documents the thinking and steps underlying over two dozen pattern-based design transformations. Along the way he offers insights into pattern differences and how to implement patterns in the simplest possible ways.Coverage includes: A catalog of twenty-seven pattern-directed refactorings, featuring real-world code examples Descriptions of twelve design smells that indicate the need for this book's refactorings General information and new insights about patterns and refactoring Detailed implementation mechanics: how low-level refactorings are combined to implement high-level patterns Multiple ways to implement the same pattern-and when to use each Practical ways to get started even if you have little experience with patterns or refactoringRefactoring to Patterns reflects three years of refinement and the insights of more than sixty software engineering thought leaders in the global patterns, refactoring, and agile development communities. Whether you're focused on legacy or “greenfield” development, this book will make you a better software designer by helping you learn how to make important design changes safely and effectively.
---
paper_title: Developing scientific applications using Generative Programming

paper_content:

Scientific applications usually involve large number of distributed and dynamic resources and huge datasets. A mechanism like checkpointing is essential to make these applications resilient to failures. Using checkpointing as an example, this paper presents an approach for integrating the latest software engineering techniques with the development of scientific software. Generative programming is used in this research to achieve the goals of non-intrusive reengineering of existing applications to insert the checkpointing mechanism and to decouple the checkpointing-specifications from its actual implementation. The end-user specifies the checkpointing details at a higher level of abstraction, using which the necessary code is generated and woven into the application. The lessons learned and the implementation approach presented in this paper can be applied to the development of scientific applications in general. The paper also demonstrates that the generated code does not introduce any inaccuracies and its performance is comparable to the manually inserted code.
---
paper_title: Requirements Analysis for Engineering Computation: A Systematic Approach for Improving Reliability

paper_content:

This paper argues that the reliability of engineering computation can be significantly improved by adopting software engineering methodologies for requirements analysis and specification. The argument centers around the fact that the only way to judge the reliability of a system is by comparison to a specification of the requirements. This paper also points to methods for documenting the requirements. In particular, a requirements template is proposed for specifying engineering computation software. To make the mathematical specification easily understandable by all stakeholders, the requirements documentation employs the technique of using tabular expressions. To clarify the presentation, this paper includes a case study of the documentation for a system for analyzing statically determinant beams.
---
paper_title: Electronic Documents Give Reproducible Research a New Meaning

paper_content:

A revolution in education and technology transfer follows from the marriage of word processing and software command scripts. In this marriage an a.uthor attaches to every figure caption a pushbutton or a name tag usable t,o recalculate the figure from all its data, parameters, and programs. This provides a concrete definition of reproducibility in computationally oriented research. Experience at the Stanford Exploration Project shows that preparing such electronic documents is lit,tle effort beyond our customary report writing; mainly, we need to file everything in a systematic way.
---
paper_title: myExperiment: social networking for workflow-using e-scientists

paper_content:

We present the Taverna workflow workbench and argue that scientific workflow environments need a rich ecosystem of tools that support the scientists. experimental lifecycle. Workflows are scientific objects in their own right, to be exchanged and reused. myExperiment is a new initiative to create a social networking environment for workflow workers. We present the motivation for myExperiment and sketch the proposed capabilities and challenges. We argue that actively engaging with a scientist's needs, fears and reward incentives is crucial for success.
---
paper_title: A Universal Identifier for Computational Results

paper_content:

Abstract We present a discipline for verifiable computational scienti_c research. Our discipline revolves around three simple new concepts — verifiable computational result (VCR), VCR repository and Verifiable Result Identifier (VRI). These are web- and cloud-computing oriented concepts, which exploit today's web infrastructure to achieve standard, simple and automatic reproducibility in computational scientific research. The VCR discipline requires very slight modifications to the way researchers already conduct their computational research and authoring, and to the way publishers manage their content. In return, the discipline marks a significant step towards delivering on the long-anticipated promises of making scientific computation truly reproducible. A researcher practicing this discipline in everyday work produces computational scripts and word processor files that look very much like those they already produce today, but in which a few lines change very subtly and naturally. Those scripts produce a stream of verifiable results, which are the same tables, figures, charts and datasets the researcher traditionally would have produced, but which are watermarked for permanent identification by a VRI, and are automatically and permanently stored in a VCR repository. In a scientific community practicing Verifiable Computational Research, exchange of both ideas and data involves exchanging result identifiers—VRIs—rather than exchanging files. These identifiers are controlled, trusted and automatically generated strings that point to publicly available result as it was originally created by the computational process itself. When a verifiable result is included in a publication, its identifier can be used by any reader with a web browser to locate, browse and, where appropriate, re-execute the computation that produced the result. Journal readers can therefore scrutinize, dispute, understand and eventually trust these computational results, all to an extent impossible through textual explanations that constitute the core of scientific publications to date. In addition, the result identi_er can be used by subsequent computations to locate and retrieve both the published result (in graphical or numerical form) and the original datasets used by its generating computation. Colleagues can thus cite and import data into their own computations, just as traditional publications allow them to cite and import ideas. We describe an existing software implementation of the Verifiable Computational Research discipline, and argue that it solves many of the crucial problems commonly facing computer-based and computer-aided research in various scientific fields. Our system is secure, naturally adapted to large-scale and cloud computations and to modern massive data analysis, yet places effectively no additional workload on either the researcher or the publisher.
---
paper_title: A literate experimentation manifesto

paper_content:

This paper proposes a new approach to experimental computer systems research, which we call Literate Experimentation. Conventionally, experimental procedure and writeup are divided into distinct phases: i.e. setup (the method), data collection (the results) and analysis (the evaluation of the results). Our concept of a literate experiment is to have a single, rich, human-generated, text-based description of a particular experiment, from which can be automatically derived: (1) a summary of the experimental setup to include in the paper; (2) a sequence of executable commands to setup a computer platform ready to perform the actual experiment; (3) the experiment itself, executed on this appropriately configured platform; and, (4) a means of generating results tables and graphs from the experimental output, ready for inclusion in the paper. Our Literate Experimentation style has largely been inspired by Knuth's Literate Programming philosophy. Effectively, a literate experiment is a small step towards the executable paper panacea. In this work, we argue that a literate experimentation approach makes it easier to produce rigorous experimental evaluation papers. We suggest that such papers are more likely to be accepted for publication, due to (a) the imposed uniformity of structure, and (b) the assurance that experimental results are easily reproducible. We present a case study of a prototype literate experiment involving memory management in Jikes RVM.
---
paper_title: Taverna , lessons in creating a workflow environment for the life sciences

paper_content:

Life sciences research is based on individuals, often with diverse skills, assembled into research groups. These groups use their specialist expertise to address scientific problems. The in silico experiments undertaken by these research groups can be represented as workflows involving the co-ordinated use of analysis programs and information repositories that may be globally distributed. With regards to Grid computing, the requirements relate to the sharing of analysis and information resources rather than sharing computational power. The myGrid project has developed the Taverna Workbench for the composition and execution of workflows for the life sciences community. This experience paper describes lessons learnt during the development of Taverna. A common theme is the importance of understanding how workflows fit into the scientists' experimental context. The lessons reflect an evolving understanding of life scientists' requirements on a workflow environment, which is relevant to other areas of data intensive and exploratory science.
---
paper_title: The Collage Authoring Environment

paper_content:

Abstract The Collage Authoring Environment is a software infrastructure which enables domain scientists to collaboratively develop and publish their work in the form of executable papers. It corresponds to the recent developments in both e-Science and computational technologies which call for a novel publishing paradigm. As part of this paradigm, static content (such as traditional scientific publications) should be supplemented with elements of interactivity, enabling reviewers and readers to reexamine the reported results by executing parts of the software on which such results are based as well as access primary scientific data. Taking into account the presented rationale we propose an environment which enables authors to seamlessly embed chunks of executable code (called assets) into scientific publications and allow repeated execution of such assets on underlying computing and data storage resources, as required by scientists who wish to build upon the presented results. The Collage Authoring Environment can be deployed on arbitrary resources, including those belonging to high performance computing centers, scientific e-Infrastructures and resources contributed by the scientists themselves. The environment provides access to static content, primary datasets (where exposed by authors) and executable assets. Execution features are provided by a dedicated engine (called the Collage Server) and embedded into an interactive view delivered to readers, resembling a traditional research publication but interactive and collaborative in its scope. Along with a textual description of the Collage environment the authors also present a prototype implementation, which supports the features described in this paper. The functionality of this prototype is discussed along with theoretical assumptions underpinning the proposed system.
---
paper_title: Automated Capture of Experiment Context for Easier Reproducibility in Computational Research

paper_content:

Published scientific research that relies on numerical computations is too often not reproducible. For computational research to become consistently and reliably reproducible, the process must become easier to achieve, as part of day-to-day research. A combination of best practices and automated tools can make it easier to create reproducible research.
---
paper_title: SHARE: a web portal for creating and sharing executable research papers

paper_content:

Abstract This paper describes how SHARE (Sharing Hosted Autonomous Research Environments) satisfies the criteria of the Elsevier 2011 Executable Paper Grand Challenge. This challenge aims at disseminating the use of systems that provide reviewers and fellow scientists a convenient way to reproduce computational results of research papers. This can involve among others the calculation of a number, the plotting of a diagram, the automatic proof of a theorem or the interactive transformation of various inputs into a complex output document. Besides reproducing the literate results, readers of an executable paper should also be able to explore the result space by entering different input parameters than the ones reported in the original text. SHARE is a web portal that enables academics to create, share, and access remote virtual machines that can be cited from research papers. By deploying in SHARE a copy of the required operating system as well as all the relevant software and data, authors can make a conventional paper fully reproducible and interactive. Shared virtual machines can also contain the original paper text— when desirable even with embedded computations. This paper shows the concrete potential of SHARE-based articles by means of an example virtual machine that is based on a conventional research article published by Elsevier recently. More generally, it demonstrates how SHARE has supported the publication workflow of a journal special issue and various workshop proceedings. Finally, it clarifies how the SHARE architecture supports among others the Elsevier challenge's licensing and scalability requirements without domain specific restrictions.
---
paper_title: Developing open source scientific practice

paper_content:

3 Routine practice 8 3.1 Version control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Execution automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.4 Readability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.5 Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
---
paper_title: Literate programming in quantum chemistry: A simple example

paper_content:

Literate programming methods have not been widely used in computational molecular structure theory. We argue that significant advantages would result from the use of literate programming methods in the computational molecular sciences and, indeed, in computational science, in general. Our arguments are illustrated by a simple example of literate programming methods in ab initio electronic structure theory is described. We distinguish text-embedded code, or literate programs, from code-embedded text and suggest that the latter may form a more useful vehicle for publication than literate programs. © 2005 Wiley Periodicals, Inc. Int J Quantum Chem, 2005
---
paper_title: Beyond accuracy: what data quality means to data consumers

paper_content:

Poor data quality (DQ) can have substantial social and economic impacts. Although firms are improving data quality with practical approaches and tools, their improvement efforts tend to focus narrowly on accuracy. We believe that data consumers have a much broader data quality conceptualization than IS professionals realize. The purpose of this paper is to develop a framework that captures the aspects of data quality that are important to data consumers.A two-stage survey and a two-phase sorting study were conducted to develop a hierarchical framework for organizing data quality dimensions. This framework captures dimensions of data quality that are important to data consumers. Intrinsic DQ denotes that data have quality in their own right. Contextual DQ highlights the requirement that data quality must be considered within the context of the task at hand. Representational DQ and accessibility DQ emphasize the importance of the role of systems. These findings are consistent with our understanding that high-quality data should be intrinsically good, contextually appropriate for the task, clearly represented, and accessible to the data consumer.Our framework has been used effectively in industry and government. Using this framework, IS managers were able to better understand and meet their data consumers' data quality needs. The salient feature of this research study is that quality attributes of data are collected from data consumers instead of being defined theoretically or based on researchers' experience. Although exploratory, this research provides a basis for future studies that measure data quality along the dimensions of this framework.
---
paper_title: The Earth System Grid: Supporting the Next Generation of Climate Modeling Research

paper_content:

Understanding the Earth's climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing rapidly. Looking toward mid-decade and beyond, we must anticipate and prepare for distributed climate research data holdings of many petabytes. The Earth System Grid (ESG) is a collaborative interdisciplinary project aimed at addressing the challenge of enabling management, discovery, access, and analysis of these critically important datasets in a distributed and heterogeneous computational environment. The problem is fundamentally a Grid problem. Building upon the Globus toolkit and a variety of other technologies, ESG is developing an environment that addresses authentication, authorization for data access, large-scale data transport and management, services and abstractions for high-performance remote data access, mechanisms for scalable data replication, cataloging with rich semantic and syntactic information, data discovery, distributed monitoring, and Web-based portals for using the system.
---
paper_title: Data Curation at Scale: The Data Tamer System

paper_content:

Data curation is the act of discovering a data source(s) of interest, cleaning and transforming the new data, semantically integrating it with other local data sources, and deduplicating the resulting composite. There has been much research on the various components of curation (especially data integration and deduplication). However, there has been little work on collecting all of the curation components into an integrated end-to-end system. In addition, most of the previous work will not scale to the sizes of problems that we are nding in the eld. For example, one web aggregator requires the curation of 80,000 URLs and a second biotech company has the problem of curating 8000 spreadsheets. At this scale, data curation cannot be a manual (human) eort, but must entail machine learning approaches with a human assist only when necessary. This paper describes Data Tamer, an end-to-end curation system we have built at M.I.T. Brandeis, and Qatar Computing Research Institute (QCRI). It expects as input a sequence of data sources to add to a composite being constructed over time. A new source is subjected to machine learning algorithms to perform attribute identication, grouping of attributes into tables, transformation of incoming data and deduplication. When necessary, a human can be asked for guidance. Also, Data Tamer includes a data visualization component so a human can examine a data source at will and specify manual transformations. We have run Data Tamer on three real world enterprise curation problems, and it has been shown to lower curation cost by about 90%, relative to the currently deployed production software.
---
paper_title: A software architecture-based framework for highly distributed and data intensive scientific applications

paper_content:

Modern scientific research is increasingly conducted by virtual communities of scientists distributed around the world. The data volumes created by these communities are extremely large, and growing rapidly. The management of the resulting highly distributed, virtual data systems is a complex task, characterized by a number of formidable technical challenges, many of which are of a software engineering nature. In this paper we describe our experience over the past seven years in constructing and deploying OODT, a software framework that supports large, distributed, virtual scientific communities. We outline the key software engineering challenges that we faced, and addressed, along the way. We argue that a major contributor to the success of OODT was its explicit focus on software architecture. We describe several large-scale, real-world deployments of OODT, and the manner in which OODT helped us to address the domain-specific challenges induced by each deployment.
---
paper_title: Verification of Computer Simulation Models

paper_content:

The problem of validating computer simulation models of industrial systems has received only limited attention in the management science literature. The purpose of this paper is to consider the problem of validating computer models in the light of contemporary thought in the fields of philosophy of science, economic theory, and statistics. In order to achieve this goal we have attempted to gather together and present some of the ideas of scientific philosophers, economists, statisticians, and practitioners in the field of simulation which are relevant to the problem of verifying simulation models. We have paid particular attention to the writings of economists who have been concerned with testing the validity of economic models. Among the questions which we shall consider are included: What does it mean to verify a computer model of an industrial system? Are there any differences between the verification of computer models and the verification of other types of models? If so, what are some of these differences? Also considered are a number of measures and techniques for testing the "goodness of fit" of time series generated by computer models to observed historical series.
---
paper_title: Automated testing of refactoring engines

paper_content:

Refactorings are behavior-preserving program transformations that improve the design of a program. Refactoring engines are tools that automate the application of refactorings: first the user chooses a refactoring to apply, then the engine checks if the transformation is safe, and if so, transforms the program. Refactoring engines are a key component of modern IDEs, and programmers rely on them to perform refactorings. A bug in the refactoring engine can have severe consequences as it can erroneously change large bodies of source code. We present a technique for automated testing of refactoring engines. Test inputs for refactoring engines are programs. The core of our technique is a framework for iterative generation of structurally complex test inputs. We instantiate the framework to generate abstract syntax trees that represent Java programs. We also create several kinds of oracles to automatically check that the refactoring engine transformed the generated program correctly. We have applied our technique to testing Eclipse and NetBeans, two popular open-source IDEs for Java, and we have exposed 21 new bugs in Eclipse and 24 new bugs in NetBeans.
---
paper_title: The Recomputation Manifesto

paper_content:

Replication of scientific experiments is critical to the advance of science. Unfortunately, the discipline of Computer Science has never treated replication seriously, even though computers are very good at doing the same thing over and over again. Not only are experiments rarely replicated, they are rarely even replicable in a meaningful way. Scientists are being encouraged to make their source code available, but this is only a small step. Even in the happy event that source code can be built and run successfully, running code is a long way away from being able to replicate the experiment that code was used for. I propose that the discipline of Computer Science must embrace replication of experiments as standard practice. I propose that the only credible technique to make experiments truly replicable is to provide copies of virtual machines in which the experiments are validated to run. I propose that tools and repositories should be made available to make this happen. I propose to be one of those who makes it happen.
---
paper_title: Managing Chaos: Lessons Learned Developing Software in the Life Sciences

paper_content:

In the life sciences, the need to balance the costs and benefits of introducing software processes into a research environment presents a distinct set of challenges due to the cultural disconnect between life sciences research and software engineering. The Institute for Systems Biology's research informatics team has studied these challenges and developed a software process to address them.
---
paper_title: A Software Chasm: Software Engineering and Scientific Computing

paper_content:

Some time ago, a chasm opened between the scientific-computing community and the software engineering community. Originally, computing meant scientific computing. Today, science and engineering applications are at the heart of software systems such as environmental monitoring systems, rocket guidance systems, safety studies for nuclear stations, and fuel injection systems. Failures of such health-, mission-, or safety-related systems have served as examples to promote the use of software engineering best practices. Yet, the bulk of the software engineering community's research is on anything but scientific-application software. This chasm has many possible causes. In this article, we look at the impact of one particular contributor in industry.
---
paper_title: A vision towards Scientific Communication Infrastructures On bridging the realms of Research Digital Libraries and Scientific Data Centers

paper_content:

The two pillars of the modern scientific communication are Data Centers and Research Digital Libraries (RDLs), whose technologies and admin staff support researchers at storing, curating, sharing, and discovering the data and the publications they produce. Being realized to maintain and give access to the results of complementary phases of the scientific research process, such systems are poorly integrated with one another and generally do not rely on the strengths of the other. Today, such a gap hampers achieving the objectives of the modern scientific communication, that is, publishing, interlinking, and discovery of all outcomes of the research process, from the experimental and observational datasets to the final paper. In this work, we envision that instrumental to bridge the gap is the construction of "Scientific Communication Infrastructures". The main goal of these infrastructures is to facilitate interoperability between Data Centers and RDLs and to provide services that simplify the implementation of the large variety of modern scientific communication patterns.
---
paper_title: SHARE: a web portal for creating and sharing executable research papers

paper_content:

Abstract This paper describes how SHARE (Sharing Hosted Autonomous Research Environments) satisfies the criteria of the Elsevier 2011 Executable Paper Grand Challenge. This challenge aims at disseminating the use of systems that provide reviewers and fellow scientists a convenient way to reproduce computational results of research papers. This can involve among others the calculation of a number, the plotting of a diagram, the automatic proof of a theorem or the interactive transformation of various inputs into a complex output document. Besides reproducing the literate results, readers of an executable paper should also be able to explore the result space by entering different input parameters than the ones reported in the original text. SHARE is a web portal that enables academics to create, share, and access remote virtual machines that can be cited from research papers. By deploying in SHARE a copy of the required operating system as well as all the relevant software and data, authors can make a conventional paper fully reproducible and interactive. Shared virtual machines can also contain the original paper text— when desirable even with embedded computations. This paper shows the concrete potential of SHARE-based articles by means of an example virtual machine that is based on a conventional research article published by Elsevier recently. More generally, it demonstrates how SHARE has supported the publication workflow of a journal special issue and various workshop proceedings. Finally, it clarifies how the SHARE architecture supports among others the Elsevier challenge's licensing and scalability requirements without domain specific restrictions.
---
paper_title: Reproducible research for scientific computing: Tools and strategies for changing the culture

paper_content:

This article considers the obstacles involved in creating reproducible computational research as well as some efforts and approaches to overcome them.
---

Title: Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming

Section 1: INTRODUCTION
Description 1: This section introduces the indispensable role of software in scientific research and outlines the benefits and challenges of its use.

Section 2: CASE STUDIES OF SOFTWARE PROCESSES
Description 2: This section reviews various case studies exploring the relationship between scientific research and software development practices.

Section 3: Scientific Programming in Practice
Description 3: This section provides an overview of published case studies focusing on general practices in scientific programming.

Section 4: Agile Methods
Description 4: Exploration of the use and adaptation of agile methods and practices in scientific programming teams is discussed in this section.

Section 5: Project Team Evolution and Software Documentation
Description 5: This section discusses the evolutionary nature of scientific software development teams and the importance of documentation.

Section 6: Best Practices
Description 6: Proposals for best practices in scientific programming based on experiences and observations in the domain are summarized in this section.

Section 7: QUALITY ASSURANCE PRACTICES
Description 7: This section addresses the importance of quality assurance practices in scientific programming and explores various techniques such as testing and inspections.

Section 8: Testing
Description 8: This section delves into the challenges and methodologies of testing scientific software to ensure reliability and accuracy.

Section 9: Inspections
Description 9: The use of inspections as a quality assurance technique in scientific programming, including methodologies and case studies, is discussed here.

Section 10: Continuous Integration
Description 10: Recent practices of continuous integration employed in scientific programming efforts are reviewed in this section.

Section 11: Formal Methods
Description 11: This section covers the application of formal methods in scientific programming and their benefits in ensuring software correctness.

Section 12: DESIGN, EVOLUTION, AND MAINTENANCE
Description 12: This section discusses the management and maintenance of long-term scientific software projects, highlighting component architectures, design patterns, and refactoring techniques.

Section 13: Component Architectures
Description 13: The integration and structuring of heterogeneous component architectures in scientific computing are outlined in this section.

Section 14: Design Patterns
Description 14: Exploration of applying software design patterns to scientific programming to manage complexity is provided in this section.

Section 15: Refactoring and Reengineering Techniques
Description 15: Techniques for refactoring and reengineering legacy scientific software to enhance maintainability are discussed in this section.

Section 16: Workflow Management and Executable Research Papers
Description 16: This section reviews tools and approaches for managing scientific workflows and the role of executable research papers in enhancing reproducibility.

Section 17: DATA QUALITY
Description 17: The challenges and methods for ensuring data quality in scientific research are discussed in this section.

Section 18: CONCLUSIONS
Description 18: The concluding section summarizes the challenges and unresolved issues in integrating software engineering practices into scientific programming and proposes future directions for research and practice.