EDOCKET NO:
                             EPA-HQ-OAR-2014-0609

                    REVIEW OF WIPP PERFORMANCE ASSESSMENT
                      COMPUTER CODE MIGRATION ACTIVITIES

                    U. S. Environmental Protection Agency
                      Office of Radiation and Indoor Air
                 Center for Waste Management and Regulations
                             Washington, DC 20460

                                   July 2015

Table of Contents
Preface	7
Executive Summary	8
Background	10
Software Qualification	13
Post CCA SQA Upgrades and Documentation	18
Overview of DOE's Computer Code Migration Activities	20
Current Status	20
DOE'S Test Methodology	22
References	23
The Agency's Review Approach	25
Summary of Individual Computer Code Migration	26
ALGEBRACDB	26
1.1.1	Introduction	26
1.1.2	Test Methodology	27
1.1.3	Test Results	27
1.1.4	The Agency's Conclusions	28
1.1.5	References	28
BLOTCDB	29
1.1.6	Introduction	29
1.1.7	Test Methodology	30
1.1.8	Test Results	30
1.1.9	The Agency's Conclusions	30
1.1.10	References	31
BRAGFLO	32
1.1.11	Introduction	32
1.1.12	Test Methodology	34
1.1.13	Test Results	35
1.1.14	The Agency's Conclusions	36
1.1.15	References	36
CCDFGF	38
1.1.16	Introduction	38
1.1.17	Test Methodology	40
1.1.18	Test Results	42
1.1.19	The Agency's Conclusions	42
1.1.20	References	43
CCDFSUM	45
1.1.21	Introduction	45
1.1.22	Test Methodology	45
1.1.23	Test Results	46
1.1.24	The Agency's Conclusions	46
1.1.25	References	46
CUTTINGS_S	47
1.1.26	Introduction	47
1.1.27	Test Methodology	49
1.1.28	Test Results	50
1.1.29	The Agency's Conclusions	50
1.1.30	References	51
DRSPALL	52
1.1.31	Introduction	53
1.1.32	Test Methodology	54
1.1.33	Test Results	55
1.1.34	The Agency's Conclusions	57
1.1.35	References	57
DTRKMF	58
1.1.36	Introduction	58
1.1.37	Test Methodology	59
1.1.38	Test Results	59
1.1.39	The Agency's Conclusions	60
1.1.40	References	60
EPAUNI	61
1.1.41	Introduction	61
1.1.42	Test Methodology	62
1.1.43	Test Results	63
1.1.44	The Agency's Conclusions	63
1.1.45	References	63
EQ3/6		64
1.1.46	Introduction	64
1.1.47	Test Methodology	65
1.1.48	Test Results	65
1.1.49	The Agency's Conclusion	66
1.1.50	References	66
FMT		66
1.1.51	Introduction	67
1.1.52	Test Methodology	68
1.1.53	Test Results	68
1.1.54	The Agency's Conclusions	68
1.1.55	References	69
GENMESH	70
1.1.56	Introduction	70
1.1.57	Test Methodology	71
1.1.58	Test Results	71
1.1.59	The Agency's Conclusions	71
1.1.60	References	72
GROPECDB	73
1.1.61	Introduction	73
1.1.62	Test Methodology	74
1.1.63	Test Results	74
1.1.64	The Agency's Conclusions	74
1.1.65	References	74
ICSET		75
1.1.66	Introduction	75
1.1.67	Test Methodology	76
1.1.68	Test Results	76
1.1.69	The Agency's Conclusions	77
1.1.70	References	77
JAS3D		78
1.1.71	Introduction	78
1.1.72	Test Methodology	79
1.1.73	Test Results	79
1.1.74	The Agency's Conclusion	79
1.1.75	References	79
LHS		80
1.1.76	Introduction	80
1.1.77	Test Methodology	81
1.1.78	Test Results	82
1.1.79	The Agency's Conclusions	83
1.1.80	References	83
MATSET	84
1.1.81	Introduction	84
1.1.82	Test Methodology	85
1.1.83	Test Results	86
1.1.84	The Agency's Conclusions	86
1.1.85	References	87
MODFLOW2000	88
1.1.86	Introduction	88
1.1.87	Test Methodology	89
1.1.88	Test Results	91
1.1.89	The Agency's Conclusions	91
1.1.90	References	91
MWT3D	92
1.1.91	Introduction	92
1.1.92	Test Methodology	93
1.1.93	Test Results	93
1.1.94	The Agency's Conclusions	93
1.1.95	References	93
NONLIN	94
1.1.96	Introduction	94
1.1.97	Test Methodology	94
1.1.98	Test Results	95
1.1.99	The Agency's Conclusions	95
1.1.100	References	95
NUTS		95
1.1.101	Introduction	96
1.1.102	Test Methodology	97
1.1.103	Test Results	97
1.1.104	The Agency's Conclusions	98
1.1.105	References	98
PANEL	100
1.1.106	Introduction	100
1.1.107	Test Methodology	101
1.1.108	Test Results	102
1.1.109	The Agency's Conclusions	102
1.1.110	References	102
PEST		104
1.1.111	Introduction	104
1.1.112	Test Methodology	107
1.1.113	Test Results	108
1.1.114	The Agency's Conclusions	108
1.1.115	References	109
POSTBRAG	110
1.1.116	Introduction	110
1.1.117	Test Methodology	111
1.1.118	Test Results	111
1.1.119	The Agency's Conclusions	111
1.1.120	References	111
POSTLHS	112
1.1.121	Introduction	112
1.1.122	Test Methodology	113
1.1.123	Test Results	114
1.1.124	The Agency's Conclusions	114
1.1.125	References	114
POSTSECOTP2D	115
1.1.126	Introduction	116
1.1.127	Test Methodology	117
1.1.128	Test Results	117
1.1.129	The Agency's Conclusions	118
1.1.130	References	118
PREBRAG	119
1.1.131	Introduction	119
1.1.132	Test Methodology	121
1.1.133	Test Results	121
1.1.134	The Agency's Conclusions	121
1.1.135	References	122
PRECCDFGF	123
1.1.136	Introduction	123
1.1.137	Test Methodology	125
1.1.138	Test Results	125
1.1.139	The Agency's Conclusions	125
1.1.140	References	125
PRELHS	126
1.1.141	Introduction	127
1.1.142	Test Methodology	128
1.1.143	Test Results	128
1.1.144	The Agency's Conclusions	129
1.1.145	References	129
PRESECOTP2D	130
1.1.146	Introduction	131
1.1.147	Test Methodology	132
1.1.148	Test Results	132
1.1.149	The Agency's Conclusions	132
1.1.150	References	132
RELATE	133
1.1.151	Introduction	134
1.1.152	Test Methodology	134
1.1.153	Test Results	135
1.1.154	The Agency's Conclusions	135
1.1.155	References	135
SECOTP2D	136
1.1.156	Introduction	136
1.1.157	Test Methodology	137
1.1.158	Test Results	137
1.1.159	The Agency's Conclusions	138
1.1.160	References	138
STEPWISE	139
1.1.161	Introduction	139
1.1.162	Test Methodology	140
1.1.163	Test Results	140
1.1.164	The Agency's Conclusions	141
1.1.165	References	141
SUMMARIZE	141
1.1.166	Introduction	142
1.1.167	Test Methodology	144
1.1.168	Test Results	146
1.1.169	The Agency's Conclusions	146
1.1.170	References	146
LIBRARIES	148
1.1.171	CAMCON_LIB	149
1.1.172	CAMDAT_LIB	152
1.1.173	CAMSUPES_LIB	154
1.1.174	PLT_LIB	156
1.1.175	SDBREAD_ LIB	158
Codes Used to Support the Inventory Report	161
1.1.176	ORIGEN2	161
1.1.177	TransOrigen	171
1.1.178	Comprehensive Inventory Database	175
Summary and Conclusions	182

Preface
The U.S. Department of Energy (DOE) is required to submit a Compliance Recertification Application (CRA) to the U.S. Environmental Protection Agency (EPA) for the Waste Isolation Pilot Plant (WIPP) facility every five years including an updated assessment of future WIPP performance. During EPA's review of DOE's CRA-2014 performance assessment (PA), events associated with the February 2014 repository fire and radionuclide release have resulted in closed portions of the underground facility. This closure has created a situation where certain parts of the underground facility could not be accessed for ground control. Panel 9 may be abandoned along with plans to install panel closures in panels 3, 4, 5 and 6.
Because the CRA performance assessments are predictions of post-closure repository performance and the EPA knows there will be modifications to the current repository design, modifying the CRA-2014 PA at this time to incorporate alternative parameter values would not add more reality to predictions of repository post-closure performance. Consequently, the EPA adopted the CRA-2014 PA as originally submitted by DOE as the baseline, rather than have DOE conduct a revised PA baseline calculation (PABC). In lieu of requesting a PABC-2014, the EPA requested that DOE conduct a set of sensitivity studies to address some of the significant technical concerns arising from the EPA's CRA-2014 review. The inputs to these sensitivity studies broadly address many of the EPA's technical concerns that could potentially impact long-term repository performance. The Agency has reviewed the results of these studies and determined that there exists an adequate level of confidence -- that is, a reasonable expectation -- that the repository will continue to comply with EPA regulations.

This report describes the results of the U.S. Environmental Protection Agency's (EPA or the Agency) review of Performance Assessment (PA) computer code development and testing activities performed by the U.S. Department of Energy (DOE or the Department) in support of their ongoing PA of the Waste Isolation Pilot Plant (WIPP).  The ability of the WIPP facility to meet the Agency's certification requirements was demonstrated, in part, through the use of a series of PA computer codes that are documented in the Department's Compliance Certification Application (CCA).

Executive Summary

The Waste Isolation Pilot Plant (WIPP), located in southeastern New Mexico, is an underground facility designed for the permanent disposal of Transuranic (TRU) defense-related waste.  The U.S. Department of Energy (the Department or DOE) operates the WIPP repository under the regulatory oversight of the U.S. Environmental Protection Agency (the Agency or EPA).  The ability of the DOE WIPP facility to continue to meet the certification requirements of the EPA is demonstrated in part through the use of a series of performance assessment (PA) computer codes. DOE must demonstrate on an ongoing basis that PA computer software is in compliance with regulations outlined in 40 CFR 194.23  -  Models and Computer Codes.  Since the Agency's certification of the DOE WIPP Compliance Certification Application (CCA), DOE has added computer hardware and upgraded the computer software.  In order to maintain compliance with Sections 194.22 and 194.23, DOE is required to conduct testing on the computer codes to ensure that they still function properly on new hardware and software whenever changes are made.  The Agency reviewed the testing performed by DOE to demonstrate continued compliance with the addition of computer hardware and upgraded software.

For the CCA, performance analyses were run on the DEC Alpha Cluster using the OpenVMS operating system, Version 6.1.  In 1999, the operating system was updated from OpenVMS 6.1 to 7.1, and a year later from OpenVMS 7.1 to 7.2.  In the summer of 2001, the FORTRAN compiler available on the cluster was upgraded to Version 7.4A.  In August 2002, the operating system was upgraded to an OpenVMS 7.3-1.

In addition to software upgrades, DOE has made hardware changes.  The DEC Alpha Cluster was the main platform for performance analyses for the WIPP during the CCA.  The cluster consisted of 11 DEC Alpha 2100 computers with 44 processors.  In September 2001, a single Compaq Alpha ES40 computer was added to the WIPP PA hardware cluster.  In August 2002, the DEC Alphas were replaced by a Compaq ES40.

In 2003, two new hardware systems were added to the PA computational cluster -- the Compaq ES45 and the Compaq Alpha 8400.  In September 2004, the Agency approved 38 (of 39) computer codes and 3 libraries for use on the Compaq ES45 and 8400 using OpenVMS 7.3-1.

In 2006, the DOE procured four Compaq ES47 machines to add to the PA computing resources of two Compaq ES40 and two Compaq ES45 machines.  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 has been upgraded to OpenVMS 8.2.  Because of these changes in the operating system and the addition of a new computing platform, DOE has conducted regression testing for each PA software code to ensure that each code continues to function correctly.  The regression test methodology uses the VMS DIFFERENCE command to compare output from the latest version of the computer code and/or operating system to the earlier versions.  The regression test cases are outlined in the Validation Document (VD) and are run using the WIPP PA run control system.  The scripts, script input files, and other files related to validation testing of the code reside in the Configuration Management System (CMS) library.  All test inputs are fetched at run time by the scripts, and test outputs/results and run logs are automatically stored by the scripts the CMS library.  The VMS DIFFERENCE command compares two files and identifies records that are different in the two files.  Records with differences are grouped into sections; a section begins with a record that is different between the two files, and ends with first subsequent record where the two files agree.  In the output of the DIFFERENCE command, sections are separated by rows of 12 asterisks; inside a section, the records from the two files are separated by a row of 6 asterisks.  At the end of the DIFFERENCE output, the utility reports the number of sections and the number of records in which differences were found.  Differences that are limited to code run dates and time, file and directory names, user names, platform names and execution statistics are acceptable.  Differences involving numerical output require analysis to determine the origin of the differences and whether the differences affect the code's performance.  Numerical differences may be determined to be acceptable based on the analysis of each difference.  If all differences are found to be acceptable, it follows that the output of the newer code version and/or operating system meets the acceptance criteria specified in the Requirements Document (RD/VVP), and the code will be considered to be validated on the platforms and operating systems that are tested.

In 2015, the Agency updated this report to document the review of two significant changes to the WIPP performance assessment since the last WIPP recertification in 2009.  First, codes changed to support the PA calculations performed on the VMS computer platform to support the 2014 recertification (CRA14) submission. Second, the migration of the WIPP PA system from the VMS platform to the Sun Solaris Blade platform will be used in future compliance calculations.

Most PA codes were not modified or re-qualified to support the CRA14. Six PA codes were modified and qualified for use in CRA14, the EPA's review findings are discussed in this report.

In 2012 and 2013, DOE had SNL migrate the WIPP performance assessment from the aging OpenVMS Alpha Cluster to a Sun Solaris Blade Server running Intel processors [3].  AP-162 [3] documents the requirements and migration process.  SNL notes, "The primary motivation for the PA migration is diminishing technical and hardware support for the OpenVMS and the Alpha servers." Migration to the Solaris Blade includes the PA codes, support libraries, and the PA parameter database. SNL's summary report [4] describes the results of the migration to the Solaris Blade.

This report presents the Agency's findings with respect to the qualification of six modified codes on the VMS platform using OpenVMS 8.2 and migration and qualification of the WIPP PA computer codes running on the Solaris Blade machines with SunOS 5.11 (Table 3.1-1). After conducting a review, the Agency concludes that the versions of the computer codes specified in Table 3.1-1 are approved, except EQ3/6, for use in PA compliance calculations running VMS platform using OpenVMS 8.2 for the 2014 CRA PA (table column "2014 CRA") and on the Solaris Blade machines with SunOS 5.11 for future WIPP PAs (table column "SOLARIS MIGRATION").
Background

This report describes the results of the U.S. Environmental Protection Agency's (EPA or the Agency) review of Performance Assessment (PA) computer code development and testing activities performed by the U.S. Department of Energy (DOE or the Department) in support of their ongoing PA of the Waste Isolation Pilot Plant (WIPP).  The ability of the WIPP facility to meet the Agency's certification requirements was demonstrated, in part, through the use of a series of PA computer codes that are documented in the Department's Compliance Certification Application (CCA).

DOE conducted a PA to show compliance with the Agency's disposal regulations as part of the WIPP certification process.  DOE must demonstrate on an ongoing basis that PA computer software is in compliance with regulations outlined in §194.22  -  Quality Assurance and §194.23  -  Models and Computer Codes.  These regulations are presented in Appendix A.  Examples of software that must meet the compliance criteria are as follows:

 Scientific or engineering software used to assess the performance of a site
 Scientific or engineering software used to analyze data for, or produce input (parameters) to, a PA calculation
 Software that is used in managing information or augmenting mission-essential decisions
 Software used to collect data (e.g., far-field, near-field, engineered barriers)

DOE executes the PA conceptual models through software applications with parameter value inputs on an infrastructure composed of computers and operating systems that must be periodically updated.  For the CCA, PA analyses were run on the DEC Alpha Cluster using the OpenVMS operating system, Version 6.1.  In 1999, the operating system was updated from OpenVMS 6.1 to 7.1, and a year later from OpenVMS 7.1 to 7.2.  In the summer of 2001, the FORTRAN compiler available on the cluster was upgraded to Version 7.4A.  In August 2002, the operating system was upgraded to an OpenVMS 7.3-1.  In 2006, SNL updated the operating systems and installed the OpenVMS 8.2 operating system. In 2012, SNL modified six performance assessment codes to support the 2014 recertification. In 2013, SNL migrated the entire WIPP PA to the Sun Solaris Blade Server hardware running the SunOS 5.11 operating system. In 2014, codes residing on other computers, such as Linux "Alice", were also migrated to the Solaris Blade.

In addition to software upgrades, DOE has made hardware changes.  The DEC Alpha Cluster was the main platform for performance analyses for the WIPP during the CCA.  The cluster consisted of 11 DEC Alpha 2100 computers with 44 processors.  In September 2001, a single Compaq Alpha ES40 computer was added to the WIPP PA hardware cluster.  In August 2002, the DEC Alphas were replaced by a Compaq ES40.  In 2003, two new hardware systems were added to the PA computational cluster; the Compaq ES45 and the Compaq Alpha 8400.  For the 2004 Compliance Recertification Application (CRA) PA, DOE used OpenVMS 7.3-1 as the operating system in conjunction with the Compaq ES40, ES45, and 8400.  In June 2003, the EPA presented their findings with respect to their review of 27 codes and 3 libraries that were migrated to the Compaq ES40.  The Agency concluded that all of the 27 codes and 3 libraries were migrated successfully to the Compaq ES40 with OpenVMS 7.3-1 and were approved for use in compliance calculations for the WIPP PA.
With respect to the Compaq ES45 and 8400 hardware systems, most of the computer codes have undergone regression testing by DOE to ensure that each code will function correctly on the ES45 and 8400 platform running OpenVMS 7.3-1.  In March 2004 (Docket A-98-49, II-B3-70), the Agency concluded that 36 (of 39) computer codes and 3 libraries migrated to the Compaq ES45 and 8400 using OpenVMS 7.3-1 were acceptable and were approved for use in compliance calculations for the WIPP PA.  In September 2004 (Docket A-98-49, II-B1- 7), the Agency published their findings with respect to the qualification of the computer codes on the Compaq ES45 and the Compaq Alpha 8400.  At that time, the Agency concluded that 38 (of 39) computer codes and 3 libraries migrated to the Compaq ES45 and 8400 using OpenVMS 7.3-1 were acceptable and approved for use in compliance calculations for the WIPP PA.  SANTOS was the only code not approved for PA calculations.  A detailed analysis of SANTOS completed in 2005 concluded that the approximations of room closure and waste compaction developed by the SANTOS model are adequate for use in WIPP PA (Docket A-98-49, II-B1-17).  In 2005, DOE made revisions to eight of the computer codes; including LHS, POSTLHS, CUTTINGS, DRSPALL, PANEL, SUMMARIZE, PRECCDFGF, and CCDFGF.  In March 2006 (Docket A-98-49, II-B1-8), the Agency concluded that these codes were qualified on the Compaq ES40 and ES45.

In 2006, the DOE procured four Compaq ES47 machines to add to the PA computing resources of two Compaq ES40 and two Compaq ES45 machines.  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 has been upgraded to OpenVMS 8.2.  Because of these changes in the operating system and the addition of a new computing platform, DOE has conducted regression testing for each PA software code to ensure that each code continues to function correctly as required.  This report documents the results of the Agency's assessment performed to determine whether the observed DOE PA code activities conform with the compliance criteria requirements for §194.22 and §194.23.  Specifically, the Agency's evaluation addresses whether these changes have materially affected the Agency's original determination that the computer codes were adequate to support the certification decision.

In the 2012 and 2013 timeframe SNL performed PA calculations to support the 2014 recertification.  All but six of the performance assessment codes remained the same for these calculations and the hardware framework described above did not change.  The six codes modified and re-qualified for the 2014 CRA are BRAGFLO, CCDFGF, PRECCDFGF, MATSET, PRELHS, and PREBRAG.  BRAGFLO and PREBRAGFLO were modified to include water balance calculations, to support case CRA14-0 in the 2014 CRA.  CCDFGF was changed in 2010 to investigate the impact of lack of mass balance in the mobilization of radionuclides. MATSET, PRELHS, and PRECCDFGF were modified to use the new parameter database software, MySQL.

In 2012 and 2013, SNL also systematically migrated the PA codes, code libraries, and parameter database to the Solaris Blade Server [3].  SNL could not simply use regression testing to verify PA code performance as done previously because VMS uses a different floating point format than the Intel Solaris OS [4]. Because of the difference in hardware architecture between the VMS platforms and the Sun Solaris, SNL had to use a combination of regression testing (comparing previous results to new results calculated on the new hardware and operating system) and code validation testing using functional requirements [3, 4]. PA codes, such as BRAGFLO, that require numerical convergence calculations were expected to show different results in some cases and may not be conducive to simple result comparison used in regression testing [4].

This report presents the Agency's findings with respect to the six codes qualified for the 2014 CRA calculation and the qualification of the most recent versions of the PA computer codes running on the Sun Solaris Blade machines with SunOS 5.11 (Table 3.1-1).  After conducting reviews, the Agency concludes that the versions of the computer codes specified in Table 3.1-1 are approved, except EQ3/6, for use in PA compliance calculations. These include the six codes for use in the 2014 CRA running on the Compaq cluster of computers with OpenVMS 8.2 and the migrated PA codes running on the Solaris Blade machines with SunOS 5.11 for the WIPP PA.

Based on the results of the Type 3 problems (Test Cases 4, 10, 11, 12, 13, and 14), EPA cannot approve the use of EQ3/6 Version 8.0a for WIPP PA calculations (see EQ3/6 section below).  EPA further suggests that EQ3/6 Version 8.0a be qualified for the WIPP PA using its own requirements and acceptance criteria specific to the code's usage to support performance assessment calculations.

This report is divided into five sections.  Following this Introduction (Section 1), a Background section (Section 2) presents the approach that DOE has taken to meet the compliance criteria requirements for the computer codes.  The Background section is followed by a summary of DOE's code migration approach and conclusions (Section 3).  Section 4 presents the general approach that the Agency followed to review DOE's code migration activities.  Section 5 summarizes each of the computer codes that were reviewed by the Agency.  Section 6 provides the summary and conclusions.  References are provided at the end of each section.

Software Qualification

To demonstrate that computer software is in compliance with disposal regulations outlined in §194.22, the DOE established a life-cycle management process for software used to support their PA.  Their qualification approach for the software follows the life-cycle phases outlined in ASME NQA-2a-1990 addenda, part 2.7, as follows:

 Planning
 Requirements
 Design
 Implementation
 Validation
 Installation and Checkout
 Maintenance
 Retirement

Life-cycle phases are implemented using an iterative or sequential approach following the process flowchart below (Figure 1).  Each phase and its associated documentation shown in Figure 1 are discussed in the following sections.

Planning Phase

A Software QA Plan (SQAP) is produced during the planning phase for new software development (Figure 1).  Software under configuration control and developed within the scope of these QA requirements does not require a stand-alone SQAP.  Following the development of the SQAP, all specified requirements for each phase must be met and not subvert the intent of the requirements.  SQAPs may be written for an individual code or a set of codes.

Figure 1.	Major Components of DOE's Software Development Process

Requirements Phase

The document produced during the requirements phase is the Requirements Document and Verification and Validation Plan (RD/VVP) (Figure 1), which is a single document identifying the computational requirements of the code (e.g., SECOFL2D must be able to simulate ground-water flow under steady-state conditions).  The RD/VVP also describes how the code will be tested to ensure that those requirements are satisfied (e.g., such as list the code functions that will be tested using test cases).

Design Phase

The Design Document (DD), produced during the design phase, provides the following information (as applicable):

 Theoretical basis (physical process represented)
 Mathematical model (numerical model)
 Control flow and logic
 Data structures
 Functionalities and interfaces of objects, components, functions, and subroutines
 Ranges for data inputs and outputs, in a manner that can be implemented in software

More than one DD may be created during software development.  For example, a high-level design may be developed to match the code design to the requirements and define the overall architecture of the code (define modules and subroutines and their purpose, data structures, subroutine-call hierarchy, code language used, etc.).  Another DD may be developed to define how the modules will function in detail (define call interfaces between routines, define data types, etc.).  A detailed design, as its name implies, is very detailed, down to the level of almost writing the code (pseudocode).  These separate DDs may be combined into a single document.

Implementation Phase

The following documents are produced during the implementation phase:

User's Manual (UM)  -  describes the code's purpose and function, mathematical governing equations, model assumptions, the user's interaction with the code (e.g., how data is input into the code), and the models and methods employed by the code.  The UM generally includes:

 The numerical solution strategy and computational sequence, including program flowcharts and block diagrams.
 The relationship between the numerical strategy and the mathematical strategy (i.e., how boundary or initial conditions are introduced).
 A clear explanation of model derivation.  The derivation starts from generally accepted principles and scientifically proven theories.  The UM justifies each step in the derivation, and notes the introduction of assumptions and limitations.  For empirical and semi-empirical models, the documentation describes how experimental data are used to arrive at the final form of the models.  The UM clearly states the final mathematical form of the model and its application in the computer code.
 Descriptions of any numerical method used in the model that goes beyond simple algebra (e.g., finite-difference, Simpson's rule, cubic splines, Newton-Raphson Methods, and Jacobian Methods).  The UM explains the implementation of these methods in the computer code in sufficient detail, so that an independent reviewer can understand them.
 The derivation of the numerical procedure from the mathematical component model.  The UM gives references for all numerical methods.  It explains the final form of the numerical model and its algorithms.  If the numerical model produces only an intermediate result, such as terms in a large set of linear equations that are later solved by another numerical model, then the UM explains how the model uses intermediate results. The documentation also indicates those variables that are input to and output from the component model.

Implementation Document (ID)  -  provides the information necessary for the re-creation of the code used in the WIPP PA calculation.  Using this information, the computer user can reconstruct the code (e.g., compile the source language) and/or install it on an identical platform to that used in the WIPP PA calculation.  In this manner, the code can be regression-tested against an Agency-approved version for subsequent PA calculations (i.e., CRAs).  The document includes the source-code listing, the subroutine-call hierarchy, and code compilation information.
Validation Phase

The validation phase consists of executing and reviewing the functional test cases identified in the previously approved VVP to demonstrate that the developed software meets the requirements defined for it in the RD.  The Validation Document (VD), produced during this phase, summarizes the results of the code functional testing activities prescribed in the RD/VVP documents for the individual codes, and provides evaluations based on those results.  The VD contains listings of input and output files from computer runs of a model.  The VD also contains reports on code verification, benchmarking, and validation, and documents the results of the QA procedures.

Installation and Checkout Phase

The following documents are produced during the installation and checkout phase:

 The Installation and Checkout (I&C) Form [e.g., Sandia National Laboratories (SNL) and Los Alamos National Laboratories (LANL) form NP 19-1-8]
 The Access Control Memorandum
 The Approved Users' Memorandum

Production Software and/or Baseline Document Change Control

When there are changes to the software baseline, the Change Control Form, SNL and LANL Form NP 19-1-9, is used to document the changes.  Types of changes that may be implemented are:

 Major changes, including new requirements, new design, new models, and new implementation, require a new baseline (i.e., SQAP, RD, DD, VVP, ID, UM, VD) to be documented.  In addition to revising every baseline document, a Change Control Form and Installation and Checkout Form are used.
 Minor changes do not affect the requirements or design and can be documented with an addendum (no more than three addenda per baseline document) or page change to the affected baseline document.  In addition to the Change Control Form, the Installation and Checkout Form must be used.
 Patch changes can be used for very small fixes to the code, usually one or two lines of source code or expanding a field's character length, etc.  Patch changes can be documented and tested with the Change Control Form and Installation & Checkout Form.

System Software and Hardware Change Control

Coding Documentation Standards.  Any change to software must be accompanied by documentation describing the change, the date the change was made, and the name of the person responsible for implementing the change.  This documentation should be clearly identified and placed in the source code (e.g., the actual written computer code text) in the vicinity of the change and at the top of the source code prior to the first executable line.  The code reviewer shall determine if this documentation is clear and sufficient.

Significant System Software or Hardware Changes.  The Code Team/Sponsor (single-user systems) or System Administrator (multi-user systems) proposes significant system software or hardware changes using the Change Control Form, SNL and LANL form NP 19-1-9.  Examples of significant changes to system software or hardware:

 Changes to the operating system, such that the version or level identifier changes
 Changes to the Central Processing Unit (CPU)
 Database management system changes

In general, changes are significant if they impact the results generated by production software or cause recompilation of production software.

Software Problem Report (SPR).  Whenever a software problem is identified, the Code Team/Sponsor evaluates the problem to determine if it is, indeed, a problem (as opposed to user error).  If it is a problem, the SPR process is followed.

The Code Team/Sponsor classifies the problem as major, if it could significantly impact previous uses of code, or if it will require significant modification to the software; otherwise it is classified as minor.  For a major problem, the Responsible Manager identifies affected users to be notified of the problem, and designates qualified personnel to identify and evaluate the impact of the software problem.  The affected analysis is revised, and the evaluation and resolution of the software problem is documented in Part II of the SPR and Evaluation Form.  For a minor problem, this evaluation can be performed by the Code Team/Sponsor.

Configuration Management (Configuration Identification and Status Accounting).  Configuration management is the process for defining the configuration of software products, establishing software configuration baselines, and tracking the status of baseline changes.  A software configuration baseline consists of the source code and baseline documents, and provides objective evidence of technical adequacy.

The Software Configuration Management (SCM) Coordinator maintains a Software Baseline List and makes it available upon request.  The SCM Coordinator performs a completeness review to ensure compliance with the procedure and that necessary components of configuration management are present.

For compliance software, the Software Baseline List contains:

 Code name and version
 Code version date
 Code Team/Sponsor name
 Code classification
 RD version
 VVP version
 DD version
 ID version
 UM version
 VD version
 List of approved users (may be listed by name, organization, group, or task, etc.)
 List of approved system software/hardware configurations
 List of outstanding Software Problem Report (SPR) numbers
 Status of approved changes that are in process
 I&C date

Retirement Phase

To retire a code:
 The Code Team/Sponsor issues a memorandum to the SCM Coordinator requesting that the code be retired and provides a reason for the retirement.
 The SCM Coordinator marks the code as retired in the baseline software list.
 The System Administrator and/or Code Team/Sponsor take action to prevent the use of the retired code.  This could involve removal of the software from the computer or the changing of execution privileges.

Post CCA SQA Upgrades and Documentation

Since the time of CCA, the DOE has implemented upgrades to the software operating systems and computer hardware, which are documented in the following reports:

 Summary of Performance Assessment System Upgrades since the CCA
 Analysis Package for AP-042 (documents the upgrade from OpenVMS operating software from Version 6.1 to Version 7.1)
 Analysis Package for Regression Testing the Upgrade to OpenVMS Version 7.2 on the WIPP DEC Alpha Cluster
 Analysis Package for Regression Testing for the Compaq Alpha ES40 Hardware Upgrade on the WIPP DEC Alpha Cluster
 Analysis Package for Regression Testing for the Upgrade of Operating System to OpenVMS 7.3-1 and Hardware to HP Alpha ES45
 Analysis Report for the ES45 Regression Tests
 Analysis Report for the 8400 Regression Tests
 Individual code regression tests for the addition of the Compaq ES40, ES45, and ES47 machines and the upgrade of the operating system to OpenVMS 8.2

Regression testing was performed on the upgraded operating systems and hardware, which run PA codes to demonstrate that the codes continue to produce acceptable output.  Regression testing, as a discipline, consists of running a set of one or more tests for a computer program and verifying that the output produced in the tests is within previously specified acceptable limits.

The Agency has reviewed the documentation that DOE has developed to assess whether the computer codes still meet the requirements specified in §194.22 and §194.23.  In addition to the references cited above, the Agency reviewed UMs, VDs, IDs, and RD/VVPs for each code.  The Agency also reviewed all of the Change Control and Software Installation and Checkout forms for code modifications made since the CCA.
Overview of DOE's Computer Code Migration Activities

Current Status

In August 2002, the operating system was upgraded to OpenVMS 7.3-1, and the DEC Alpha 2100s was replaced by a Compaq ES40.  In June 2003, the Agency approved the qualification of the computer codes on the Compaq ES40 and the use of the PA computer codes on this computer.  With the exception of NUMBERS, the Agency concluded that all of the remaining 38 codes and 3 libraries migrated to the Compaq ES40 using OpenVMS 7.3-1 were approved for use in compliance calculations for the WIPP PA.  NUMBERS 1.19 has since been approved for use by the Agency (2006; DOCKET NO: A-98-49 II-B1-7).

In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, both running OpenVMS 7.3-1.  This configuration was used for preparing the 2004 CRA.  Because of these changes, regression testing was conducted by DOE for the software codes and three libraries on the Compaq ES45 and 8400 using the OpenVMS 7.3-1 operating system to ensure that each code continues to satisfy all the criteria in its RDs.  In September 2004, the Agency published their findings with respect to the qualification of the computer codes on the Compaq ES40 and the Compaq Alpha 8400 [1].  At that time, the Agency concluded that 38 (of 39) computer codes and 3 libraries migrated to the Compaq ES45 and 8400 using OpenVMS 7.3-1 were approved for use in compliance calculations for the WIPP PA (as noted above, NUMBERS 1.19 has since been approved).

 In 2005, DOE made revisions to eight of the computer codes, including LHS, POSTLHS, CUTTINGS, DRSPALL, PANEL, SUMMARIZE, PRECCDFGF and CCDFGF.  In March 2006 (Docket A-98-49, II-B1-8), the Agency concluded that these codes were qualified on the Compaq ES40 and ES45.

In 2006, the DOE procured four Compaq ES47 machines to add to the PA computing resources of two Compaq ES40 and two Compaq ES45 machines.  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2.  Because of these changes in the operating system and the addition of a new computing platform, DOE conducted regression testing for each PA software code to ensure that each code continues to function correctly.

In 2012 and 2013, SNL performed the performance assessment calculations to support the submission of the 2014 CRA.  Six PA codes were modified and re-qualified for use in the 2014 CRA PA calculations. Also, during the same time period, DOE/SNL procured Sun Solaris Blade Servers with SunOS 5.11 operating system.  SNL migrated the PA codes, code libraries, and parameter database to the Solaris [3, 4].  The migration of the WIPP PA required extensive qualification using regression testing as well as validation against the acceptance criteria of the VVP [4].

This report presents the Agency's findings with respect to the qualification of the computer codes used in the 2014 CRA and PA code migration to the Sun Solaris Blade Servers with SunOS 5.11 as the operating system.  The versions of the computer codes and libraries that are approved for WIPP PA use in PA calculations are presented in Table 3.1-1.

Table 3.1-1.	Computer Codes and Libraries Reviewed and Approved by the Agency

                                 Computer Code
                            Agency Approved Version

                                      CCA
                              2004 CRA 2004 PABC
                              2009 CRA 2009 PABC
                               2014 CRA[a,] [b]
                                   SOLARIS
                                   MIGRATION
ALGEBRACDB
                                     2.35
                                     2.35
                                     2.35
                                      ---
                                     2.36
BLOTCDB
                                      --
                                      --
                                     1.37
                                      ---
                                     1.38
BRAGFLO
                                      4.0
                                      5.0
                                      6.0
                                   6.0/6.02
                                     6.03
CCDFGF
                                     1.01
                                     5.0A
                                     5.02
                                   6.0/6.02
                                     7.01
CCDFSUM
                                     1.01
                                     2.00
                                     2.00
                                      ---
                                   Replaced
CUTTINGS_S
                                     5.03
                                     5.04A
                                     6.02
                                      ---
                                     6.03
DRSPALL
                                      NA
                                      1.0
                                     1.10
                                      ---
                                     1.22
DTRKMF
                                      NA
                                      1.0
                                      1.0
                                      ---
                                     1.01
EPAUNI
                                     1.14
                                     1.15A
                                     1.15A
                                      ---
                                     1.16
EQ3/6  - not approved for PA

                                 8.0A Windows
FMT
                                      NA
                                     2.40
                                     2.40
                                      ---
                                   Replaced
GENMESH
                                     6.08
                                     6.08
                                     6.08
                                      ---
                                     6.09
GROPECDB
                                     2.12
                                     2.12
                                     2.12
                                      ---
                                     2.13
ICSET
                                     2.21
                                     2.22
                                     2.22
                                      ---
                                     2.23
JAS3D

                                  2.4.C-LINUX
LHS
                                    2.32Z0
                                     2.41
                                     2.42
                                      ---
                                     2.43
MATSET
                                      9.0
                                     9.10
                                     9.10
                                     9.20
                                     9.21
MODFLOW2000
                                      NA
                                     1.60
                                     1.60
                                      ---
                                     1.70
MWT3D

                                     2.50
NONLIN

                                     2.01
                                      ---
                                     2.02
NUTS
                                     2.02
                                     2.05A
                                     2.05C
                                      ---
                                     2.06
PANEL
                                      3.6
                                     4.02
                                     4.03
                                      ---
                                     4.04
PEST
                                      NA
                                     5.51
                                     9.11
                                      ---
                                     9.12
POSTBRAG
                                     4.00
                                     4.00
                                     4.00
                                      ---
                                     4.02
POSTLHS
                                     4.07
                                     4.07
                                     4.07A
                                      ---
                                     4.08
POSTSECOTP2D
                                     1.02
                                     1.04
                                     1.04
                                      ---
                                     1.05
PREBRAG
                                      6.0
                                     7.00
                                     8.00
                                   8.00/8.02
                                     8.03
PRECCDFGF
                                      1.0
                                     1.00B
                                     1.01
                                      2.0
                                   1.06/2.01
PRELHS
                                     2.10
                                     2.10
                                     2.30
                                     2.40
                                     2.41
PRESECOTP2D
                                     1.20
                                     1.22
                                     1.22
                                      ---
                                     1.23
RELATE
                                     1.43
                                     1.43
                                     1.43
                                      ---
                                     1.45
SANTOS2
                                     2.17
                                     2.17
                                     2.17
                                      ---
                                   Replaced
SECOTP2D
                                     1.30
                                     1.41
                                     1.41A
                                      ---
                                     1.43
STEPWISE
                                     2.20
                                     2.21
                                     2.21
                                      ---
                                     2.22
SUMMARIZE
                                     2.10
                                     2.20
                                     3.01
                                      ---
                                     3.02

Libraries

CAMCON_LIB
                                     2.16
                                     2.20
                                     2.21
                                      ---
                                 Section 5.30
CAMDAT_LIB
                                     1.22
                                     1.25
                                     1.25
                                      ---
                                 Section 5.30
CAMSUPES_LIB
                                     2.18
                                     2.21
                                     2.22
                                      ---
                                 Section 5.30
PLT_LIB
                                     1.02
                                     2.04
                                     2.06
                                      ---
                                 Section 5.30
SDBREAD_LIB
                                     3.10
                                     3.11
                                     3.12
                                     4.00
                                 Section 5.30

Databases

ORIGIN2
                                      NA
                                      2.2
                                      2.2
                                 ORIGEN-S 2.2
                                      NA
TRANSORIGIN
                                      NA
                                      2.2
                                      2.2
                                    In CID
                                      NA
CID
                                      NA
                                      1.0
                                      1.0
                                     2.02
                                      NA
[a] 2014 CRA  -  PA was performed on the VMS computer platforms before the PA was migrated to Solaris.
[b] Most of the 2014 CRA PA codes have not changed (noted "---") and are the same versions.

The RD for each software code specifies the validation criteria for the code and the test cases that demonstrate compliance with these criteria.  SNL used qualification against acceptance criteria and regression testing to determine whether each code could satisfy the criteria in its RDs when run in the current computing configuration (i.e. Intel Solaris Blade Computer Servers).

DOE'S Test Methodology

The test methodology and acceptance criteria described in AP-089 [2] were implemented by the DOE for these regression tests, and the results are presented in Section 5 of this document.  The regression tests were conducted by running every validation test for each code in the most recent computing configuration (Compaq ES40, ES45, and ES47 running OpenVMS 8.2) and comparing the code's output to the output from the code's previously approved validation tests (OpenVMS 7.3-1 running on the Compaq ES40).  The differences between the two sets of outputs were then analyzed.  Any numerical differences between code outputs were evaluated to determine if the code output met the code's acceptance criteria.

In each case, the regression test methodology used the VMS DIFFERENCE command to compare outputs from the regression testing to outputs from previous validations.  The
DIFFERENCE command compares two files and identifies records that are different in the two files.  The DIFFERENCE command was not used to compare binary output data.  Binary output data from both the regression testing and previous validations were often processed through other software codes to produce ASCII files that could then be compared using the DIFFERENCE command.

Differences that involve dates and times, file and directory names, user names, platform names, system version numbers, and execution statistics were termed acceptable.  Differences in numerical output required analysis to determine the origin of the differences and whether the differences affect the code's performance.  Numerical differences were determined to be acceptable if the analyst judged that the output, although different, still met the acceptance criteria for the code.

After DOE ran the tests outlined in the Validation Plan and it was concluded that a code met the acceptance criteria specified in its RDs, a Software I&C form was completed.  The I&C form documents that a code's regression test results met the acceptance criteria specified in its RDs, management's approval of the installation of the software, and the SCM Coordinator's approval of the release of the code as production baseline software.

3.2.1 MIGRATION TO SOLARIS BLADE PLATFORM

DOE/SNL modified the test methodology using regression testing described above because of the differences between the VMS platforms and the Solaris computers. Solaris uses a different floating point format than the VMS platform and many of the PA codes were converted to double precision number formats on the Solaris [4].  Similar results were expected for utility codes, therefore regression testing was applicable for their migration.  However, for codes solving differential equations and using numerical solvers, regression testing sometimes failed and numeric results were not always the same. SNL sometimes used regression testing in a screening mode, but in other cases SNL used acceptance criteria documented in the VVP to qualify the code for WIPP PA calculations. (See AP-162 ERMS #561457, page 7.)

SNL developed the RPD (relative percent differences of 1x10[-4] or less is considered insignificant) to determine if a numeric difference between the two platforms was significant to assist in evaluating reasonable differences.  For values greater than the RPD, SNL was required to provide an adequate explanation or use VVP criteria to verify adequate PA code performance. SNL was not always able to use the SunOS difference command (UNIX diff) but developed utility codes that used the RPD to determined values needing additional review and explanation.  These differences in approach are discussed in Section 5.0 below for each code.

The Solaris uses a difference configuration management system than the VMS platform [4]. This system is called CVS (Code Versioning System) on the Solaris platform.  CVS is used to manage all activities related to WIPP PA calculations.

<IP_ADDRESS> SOLARIS MIGRATION  -  INTEGRATION TESTS

In addition to the qualification of the codes on the Solaris discussed above, SNL/DOE determined that to enhance confidence in the new PA system on the Solaris, "...a comparison of results from one or more PAs runs on VMS and Solaris" was required [4].  Two integration tests were performed consisting of running the PABC09 and CRA14 analyses on the Solaris and then comparing the releases projected by CCDFGF from the VMS and Solaris runs [4]. SNL compared the mean releases and releases by vector to verify that PA calculations performed on the Solaris produced similar results.  The section titled "Integration Tests" in the AP-164 Summary Report [4] describes the integration test process and compared results between the VMS and Solaris PA calculations.

References

 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.

 Analysis Plan (AP-089) 2002.  "Upgrade of Operating System to OpenVMS 7.3-1 and Hardware to HP Alpha ES45."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #523491.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
 WIPP PA - "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Analysis Plan for the 2014 WIPP Compliance Recertification Application Performance Assessment, AP-164, dated January 30, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #559198.

The Agency's Review Approach

The Agency's review has been conducted by a team consisting of Agency and contractor personnel.  The review was initiated with preparatory activities and assembly of background information.  With the exception of computer codes associated with the inventory database, for which LANL is responsible, all of the computer codes are maintained in Carlsbad by the Department's WIPP science advisor, SNL.

The Agency's review has been conducted in several stages, recognizing that changes in the assessment approach might be required, depending upon results obtained.  The following preparatory activities were conducted before conducting on-site reviews:

 Tabulation of the Agency's code acceptance criteria developed by the Agency during the CCA
 Preparation of a list of computer code life-cycle documentation (e.g., VVPs, Change Control and Error Reporting forms) that the Agency needed to review
 Preparation of a draft checklist for reviewing the ability of the PA codes to meet the QA criteria

The following on-site review activities were conducted by the Agency:

 Received overview presentations by SNL and LANL personnel describing the computer code migration activities
 Obtained and reviewed the adequacy of documentation describing the computer code migration activities
 Reviewed the adequacy of testing performed to demonstrate consistency of code output under different operating/hardware systems
 Reviewed and evaluated the traceability of the code migration information
 Reviewed the ability of PA codes to accurately reproduce output obtained under the software/hardware configurations in place during the CCA and subsequent CRAs

In addition to the on-site reviews, off-site reviews were conducted that included relevant documents (e.g., Change Control and Error Reporting Forms, Code Tracking Sheets, Validation Documents) and the DIFFERENCE files for all tests cases for each of the computer codes and libraries that DOE has tested.  The results of these activities are summarized in Section 5 below.
Summary of Individual Computer Code Migration

The following section presents the results of the Agency's computer code migration analysis for each individual code examined (Table 3.1-1).  Specific software and hardware configurations are reviewed, followed by the regression test methodology, the Agency's analysis of the testing, and the Agency's conclusion.

ALGEBRACDB

This section presents the regression test results for ALGEBRACDB.  ALGEBRACDB is a utility code that adds, removes, or manipulates data on CAMDAT database (CDB) files.  The data manipulations to be performed are expressed as algebraic equations involving the existing and/or newly created data.

Introduction

ALGEBRACDB 2.35 was used in the WIPP Compliance Certification Application (CCA) PA.  ALGEBRACDB 2.35 was validated in January 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of 10 Test Cases (1 through 10) met the acceptance criteria defined in the RD/VVP for ALGEBRACDB 2.35 (document Version 1.00) [2].  In January 1997, ALGEBRACDB was re-evaluated, and DOE determined that several requirements, previously identified as "Functionality Not Tested" in the RD/VVP (document Version 1.00), were, in fact, in need of testing.  DOE generated five additional Test Cases (11 through 15) to address these parameters and validated on a DEC Alpha 2100 with OpenVMS 6.1 [3], by demonstrating that the results met the acceptance criteria defined in the RD/VVP for ALGEBRACDB 2.35 (document Version 1.01) [3].  In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from ALGEBRACDB 2.35 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of ALGEBRACDB 2.35 run on a DEC Alpha 2100 with OpenVMS 6.1 [1].  In June 2003, the Agency completed a report documenting the Agency's approval of ALGEBRACDB 2.35 [7] on the ES40 and DEC Alpha 2100 with OpenVMS 6.1.  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [8, 9].  In March 2004, the Agency completed a report documenting the Agency's approval of ALGEBRACDB 2.35 on the Compaq Alpha ES45 and 8400 for use in the CRA 2004 [10].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [11].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [11].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for ALGEBRACDB 2.35 to ensure that it continues to function correctly.

In 2013, SNL migrated the WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [13, 15]. SNL, in 2012, executed regression testing to verify that ALGEBRACDB 2.36 continues to perform WIPP PA calculations correctly [15].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to ALGEBRACDB 2.36 running on the Solaris Blade with SunOS 5.11.

Test Methodology

The tests for this code comprised the 15 test cases described in the Requirements Document & Verification and Validation Plan for ALGEBRACDB  2.35 (RD/VVP) (both document Versions 1.00 [3] and 1.01 [4]).  The first 10 tests are described in document Version 1.00, and the remaining 5 cases are included in document Version 1.01.  Regression test results from ALGEBRACDB 2.35 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of ALGEBRACDB 2.35 run on a DEC Alpha 2100 with OpenVMS 6.1, as documented in the Validation Document for ALGEBRACDB  2.35 (VD) (both document Versions 1.00 [5] and 1.01 [6]).  In January 2003, regression test results from ALGEBRACDB 2.35 run on the ES45 and 8400 with OpenVMS 7.3-1 were compared to results from the validation tests of ALGEBRACDB 2.35 run on a Compaq ES40 with OpenVMS 7.3-1 [8, 9].  In 2006, the regression test methodology used the VMS DIFFERENCE command to compare output from ALGEBRACDB 2.35 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2, to results from the validation tests of ALGEBRACDB 2.35 run on the ES40 with OpenVMS 7.3-1 [12].

CAMDAT database files (CDB) are produced in 14 of the ALGEBRACDB test cases.  The output CDB files are converted from a binary, CDB, file format to an ASCII file format for comparison during the validation process.  In the previous ALGEBRACDB validation, the CDB files were converted using GROPECDB 2.12.  GROPECDB 2.12 was validated in April 2006, on the Compaq ES40, ES45, and ES47 running OpenVMS 8.2 (Section 5.11, GROPECDB).  For this regression test, GROPECDB 2.12 on the Compaq ES40, ES45, and ES47 running OpenVMS 8.2 is used to convert the CDB output files from ALGEBRACDB.

GROPECDB 2.12 was validated on the Compaq ES40, ES45, and ES47 with OpenVMS 8.2 as part of the hardware regression test (see Section 5.11, GROPECDB).  For DOE's regression test, GROPECDB 2.12 is used to convert the CDB output files from ALGEBRACDB 2.35 in OpenVMS 8.2.

In 2012, SNL performed regression testing of ALGEBRACDB 2.36 run on the Solaris Blade with SunOS 5.11 [14].  Results were compared to ALGEBRACDB 2.35 results run on the VMS Compaq E47 with OpenVMS 8.2. The VMS results were transferred to the Solaris, converted as needed, and compared using the UNIX diff command [14].

Test Results

The 15 test cases for ALGEBRACDB 2.36 were executed on the Solaris Blade with SunOS 5.11.  Output files from the test cases were compared to the corresponding output files from the validation of ALGEBRACDB 2.35 on the Compaq ES47 with OpenVMS 8.2 by using the UNIX diff command.  The comparison found that all differences found in the output are limited to code run date and time, platform names, system version numbers, the directory, file names, and very minor numeric differences.

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The comparison found only minor differences in the numerical output of ALGEBRACDB 2.36.  The Agency concludes that ALGEBRACDB 2.36 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 Analysis Report 2003.  "Analysis Report for the VMS 7.3-1 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  WIPP:<IP_ADDRESS>.1: SFT: QA-L 525277.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for ALGEBRACDB Version 2.35" (document Version 1.00).  Sandia National Laboratories.  Sandia WIPP Central Files WPO #28109.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for ALGEBRACDB Version 2.35" (document Version 1.01).  Sandia National Laboratories.  Sandia WIPP Central Files WPO #41863.
 WIPP PA (Performance Assessment) 1995.  "Validation Document for ALGEBRACDB Version 2.35" (document Version 1.00).  Sandia National Laboratories.  Sandia WIPP Central Files WPO #28112.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for ALGEBRACDB Version 2.35" (document Version 1.01).  Sandia National Laboratories.  Sandia WIPP Central Files WPO #41865.
 WIPP PA  -  Validation (Performance Assessment) 1996.  "Validation Document for GROPECDB Version 2.12, May 17, 1996."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37497.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of ALGEBRACDB Version 2.35 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 6, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543463.
 WIPP PA - "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories, Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for ALGRBRACDB Version 2.36 Regression Testing for the Solaris Blade, date October 31, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557796.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

BLOTCDB

BLOTCDB is a code to plot the mesh and results from finite-element and finite-difference analysis programs. BLOTCDB plots all intermediate and final results from all main modules used to perform the WIPP PA. BLOTCDB directly reads a CAMDAT (CDB) file and plots: (1) the computational mesh with contoured analysis results, (2) grid distance versus any variable, and/or (3) any variable versus any other variable. BLOTCDB produces mesh plots with various representations of the analysis output variables and can also produce X-Y curve plots of the analysis variables [13].

Introduction

In 2013, SNL migrated the WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade with SunOS 5.11 [13].  SNL notes: "Small changes in the device drivers and the modifications to the test cases make regression testing impractical. The test cases will be fully validated according to the acceptance criteria presented in this document" ([13] page 14).

BLOTCDB 1.38 validation includes six test cases that verify that the code satisfies various requirements listed in Section 2.0 of the VVP/VD ([13] page 7).  The code has twelve Functional Requirements (R), five External Interface Requirements (R), and twenty-nine Additional Functionality (A) requirements that are examined in the six test cases ([13] page 15). Table 4-1 list the requirements tested by the various test cases. Each test case describes the acceptance criteria for that test, as shown in Section 4.0 of the VVP/VD ([13] page 14). SNL states, "Most criteria involve visual inspection of the plot to assure that the relevant CAMDAT data is plotted as specified in the input command file. Usually the tester is required to locate points or lines on a plot. The accuracy of these visual measurements rests on the "best judgment" of the tester" ([13] page 14).

Test Methodology

Each test case produces plots that are manually examined to verify that the plots are correct, represent input data, and satisfy the acceptance criteria for that test case (see each test case for details in the VVP/VD ([13] page 17)).
Test Case	Functional Requirements Tested (see [13], Section 4.0)
  #1		BLOTCDB can display a wireframe, a mesh and vector plots.  Test R.1, R.5, R.6, R.14, R.15, R.17, A.3 through A.11, A.25, A.26, A.28 and A.29.
  #2		Code can display line and painted contours.  Test R.1, R.2, R.3, R.4, R.14, R.15, R.17, R.18, A.2, A.3, A.4, A.6, A.7,A.13, A.14, A.15, A.16, A.27 and A.28.
  #3		BLOTCDB can display pathlines for particle tracking.  Test R.7, R.14, R.15, R.17, A.3, A.7, and A.18.
  #4		Code can plots time history plots and variable-verses-variable plots. Test R.1, R.8, R.9, R.10, R.14, R.15, R.16, R.17, A.1, A.3, A.18 through A.24, and A.28.
  #5		Code can plot variable-verse-distance plots.  Test R.1, R11, R.12, R.14, R.15, R.16, R.17, A.3, A.18 through A.24 and A.28.
  #6		BLOTCDB can display data on a large mesh. Test R.1, R.2, R.3 R.6 R.14, R.15, R.17, R.18, A.1, A.7, A.8, A.12 and A.13.
Each test case writes the BLOTCDB input data to a plot file that is printed for the tester to examine.  The acceptance criterion for each test describes the expected appearance of each plot.  Acceptance criteria involve visual inspection of the tester to verify that the input command file data is plotted correctly.  The tester is required to locate points on the plots to verify the input information is plotted correctly.
Test Results

SNL used a review process of manual inspection to validate BLOTCDB 1.38, as described in Section 4.0 of the VVP/VD ([13] page 14).  Results of each test are examined against the acceptance criteria in the VVP/VD [13] to verify that acceptance criteria are satisfied.
The Agency's Conclusions

The Agency thoroughly examined SNL's examination of the results of each test case and verified that manual evaluation is adequate and that their conclusions are reasonable.  The Agency concludes that BLOTCDB 1.38 meets the acceptance criteria in the RD ([15] page 4) and the VVP/VD ([13], page 15) and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.
References

[1] Long, J. J. 2012. Nuclear Waste Management Procedure NP 19-1 Software Requirements,
Revision 14. Sandia National Laboratories, Carlsbad, NM. ERMS #558215.

[2] WIPP PA. 2013. Requirements Document for BLOTCDB Version 1.38. Sandia National
Laboratories, Carlsbad, NM. ERMS #560360.

[3] WIPP PA. 1996. User's Manual for BLOTCDB Version 1.37. Sandia National
Laboratories, Carlsbad, NM. ERMS #237501.

[4] WIPP PA. 2013. Addendum to User's Manual for BLOTCDB Version 1.38. Sandia
National Laboratories, Carlsbad, NM. ERMS #560363.

[5] Kirchner, T. B. 2012. AP-162: Analysis Plan for Migration of the Performance Assessment
Codes to the Sun Solaris Blade Server running with Intel Processors, Revision O. Sandia
National Laboratories, Carlsbad, NM. ERMS #557765.

[6] WIPP PA. 1996. Requirements Document & Verification and Validation Plan for
BLOTCDB (Version 1.37). Sandia National Laboratories, Carlsbad, NM. ERMS #237499.

[7] WIPP PA. 1996. Validation Document for BLOTCDB (Version 1.37). Sandia National
Laboratories, Carlsbad, NM. ERMS #237502.

[8] WIPP PA. 1996. User's Manual for GROPECDB Version 2.12. Sandia National
Laboratories, Carlsbad, NM. ERMS #237496.

[9] WIPP PA. 2012. Addendum to User's Manual for GROPECDB Version 2.13. Sandia
National Laboratories, Carlsbad, NM. ERMS #557792.

[10] WIPP PA. 1996. User's Manual for ALGEBRACDB Version 2.35, document version 1.01.
Sandia National Laboratories, Carlsbad, NM. ERMS #241864.

[11] WIPP PA. 2012. Addendum to User's Manual for ALGEBRACDB Version 2.36. Sandia
National Laboratories, Carlsbad, NM. ERMS #557795.

[12] WIPP PA - "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #561457.

[13] WIPP PA  -  "Verification and Validation Plan/Validation Document for BLOTCDB Version 1.38 dated July 18, 2013."  Sandia National Laboratories. Sandia WIPP Central Files ERMS #560361.

[14] WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

[15] WIPP-PA "Requirements Document Criteria dated July 22, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #560360.

BRAGFLO

This section presents the qualification and regression test results for the BRAGFLO.  BRAGFLO is a program used to study two-phase (brine and gas), three-dimensional isothermal flow in porous media.  It has been developed specifically for use in assessing the performance of the WIPP, particularly the flow behavior in the immediate vicinity of the repository.  The physical model is described by material balance equations for brine and gas, Darcy's law, and two-phase fluid properties.  The numerical model includes a cell-centered finite difference discretization, Newton solution of the nonlinear constitutive equations, and linear equation solvers necessary for the Newton iteration.  Various sub-models specific to WIPP include a pressure-induced fracture treatment, creep closure of the repository, and gas generation resulting from corrosion and biodegradation of waste components.

Introduction

Since the CCA PA, the BRAGFLO code has undergone a series of revisions.  Versions 4.00 and 4.01 of BRAGFLO were used in the WIPP CCA.  BRAGFLO 4.00 was used to calculate Salado flow; BRAGFLO 4.01 was used to calculate direct brine releases.  These codes were validated on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of each test case met the acceptance criteria defined in the RD/VVP/VDs [3, 4, 5, 6].

In 1997, BRAGFLO 4.10 was created to combine the capabilities of both BRAGFLO 4.00 and BRAGFLO 4.01 into a single code version [2].  No new functionality was added [1].  BRAGFLO 4.10 was validated on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of each test case met the acceptance criteria defined in the RD/VVP [3].  Several changes were made to BRAGFLO 4.10 during its revision to BRAGFLO 5.0 including removing a number of parameter assignments from embedded data to input data; moving the porosity surface from embedded data to input data; and changing the input-output format [16].

DOE ran the OpenVMS 7.3-1 tests using the FORTRAN 7.3 Run-Time Library (RTL) instead of the RTL, Version 7.4A [9].  The date and time functions in the RTL changed between Version 7.3 and 7.4A, and BRAGFLO 4.10 does not run with the new date and time functions.  Accordingly, BRAGFLO 4.10 is run using the FORTRAN 7.3 RTL by implementing the procedure described in [7].  BRAGFLO 4.10 had one problem report that has since been resolved [8].

In June 2003, the Agency completed a report documenting the Agency's approval of BRAGFLO 4.10 on the ES40 and DEC Alpha 2100 with OpenVMS 6.1 [13].  For the 2004 CRA, the DOE modified BRAGFLO 4.10 to produce BRAGFLO 5.0 to allow the user to input information that was previously included in the BRAGFLO executable file [10, 16].  Beginning with BRAGFLO 5.0, the user provides various constants and molecular weights, as well as information defining the porosity surface, which comes from the SANTOS software.  Changes from BRAGFLO 4.10 to BRAGFLO 5.0 involved input/output issues.

In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which were both running OpenVMS 7.3-1 [14, 15].  In March 2004, the Agency completed a report documenting the Agency's approval of BRAGFLO 5.0 on the Compaq Alpha ES45 and 8400 that were both running OpenVMS 7.3-1 [19].
In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [20].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 has been upgraded to OpenVMS 8.2 [20].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for BRAGFLO 5.0 to ensure that it continued to function correctly [21].

In 2007, a number of changes were made to BRAGFLO from version 5.00 to version 6.00 in accordance with a paper by Hansen and Stein (2005), which describes changes that should be made to PA models to accommodate a more realistic evolution of the WIPP underground [22].  The changes made to BRAGFLO included changes to the disturbed rock zone, brine availability, magnesium oxide (MgO) precipitation, room closure, and formulations pertaining to the capillary pressure versus saturation, which impact the physical and chemical characteristics of the WIPP disposal rooms [23].

In 2013, SNL developed BRAGFLO 6.02 to support Case CRA14-0 of the 2014 CRA PA calculations.  In particular, improved water balance calculations were added [30]: "The primary purpose for revision of BRAGFLO 6.0 is to incorporate the conversion of
hydromagnesite to magnesite. This additional reaction is added to improve the modeling of
water production and consumption. In Version 6.0, hydrated MgO reacts with C02 generated by
microbial degradation of CPR material to form either hydromagnesite or magnesite. For version
6.02, an additional chemistry step of the hydromagnesite conversion to magnesite was added.
This reaction is included to improve the calculation of saturation in the waste area." To qualify BRAGFLO 6.02, SNL developed a new Test Case 14 to verify that the water balance modification works properly.  Test Case 14 was compared to the acceptance criteria in Section 9.14.4 of the RD/VVP [30].

Also in 2013, SNL migrated WIPP PA software from the VMS platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [26, 28]. Because of the changes in computer hardware and operating system, qualification testing and regression testing were completed to verify that BRAGFLO 6.03 continues to perform WIPP PA calculations correctly [27].

The discussion below documents the test methodology, regression test results and qualification results, and the Agency's conclusions with respect to qualification of BRAGFLO 6.02 running on the Compaq ES45 and ES47 with OpenVMS 8.2, and to BRAGFLO 6.03 running on the Solaris Blade computers with SunOS 5.11.

Test Methodology

The tests for Version 4.10 of this code comprised all 12 test cases described in the Requirements Document & Verification and Validation Plan for BRAGFLO Version 4.10 (RD/VVP) [1, 12].  Results of regression tests performed on BRAGFLO 4.10 run on the ES40 with OpenVMS 7.3-1 and on the DEC Alpha 2100 with OpenVMS 6.1 were documented by the EPA [13].  In January 2003, regression test results from BRAGFLO 4.10 run on the ES45 and 8400 with OpenVMS 7.3-1 were compared to results from the validation tests of BRAGFLO 4.10 run on a Compaq ES40 with OpenVMS 7.3-1 [14, 15].

In 2003, BRAGFLO 5.0 was tested by performing all 12 test cases presented in the RD\VVP and comparing the results to the acceptance criteria [11, 17].  This testing was followed in 2004 by regression testing to compare output from BRAGFLO 5.0 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 to the output from the validation of BRAGFLO 5.0 on the Compaq ES40 with OpenVMS 7.3-1 [18].

In 2006, the regression test methodology used the VMS DIFFERENCE command to compare output from BRAGFLO 5.0 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 to results from the validation tests of BRAGFLO 5.0 run on the ES40 with OpenVMS 7.3-1 [21].

Test Case 7 required the use of other WIPP PA codes: POSTBRAG 4.00 and SUMMARIZE 3.01.  These codes have been validated on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 as part of the hardware regression tests.

In 2007, regression testing was used to determine whether BRAGFLO 6.0 satisfied the acceptance criteria of the RD/VVP for Test Cases 1 - 13 [24].  Test Case 14 was validated in this analysis by evaluating BRAGFLO results with respect to the acceptance criteria specified in the RD/VVP, since it is a new test case for the new requirements in BRAGFLO 6.0 [24].
Regression analyses are accomplished in this validation work by comparing results from BRAGFLO 6.0 to the corresponding results from BRAGFLO 5.0.  BRAGFLO 5.0 was validated on the ES40, ES45 and ES47 with Open VMS 8.2 [25].  The VMS Difference command was used to compare ASCII output files from BRAGLO 6.0 with the corresponding output files from BRAGFLO 5.0.

In 2013, BRAGFLO 6.02 was developed to support the 2014 recertification.  A total of 14 test cases were used to qualify the code for use in the WIPP performance assessment.  The original 13 test cases were regression tested as previously performed, as described above. Test Case 14 was developed to verify the performance of the changes made to the code and test results were compared to the acceptance criteria in the VVP [30].

In 2013, SNL migrated the WIPP PA and performed qualification and regression testing of BRAGFLO 6.03 on the Solaris Blade platform with SunOS 5.11 on all fourteen test cases to verify that the code performs PA calculations on the new computer system correctly.  SNL used a two-phased approach to execute these tests. First, BRAGFLO 6.03 was regression tested against BRAGFLO 6.02 using results obtained on the Compaq computer cluster running OpenVMS 8.2. These files were moved to the Solaris, converted to Solaris format as needed, and then compared using the UNIX diff command.  If the differences recorded satisfied the Regression Test criteria ([27] Section 3.2), such as acceptable differences (i.e. file names, version number differences, etc.) and "Small numeric differences" as expected ([27] page 9) because of different characteristics of the two computer systems, then BRAGFLO 6.03 was considered to be qualified for use on the Solaris Blade with SunOS 5.11.

However, if the "number of differences is large" ([27] page 9) then the BRAGFLO 6.03 was validated using acceptance criteria expressed in the VVP document.  Four Test Cases 2, 6, 7, 14, required the use of VVP acceptance criteria to be validated.  Test Case 13, input error reporting, was also qualified using the VVP acceptance criteria ([27] page 10).

Test Results

The 14 test cases for BRAGFLO 6.02 were executed on the Compaq ES45 and ES47 with OpenVMS 8.2. As noted by SNL in [29], the following procedure is performed for Test Cases 1 to 13.

   "1) Validation test results from BRAGFLO 6.02 run on the ES47 with OpenVMS 8.2 are compared to results from the validation tests of BRAGFLO 6.0 run on the ES47 with
          OpenVMS 8.2.

    2) Validation test results from BRAGFLO 6.02 run on the ES45 with OpenVMS 8.2 are compared to results from the validation tests of BRAG FLO 6.02 run on the ES47 with Open VMS 8.2.

    The VMS DIFFERENCE command is used to compare the output file from BRAGFLO 6.02 to the corresponding output file as outlined above."

The results for Test Case 14 are compared to the acceptance criteria described in the VVP [30].  For Test Cases 1 through 13, SNL found acceptable changes such as date, time, file and directory names, platform names, code version, execution statistics, and changes to input parameters used to support the water balance changes in the code during the validation of BRAGFLO 6.02.

The 14 test cases for BRAGFLO 6.03 were executed on the Solaris Blade with SunOS 5.11.  Each test case generated output files, which were compared to the output files from the BRAGFLO 6.02 validation tests, executed on the Compaq cluster with OpenVMS 8.2. All test cases except 2, 6, 7, 13, and 14 showed acceptable textural differences and small numeric differences between BRAGFLO 6.03 and BRAGFLO 6.02 test case output files.

Test Cases 2, 6, 7, 13, and 14 were evaluated against the VVP acceptance criteria.  This evaluation verified that these test cases satisfy the acceptance criteria.

The Agency's Conclusions

The Agency found that the regression testing of BRAGFLO 6.02 Test Cases 1 through 13 are acceptable and that changes found in the output files are appropriate. The Agency also found that the qualification of Test Case 14 was complete and adequate.  The Agency concludes that BRAGFLO 6.02 meets acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Compaq ES45 and ES47 with OpenVMS 8.2.

The Agency found for the BRAGFLO code migration that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The Agency also found that for test cases not regression-tested, the acceptance criteria were satisfied.   The Agency concludes that BRAGFLO 6.03 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade platform with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1997.  "Requirements Document & Verification and Validation Plan for BRAGFLO Version 4.10."  Sandia National Laboratories.  Sandia WIPP Central Files, WPO #45227.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for BRAGFLO Version 4.10."  Sandia National Laboratories.  Sandia WIPP Central Files, WPO #45242.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for BRAGFLO Version 4.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30702.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for BRAGFLO Version 4.00."  Sandia National Laboratories.  Sandia WIPP Central Files, WPO #30705.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for BRAGFLO Version 4.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #38122.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for BRAGFLO Version 4.01."  Sandia National Laboratories.  Sandia WIPP Central Files, WPO #38135.
 WIPP PA (Performance Assessment) 2001.  "Change Control Form, Compaq/DEC Alpha FORTRAN 7.4A Compiler Upgrade."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #519716.
 WIPP PA (Performance Assessment) 2001.  "Software Problem Report 01-002 for PREBRAG 6.00 and BRAGFLO 4.10."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #519714.
 Digital Equipment Corporation 1996.  "OpenVMS 7.1 Release Notes, Section 5.8.  Digital Equipment Corporation, Maynard Massachusetts, November 1996."  Order number AA-QSBTA-TE.
 WIPP PA (Performance Assessment) 1998.  "WIPP PA Analysis Package for AP-042."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #49786.
 WIPP PA (Performance Assessment) 2003.  "BRAGFLO Test Case 7 Results for the OpenVMS 7.3-1 Regression Test.  Sandia WIPP Central Files.  Records Package #525277.
 Validation Report for BRAGFLO Version 4.10 Test Case 6 Using OpenVMS 7.3.1.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA (Performance Assessment) 2003.  "Change Control Form, BRAGFLO."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #525869.
 WIPP PA (Performance Assessment) 2003.  "Validation Document for BRAGFLO Version 5.00."  Sandia National Laboratories.  Sandia WIPP Central Files, WPO #525703.
 WIPP PA (Performance Assessment) 2004.  "Results of Regression Testing for BRAGFLO Version 5.00 Running on the Compaq ES45 and 8400 Platforms."  Sandia National Laboratories.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of BRAGFLO Version 5.0 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 6, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543805.
 Hansen, F. D. and J. S. Stein, 200S.  "WIPP Room Evolution and Performance Assessment Implications."  Milestone Report, February 26, 2005.  Carlsbad, New Mexico:  Sandia National Laboratories.  ERMS #538870.
 WIPP PA 2007.  "BRAGFLO Version 6.00 Software QA Plan" Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #545013.
 Nemer, M. B., 2007.  Requirements Document and Verification and Validation Plan forBRAGFLO Version 6.0.  Document Version 6.00, ERMS# 545014. Sandia National Laboratories, Albuquerque, New Mexico.
 Nemer, M. B., 2006.  Regression Testing Report of BRAGFLO Version 5.00 on the Compaq ES40, ES45 and ES47 Platforms using Open VMS 8.2.  Document Version l.0, ERMS #543805.  Sandia National Laboratories, Albuquerque, New Mexico.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories, Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Validation Document for BRAGFLO Version 6.03, dated February 1, 2013."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #558352.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
 WIPP PA  -  "Validation Document for BRAGFLO, Version 6.02, dated January 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #558661.
 WIPP PA  -  "Requirements Document & Verification and Validation Plan for BRAGFLO, Version 6.02, dated January, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #558659.

CCDFGF

This section presents the regression test results for CCDFGF.  The CCDFGF code assembles the results calculated by other codes in the WIPP PA system to produce Cumulative Complementary Distribution Functions (CCDFs) of releases.

Introduction

Since the CCA PA, the CCDFGF code has undergone a series of revisions.  CCDFGF 1.01 was used in the WIPP CCA.  Version 1.01 was validated on a DEC Alpha 2100 running OpenVMS 6.1 [3].  The validation demonstrated that the results of the four test cases met the acceptance criteria defined in the VVP for Version 1.01 [4].  In 1996, CCDFGF was revised to Version 2.01 to improve and clarify the algorithm by which releases to the Culebra were calculated.  CCDFGF 2.01 was validated on a DEC Alpha 2100 running OpenVMS 6.1 [5].  Test Cases 1 - 4, for the validation of CCDFGF 2.01, were identical to the test cases for the validation of CCDFGF 1.01 [6].  The acceptance criteria for these test cases were satisfied by showing that the output from CCDFGF 2.01 was identical to the output of the CCDFGF 1.01 validation tests.

In 1997, CCDFGF was revised to Version 3.00 to correct an error found in Version 2.01 and to add functionality required for the Performance Assessment Verification Test (PAVT).  CCDFGF 3.00 was validated on a DEC Alpha 2100 running OpenVMS 6.1 [7].  Test Cases 1 - 4, for the validation of CCDFGF 3.00, were not identical to the test cases for the validation of CCDFGF 2.01 [6].  Rather, the test cases for CCDFGF 3.00 were modifications of those used for CCDFGF 2.01.  The modified test cases examined the features added to CCDFGF for Version 3.00 and specified additional acceptance criteria for these features.  CCDFGF 3.00 was validated by the DOE's analysis, and the additional acceptance criteria were met.  Consequently, the validation of CCDFGF 3.00 relies on the combination of the validation of CCDFGF 2.01 and on the extensions to the test cases for CCDFGF 3.00.

CCDFGF was revised again in 1997 to Version 3.01 to add the capability of producing intermediate results for releases to and from the Culebra.  Test Case 5 was added to validate this additional capability [2].  Since the revision consisted only of code to consolidate existing output of CCDFGF and previous testing had validated the existing output, the validation of CCDFGF 3.01 only examined Test Case 5 [8].  Consequently, the validation of CCDFGF 3.01 relies on the combination of the validation of CCDFGF 2.01, the extensions to the test cases for CCDFGF 3.00, and the additional test case for CCDFGF 3.01.

In June 2003, the Agency completed a report documenting the Agency's approval of CCDFGF 3.01 [8] on the Compaq ES40.  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which were both running OpenVMS 7.3-1 [9, 10, 11].  In August 2003, CCDFGF Version 3.01 was upgraded to Version 5.0.  In March 2004, the format of the open statements was changed and the version number of CCDFGF was upgraded from 5.0 to 5.0A [12].  In September 2004, the Agency concluded that CCDFGF 3.01, 5.0, and 5.0A met the acceptance criteria specified in the VVP [1, 2, 13], and thus was validated on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [14].  Version 5.0A was used to support the 2004 CRA.

In June 2004, the code was changed from Version 5.0A to Version 5.01 to reflect changes involving the confidence intervals assigned to the drilling rate that were changed from 90% to 99.5% [17, 18, 19].  Another version of the code, however, was issued in December 2004 (Version 5.02).  This new version includes changes to the Function FindSeries (i.e., a block in the IF-THEN-ELSE construction was removed and a check is made within the remaining block to ensure that 0 is never returned) [17, 20].  In March 2006, the Agency completed a report documenting the Agency's approval of CCDFGF 5.01 and 5.02 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [21].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [22].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [22].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for CCDFGF 5.02 to ensure that it continued to function correctly [23].

In 2010 CCDFGF was modified to investigate the impact of mass balance problems in mobilization of radionuclides calculations. SNL used three test cases to qualify CCDFGF 6.00 on the Compaq cluster with OpenVMS 8.2 [28].

In 2013, WIPP PA software was migrated from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [25, 27]. SNL performed regression testing and validation testing to verify that CCDFGF 7.01 continues to perform WIPP PA calculations correctly [26].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to CCDFGF 6.00 and 7.01.
Test Methodology

The code CCDFGF was upgraded from Version 5.01 to Version 5.02 in December 2004.
The tests for this code comprised the three test cases described in the VVP [15] and an additional test case for CCDFGF 5.02 described in the Addendum to the CCDFGF 5.01 VD [16].  This additional test case was designed to check that the new interpolation of the drilling rate (90% versus 99.5%) works properly.

Test Case 1 tests CCDFGF to ensure that it evaluates the different release mechanisms correctly for one future.  This test case includes five parts; one for each of the following release mechanisms:
 Cuttings and cavings
 Spallings
 Direct brine releases (DBR)
 Culebra releases
 Total releases

The input files for this test case specify only eight vectors, with two futures each.  Using only a few vectors and futures allows manual calculation of the correct results.  Moreover, each vector tests a different release mechanism, further simplifying the manual calculation.  The manual calculation is then compared to the computer-generated results.

The objective of Test Case 2 is to ensure that the random number generator is producing a statistically random sequence of numbers and that the sequence is reproducible.  For this test case, CCDFGF 5.02 is run first using the input files for Test Case 2.  Next, two utility programs are run: CCGF_CHISQ_TEST.FOR for the Chi-Squared Goodness of Fit test, and CCGF_SERIAL_TEST.FOR for the Serial test.  These codes compute the statistics for the output of the random number generator in CCDFGF.

Test Case 2 produces an output file CCGF_QA0501_RND_TEST2.BIN written to the logical RANDOMNUMBERS$BIN.  This binary file contains the sequence of random numbers generated by the random number generator in CCDFGF.

Each utility program uses a single input file assigned to the logical CCGF_RANDOM$INP.  For this test case, the logical CCGF_RANDOM$INP should have the value CCGF_QA0501_RND_TEST2.BIN.  This file is unformatted binary data written by CCDFGF, consisting of a sequence of data pairs; each data pair consists of a double precision real number and a three-character string.

Each utility code produces an output file assigned to the logicals CCGF_CHISQ_TEST$OUT and CCGF_SERIAL_TEST$OUT.  For this test, the logicals were assigned values of CCGF_QA0501_TEST2_CHISQ.OUT and CCGF_QA0501_TEST2_SERIAL.OUT, respectively.  These files contain the results of each test by reporting the chi-squared value for each repetition and whether or not the statistic exceeds the critical value.  These files can be found in the library LIBCCGF.
Test Case 3 evaluated the statistical correctness of stochastic futures modeled by CCDFGF.  Test Case 3 runs 1,000 futures for each of 8 vectors and examines the following observed statistics:

 Probability of selecting each CH waste stream
 Probability of brine pocket intrusion
 Probability of an intrusion hitting the excavated region
 Percentage of intrusions hitting the excavated region that hit CH waste
 Probability of each plugging pattern

In addition, Test Case 3 examined the average number of drilling events in each future and the distribution of mining times.  When choosing the option to use the volume fraction as a probability for the release of cuttings, Test Case 3 examines the probability of CH waste intrusions that encounter CH waste as well.  The input files for this test case specified only eight vectors, with 1,000 futures each.  The release data for each vector are the same as in Test Case 1. Unlike Test Case 1, the futures for each vector were determined randomly.

In 2004, the regression test methodology used the VMS DIFFERENCE command to compare output from CCDFGF 5.02 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 to the output from CCDFGF 5.02 on the Compaq ES40 with OpenVMS 7.3-1 [24].

In 2006, all three of the tests described in the VD were performed to compare output from CCDFGF 5.02 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 to results from the validation tests of CCDFGF 5.02 run on the ES40 with OpenVMS 7.3-1 [23].

In 2010, SNL performed validation against the VVP requirements and regression testing to qualify CCDFGF 6.00 on the Compaq ES47 computer. Three test cases were used.  In Test Case 1, the code was evaluated for determining release mechanisms correctly.  In Test Case 3, the code calculates random numbers that were evaluated against known solutions in the VD acceptance criteria in Sections 5.1 and 5.3 [28].  In Test case 2, the code calculates multiple futures and was regression-tested against CCDFGF 5.02 output files in Section 5.2 of the VVP [28].  CCDFGF 6.00 was qualified on the Compaq ES45 using regression testing against the Compaq ES47 output files [28].

In 2013, regression testing of CCDFGF 7.01 on the Solaris Blade platform with SunOS 5.11 was performed on the original three VMS test cases and two new test cases to validate the codes performance [26].  The original Test Cases 1, 2, and 3 were compared to the results obtained using CCDFGF 7.00, which had been previously validated using CCDFGF 5.02 on the Compaq computers with OpenVMS 8.2 to verify that CCDFGF 7.01 correctly inserts data into the performance assessment results database on the Solaris platforms.  Comparison of these test cases verify that CCDFGF 7.01 generates the binary random file correctly on the Solaris platform [26].

Two new test cases, case 5 and 6, were developed to read the database data inserted into the PA results database by CCDFGF 7.01 to verify that the code correctly inserts data into the PA database [26].  Test case 5 compares the binned data and test case 6 compares the statistics data inserted by CCDFGF 7.01 when test cases 1 and 3 are executed.

Test Results

Test Cases 1, 2 and 3 for CCDFGF 6.00 were executed on the Compaq ES45 and ES47 with OpenVMS 8.2 to qualify the code for WIPP PA [28]. SNL verified that Test Cases 1 and 3 compare satisfactorily against known solutions in the VVP acceptance criteria. Test Case 2 was adequately compared to CCDFGF 5.02 output files using the VMS DIFFERENCE command.

Test Cases 1, 2, 3, 5, and 6 for CCDFGF 7.01 were executed on the Solaris Blade with SunOS 5.11.  Each test case generated output files, which were compared to the output files from the CCDFGF 7.00 validation tests executed on Compaq ES40, ES45, ES47 with OpenVMS 8.2.  The differences are generally limited to code run date and time, file and platform names. Results of Test Cases 1, 2, and 3 showed no differences or acceptable numeric differences.  SNL notes that, "All show three significant digits that differ by one in the final digit (with rounding), so all these differences are acceptable" [26].

Test Cases 5 and 6 compared the CCDFGF output data that was inserted into the PAResults database by CCDFGF 7.01.  Test Cases 1 and 3 were compared to the data in the output files from the CCDFGF 7.0 test on VMS.  No significant differences were found and the data was written correctly into the PA Results database [26].

The Agency's Conclusions

The Agency found that CCDFGF 6.00 was adequately qualified for testing and use in the 2014 CRA performance assessment calculations.  The Agency concludes that CCDFGF 6.00 meets the acceptance criteria in the RD/VVP, and is qualified for WIPP PA use on the Compaq ES45 and ES47 with OpenVMS 8.2.

The Agency also found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names, and rounding differences between the VMS computers and the Solaris computers.  The comparison found minor differences in the numerical output of CCDFGF 7.01.  The Agency concludes that CCDFGF 7.01 meets the acceptance criteria in the RD/VVP, and is qualified for WIPP PA use on the Solaris Blade platform with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1997.  "Verification and Validation Plan for CCDFGF Version 3.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45412.
 WIPP PA (Performance Assessment) 1997.  "Addendum to Verification and Validation Plan for CCDFGF Version 3.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45412.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for CCDFGF Version 1.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42042.
 WIPP PA (Performance Assessment) 1996.  "Verification and Validation Plan for CCDFGF Version 1.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42043.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for CCDFGF Version 2.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42772.
 WIPP PA (Performance Assessment) 1996.  "Verification and Validation Plan for CCDFGF Version 2.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42768.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for CCDFGF Version 3.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45415.
 WIPP PA (Performance Assessment) 1997.  "Addendum to Validation Document for CCDFGF Version 3.01."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45415.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA  -  "Analysis Report for CCDFGF Version 5.00A Regression Testing for the Compaq ES45 and 8400 Platforms," 2004.  Sandia National Laboratories.
 WIPP PA  -  "Change Control Form from 5.00 to 5.00A, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #531461.
 WIPP PA (Performance Assessment) 2003.  "Validation Document for CCDFGF Version 5.0."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #530042.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration Activities  - Version 2."  September 2004.  WIPP-PA (Performance Assessment) 2004.
 WIPP PA (Performance Assessment) Verification and Validation Plan for CCDFGF Version 5.01."  Sandia National Laboratories.  Sandia WIPP Central Files. ERMS #535736.
 WIPP PA (Performance Assessment) Addendum to the CCDFGF Version 5.01 Validation Document ERMS #535737.
 WIPP PA (Performance Assessment) 2005.  `Change Control Form' for CCDFGF 5.01, Sandia National Laboratories.  Sandia.  ERMS #538177.
 WIPP PA (Performance Assessment) 2003.  "Validation Document for CCDFGF Version 5.01."  Sandia National Laboratories.  Sandia WIPP Central Files. ERMS #535737.
 WIPP PA (Performance Assessment) 2004.  "Addendum to the CCDFGF Version 5.01 Validation Document."  Sandia National Laboratories.  Sandia WIPP Central Files. ERMS #538167.
 USEPA  -  "Technical Support Document for Section 194.23:  Models and Computer Codes-PABC Codes Changes Review."  March 2006.  Docket No. A-98-49/II-B1-8.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files. WPO #542680.
 WIPP PA  -  "Regression Testing Report of CCDFGF Version 5.01 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May17, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543452.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013."  Sandia National Laboratories.  Sandia WIPP Central Files. ERMS #561457.
 WIPP PA  -  "Addendum to Requirements Document, User's Manual, Verification and Validation Plan, and Validation Document for CCDFGF Version 7.01, dated November 2013." Sandia National Laboratories.  Sandia WIPP Central Files. ERMS #561130.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012."  Sandia National Laboratories. Sandia WIPP Central Files. ERMS #557765
 WIPP PA  -  "Validation Document for CCDFGF (Version 6.00), dated April 2010." Sandia National Laboratories. Sandia WIPP Central Files. ERMS #552387.

CCDFSUM

**(Please note CCDFSUM functionality is replaced by database functions. The text in this section is unchanged to preserved historical context.)

This section presents the regression test results for CCDFSUM.  The CCDFSUM code plots the cumulative complementary distribution functions (CCDFs) for the releases calculated by the code CCDFGF.

Introduction

Since the CCA PA, the CCDFSUM code has undergone a series of revisions.  CCDFSUM 1.01 was used in the CCA.  Version 1.01 was validated on a DEC Alpha 2100 running OpenVMS 6.1 under the requirements of SNL QAP 9-1 (now SNL NP 9-1) [2].  In 1996, CCDFSUM was revised to Version 2.00 to accommodate changes made in CCDFGF 3.00.  CCDFSUM 2.00 was validated on a DEC Alpha 2100 running OpenVMS 6.1 [3].

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from CCDFSUM 2.00 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of CCDFSUM 2.00 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of CCDFSUM 2.00 [4] on those operating systems.  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [5, 6].  In March 2004, the Agency completed a report documenting the Agency's approval of CCDFSUM 2.00 on the Compaq Alpha ES45 and 8400 running OpenVMS 7.3-1 [7].  CCDFSUM 2.00 was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [8].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 has been upgraded to OpenVMS 8.2 [8].  Because of these changes in the operating system and the addition of a new computing platform, regression testing has been conducted for CCDFSUM 2.00 to ensure that it continues to function correctly [9].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to CCDFSUM 2.00 running on the Compaq ES40, ES45, and ES47 machines with OpenVMS 8.2.

Test Methodology

The tests for this code comprised the test cases described in the Verification and Validation Plan for CCDFSUM Version 2.00 (VVP) [1].  In 2003, regression test results from CCDFSUM 2.00 run on the ES45 and 8400 with OpenVMS 7.3-1 were compared to results from the validation tests of CCDFSUM 2.00 run on the ES40 with OpenVMS 7.3-1 [6].  CCDFSUM 2.00 was used to support the 2004 CRA.

In 2006, the regression test methodology used the VMS DIFFERENCE command to compare output from CCDFSUM 2.00 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 to results from the validation tests of CCDFSUM 2.00 run on the ES40 with OpenVMS 7.3-1 [9].

Test Results

The VVP for CCDFSUM 2.00 lists a total of nine test cases; however, CCDFSUM is run only in the first test case.  The other eight test cases specify comparison of the output of the first test case with different criteria.  These test cases do not exercise any function of the code.  For this regression test, DOE believes (and the Agency agrees) that it is sufficient to run only the first test case and compare its output with the output of the previous validation test.

The first test case was executed on the Compaq ES40, ES45, and ES47 with OpenVMS 8.2 [9].  The test case generated output files, which were compared to the output files from the CCDFSUM 2.00 validation tests, executed on Compaq ES40 with OpenVMS 7.3-1.  The differences are limited to code run date and time, file and platform names.  The comparison found that all differences in output are limited to code run date and time, file and platform names.

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The comparison found no differences in the numerical output of CCDFSUM 2.00.  The Agency concludes that CCDFSUM 2.00 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the ES40, ES45, and ES47 with OpenVMS 8.2.

References

 WIPP PA (Performance Assessment) 1997.  "Verification and Validation Plan for CCDFSUM Version 2.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #43920.
 WIPP PA 2001.  "Nuclear Waste Management Program Procedure 9-1, Analyses."  Sandia National Laboratories.
 WIPP PA 1997.  "Validation Document for CCDFSUM Version 2.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #43925.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of CCDFSUM 2.00 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 31, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543594.

CUTTINGS_S

This section presents the regression test results for CUTTINGS_S.  The CUTTINGS_S (CUSP) code was written to calculate the quantity of radioactive material (in curies) brought to the surface from a radioactive waste disposal repository as a consequence of an inadvertent human intrusion through drilling.  The code determines the amount of material removed from the repository by several release mechanisms, and decays the material to the time of intrusion.

Introduction

Since the CCA PA, the CUTTINGS_S code has undergone a series of revisions.  CUTTINGS_S 5.03 was used in the WIPP CCA.  Version 5.03 was validated in May 1996 on a DEC Alpha 2100 with OpenVMS 6.1.  The validation was accomplished by demonstrating that results of the six test cases met the acceptance criteria defined in the RD/VVP [4] and Validation Document [7].

In July 1997, CUTTINGS_S was revised to Version 5.04 and was validated on a DEC Alpha 2100 with OpenVMS 6.1.  Test Cases 1 - 6 for the validation of CUTTINGS_S 5.04 were identical to test cases for the validation of CUTTINGS_S 5.03.  The acceptance criteria for these test cases were satisfied by showing that the output from CUTTINGS_S 5.04 was identical to the output of the CUTTINGS_S 5.03 validation tests.  New Test Cases 7 - 9 were validated by demonstrating the output of Test Cases 7 - 9 met the acceptance criteria defined in the RD/VVP for CUTTINGS_S 5.04 [1, 3].

In January 2001, CUTTINGS_S was revised to Version 5.04A to remove references to unused libraries.  Although SDBREAD_LIB and the INGRES library are not used in PA calculations, CUTTINGS_S 5.04 checked for their availability and will not run if they are absent.  Since these libraries were no longer present on the system, it was necessary to eliminate the linkages.  The following quotations from the Change Control form explain the revisions:
      CUTTINGS_S Version 5.04 was mistakenly linked with SDBREAD_LIB and an INGRES library.  Although SDBREAD_LIB and the INGRES library are not used, the INGRES system must be installed on the system for Version 5.04 to run.  The linked software is no longer available on the system, so CUTTINGS_S will be relinked to remove these libraries.

      There were no source changes between CUTTINGS_S Version 5.04A and Version 5.04.  The only difference is that CUTTINGS_S Version 5.04A will not be linked with SDBREAD_LIB and the INGRES library.  The code will now be linked with the standard libraries CAMDAT_LIB, CAMCON_LIB, and CAMSUPES_LIB. (The library.OLB files that were used for Version 5.04 will not be used for Version 5.04A. [5]

In order to test the new operating systems that were added in 2002 - 2003 (Section 1), regression test results from CUTTINGS_S 5.04A run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of CUTTINGS_S 5.04A run on a DEC Alpha 2100 with OpenVMS 6.1 [2, 6].  In June 2003, the Agency completed a report documenting the Agency's approval with respect to the migration and verification of CUTTINGS_S 5.04A [8].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [9, 10].  In September 2004, EPA concluded that that CUTTINGS_S 5.04A met the acceptance criteria specified in the RD/VVP [3], and thus was validated on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [11].  CUTTINGS_S 5.04A was used to support the 2004 CRA.

The CUTTINGS_S code was upgraded from Version 5.10 to Version 6.00 in January 2005 [12, 18].  CUTTINGS_S was modified to remove unneeded functionality and improve maintainability [13].  The changes reduce the number of input files needed to run CUTTINGS_S and improved the traceability of input parameters to CUTTINGS_S.  The new CUTTINGS_S can produce output values for multiple intrusions and cavities, resulting in fewer code executions.

The validation of CUTTINGS_S Version 6.00 was conducted on a Compaq ES40 platform and documented in the Verification and Validation Plan and Validation Document for CUTTINGS_S Version 6.00 (VVP/VD) [12].  For completeness, regression testing was also conducted by DOE to demonstrate the validity of the code on the Compaq ES45 platform [14, 19].

In April 2005, the CUTTINGS_S code was upgraded from Version 6.00 to Version 6.1 [16].
In CUTTINGS_S Versions 5.10 and 6.00, the input parameter RNDSPALL was required for use of spall model 4.  In Version 6.01 use of this parameter is optional.  If RNDSPALL is not specified in the input control file, the first spallings vector volume in the spall input file will be used for calculation of spallings releases for the first vector output, the second spallings vector volume will be used for calculation of spallings releases for the second vector output, and so on.

The validation of CUTTINGS_S Version 6.01 was conducted on a Compaq ES40 platform and documented in the Verification and Validation Plan and Validation Document for CUTTINGS_S Version 6.01 (VVP/VD) [16].  For completeness, regression testing was also conducted by DOE to demonstrate the validity of the code on the Compaq ES45 platform [17].

In June 2005, the CUTTINGS_S code was upgraded from Version 6.01 to Version 6.02 [20].  In Version 6.01, when the flow is turbulent, the subroutine DRILL attempted to calculate the radius at which the flow becomes laminar.  It is required, both physically and computationally, that ROUTER remains larger than the constant RINNER.  In Version 6.02, the subroutine DRILL was modified by adding an IF statement that assigns a value of RORIG to ROUTER if ROUTER becomes less than RINNER.  In March 2006, the Agency completed a report documenting the Agency's approval of CUTTINGS_S Version 6.02 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [20].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [22, 24].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [22].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for CUTTINGS_S Version 6.02 to ensure that it continues to function correctly [23].

In 2013, SNL migrated the WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [26, 28].  Because of these changes, regression testing was done to verify that CUTTINGS_S 6.03 continues to perform calculations correctly based on the WIPP PA [27].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to CUTTINGS_S Version 6.03 running on the Solaris Blade computers with SunOS 5.11.

Test Methodology

The tests for CUTTING_S Version 6.02 comprise the 12 test cases described in the Verification and Validation Plan / Validation Document CUTTINGS_S Version 6.01 (VVP/VD) [16] and an additional test case for CUTTINGS_S 6.02 described in the Addendum to the CUTTINGS_S Version 6.01 Verification and Validation Plan / Validation Document (VVP/VD Addendum) [21].  Per the VVP/VD [16], Test Cases 1, 2, 3, 6, and 9 are obsolete, leaving seven test cases as described in the VVP/VD [16] and its addendum [21].

In 2003, the regression testing of CUTTINGS_S 6.02 on the ES40 platform with OpenVMS 7.3-1 was conducted for 7 test cases.  Test results were compared to the results of the validation testing conducted for CUTTINGS_S 6.01 on the ES40 platform with OpenVMS 7.3-1 [15, 16].  Test Case 13, added when CUTTINGS_S was upgraded to Version 6.02, was subject to unit testing, rather than regression testing.  Results of the unit testing for CUTTINGS_S 6.02 Test Case 13 on the ES40 platform with OpenVMS 7.3-1 can be found in VVP/VD Addendum [21].

In 2004, the regression testing of CUTTINGS_S 6.02 on the ES45 platform with OpenVMS 7.3-1 was conducted for all seven test cases.  Test results for the seven test cases (above) were compared to the results of the regression testing conducted for CUTTINGS_S 6.02 on the ES40 platform with OpenVMS 7.3-1.  Test results for Test Case 13 were compared to the results of the unit testing conducted for CUTTINGS_S 6.02 [16, 23].

In 2006, the regression test methodology used the VMS DIFFERENCE command to compare output from CUTTINGS_S 6.02 for all of the tests described in the VVP/VD [16].  The tests were run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 and compared to results from the validation tests of CUTTINGS_S 6.02 run on the ES40 with OpenVMS 7.3-1 [24].  Test results for Test Case 13, however, were not compared to the results of the unit testing conducted for CUTTINGS_S 6.02 on the Compaq ES40 with OpenVMS 7.3-1.

In 2013, regression testing of CUTTINGS_S 6.03 on the Solaris Blade platform with SunOS 5.11 was performed on all seven test cases examined on the VMS platform.  Test results for the seven test cases were compared to the results obtained executing CUTTINGS_S 6.02 on the Compaq computers with OpenVMS 8.2 [27]. Because of the differences in the computer architecture of the Compaq and Solaris computer, such as single precision versus double precision computations, SNL expected minor differences in results of regression testing [27].  SNL used the UNIX diff command and SNL developed a Python utility, Regr_Diff.py, to compare results.

Six test cases showed no or minor numeric differences between the tests.  Test Case 11 showed "significant numeric differences" [27] on the Solaris Blade platform. SNL did a thorough review and concluded that these results were from the greater numerical precision of the double precision calculations done on the Solaris platform [27].

Test Results

Seven test cases for CUTTINGS_S 6.03 were executed on the Solaris Blade with SunOS 5.11.  All tests output files were compared with CUTTINGS_S 6.02 test case runs executed on Compaq cluster with OpenVMS 8.2.  The comparison found that all differences in output are limited to code run date and time, file and platform names, minor numerical differences and a few greater differences caused by the use of double precision computations on the Solaris Blade [27].

The Agency's Conclusions