In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from NUTS 2.05A run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of NUTS 2.05A run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of NUTS 2.05A on those operating systems [19].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [20, 21].  In March 2004, the Agency completed a report documenting the Agency's approval of NUTS 2.05A on the Compaq Alpha ES45 and 8400 with OpenVMS 7.3-1 [21].

As indicated above, NUTS 2.05A was validated on OpenVMS 7.3-1.  When NUTS 2.05A was run on OpenVMS 8.2, however, it aborted because the time argument of DATE_AND_TIME was too short [22].  Therefore, the CAMSUPES_LIB routines EXDATE and EXTIME were substituted for the DATE_AND_ TIME call and the date was expanded to 10 characters, while time remains at 8 characters.  The only change in output is the format of the date and time.  Since there is already a NUTS 2.05B, this change to NUTS V2.05A resulted in NUTS 2.05C.  (NUTS 2.05B was qualified on the Compaq Alpha ES45 and 8400 that were both running OpenVMS 7.3-1, but never used for any analyses). NUTS 2.05A was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [23].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [23].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for NUTS 2.05C to ensure that it continued to function correctly.

In 2013, regression testing of NUTS 2.06 on the Solaris Blade platform with SunOS 5.11 was performed on Test Cases 1 through 12 and Test Case 14 (13 was never defined).  Test results for these thirteen test cases were compared to the results obtained executing NUTS 2.05C on the Compaq ES47 with OpenVMS 8.2 [27]. SNL expected numeric differences "due to the change to double precision floating point variables in the code and differences in the two platforms" [27].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to NUTS 2.06 running on the Solaris Blade platform with SunOS 5.11.

Test Methodology

SNL converted the text VMS files to UNIX for comparison on the Solaris platform.  Binary files on the VMS platform were converted to text format then transferred to the Solaris and convert back to binary as needed for comparison of the test case results.  File name convention is different on the Solaris platform; therefore, file names were converted to make the files compatible with the Solaris requirements [27]. File comparison of the VMS and Solaris results were considered acceptable if text difference were as expected, such a file names and executable names, etc.  Numerical differences were considered to be acceptable if the RPD, relative percent difference, was less than 1E-4.  Numerical differences above this RPD were more closely examined and a complete explanation was provided by SNL staff.

The tests for NUTS 2.06 comprised all 13 test cases described in the Requirements Document & Verification and Validation Plan for NUTS Version 2.05 RD/VVP) [3].  The NUTS 2.06 regression test methodology uses a Python utility code, called Regr_Nuts.py [27] developed by SNL to compare files and determine differences above the RPD threshold. This was performed to compare output from NUTS 2.06 on the Solaris Blade with SunOS 5.11 to the output from validation tests of NUTS 2.05C on the Compaq ES47 with OpenVMS 8.2 [27].

Test Results

Based on the results of the tests described above, only very minor differences (e.g., spacing, version number, numerical differences below the RPD threshold) were found for ten of the test cases.  Three cases, 7, 12 and 14 [27] showed differences above the RPD threshold and required additional explanation. SNL concluded that the differences in these cases were insignificant and were easily explained [27]. SNL concluded that NUTS 2.06 is validated for WIPP PA used on a Solaris Blade with SunOS 5.11.

The Agency's Conclusions

The Agency found that differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names and minor numerical differences for ten of the test cases.  EPA closely examined Test Cases 7, 12 and 14 to verify that SNL's explanations are complete and reasonable.   The Agency concludes that NUTS 2.06 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 Analysis Plan (AP-042) 1998.  "Regression for the Upgrade to OpenVMS Version 7.1 on the WIPP COMPAC Alpha Cluster."  Sandia National Laboratories.
 Analysis Plan (AP-065) 2000.  "Regression for the Upgrade to OpenVMS Version 7.2 on the WIPP DEC Alpha Cluster."  Sandia National Laboratories.
 WIPP PA (Performance Assessment) 1997.  "Requirements Document & Verification and Validation Plan for NUTS Version 2.05."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45999.
 WIPP PA (Performance Assessment) 2001.  "Installation and Checkout Form (NUTS 2.05A)."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #516515.
 WIPP PA (Performance Assessment) 1996.  "User's Manual for NUTS Version 2.02."   Sandia National Laboratories.  Sandia WIPP Central Files WPO #37927.
 WIPP PA (Performance Assessment) 1997.  "Change Control Form for NUTS, Version 2.03."  Memorandum to Distribution, J.J. Loukota, Sandia National Laboratories.  Sandia WIPP Central Files WPO #43730.
 WIPP PA (Performance Assessment) 1997.  "Change Control Form for NUTS, Version 2.05."  Memorandum to Distribution, J.J. Loukota, Sandia National Laboratories.  Sandia WIPP Central Files WPO #46624.
 WIPP PA (Performance Assessment) 2001.  "Change Control Form for NUTS, Version 2.05A."  Memorandum to Distribution, P. Painter, Sandia National Laboratories.  Sandia WIPP Central Files ERMS #515790.
 Analysis Plan (AP-089) 2002.  "Upgrade of Operating System to OpenVMS 7.3-1 and Hardware to HP Alpha ES45."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #523491.
 WIPP PA (Performance Assessment) 1999.  "Software Problem Report 99-001 for NUTS Version 2.05."  Memorandum to Distribution, J.J. Loukota, Sandia National Laboratories.  Sandia WIPP Central Files ERMS #504773.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for NUTS Version 2.05."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #46003.
 WIPP PA (Performance Assessment) 1999.  "Addendum to NUTS Version 2.05 Validation Document  -  Analytical Solution Test Results for Part 1 of Test Case #14 and Additional Test Problem Results."  Memorandum to Distribution, A. Treadway Analysis Report for the VMS 7.3-1 Regression Test Appendix NUTS Page 33 of 121 and M. Lord, Sandia National Laboratories.  Sandia WIPP Central Files ERMS #503096.
 WIPP PA (Performance Assessment) 1997.  "Requirements Document & Verification and Validation Plan for NUTS Version 2.02."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37924.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for NUTS Version 2.02."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37929.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for NUTS Version 2.03."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42618.
 WIPP PA (Performance Assessment) 1996.  "Software Problem Report 96-012 for NUTS Version 2.02."  Memorandum to Distribution, J.J. Loukota, Sandia National Laboratories.  Sandia WIPP Central Files WPO #41769.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for NUTS Version 2.03."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42619.
 Digital Equipment Corporation. 1996.  "OpenVMS 7.1 Release Notes, Section 5.8."  Digital Equipment Corporation, Maynard Massachusetts.  Order number AA-QSBTB-TE.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA (Performance Assessment) 2006.  "Change Control Form for NUTS, Version 2.05A."  Sandia WIPP Central Files WPO #43730.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of NUTS 2.05A on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 31, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543598.
 WIPP PA - "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for NUTS Version 2.06, Regression Testing for the Solaris Blade, dated June 19, 2013." Sandia National Laboratories, Sandia WIPP Central Files ERMS #559596.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

PANEL

This section presents the regression and validation test results for PANEL.  PANEL takes the source term data and computes the solubility of the elements needed.  PANEL also takes brine flow and repository volume data from a CAMDAT database (CDB) file and computes the amount of mobilized radioisotopes that leave the repository.

Introduction

PANEL 3.50ZO was initially validated in September 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of two test cases met the acceptance criteria defined in the RD/VVP for PANEL 3.50ZO [4, 5].

In May 1996, PANEL was revised to Version 3.60 and was validated on a DEC Alpha 2100 with OpenVMS 6.1.  Test cases identical to the two test cases for the validation of PANEL 3.50ZO were run.  The acceptance criteria for these test cases were satisfied by showing that the output from PANEL 3.60 was identical to the output of the PANEL 3.50ZO validation tests [6, 7].  PANEL 3.60 was used in the CCA.  In June 1998, PANEL was revised to Version 4.00 and was validated on a DEC Alpha 2100 with OpenVMS 7.1 [8].  In addition to the two test cases from the previous validation, five more test cases were added to the RD/VVP for Version 4.00 [3].  The acceptance criteria for Test Cases 1 and 2 were satisfied by showing that the output from PANEL 4.00 was identical to the output of the PANEL 3.60 validation tests [7].  Test Cases 3 - 7 were validated by demonstrating that the output from PANEL 4.00 met the acceptance criteria defined in the RD/VVP for PANEL 4.00 [3].

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from PANEL 4.00 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of PANEL 4.00 run on a DEC Alpha 2100 with OpenVMS 6.1 [1].  In March of 2003, several modifications were made to PANEL and the version number was changed from 4.00 to 4.02 [13].  The test set used for PANEL 4.02 consists of all nine of the test cases presented in Section 9 of the RD/VVP [14].

In June 2003, the Agency completed a report documenting the Agency's approval with respect to the migration and verification of PANEL 4.00 on those operating systems [10].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [11, 12, 15]. In September 2004, the Agency concluded that PANEL 4.02 met the acceptance criteria specified in the RD/VVP [3], and thus was considered validated on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [16]. PANEL 4.02 was used to support the 2004 CRA.

In April 2005, the PANEL code was revised to Version 4.03 in order to be able to set the default panel brine volume via MATSET [17, 18].  To ensure that this version was working properly, the DOE regression-tested Version 4.03 against Version 4.02 on the Compaq ES40 and ES45 with OpenVMS 7.3-1 [2].  In March 2006, the Agency completed a report documenting the Agency's approval of PANEL 4.03 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [19].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [21].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [21].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for PANEL 4.03 to ensure that it continued to function correctly [20, 22].

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [23, 25].  SNL did not perform a regression test as before but determined that a full validation of PANEL 4.04 was necessary [24].

The discussion below documents the test methodology, qualification results, and the Agency's conclusions with respect to PANEL 4.04 running on the Solaris Blade computers with SunOS 5.11.

Test Methodology

The test suite for PANEL, as described in the VVP [14], consists of Test Cases 1 through 9, with Test Case 4 requiring two executions.

PANEL needs the following input files:  a CAMDAT file containing source term and inventory data, and an optional CAMDAT file containing brine flow and volume data.  PANEL also needs a run type option input on the command line.  All input files and options used to execute the PANEL 4.03 tests were the same files and options used in the previous validation of PANEL 4.02.  In 2004, the regression test methodology used the VMS DIFFERENCE command to compare output from PANEL 4.03 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 to the output from PANEL 4.03 on the Compaq ES40 with OpenVMS 7.3-1 [18, 16].

Each successful execution of PANEL generates the following output files:  a binary output CAMDAT file and a debug file.  The regression test methodology uses the VMS DIFFERENCE command to compare output from the current execution to the output from a validation of PANEL.  The output CAMDAT files are binary files and cannot be compared with the VMS DIFFERENCE command.  The GROPECDB utility [4, 9] is used to write portions of the CAMDAT files as text, so that they can be compared.  Thus, the debug file and the GROPECDB output from the output CAMDAT file are differenced.  In 2006, the output from PANEL 4.03 on the Compaq ES40 with OpenVMS 7.3-1was compared using the VMS DIFFERENCE command to the output from PANEL 4.03 on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [22].

In 2012, SNL did not use regression testing but fully qualified PANEL 4.04 ([24] Section 3.0) using Functional Requirements (R1, R2, R3, R4, R5, R6 and R7) documented in the PANEL 4.02 RD/VVP [14].  PANEL's validation testing consisted of nine test cases which validated that the code adequately performs WIPP performance assessment calculations on the Solaris.  External Interface Requirements include three requirements (R8, R9, and R10) that PANEL 4.04 can input and output results correctly on the Solaris Blade computers.  Each test case was evaluated with respect to the acceptance criteria as documented with each test case in the PANEL 4.04 VD [24].

Test Results

SNL executed nine test cases as noted in Section 3 of the VD [24].  Each functional requirement and external interface requirements were tested and examined to verify that all acceptance criteria were satisfied for the qualification of PANEL 4.04 on the Solaris platform.

The Agency's Conclusions

The Agency examined SNL's qualification process and results for PANEL 4.04.  EPA found that each test case satisfied the acceptance criteria for each requirement tested.  The Agency also found that SNL's analysis was complete and thorough.

The Agency concludes that PANEL 4.04 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade platform with SunOS 5.11.

References

 Analysis Plan (AP-042) 1998.  "Regression for the Upgrade to OpenVMS Version 7.1 on the WIPP COMPAC Alpha Cluster."  Sandia National Laboratories.
 Analysis Plan (AP-065) 2000.  "Regression for the Upgrade to OpenVMS Version 7.2 on the WIPP DEC Alpha Cluster."  Sandia National Laboratories.
 WIPP PA (Performance Assessment) 1998.  "Requirements Document and Verification and Validation Plan for PANEL Version 4.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #48787.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document and Verification and Validation Plan for PANEL Version 3.50ZO" Sandia National Laboratories.  Sandia WIPP Central Files WPO #24326.
 WIPP PA (Performance Assessment) 1995.  "Validation Document for PANEL Version 3.50ZO" Sandia National Laboratories.  Sandia WIPP Central Files WPO #24328.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document and Verification and Validation Plan for PANEL Version 3.60."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37358.
 WIPP PA (Performance Assessment) 1998.  "Validation Document for PANEL Version 3.60."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37362.
 WIPP PA (Performance Assessment) 1998.  "Validation Document for PANEL Version 4.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #48791.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for GROPECDB Version 2.12."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37497.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA (Performance Assessment) 2003.  "Change Control Form for PANEL" Sandia National Laboratories.  Sandia WIPP Central Files WPO #526499.
 WIPP PA (Performance Assessment) 2003.  "Requirements Document and Verification and Validation Plan for PANEL Version 4.02."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #526649.
 WIPP PA (Performance Assessment) 2003.  "Analysis Report for PANEL Version 4.02 Regression Testing for the ES45 and 8400 Platforms" Sandia National Laboratories.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration Activities  - Version 2."  September 2004.
 WIPP PA (Performance Assessment) 2005.  "Change Control Form for PANEL, Version 4.03 Sandia National Laboratories.  Sandia WIPP Central Files WPO #539452.  WIPP PA (Performance Assessment) 2004.
 WIPP PA (Performance Assessment) 2005.  "Installation and Checkout for PANEL Version 4.03 Regression Testing for the Compaq ES40 and ES45 Platforms."  Sandia National Laboratories.
 USEPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 USEPA  -  "Technical Support Document for Section 194.23:  Models and Computer Codes-PABC Codes Changes Review."  March 2006.  Docket No. A-98-49/II-B1-8.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of PANEL 4.03 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 19, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543600.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Validation Document for PANEL (Version 4.04) on the SOLARIS System dated November 27, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557733.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server System Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

PEST

This section presents the verification and validation of PEST.  PEST is an acquired code that solves the problem of parameter estimation for any mathematical model, but with specific application to optimizing T-fields using pilot points in conjunction with the MODFLOW2000 groundwater flow model.  In the context of the Culebra T-fields, PEST is used to iteratively optimize a spatially correlated residual field that is then added to the original mean T-field to produce the final T-field. The PEST code is freely available on the web at: http://www.sspa.com/PEST/.

Introduction

PEST is a parameter estimation program that can be used with other models to calibrate parameters quickly using a set of known observations.  Models produce numbers and if there are field or laboratory measurements corresponding to some of these numbers, PEST can adjust model parameter and/or excitation data to reduce to a minimum the discrepancies between the pertinent model-generated numbers and the corresponding measurements.  It does this by taking control of the model and running it as many times as is necessary in order to determine the optimal set of parameters and/or excitations.  The modeler must inform PEST of where the adjustable parameters and excitations are to be found on the model input files.  Once PEST is provided with this information, it can rewrite these model input files using whatever parameters and excitations are appropriate at any stage of the optimization process.  Files are constructed so that PEST can identify those numbers on the model output files that correspond to the actual observations that have been made.  Thus, each time PEST runs the model it is able to read those model outcomes that must be matched to field or laboratory observations.  The difference that PEST calculates between the measured and observed values is called the "residual error".  PEST continues to perform iterations in order to minimize the residual errors.  Once the errors are below the pre-defined error criteria, the model will stop and create a transmissivity field.

PEST was not used by DOE for the CCA, however, PEST Version 5.51 was used to create transmissivity fields for the 2004 CRA PA and the PABC-2004.  In 2004, PEST 5.51 was approved by the Agency on the Linux operating system [1].

In 2005, DOE proposed that additional stochastic inverse calibration functionality be added to PEST and that the version be updated to 9.11 [2].  This additional functionality includes:

 Truncated singular value decomposition ("truncated SVD") as a parameter regularization methodology.  This additional regularization approach complements the previously used means of enforcing regularization termed "Tikhonov" regularization through solution of a constrained minimization problem.  The truncated SVD approach allows for fewer model iterations to achieve a model calibration by using a subspace decomposition of the estimated parameters.  A disadvantage to this approach over the "Tikhonov" regularization, however, is that it can introduce numerical artifacts.  A solution to this problem is to combine components of the "Tikhonov" regularization with the truncated SVD approach into the "SVD Assist" inversion approach within PEST.

 "SVD-Assist" regularized inversion methodology.  The "SVD-assist" method combines the strengths of both of the above regularization methods.  The result is a scheme that is numerically stable, very efficient, and produces intuitively realistic parameter fields.  This approach uses parameter sensitivity information as identified in a Jacobian matrix and calculated using Tikhonov regularization.  The principal components are determined by all the parameters that are subsequently used in the truncated SVD inverse approach.

The two changes mentioned above are fundamental changes to the way the inverse problem is formulated and solved.  In addition to these two changes, several more practical changes were also implemented.

 The calibration process is now able to use calibration with pilot points to estimate more than one parameter at each pilot point.  In previous applications of the pilot point method to the Culebra transmissivity field problem, only one transmissivity had been estimated at each pilot point.  PEST has been changed to allow for the estimation of multiple parameters at each pilot point, and this multi-variate estimation also works under the more efficient SVD assist inverse procedure.  For the transmissivity field calibrations, only two spatially varying parameters will be estimated in the Culebra; transmissivity and storage capacity.  Both are parameterized using pilot points.

 A second practical change to PEST was made to allow for the use of different regularization and spatial variation parameters within different predefined zones.  The model domain is subdivided into zones of differing geostatistical structure.  Kriging factors and regularization constraints employed in the inversion process can be defined as zone specific and reflect these different structures.  Spatial interpolation from pilot points to the model grid is strictly zone-specific.  The number of distinct zones possible within PEST is theoretically unlimited; however, in practical situations, there will only be a small number of distinct zones.

In 2005, the DOE developed a QA plan for PEST Version 9.11 [3].  The QA Plan indicated that 6 primary documents be developed, reviewed, and maintained to meet SNL NP 19-1 Software Requirements for PEST Version 9.11.  These documents are the QA plan, RD [4], DD [5], UM [7], ID [8], VVP [9], and VD [10].

Documentation for PEST Version 9.11 consists of the baseline documents.  Software changes are documented in addenda for each of the documents.  Configuration control is maintained through completion of Installation & Checkout (I&C) documentation for all changes made to PEST, and system software and/or system hardware.  In addition, Change Control (CC) and Software Problem Report (SPR) documents will be developed as needed.  As defined in SNL procedure NP 19-1, an I&C, software problem reporting, change control, software configuration control, and appropriate revisions to the quality assurance documents will be prepared, reviewed, and maintained for each change during the software lifecycle.

The RD for PEST 9.11 outlines requirements that need to be tested [4].  These include requirements related to performance, attributes, external interface, and 13 functional requirements.  The DD for PEST, however, compresses requirements 10 through 13 into a single requirement pertaining to the optimization of pilot points [6].  The remaining functional requirements include capabilities related to numerical algorithms, screen input and output, user intervention, statistical calculations, regularization, pilot points, parallel processing, predictive analysis and utility programs.

The DD and DD Addendum for PEST 9.11 describe the design considerations necessary to incorporate the functional requirements [5, 6].  The Addendum also outlines the control flow and logic describing how the parallel processing will be performed.  Although the User's Manual was developed by Watermark Computing, DOE has added a table that maps the quality assurance requirements to the relevant sections in the manual.  The PEST 9.11 Implementation Document focuses on the compilation of the source code [8].

In 2014, SNL migrated PEST 9.12 from the "Alice" cluster of computers with Linux operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [14, 15].  SNL performed qualification testing and migration testing to ensure that PEST 9.12 continues to perform PA calculations correctly [16].

The test methodology, results and Agency findings pertaining to the qualification of PEST 9.12 are discussed below.

Test Methodology

The purpose of VVP is to describe the testing procedures that will be used to demonstrate that the code requirements outlined in the RD are met.  The VVP for PEST 9.11 presents 31 tests designed to validate all of the code requirements [9].  The only functional requirement for which a test is intentionally not designed is related to predictive analysis.  This requirement is described in the VVP as the following:

      PEST shall be capable of maximizing/ minimizing a key model outcome (defined as a "prediction") while maintaining the goodness of fit between all other model outcomes and corresponding field data within a user-defined tolerance.  Implementation shall not rely on an assumed linear relationship between model parameters and model outputs; PEST shall calculate the true nonlinear maximum or minimum prediction based on theory.

The VVP indicates that it is not anticipated that the predictive analysis requirement will be needed for the WIPP PA, and will not be tested at this time.

In the VVP, a series of tests are presented based on calibration of a synthetic model for a single layer confined aquifer.  In DOE's calibration exercise, PEST's SVD-assist functionality is employed to estimate values for 296 parameters, 148 of which represent storage capacity, the other half of which represent transmissivity.  All parameters pertain to pilot points, 148 of which are deployed throughout the model domain.  Synthetic transmissivity and storage capacity fields are heterogeneous, being generated on the basis of exponential variograms.  However, the model domain is zoned, and DOE employs different variograms for field generation within each of two different zones; hydraulic properties are discontinuous at the zone boundary.  Of the 148 pilot points employed as a basis for spatial parameterization, 79 of these are allocated to one zone, while 69 are allocated to the other.

Use of this test case allows DOE to verify the pilot point interpolation and regularization procedures that vary between zones.  DOE notes, however, that this is not a fundamental requirement for the application of regularized inversion.

The validation and verification tests are divided into four categories.  The "M" tests are used for verification that the model and all of its components (including utility programs that support the use of pilot points) run correctly and that all input files for the current set of tests have been properly installed and contain no errors.  The "P" tests pertain to operations undertaken in preparation for an SVD-assisted PEST run, including the introduction of regularization constraints to a PEST control file.  They are used for verification of the operation of a number of PEST utilities, and of PEST itself when using truncated singular value decomposition as a parameter estimation mechanism.  The "S" tests verify correct operation of PEST in undertaking SVD-assisted parameter estimation, and correct operation of a number of its utility programs in the post-processing of an SVD-assisted PEST run.  The "PP" tests verify correct operation of Parallel PEST in implementing SVD-based inversion.  A complete table showing the procedures and the requirements that they fulfill are presented in Chapter 7 of the VVP [9].

The VD indicates that validation tests were performed on Intel Xeon PCs running a Kernel 2.69-11 operating system [10].  The cluster is configured so that each of the client nodes uses the same hard disk, mounted from the server.  In addition, because the individual nodes on the cluster have identical hardware and operating systems, testing need only be done once to validate the entire cluster.  DOE notes that future installations and checkouts will be regression tested against the results obtained on this original validation platform.  DOE indicates that the same requirements regarding the clustered environment will apply to future platforms; namely, each different hardware and/or operating system configuration will be tested once for a set of cloned machines.

In addition to PEST 9.11 being tested on the Intel Xeon PCs, it was also tested on the "Geo-Hydro Linux Cluster."  This cluster is comprised of three different hardware groups, each with a group name; (1) Eleionomae, (2) Pegaeae, and (3) Crinaeae.  These AMD Athlon 64 computers running a Linux Kernel 2.6.18 operating system are connected to a job control server, "tethys.sandia.gov", which is not used for execution of codes.

To qualify PEST 9.11 on these platforms, all of the test problems described in the VD [10] was first performed on the "Eleionomae" group [11].  The same suite of test problems were subsequently run on the "Pegaeae" and "Crinaeae" systems, and the UNIX diff command was used to compare the results to the validation results for "Eleionomae" [12, 13].

In 2014 SNL performed qualification of PEST 9.12 on the Solaris Blade with SunOS 5.11.  The validation document [16] notes that the same test suite used to qualify PEST 9.11 is used for PEST 9.12, therefore the detailed description of the testing provided above still applies.

Test Results

The acceptance criteria for each of the tests are detailed in Section 3 of the VD [10].  All the results from testing on the Intel Xeon PCs were within the expected limits of accuracy and uniqueness and met all specified criteria.  The results show that PEST performed as expected on the Intel Xeon PCs and on the "Eleionomae" group.

The regression testing of the "Pegaeae" and "Crinaeae" systems against the results obtained from the "Eleionomae" group indicates that all differences in output are limited to code run date and time, file and platform names.

The validation of PEST 9.12 verifies that the code satisfies the acceptance criteria on the Solaris Blade with SunOS 5.11 [16].

The Agency's Conclusions

The Agency closely examined the testing of PEST 9.12 and found that all test results run on the Solaris Blade with SunOS 5.11 operating system meets the functional requirements of the acceptance criteria specified in Section 2.0 of the VD [16].

The Agency concludes that PEST Version 9.12 is verified for use on the Solaris Blade with SunOS 9.12UNIX.

References

 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA (Performance Assessment) 2005.  "Change Control Form for PEST, Version 5.51 Sandia National Laboratories.  Sandia WIPP Central Files WPO #539290.  WIPP PA (Performance Assessment).
 WIPP PA (Performance Assessment) 2005.  Software QA Plan for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #539280.
 WIPP PA (Performance Assessment) 2008.  Requirements Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #548334.
 WIPP PA (Performance Assessment) 2005.  Design Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #539283.
 WIPP PA (Performance Assessment) 2008.  Addendum to Design Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #548335.
 Watermark Computing. 2005.  User's Manual for PEST.  Sandia WIPP Central Files WPO #539287.
 WIPP PA (Performance Assessment) 2008.  Implementation Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #539286.
 WIPP PA (Performance Assessment) 2008.  Verification and Validation Plan/Validation Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #539282.
 WIPP PA (Performance Assessment) 2008.  Validation Document for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #539284.
 WIPP PA (Performance Assessment) 2009.  Validation Document for PEST on the "Eleionomae" nodes of the GeoHydro Linux Cluster for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #550914.
 WIPP PA (Performance Assessment) 2009.  Validation Document for PEST on the "Pegaeae" nodes of the GeoHydro Linux Cluster for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #550915.
 WIPP PA (Performance Assessment) 2009.  Validation Document for PEST on the "Crinaeae" nodes of the GeoHydro Linux Cluster for PEST Version 9.11.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #550916.
 WIPP PA  -  "Summary Report on the Migration of VMS files from the HP Alpha Cluster to the Sun/Solaris Cluster and Qualification of Codes from the Alice Linux Cluster on the Sun Solaris Cluster, AP-168, dated December 12, 2014." Sandia National Laboratories. Sandia WIPP Central Files ERMS #563103.
 WIPP PA  -  "AP-168 Analysis Plan for Migration of VMS files from the HP Alpha Cluster to the Sun/Solaris Cluster and Qualification of Codes from the Alice Linux Custer on the Sun Solaris Cluster, Revision 0 dated April 21, 2014." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561953.
 WIPP PA  -  "Validation Document for PEST Version 9.12, dated August 15, 2014." Sandia National Laboratories. Sandia WIPP Central Files ERMS #562466.

POSTBRAG

This section describes the regression test results for POSTBRAG.  POSTBRAG is a utility code that takes the binary output file generated by BRAGFLO and puts it into the CAMDAT (CDB) output file format.

Introduction

For WIPP PA, POSTBRAG is used to create CAMDAT files, which are examined with BLOTCDB and/or GROPECDB [5].  CAMDAT database files may also be referred to as CDB files.  POSTBRAG 4.00 was validated in February 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of two test cases met the acceptance criteria defined in the RD/VVP [1] for POSTBRAG 4.00.  POSTBRAG 4.00 was to support the CCA.  The code has not been revised since this validation.  Previous to this version, POSTBRAG Version 3.05ZO had a single test case validated to the acceptance criteria defined in the RD/VVP [2, 3, 4] for POSTBRAG 3.05ZO.

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from POSTBRAG 4.00 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of POSTBRAG 4.00 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of POSTBRAG 4.00 on those operating systems [6].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [7, 8].  In March 2004, the Agency completed a report documenting the Agency's approval of POSTBRAG 4.00 on the Compaq Alpha ES45 and 8400 with OpenVMS 7.3-1 [9]. POSTBRAG 4.00 was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [10].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [10].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for POSTBRAG 4.00 to ensure that it continued to function correctly.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [12, 14].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to POSTBRAG 4.02 running on the Solaris Blade with SunOS 5.11 [13].

Test Methodology

The tests for this code comprised the two test cases described in the Requirements Document & Verification and Validation Plan for POSTBRAG Version 4.00 RD/VVP) [2, 3].  Regression test results from POSTBRAG 4.02 run on the Solaris Blade with SunOS 5.11 were compared to results from the validation tests of POSTBRAG 4.00A run on a Compaq ES47 with OpenVMS 8.2 [11].  The UNIX diff command was used to compare test results.  SNL noted that numeric differences were expected due to the change to double precision and platform differences.  The two test cases show very minor numeric differences of one digit in a few time steps [13].

Test Results

The results of the tests described above are that only very minor differences (e.g., spacing, version number, very minor numeric differences) were found for the two test cases.  The comparison found that all differences found in the output are generally limited to code run date and time, platform names, system version numbers, the directory, and file names.

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names, and very minor numeric differences. The Agency concludes that POSTBRAG 4.04 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for POSTBRAG Version 4.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30681.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for POSTBRAG Version 4.00", Sandia National Laboratories.  Sandia WIPP Central Files WPO #30685.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for POSTBRAG Version 3.05ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #23603.
 WIPP PA (Performance Assessment) 1995.  "Validation Document for POSTBRAG Version 3.05ZO", Sandia National Laboratories.  Sandia WIPP Central Files WPO #23604.
 WIPP PA (Performance Assessment) 2002.  "Software Problem Report" (SPR) for BLOTCDB Version 1.37.  SPR #02-004, Sandia National Laboratories.  Sandia WIPP Central Files ERMS #.525354.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of POSTBRAG 4.00 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated July 5, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543802.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for POSTBRAG Version 4.02 Regression Testing for the Solaris Blade, dated January 31, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557831.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

POSTLHS

This section presents the regression test results for POSTLHS.  The statistical code, POSTLHS, evaluates parameter importance by reporting the partial correlation coefficients (PCC) and standardized regression coefficients (SRCs) on either the raw or ranked data.  The absolute values of the standardized regression coefficients (or mathematically-related partial correlation coefficients) can be used to measure parameter importance.
Introduction

POSTLHS Version 4.06ZO was validated in October 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of two test cases met the acceptance criteria defined in the RD/VVP for POSTLHS 4.06ZO [1].  In February 1996, POSTLHS was revised to Version 4.07 and was validated on a DEC Alpha 2100 with OpenVMS 6.1.  Test cases identical to the test cases for the validation of POSTLHS 4.06ZO were run.  The acceptance criteria for these test cases were satisfied by showing that the output from POSTLHS 4.07 was identical to the output of the POSTLHS 4.06ZO validation tests [2].  POSTLHS 4.07 was used in the CCA.

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from POSTLHS 4.07 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of POSTLHS 4.07 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's approval with respect to the migration and verification of POSTLHS 4.07 on those operating systems [4].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [5, 6].  POSTLHS 4.07 was used to support the 2004 CRA.

In September 2004, the Agency concluded that POSTLHS 4.07 met the acceptance criteria specified in the RD/VVP [3], and thus is considered as validated on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [7].  In April 2005, the POSTLHS code was revised to Version 4.07A in order to re-index the CAMDAT output files [8].  To ensure that this version was working properly, the DOE regression tested Version 4.07 against Version 4.07A on the COMPAQ ES40 and ES45 with OpenVMS 7.3-1 [9].  In March 2006, the Agency completed a report documenting the Agency's approval of POSTLHS 4.07A on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [10].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [11].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [11].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for POSTLHS 4.07A to ensure that it continued to function correctly [12].

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [14, 16].

The discussion below documents the test methodology, validation testing, and the Agency's conclusions with respect to POSTLHS 4.08 running on the Solaris Blade with SunOS 5.11 [15].

Test Methodology

The test suite for POSTLHS, as described in the VVP [1], consists of Test Cases 1 and 2.  The entire test suite for POSTLHS Version 4.07A was executed on the Compaq ES40 platform with OpenVMS 7.3 1, and then executed again on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [13].

POSTLHS needs the following input files: an LHS output file, an input control file, and an input CAMDAT file.  All input files used to execute the POSTLHS 4.07A tests were the same files used in the previous validation of POSTLHS 4.07A.

Each successful execution of POSTLHS generates the following output files: a set of binary output CAMDAT files and a debug file.  The regression test methodology uses the VMS DIFFERENCE command to compare output from the respective platforms.

The output CAMDAT files are binary files and cannot be compared with the VMS DIFFERENCE command.  The GROPECDB utility is used to write portions of the CAMDAT files as text, so that they can be compared.  Thus, the debug file and the GROPECDB output from each output CAMDAT file are differenced.

In 2013, SNL validated POSTLHS 4.08 on the Solaris Blade using SunOS 5.11.  SNL notes in the VVP/VD that: "Regression testing would be difficult because the input now comes from a database and many cosmetic changes have been made to the text debug file to make it more readable and to make verification easier. Test Cases 1 and 2 have been removed from the test suite. A new test case, Test Case 3, has been designed to better reflect realistic LHS input" [15].

Test Results

SNL documents their review process in [15] Section 4.1.6.  The evaluation verifies that POSTLHS:

 correctly reads the LHS data from the database,
 identifies corresponding CAMDAT material properties, and
 transfers the LHS sampled parameter values to the output CAMDAT file for each vector.

Test Case 3 verifies that POSTLHS 4.08 satisfies the functional, external interface, and additional requirements on the Solaris.

The Agency's Conclusions

The Agency examined SNL's qualification process and results for POSTLHA 4.08.  EPA found that Test Case 3 satisfies the acceptance criteria for each requirement tested.  The Agency also found that SNL's analysis was complete and thorough.

 The Agency concludes that POSTLHS 4.08 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1995.  "A Requirements Document & Verification and Validation Plan for POSTLHS Version 4.06ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #23552.
 WIPP PA (Performance Assessment) 1996.  "Software Installation and Checkout Form, POSTLHS Version 4.07."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #230717.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for GROPECDB Version 2.12."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37497.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration Activities  - Version 2."  September 2004.
 WIPP PA (Performance Assessment) 2005.  "Change Control Form for POSTLHS, Version 4.07."  Sandia National Laboratories.
 WIPP PA (Performance Assessment) 2005.  "Installation and Checkout for POSTLHS Version 4.07A Regression Testing for the Compaq ES40 and ES45 Platforms."  Sandia National Laboratories.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 USEPA  -  "Technical Support Document for Section 194.23:  Models and Computer Codes-PABC Codes Changes Review."  March 2006.  Docket No. A-98-49/II-B1-8.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of POSTLHS 4.07A on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 19, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543782.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Verification and Validation Plan/Validation Document for POSTLHS Version 4.08 dated March 5, 2013." Sandia National Laboratories. Sandia WIPP Central Files #559272.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
POSTSECOTP2D

This section presents the regression test results for POSTSECOTP2D.  POSTSECOTP2D creates a new CAMDAT database (the WIPP PA computational database) from the output of the SECOTP2D computer program and the previous CAMDAT file.  The program appends the computational database with ANALYSIS information output from the SECOTP2D code.  Specifically, for each time step of SECOTP2D output, POSTSECOTP2D writes values to the CAMDAT file.  These values are written to the "Analysis Results" section of the CAMDAT file: TIME, HIFLAG(=0), and ELEMENT variables (Species Concentrations and Darcy flow velocities).

Introduction

Since the CCA, the POSTSECOTP2D code has undergone a series of revisions.  POSTSECOTP2D Version 1.02, which was used in the WIPP CCA, was validated in June 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of a test case met the acceptance criteria defined in the RD/VVP for POSTSECOTP2D 1.02 [3, 4].

Validation was accomplished by demonstrating that the input data into POSTSECOTP2D 1.02 is the same as the output file to the CAMDAT database.  The program ST2D3_VERIFY_RES (compiled and linked from ST2D3_VERIFY_RES.FOR) was executed to extract data corresponding to the data extracted from the output CAMDAT database file, ST2D3_SECOTP_TEST.CDB, and the data were compared and showed that POSTSECOTP2D correctly transfers data from the binary output file to the CAMDAT database.  Only selected portions of each array written to the database were compared by DOE.  If the entire contents of the database were compared to the results on the binary output file, a manual inspection of tens of thousands of numbers would have to be made.  DOE points out that the magnitude of this task would be overwhelming.

In June 1997, POSTSECOTP2D was revised to Version 1.04 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 [1, 2].  Validation was accomplished by demonstrating that the results of the two test cases met the acceptance criteria defined in the RD/VVP for POSTSECOTP2D 1.02.  Both test cases were different than had been used in the previous validation.  Otherwise the methodology was the same as described above for Version 1.02.
In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from POSTSECOTP2D 1.04 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of POSTSECOTP2D 1.04 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of POSTSECOTP2D 1.04 on those operating systems [6].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [7, 8].  POSTSECOTP2D 1.04 was used to support the 2004 CRA.

In March 2004, the Agency completed a report documenting the Agency's approval of POSTSECOTP2D 1.04 on the Compaq Alpha ES45 and 8400 which were both running OpenVMS 7.3-1 [9].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [10].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [10].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for POSTSECOTP2D 1.04 to ensure that it continued to function correctly.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [12, 14].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to POSTSECOTP2D 1.05 running on the Solaris Blade with SunOS 5.11 [13].

Test Methodology

The tests for this code comprised the two test cases described in the Requirements Document & Verification and Validation Plan for POSTSECOTP2D Version 1.04 (RD/VVP) [1].  Regression test results from POSTSECOTP2D 1.04 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of POSTSECOTP2D 1.04 run on a Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [11].  The regression test methodology uses the VMS DIFFERENCE command to compare output from the respective platforms.

CAMDAT database files (CDB) are produced in each of the two POSTSECOTP2D test cases.  The output CDB files are converted from a binary, CDB, file to an ASCII file for comparison during the validation process.  In the previous POSTSECOTP2D 1.04 validation, the CDB files were converted using GROPE 2.10.  GROPE has since been revised to Version 2.12.  GROPE 2.12 was validated in June 1996 on a DEC Alpha 2100 with OpenVMS 6.1 [5].  GROPE 2.12 has been validated on a Compaq ES40, ES45, and the ES47 with OpenVMS 8.2, as part of the hardware regression test (see Section 5.10).  For this regression test, GROPE 2.12 is used to convert the CDB output files from POSTSECOTP2D 1.04 in OpenVMS 8.2.

In 2013, SNL used two Test Cases, 1 and 2, to regression test POSTSECOTP2D 1.05 on the Solaris Blade SunOS 5.11 comparing the results to POSTSCEOTP2D 1.04 run on the Compaq ES47 with OpenVMS 8.2 ([13] Section 1.0). The VMS results were transferred to the Solaris and converted as needed.

The UNIX diff command was used to compare test results. SNL notes: "For POSTSECOTP2D, double precision numeric values may differ by one in the final digit (with rounding)." ([13] page 7) The two test cases show very minor numeric differences of one digit in the final digit ([13] Section 5).

Test Results

The results of the tests described show that only very minor differences (e.g., spacing, version number) were found for the two test cases and minor differences in the final digit ([13] Sections 1.0 and 5.0).  The comparison found that all differences in the output are limited to code run date and time, platform names, system version numbers, the directory, file names, and minor numeric differences ([13] Section 5.0).

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names, and minor numeric differences.  The Agency concludes that POSTSECOTP2D 1.05 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1997.  "Requirements Document and Verification and Validation Plan for POSTSECOTP2D Version 1.04."  Sandia National Laboratories. Sandia WIPP Central Files WPO #45696.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for POSTSECOTP2D Version 1.04."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45699.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document and Verification and Validation Plan for POSTSECOTP2D Version 1.02."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37304.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for POSTSECOTP2D Version 1.02."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37370.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for GROPECDB Version 2.12."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37497.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of POSTSECOTP2D 1.04 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 6, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543782.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for POSTSECOTP2D Version 1.05 Regression Testing for the Solaris Blade dated July 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #559789.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

PREBRAG

This section presents the qualification and regression test results for PREBRAG.  PREBRAG is used to create BRAGFLO input files.  PREBRAG reads specific data from an input CAMDAT file and, through instructions supplied in an ASCII input file, generates an ASCII BRAGFLO input file.

Introduction

Prior to the CCA, the PREBRAG code had undergone a single revision.  PREBRAG 5.05ZO was validated [3] in September 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by acceptance testing a single test case, the output of which met the acceptance criteria defined in the RD/VVP for PREBRAG 5.05ZO [4].

In February 1996, PREBRAG was revised to Version 6.00 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 [1, 2].  PREBRAG 6.00 was used in the WIPP CCA.  The validation test included the original test case defined for Version 5.05ZO, and two additional test cases.

Acceptance of the added Test Cases 2 and 3, described in the RD/VVP for Version 6.00 [1], were satisfied by comparing output of the second test to the output of Test Case 1, while the acceptance criteria for Test Case 3 was satisfied by comparing its output to that of Test Case 2.

PREBRAG 6.00 has one open problem report [5].  PREBRAG 6.00 uses an outdated list-directed I/O format that allows space-padded fields.  The output files from PREBRAG 6.00 validation and the VMS7.3-1 test include space-padded fields and cannot be read by BRAGFLO 4.10.  There is no requirement for test output of PREBRAG to be read as input to BRAGFLO.  To allow BRAGFLO 4.10 to read input files created by PREBRAG 6.00, a conversion script, EVAL_BF2_CONVERT_INPUT.COM, removes extraneous spaces from the input file.  Use of this conversion script is not necessary for this regression test of PREBRAG 6.00.  In March 2003, several modifications were made to PREBRAG 6.0 primarily to remove the "hardwiring of parameter values" and the code was updated to PREBRAG 7.0 [11].

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from PREBRAG 6.0 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of PREBRAG 6.0 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of PREBRAG 6.0 on those operating systems [6].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [7, 8, 12].  Testing of PREBRAG 7.0 consisted of conducting the three functional test cases described in Section 6 of the VD [9].  These tests were conducted on the Compaq ES45 and 8400 platforms with OpenVMS 7.3-1.  Output files from these test cases were compared to the corresponding output files from the validation of PREBRAG 7.0 on the Compaq ES45 with OpenVMS 7.3-1.  In March 2004, the Agency completed a report documenting the Agency's approval of PREBRAG 7.0 on the Compaq Alpha ES45 and 8400 that were both running OpenVMS 7.3-1 [13].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [14].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 has been upgraded to OpenVMS 8.2 [14].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for PREBRAG 7.0 to ensure that it continues to function correctly. The approach used to validate PREBRAG Version 7.00 involved 3 test cases to satisfy the 17 Functional Requirements presented in Section 9 of the RD\VVP [10].  Testing of PREBRAG 7.00 consisted of conducting the three functional test cases described in Section 6 of the VD [9]. Regression test results from PREBRAG 7.00 run on the Compaq ES45 with OpenVMS 7.3-1 were compared using the VMS DIFFERENCE command to results from the validation tests of PREBRAG 7.00 run on a Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [15]. The results of these tests found that only very minor differences (e.g., spacing, version number) were found for the three test cases.  All differences found in the output are limited to code run date and time, platform names, system version numbers, the directory, and file names. The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The comparison found no differences in the numerical output of PREBRAG 7.00.  The Agency concluded that PREBRAG 7.00 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the ES40, ES45, and ES47 with OpenVMS 8.2.  PREBRAG 7.00 was used to support the 2004 CRA.

In 2007, PREBRAG 7.0 was updated to PREBRAG 8.0 to provide new input needed for BRAGFLO 6.0.  This new input involves new keywords in the input control file and writes new data to the BRAGFLO input file.  The routine WASTE_POROS was removed (which calculates values of the initial conditions of iron and CH2O and saturated brine of old waste areas and reaction rates RKCOR and RKBIO.  The old waste area values are no longer set in PREBRAG. The reaction rates are now input directly with a PREBRAG command.

In 2012, PREBRAG 8.00 was modified to PREBRAG 8.02 to support the 2014 recertification, in particular to calculate CRA14 Case14-0 which includes water balance computations [21].  The BRAGFLO input file was modified to accommodate addition input and control parameters.  PREBRAG was modified to build the BRAGFLO input file with the correct data.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [17, 19].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to PREBRAG 8.02 running on the Compaq ES45 and ES47 with OpenVMS 8.2 and PREBRAG 8.03 running on the Solaris Blade with SunOS 5.11 [18].

Test Methodology

SNL performed regression testing using three test cases, Test Cases 1, 3, and 4, to verify that PREBRAG 8.02 performed as expected and that new variable input computer code was correct [20].  SNL used the VMS DIFFERENCE command to compare the PREBRAG 8.02 to PREBRAG 8.00 output files on the Compaq ES45 and ES47 machines to qualify the code.  SNL also examined the output files to verify that new input data and control commands were included in the BRAGFLO input files correctly.

Testing of PREBRAG 8.03 consisted of conducting three test cases, Test Cases 1, 3, and 4. Regression test results from PREBRAG 8.03 run on the Solaris Blade with SunOS 5.11were compared using the python difference utility, Regr_Diff.py, (this code compares results line-by-line and calculates the relative percentage difference (RPD)), command to results from the validation tests of PREBRAG 8.02  run on a Compaq  ES47 with OpenVMS 8.2 [18].  If the RPD value calculated is less than 1E-4 then SNL considers the numeric differences acceptable.

Test Results

PREBRAG 8.02 testing shows only minor differences in the VMS DIFFERENCES files [20].  The expected input data and control commands were also verified to be included in the BRAGFLO input files.

The PREBRAG 8.03 test results described above showed only very minor differences (e.g., spacing, version number, and some numeric differences) were found for the three test cases.  All differences found in the output are limited to code run date and time, platform names, system version numbers, the directory, file names, and minor numeric differences [18].

The Agency's Conclusions

The Agency found that the testing of PREBRAG 8.02 to be adequate and verifies that the code performs properly.  The Agency concludes that PREBRAG 8.02 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Compaq ES45 and ES47 computers with OpenVMS 8.2 for the 2014 CRA PA calculations.

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names, and minor numeric differences below the RPD threshold [18].  The Agency concludes that PREBRAG 8.030 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for PREBRAG Version 6.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30676.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for PREBRAG Version 6.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30679.
 WIPP PA (Performance Assessment) 1995.  "Validation Document for PREBRAG Version 5.05ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #23596.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for PREBRAG Version 5.05ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #23594.
 WIPP PA (Performance Assessment) 2001.  "Software Problem Report 01-002 for PREBRAG 6.00 and BRAGFLO 4.10."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #519714.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA (Performance Assessment) 2003.  "Validation Document for PREBRAG Version 7.00."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #526627
 WIPP PA (Performance Assessment) 2003.  "Requirements Document & Verification and Validation Plan for PREBRAG Version 7.0."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #526625.
 WIPP PA (Performance Assessment) 2003.  "Change Control Form Version 7.0."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #526257.
 WIPP PA (Performance Assessment) 2004.  "Analysis Report for PREBRAG Version 7.00 Regression Testing for the Compaq ES45 and 8400 Platforms."  Sandia National Laboratories.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of PREBRAG 7.00 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated July 5, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543801.
 WIPP PA (Performance Assessment) 2007.  "Change Control Form Version 8.0."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #545263.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for PREBRAG Version 8.03 Regression Testing for the Solaris Blade dated March 18, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557818.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
 WIPP PA  -  "Addendum to User's Manual and Verification and Validation Plan/Validation Document for PREBRAG Version 8.02 dated January 14, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #558655.
 WIPP PA  -  "AP-164 Analysis Plan for the 2014 WIPP Compliance Recertification Application Performance Assessment, dated January 31, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #559198.

PRECCDFGF

This section presents the regression test results for PRECCDFGF.  PRECCDFGF collates output from all other WIPP PA codes and formats this output into the RELTAB input file for CCDFGF.

Introduction

Version 1.0 of PRECCDFGF was used to support calculations during the CCA.  In 2003, Version 1.0 was upgraded to Version 1.0A and this new version of the code was tested by following the procedures outlined by the single test included in the Validation Plan on the COMPAQ ES40 and 8400 with OpenVMS 7.3-1 [1].  In September of 2003, minor changes were made in the codes output reporting and the version number was changed from 1.0A to 1.0B.  To ensure that this version was working properly, DOE regression tested Version 1.0B against Version 1.0A on the COMPAQ ES40 and 8400 with OpenVMS 7.3-1 [2].  In September 2004, the Agency concluded that PRECCDFGF 1.00B meets the acceptance criteria specified in the VVP [1], and thus is considered as validated on the COMPAQ ES45 and 8400 platforms with OpenVMS 7.3-1 [4].  PRECCDFGF 1.00B was used to support the 2004 CRA.

In July 2005, PRECCDFGF was updated to Version 1.01 with three significant modifications [5].

The first is that PRECCDFGF 1.01 will read the text output file of release tables written directly by CUTTINGS_S 6.00 or higher.  This file replaces a set of files that were created by SUMMARIZE from the output of the previous versions of CUTTINGS_S.  Secondly, when reading the release tables from the other codes written by SUMMARIZE and the CUTTINGS_S text output file, PRECCDFGF 1.01 reads the header records at the start of the files to ensure that the fields in the files correspond to the values expected by the READ statements in the code itself.  Failure of the header to match the expected text will cause PRECCDFGF to abort its execution with appropriate error messages being logged.  Finally, whereas PRECCDFGF Version 1.00B read a text output file from LHS to get data about the sampled parameter GLOBAL:PBRINE, Version 1.01 reads a set of CAMDAT files generated by POSTLHS to obtain this data.  Thus, PRECCDFGF is no longer required to read output files from LHS.

In 2003, Version 1.01 of the code was validated to run on the Compaq ES40 with OpenVMS 7.3-1 [5].  In September 2004, the Agency concluded that PRECCDFGF 1.01 met the acceptance criteria specified in the VVP [1, 2, 3], and thus was validated on the Compaq ES40 with OpenVMS 7.3-1 [7].  Following the validation of the code on the ES40, it was regression tested in 2005 to run on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [6].  In March 2006, the Agency completed a report documenting the Agency's approval of PRECCDFGF 1.01 on the Compaq ES45 and 8400 with OpenVMS 7.3-1 [8].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [9].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [9].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for PRECCDFGF 1.01 to ensure that it continued to function correctly [10].

In 2010, SNL modified PRECCDFGF code for use in the 2014 CRA PA to include, "The capability to assemble direct brine release data resulting from single or multiple brine volumes has been implemented in PRECCDFGF version 2.0." [14].  SNL also modified the code to correct an error message when one of the SUMMARIZE input files has an invalid header.  SNL developed two test cases, Test Cases 1 and 2, to validate these changes included in PRECCDFGF 2.0.  The discussion below documents the qualification of PRECCDFGF 2.0 for use on the Compaq cluster using OpenVMS 8.2.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [11, 13]. The discussion below also documents the test methodology, regression test results, and the Agency's conclusions with respect to PRECCDFGF 1.06/2.01 running on the Solaris Blade with SunOS 5.11 [12].

Test Methodology

The validation of PRECCDFGF 2.0 uses two test cases to evaluate the changes made to the code. They verify that the code performs the changes properly.

The regression tests for PRECCDFGF Version 1.06/2.01 comprise the two test cases described in the VVP [1].  Test results from PRECCDFGF 1.062.01 run on the Solaris Blade with SunOS 5.11 were compared using the UNIX diff command to results from the validation tests of PRECCDFGF 1.01 on the Compaq  ES47 with OpenVMS 8.2.  VMS results were moved to the Solaris and converted as needed [12].

Test Results

The results of the tests for PRECCDFGF 2.0 on the Compaq cluster verify that the code satisfies the criteria in the VVP [15].

The results of the tests for PRECCDFGF 1.06/2.01 on the Solaris described above are that only very minor differences (e.g., spacing, version number) were found for the two test cases.  The only differences found in the output are limited to code run date and time, platform names, system version numbers, the directory, and file names [12].

The Agency's Conclusions

The Agency found that PRECCDFGF 2.0 test cases verify that the code performs correctly.  The Agency concludes that PRECCDFGF 2.0 meets the acceptance criteria in the VVP and is validated for the 2014 CRA WIPP PA use on the Compaq computer cluster with OpenVMS 8.2.

The Agency also found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The comparison found no differences in the numerical output of PRECCDFGF 1.06/2.01. The Agency concludes that PRECCDFGF 1.06/2.01 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 2003.  "Verification and Validation Plan for PRECCDFGF Version 1.00A Document Version 1.02."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #530467.
 WIPP PA (Performance Assessment) 2003.  "Regression testing of PRECCDFGF 1.00B (Addendum to Validation Document for PRECCDFGF 1.00A)."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #531450.
 WIPP PA (Performance Assessment) 2004.  "Analysis Report for PRECCDFGF Version 1.00B Regression Testing for the ES45 and 8400 Platforms."  Sandia National Laboratories.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration Activities  - Version 2."  September 2004.
 WIPP PA (Performance Assessment) 2003.  "Validation Document for PRECCDFGF Version 1.01 Document Version 1.05."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #539296.
 WIPP PA (Performance Assessment) 2005.  "Installation and Checkout for PRECCDFGF Version 1.01 Regression Testing for the Compaq ES45 Platforms."  Sandia National Laboratories.
 USEPA  -  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 USEPA  -  "Technical Support Document for Section 194.23:  Models and Computer Codes-PABC Codes Changes Review."  March 2006.  Docket No. A-98-49/II-B1-8.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of PRECCDFGF 1.01 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 17, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543451.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for PRECCDFGF Version 1.06/2.01 Regression Testing for the Solaris Blade dated November 19, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557822.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
 WIPP PA  -  "Validation Document for PRECCDFGF Version 2.0, dated June 2010." Sandia National Laboratories. Sandia WIPP Central Files ERMS #552577.
 WIPP PA  -  "Verification and Validation Plan for PRECCDFGF Version 2.0, dated May 2010. Sandia National Laboratories.  Sandia WIPP Central Files #552575.
PRELHS

This section presents the validation and regression test results for PRELHS.  The PRELHS program extracts parameter distribution data requested by the user from the PAPDB and sets up the LHS (Latin Hypercube Sampling) input control file.

Introduction

Since the CCA, the PRELHS code has undergone a series of revisions.  PRELHS Version 2.10 was used in the WIPP CCA.  PRELHS 2.10 was validated in February 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of eight test cases met the acceptance criteria defined in the VVP/VD for PRELHS 2.10 [4, 5].
In August 1997, PRELHS was revised to Version 2.20 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of the eight test cases met the acceptance criteria defined in the VVP/VD [1, 6, 7].

In August 2001, PRELHS was revised to Version 2.24 and was validated on a DEC Alpha 2100 with OpenVMS 7.2-1 [2, 8, 9].  The validation test included three new test cases defined for Version 2.24.  Previous versions of PRELHS accessed the old view-based Parameters Database.  PRELHS 2.24 accesses the new procedure-based Parameters Database.  The two databases are not compatible (i.e., PRELHS 2.24 cannot read a view-based Parameters Database), and the parameter entries that were created for testing the previous versions of PRELHS do not exist in the procedure-based Parameters Database.  Therefore, the test cases used to test previous versions of PRELHS (Test Cases 1 through 8) were discarded, and three new test cases (Test Cases 9 through 11) were used to test PRELHS 2.24.

In November 2001, PRELHS was revised to Version 2.30 and was validated on a DEC Alpha 2100 with OpenVMS 7.2-1 [3, 10].  PRELHS 2.30 accesses the new procedure-based PAPDB).  It cannot read the databases accessed by previous versions of PRELHS.  The primary difference between the PAPDB and the old database is the manner in which parameter entries are identified. In the old database, a parameter entry was uniquely identified by material and property, and its compliance type and calculation.  Each parameter entry in the PAPDB is uniquely identified by its material and property, and the associated analysis, computational code, and retrieval number.  Therefore, Test Cases 9 through 11 were discarded by DOE and three new test cases (Test Cases 12 through 14) were designed to verify that PRELHS satisfies all of the requirements and additional functionality specified in the VVP/VD [3].
In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from PRELHS 2.3 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of PRELHS 2.3 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of PRELHS 2.3 on those operating systems [11].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [12, 13, 14].  In March 2004, the Agency completed a report documenting the Agency's approval of PRELHS 2.3 on the Compaq Alpha ES45 and 8400 with OpenVMS 7.3-1 [15].  PRELHS 2.3 was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [16].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [16].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for PRELHS 2.3 to ensure that it continued to function correctly.

In 2012, SNL moved the WIPP PA parameter database to new database software, MySQL [21].  PRELHS uses the parameter database and required qualification to verify that the code satisfies the three functional requirements, five external interface requirements, and one additional functionality requirement listed in the VVP/VD.  SNL developed three new test cases, Test Cases 15, 16, and 17, to validate PRELHS 2.40 for use in CRA 2014.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [18, 20].

The discussion below documents the test methodology, validation testing, regression test results, and the Agency's conclusions with respect to PRELHS 2.41 running on the Solaris Blade with SunOS 5.11 [19].

Test Methodology

In 2012, SNL validated PRELHS 2.40 on the Compaq computer cluster with OpenVMS 8.2 [21]. PRELHS 2.40 was validated against the criteria discussed in the VVP/VD.  SNL developed three new test cases to verify that the code can properly access the new parameter database (PAPDB 2.0) using the MySQL database software.

In 2013, SNL validated PRELHS 2.41 on the Solaris with SunOS 5.11 [19].  The validation includes three test cases, Test Cases 15, 16 and 17 ([19] Section 4.0), that verify that the code can satisfy the requirements listed in [19] Table 4-1.  PRELHS 2.41 testing consisted of one regression test, Test Case 15 that tests existing requirements; and two validation tests (16 and 17). Test Case 16 was modified to verify two new requirements (R.9-parameter database debug output and A.2-parameter retrieval logging) as well as many existing requirements. Test Case 17 evaluates modifications to error message formatting and existing requirements.

Test Case 15 results were compared to VMS results using the UNIX diff command and results for Test Cases 16 and 17 were evaluated against the acceptance criteria in the VVP/VD [19].

Test Results

SNL verified that PRELHS 2.40 performs various requirements listed in the VVP/VD properly and can adequately access PAPDB 2.0 on the Compaq cluster.

Test Case 15 regression results show acceptable differences, such as run date/time, etc. Test Case 16 validates PRELHS 2.41 by showing PRELHS can process: the R.2-distribution types listed, R.9-parameter entry retrieval logging, and A.2-debug database output. Test Case 17 was not regression-tested because the format of error messages was changed, therefore requiring validation against the original acceptance criteria [19].

SNL's review verifies that PRELHS 2.41 satisfies the requirements listed in Table 4-1 of the VVP/VD [19].

The Agency's Conclusions

EPA reviewed SNL qualification of PRELHS 2.40 and found that the code is adequately tested.  The Agency concludes that PRELHS 2.40 meets the acceptance criteria in the VVP/VD and is validated for use on the Compaq computer cluster with OpenVMS 8.2 and can be used for the 2014 CRA PA.

The Agency closely examined SNL's validation of PRELHS 2.41 and verifies that the code requirements are adequately tested.  The Agency concludes that PRELHS 2.41 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 Analysis Plan (AP-042) 1998.  "Regression for the Upgrade to OpenVMS Version 7.1 on the WIPP COMPAC Alpha Cluster."  Sandia National Laboratories.
 Analysis Plan (AP-065) 2000.  "Regression for the Upgrade to OpenVMS Version 7.2 on the WIPP DEC Alpha Cluster."  Sandia National Laboratories.
 WIPP PA (Performance Assessment) 2001.  "Requirements Document & Verification and Validation Plan for PRELHS Version 2.30."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #519721.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for PRELHS Version 2.10."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30712.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for PRELHS Version 2.10."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #30716
 WIPP PA (Performance Assessment) 1997.  "Requirements Document & Verification and Validation Plan for PRELHS Version 2.20."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #43935.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for PRELHS Version 2.20."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #43938.
 WIPP PA (Performance Assessment) 2001.  "Verification and Validation Plan for PRELHS Version 2.24."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #513612.
 WIPP PA (Performance Assessment) 2001.  "Validation Document for PRELHS Version 2.24."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #518675.
 WIPP PA (Performance Assessment) 2002.  "Verification and Validation Plan, Validation Document and Criteria Forms for PRELHS, Version 2.30."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #519722.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA (Performance Assessment) 2003.  "Addenda to Verification and Validation Plan/Validation Document for PREHLHS Version 2.30."  Sandia National Laboratories.  Sandia WIPP Central Files ERMS #525224.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of PRELHS 2.30 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 31, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543595.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Verification and Validation Plan/Validation Document for PRELHS Version 2.41 dated March 7, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #559260.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.
 WIPP PA  -  "Verification and Validation Plan/Validation Document for PRELHS Version 2.40, dated March 12, 2012." Sandia National Laboratories. Sandia WIPP Central Files ERMS #556630.

PRESECOTP2D

This section presents the regression test results for PRESECOTP2D.  The purpose of PRESECOTP2D 1.22 is to create all the input files required to run the code SECOTP2D.  Material properties, grid information, and source term information are obtained from CAMDAT databases.  The velocity field is obtained from a transfer file written by PRESECOFL2D.  Since the CCA, the PRESECOTP2D code has undergone a series of revisions.  PRESECOTP2D Version 1.11ZO was used in the WIPP CCA.  PRESECOTP2D 1.11ZO was validated in September 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by regression testing to a validated primitive package [3].  Regression testing demonstrated that the results of two test cases (2 and 3) met the acceptance criteria defined in the RD/VVP for PRESECOTP2D 1.11ZO [4].

Introduction

In August 1996, PRESECOTP2D was revised to Version 1.20 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 [5, 6].  Test Case 1 for the validation of PRESECOTP2D 1.20 was identical to the test case for the validation of PRESECOTP2D 1.11ZO.  The acceptance criteria for this test case were satisfied by showing that the output from PRESECOTP2D 1.20 was identical to the output of the PRESECOTP2D 1.11ZO validation tests.  Test Cases 2 and 3 were modified to test code functionality that changed from Version 1.11ZO to version 1.20.  In these test cases, the acceptance criteria were satisfied by analysis of the output of PRESECOTP2D 1.20.  PRESECOTP2D 1.20 was used to support the CCA.

In June 1997, PRESECOTP2D was revised to Version 1.22 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 [1, 2].  The validation test included the three test cases defined for Version 1.20, and an additional Test Case 4.  Test Case 4 was added to verify the variable time step functionality by showing the time increments produced by the code match those produced by the algorithm in the user's manual [7].  Acceptance criteria for Test Cases 1 - 3 were satisfied by comparing output of PRESECOTP2D 1.22 to the output of PRESECOTP2D 1.20, while the acceptance criteria for Test Case 4 were satisfied by analysis of the output of PRESECOTP2D 1.22.

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from PRESECOTP2D 1.22 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of PRESECOTP2D 1.22 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of PRESECOTP2D 1.22 on those operating systems [8].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which were both running OpenVMS 7.3-1 [9, 10].  In March 2004, the Agency completed a report documenting the Agency's approval of PRESECOTP2D 1.22 on the Compaq Alpha ES45 and 8400 both running OpenVMS 7.3-1 [11].  PRESECOTP2D 1.22 was used to support the 2004 CRA.

In 2006 SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [12].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [12].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for PRESECOTP2D 1.22 to ensure that it continued to function correctly.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [14, 16].

The discussion below documents the test methodology, validation test results, regression test results, and the Agency's conclusions with respect to PRESECOTP2D 1.23 running on the Solaris Blade with SunOS 5.11 [15].

Test Methodology

Testing of PRESECOTP2D 1.23 uses four regression tests that verify that the code satisfies acceptance criteria listed in the RD/VVP [15].  One test was also run that verifies that the code inputs MODFLOW files correctly [15].  PRESECOTP2D 1.22 files on the Compaq ES47 computers were converted to text as needed, moved to the Solaris Blade, and compared.  Comparison is done using the Python utility (Regr_Diff.py-used to compare text files) which calculates the RPD, relative percent differences [15].  RPD values less than 1E-4 are acceptable.

Test Results

Test Cases 1, 2, and 3 show expected textural differences (e.g. file names, version numbers, etc.) and minor numeric differences below the RPD threshold.  However, Test Case 4 had one numeric difference above the RPD threshold. SNL reviewed and explained this difference showing that the difference in time step would not significantly impact computations.  All other Test Case 4 results were acceptable [15].

Section 5.0 of the VD/VVP [15] describes the unit test that verifies that PRESECOTP2D 1.23 "...can read an actual MODFLOW file..." by manually converting binary results and comparing them using the difference utility code.  No results were above the RPD threshold.

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names, and minor numeric difference.  EPA closely examined Test Case 4 that had one RPD greater than 1E-4 and agrees with SNL's explanation.  EPA also closely looked at the unit test and concludes that it is adequate.  The Agency concludes that PRESECOTP2D 1.23 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.
References

 WIPP PA (Performance Assessment) 1997.  "Requirements Document & Verification and Validation Plan for PRESECOTP2D Version 1.22."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45957.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for PRESECOTP2D Version 1.22."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45965.
 "Primitive Data Package" for PRESECOTP2D, Version 1.11ZO, version date 8/16/93, Binder with the following information:  Abstract User Manual, On-Line Help, Verification, Review, Driver, and Source.  Sandia National Laboratories.  Sandia WIPP Central Files Record #220278.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for PRESECOTP2D Version 1.11ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #23324.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for PRESECOTP2D Version 1.20."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37295.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for PRESECOTP2D Version 1.20."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #40254.
 WIPP PA (Performance Assessment) 1997.  "User's Manual for PRESECOTP2D Version 1.22."  Sandia National Laboratories.  Sandia Central Files WPO #45963.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test," Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of PRESECOTP2D 1.04 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 31, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543591.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Validation Document for PRESECPTP2D Version 1.23 dated May 2, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #559779.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #557765.

RELATE

This section presents the regression test results for RELATE.  RELATE transfers information from one CAMDAT database file (the "Reference" database) to another CAMDAT database file (the "Object" database) using either the relative positions of the meshes defined on the reference and object databases, or a symbolic mapping between the material and property names on the reference database and the material and property names on the object database.  CAMDAT database files are also referred to as CDB files.

Introduction

RELATE Version 1.42ZO was validated in October 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of three test cases met the acceptance criteria defined in the RD/VVP for RELATE 1.42ZO [1].

In March 1996, RELATE was revised to Version 1.43 and was validated on a DEC Alpha 2100 with OpenVMS 6.1.  Test cases identical to the test cases for the validation of RELATE 1.42ZO were run.  The acceptance criteria for these test cases were satisfied by showing that the output from RELATE 1.43 was identical to the output of the RELATE 1.42ZO validation tests.  RELATE 1.43 was used to support the CCA.

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from RELATE 1.43 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of RELATE 1.43 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the Agency completed a report documenting the Agency's conclusions with respect to the migration and verification of RELATE 1.43 on those operating systems [3].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [4, 5].  In March 2004, the Agency completed a report documenting the Agency's approval of RELATE 1.43 on the Compaq Alpha ES45 and 8400 [6].  RELATE 1.43 was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [7].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [7].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for RELATE 1.43 to ensure that it continues to function correctly.

In 2012, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [9, 11].

The discussion below documents the 2012 test methodology, regression test results, and the Agency's conclusions with respect to RELATE 1.45 running on the Solaris Blade with SunOS 5.11 [10].

Test Methodology

The tests for this code comprised the three test cases described in the Requirements Document & Verification and Validation Plan for RELATE Version 1.42Z0 (RD/VVP) [1].  The regression test results from RELATE 1.43 run on the Compaq ES40 with OpenVMS 7.3-1 used the VMS DIFFERENCE command to compare results from the validation tests of RELATE 1.43 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2.

CAMDAT database files (CDB) are produced in each of the three RELATE test cases.  The output CDB files are converted from a binary, CDB, file to an ASCII, file for comparison during the validation process.  In the previous RELATE 1.43 validation, the CDB files were converted using GROPE 2.10.  GROPE has since been revised to Version 2.12.  GROPE 2.12 was validated in June 1996 on a DEC Alpha 2100 with OpenVMS 6.1 [2].  GROPECDB 2.12 has also been validated on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 as part of the hardware regression test (see Section 5.11 above).

In 2012, SNL regression test RELATE 1.45 using a similar approach as described above.  SNL used three test cases to preform regression testing of RELATE 1.45 on the Solaris Blade with SunOS 5.11 compared to the results of RELATE 1.43 on the VMS Compaq E47 with OpenVMS 8.2. The VMS results were transferred to the Solaris and compared using the UNIX diff command [10].

Test Results

The results of the tests referenced above show that only very minor differences (e.g., spacing, version number) were found for the three test cases.  The comparison found that all differences in the output are limited to code run date and time, platform names, system version numbers, the directory, and file names.

The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, and file names.  The comparison found no differences in the numerical output of RELATE 1.45.  The Agency concludes that RELATE 1.45 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for RELATE Version 1.42ZO."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #24184.
 WIPP PA (Performance Assessment) 1996.  "Validation Document for GROPECDB Version 2.12."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #37497.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of RELATE 1.43 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 6, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543592.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Installation and Checkout for RELATE Version 1.45 Regression Testing for the Solaris Blade, dated October 25, 2012."  Sandia National Laboratories. Sandia WIPP Central Files ERMS #557826.
 WIPP PA - "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories. Sandia WIPP Central Files ERMS #557765.

SECOTP2D

This section presents the validation and regression test results for the SECOTP2D.  SECOTP2D performs single or multiple component radionuclide transport in fractured or granular aquifers.  Fractured porous media are represented using a dual porosity model.  The code uses total variation diminishing (TVD) schemes to model the advective part of the transport equation.

Introduction

Since the CCA the SECOTP2D code has undergone a series of revisions.  SECOTP2D Version 1.30 was used in the WIPP CCA.  SECOTP2D 1.30 was validated in April 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of three test cases met the acceptance criteria defined in the RD/VVP for SECOTP2D 1.30 [3, 4]. SECOTP2D 1.30 was used to support the CCA.

In July 1997, SECOTP2D was revised to Version 1.41 and was validated on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of six new test cases met the acceptance criteria defined in the RD/VVP for SECOTP2D 1.41 [1, 2].

In order to test new operating systems that were added in 2002 - 2003, regression test results from SECOTP2D 1.41 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of SECOTP2D 1.41 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the EPA completed a report documenting the Agency's conclusions with respect to the migration and verification of SECOTP2D 1.41 on those operating systems [5].  SECOTP2D 1.41 was used to support the 2004 CRA.

In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [6, 7].  Modifications were also made to SECOTP2D 1.4 and the version number was changed from 1.4 to 1.4A [8].  In March 2004, the Agency completed a report documenting the Agency's approval of SECOTP2D 1.41A on the Compaq Alpha ES45 and 8400 with OpenVMS 7.3-1 [9].

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [10].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [10].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for SECOTP2D 1.41A to ensure that it continued to function correctly.

In 2013, SNL migrated WIPP PA software from VMS platform on the Compaq cluster of computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [12, 14].  SNL performed validation testing and regression testing to verify that SECOTP2D 1.43 continues to perform PA calculations correctly ([13] page 7).

The discussion below documents the test methodology, regression test results, qualification test results and the Agency's conclusions with respect to SECOTP2D 1.43 running on the Solaris Blade with SunOS 5.11.

Test Methodology

The tests for this code comprised the four test cases described in the Requirements Document & Verification and Validation Plan for SECOTP2D Version 1.41 (RD/VVP) [1].  Regression test results from SECOTP2D 1.41A run on the Compaq ES40 with OpenVMS 7.3-1 used the VMS DIFFERENCE command to compare results from the validation tests of SECOTP2D 1.41A run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [11].
In 2013, SNL performed validation and regression testing of SECOTP2D 1.43 on the Solaris Blade with SunOS 5.11 on all six test cases (1, 2, 3, 4, 6A, 6B ([13] Section 3.0)) to verify that the code performs PA calculations correctly.  These test cases verify that SECOTP2D 1.43 satisfies the acceptance criteria of the thirteen Functional Requirements (R.1 through R.13), three External Interface Requirements (R.14 through R.16), and eight Additional Functionality Requirements (A.1 through A.8) as noted in [13] Table 3-1 and are described in the RD/VVP [2]. SNL states, "Where possible, regression testing was used to verify that SECOTP2D Version 1.43 test results.  Where regression testing results in many differences [RPD above 1E-4], the SECOTP2D 1.43 test results were verified using the acceptance criteria from the SECOTP2D 1.41 RD/VVP [2]." ([13] page 7) SNL applied this approach to specific output values as needed.
Test Results

The six test cases for SECOTP2D 1.43 were executed on the Solaris Blade with SunOS 5.11.  SNL used a combination of regression testing and acceptance criteria test to verify that the code continues to perform the WIPP PA correctly ([13] page 7).  Mainly the differences were textual or data values were below the RPD threshold.  Other requirements were verified against the acceptance criteria specified in the VD ([3] page 7) to verify continued PA performance of the code.

SNL used the Regr_Diff.py Python utility which does a line-by-line comparison and calculates the RPD for input files ([13] page 9). Any difference with RPD greater than 1E-4 must be explained, this was usually done comparing the results to the acceptance criteria for these values ([13] Section 4.0). Test Cases 3 and 4 had differences below the RPD limit and were considered by SNL to satisfy the acceptance criteria.  Most of the differences for Test Cases 1 and 2 were below the RPD limit, however a few output values required further review against the acceptance criteria. For Test Cases 6A and 6B, SNL used the acceptance criteria to validate these tests [13].

The Agency's Conclusions

The Agency closely examined SNL's code validation process and found that it adequately complies with the approach described in the VD ([13] Section 3.0) and satisfies acceptance criteria.  EPA found that numerical differences were the result of conversion to double precision numeric format on the Solaris and that new functionality of the code was tested and reviewed reasonably. The Agency concludes that SECOTP2D 1.43 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade platform with SunOS 5.11.
References

 WIPP PA (Performance Assessment) 1997.  "Requirements Document and Verification and Validation Plan for SECOTP2D Version 1.41."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45732.
 WIPP PA (Performance Assessment) 1997.  "Validation Document for SECOTP2D Version 1.41."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #45735.
 WIPP PA (Performance Assessment) 1996.  "Requirements Document and Verification and Validation Plan for SECOTP2D Version 1.30."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #36693.
 WIPP PA (Performance Assessment) 1996. `Validation Document for SECOTP2D Version 1.30."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #36694.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 WIPP PA  -  "Change Control Form - SECOTP2D Version 1.41 to 1.41A" 2003.  Sandia National Laboratories.  Sandia WIPP Central Files WPO #526257.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  Validation (Performance Assessment) 2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of SECOTP2D 1.41A on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated June 1, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543596.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162, dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA _ "Validation Document for SECOTP2D Version 1.43, dated August 5, 2013. Sandia National Laboratories.  Sandia WIPP Central Files ERMS #559784.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories. Sandia WIPP Central Files ERMS #557765.

STEPWISE

This section presents the regression test results for STEPWISE.  STEPWISE is a statistical code that evaluates variable importance by developing regression models between the observed response and input variables using either a forward, backward, or stepwise regression procedure on the raw or ranked data.  STEPWISE 2.20 was validated in November 1995 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of three test cases met the acceptance criteria defined in the RD/VVP for STEPWISE 2.20 [2].

Introduction

In November of 1996, STEPWISE was revised from Version 2.20 to Version 2.21.  Version 2.21 was validated on the DEC Alpha 2100 with OpenVMS 6.1 by a combination of acceptance and regression testing.  Test Cases 1 - 3 were validated through regression testing while Test Cases 4 and 5 underwent acceptance testing.  Test Cases 4 and 5 were created to illustrate the correction of errors found in Version 2.20, and were validated by comparing output results to the acceptance criteria defined in the RD/VVP for STEPWISE 2.21 [1]. STEPWISE 2.20 was used to support the CCA.

In order to test new operating systems that were added in 2002 - 2003 (Section 1), regression test results from STEPWISE 2.21 run on the ES40 with OpenVMS 7.3-1 were compared to results from the validation tests of STEPWISE 2.21 run on a DEC Alpha 2100 with OpenVMS 6.1.  In June 2003, the EPA completed a report documenting the Agency's conclusions with respect to the migration and verification of STEPWISE 2.21 on those operating systems [3].  In January 2003, two new hardware systems were added to conduct PAs for the WIPP; a Compaq ES45 and a Compaq Alpha 8400, which are both running OpenVMS 7.3-1 [4, 5].  In March 2004, the Agency completed a report documenting the Agency's approval of STEPWISE 2.21 on the Compaq Alpha ES45 and 8400, which were both running OpenVMS 7.3-1 [6].  STEPWISE 2.21 was used to support the 2004 CRA.

In 2006, SNL procured four Compaq ES47 machines to add to the computing resources of two Compaq ES40 and two Compaq ES45 machines [7].  In addition to the hardware upgrades, the operating system OpenVMS 7.3-1 was upgraded to OpenVMS 8.2 [7].  Because of these changes in the operating system and the addition of a new computing platform, regression testing was conducted for STEPWISE 2.21 to ensure that it continued to function correctly.

In 2013, SNL migrated WIPP PA software from the VMS Alpha platform on a cluster of Compaq computers with OpenVMS 8.2 operating system to the Solaris Blade platform on Sun hardware with Intel based processors with SunOS 5.11 operating system [9, 11].

The discussion below documents the test methodology, regression test results, and the Agency's conclusions with respect to STEPWISE 2.22 running on the Solaris Blade with SunOS 5.11 [10].
Test Methodology

The tests for this code comprised the five test cases described in the Requirements Document & Verification and Validation Plan for STEPWISE Version 2.21 RD/VVP) [1].  Regression test results from STEPWISE 2.21 run on the Compaq ES40 with OpenVMS 7.3-1 used the VMS DIFFERENCE command to compare results from the validation tests of STEPWISE 2.21 run on the Compaq ES40, ES45, and the ES47 with OpenVMS 8.2 [8].

In 2013, SNL regression tested (compared) STEPWISE 2.22 results executed on the Solaris Blade with SunOS 5.11 and STEPWISE 2.21 executed on the Compaq ES47 with OpenVMS 8.2 [10].  Results on VMS were transferred to the Solaris, converted to Solaris platform format as needed, and compared to the Solaris results using the UNIX diff command and plotted for visual comparison as needed [10].

SNL noted: "Small numeric differences are expected due to slight differences in the two platforms. For STEPWISE, differences in the final digit of numeric values (with rounding) are acceptable." and "Small variations in the plots are acceptable, but any difference in the location of data on the plots must be explained" [10].

Test Results
All test cases showed only minor differences (e.g., spacing, version number) [10].  The comparison found that all differences in the output are limited to code run date and time, platform names, system version numbers, the directory, and file names.  Plots results showed minor differences for all test cases.  Test Case 2 showed minor numeric differences in the final digit.  Test Case 1 showed one digit with the last two digits different, SNL provided a reasonable explanation of this difference [10].
The Agency's Conclusions

The Agency found that all differences in output are acceptable; namely, that the differences are limited to code run date and time, platform names, system version numbers, the directory, file names and minor numeric difference.  EPA reviewed SNL's explanation of insignificant numeric differences and found them to be reasonable.  The Agency concludes that STEPWISE 2.22 meets the acceptance criteria in the RD/VVP and is validated for WIPP PA use on the Solaris Blade with SunOS 5.11.

References

 WIPP PA (Performance Assessment) 1996.  "Requirements Document & Verification and Validation Plan for STEPWISE Version 2.21."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #42250.
 WIPP PA (Performance Assessment) 1995.  "Requirements Document & Verification and Validation Plan for STEPWISE Version 2.20."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #27767.
 EPA 2003.  "Review of WIPP Performance Assessment Computer Code Migration, June 10, 2003."  Environmental Protection Agency.
 WIPP PA  -  "Analysis Report for the ES45 Regression Test, March 6, 2003."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #530290.
 WIPP PA  -  "Analysis Report for the 8400 Regression Test."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #527280.
 EPA 2004.  "Review of WIPP Performance Assessment Computer Code Migration, March 31, 2004."  Environmental Protection Agency.
 WIPP PA  -  2006.  "Installation of OpenVMS Version 8.2-1 on the WIPP Alpha Cluster and Regression Testing, dated March 16, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files WPO #542680.
 WIPP PA  -  "Regression Testing Report of STEPWISE 2.21 on the Compaq ES40, ES45, and ES47 Platforms Using OpenVMS 8.2 dated May 31, 2006."  Sandia National Laboratories.  Sandia WIPP Central Files.  ERMS #543589.
 WIPP PA  -  "Summary Report on the Migration of the WIPP PA Codes From VMS to Solaris, AP-162 dated December 23, 2013." Sandia National Laboratories.  Sandia WIPP Central Files ERMS #561457.
 WIPP PA  -  "Validation Document for STEPWISE Version 2.22 dated July 30, 2013." Sandia National Laboratories. Sandia WIPP Central Files ERMS #560366.
 WIPP PA  -  "AP-162 Analysis Plan for Migration of the Performance Assessment Codes to the Sun Solaris Blade Server Running with Intel Processors, Revision 0 dated June 12, 2012." Sandia National Laboratories. Sandia WIPP Central Files ERMS #557765.

SUMMARIZE

This section presents the regression test results for the SUMMARIZE code.  SUMMARIZE reads ordered sets of sampled CAMDAT (CDB) data files.  For the current regulatory calculations, there are 100 different input CAMDAT files in each ordered set.  These files would normally constitute the principal output from one of the WIPP-PA analytical codes.  SUMMARIZE can extract a single item (or set of items) from each of its 100 input files and write the 100 item values to a single output file.  If the input data constitute time histories, and they often do, SUMMARIZE will select the one value from each input file that corresponds most closely to a user-selected time or set of times from the time history.  If requested, SUMMARIZE can interpolate input data to the exact time specified by the user.  As a result of this process, selected data that originally resided on 100 different binary CAMDAT files are rearranged and reported on a single ASCII output file.

SUMMARIZE reports its results in any of several convenient ASCII formats that can be read by commercial and WIPP-PA analysis and plotting codes.  SUMMARIZE's output format is normally selected by the user so as to be compatible with the analysis and/or plotting code that will be applied next in the run sequence.

Introduction

Since the CCA, the SUMMARIZE code has undergone a series of revisions.  SUMMARIZE Version 2.10 was used in the WIPP CCA.  SUMMARIZE 2.10 was validated in May 1996 on a DEC Alpha 2100 with OpenVMS 6.1 by demonstrating that the results of seven test cases met the acceptance criteria defined in the RD/VVP for SUMMARIZE 2.10 [2, 3].