Kriging metamodeling in simulation: A review
---
paper_title: Use of Kriging Models to Approximate Deterministic Computer Models

paper_content:

The use of kriging models for approximation and metamodel-based design and optimization has been steadily on the rise in the past decade. The widespread use of kriging models appears to be hampered by 1) computationally efficient algorithms for accurately estimating the model's parameters, 2) an effective method to assess the resulting model's quality, and 3) the lack of guidance in selecting the appropriate form of the kriging model. We attempt to address these issues by comparing 1) maximum likelihood estimation and cross validation parameter estimation methods for selecting a kriging model's parameters given its form and 2) an R 2 of prediction and the corrected Akaike information criterion assessment methods for quantifying the quality of the created kriging model. These methods are demonstrated with six test problems. Finally, different forms of kriging models are examined to determine if more complex forms are more accurate and easier to fit than simple forms of kriging models for approximating computer models.
---
paper_title: Experimental Design for Sensitivity Analysis, Optimization, and Validation of Simulation Models

paper_content:

This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.
---
paper_title: Principles of geostatistics

paper_content:

Knowledge of ore grades and ore reserves as well as error estimation of these values, is fundamental for mining engineers and mining geologists. Until now no appropriate scientific approach to those estimation problems has existed: geostatistics, the principles of which are summarized in this paper, constitutes a new science leading to such an approach. The author criticizes classical statistical methods still in use, and shows some of the main results given by geostatistics. Any ore deposit evaluation as well as proper decision of starting mining operations should be preceded by a geostatistical investigation which may avoid economic failures.
---
paper_title: Noisy Aerodynamic Response and Smooth Approximations in HSCT Design

paper_content:

Convergence difficulties were encountered in our recent efforts towards a combined aerodynamic-structural optimization of the High Speed Civil Transport (HSCT). The underlying causes of the convergence problems were traced to numerical noise in the calculation of aerodynamic drag components for obstacles to convergence. The first technique employed a sequential approximation optimization method which used large initial move limits on the design variables. This helped dislodge the optimizer out of the local minima in the design space created by the noisy drag data. The second method utilized the aircraft. Two techniques were developed to circumvent the response surface methods to construct smooth approximations to the noisy data. The response surfaces were formed by analyzing several individual HSCT configuration and then fitting polynomial functions to selected objective function data. A simplified example design problem was used to demonstrate the response surface technique and to investigate various other issues relating to the construction of the response surfaces.
---
paper_title: Discrete-Event System Simulation

paper_content:

This book provides a basic treatment of one of the most widely used operations research tools: discrete-event simulation. Prerequisites are calculus, probability theory, and elementary statistics. Contents, abridged: Introduction to discrete-event system simulation. Mathematical and statistical models. Random numbers. Analysis of simulation data. Index.
---
paper_title: The Design and Analysis of Computer Experiments

paper_content:

Physical Experiments and Computer Experiments.- Basic Elements of Computer Experiments.- Analyzing Output from Computer Experiments-Predicting Output from Training Data.- Space Filling Designs for Computer Experiments.- Criteria Based Designs for Computer Experiments.- Other Issues.
---
paper_title: Kriging for interpolation in random simulation

paper_content:

Whenever simulation requires much computer time, interpolation is needed. There are several interpolation techniques in use (for example, linear regression), but this paper focuses on Kriging.This technique was originally developed in geostatistics by D.G.Krige, and has recently been widely applied in deterministic simulation.This paper, however, focuses on random or stochastic simulation.Essentially, Kriging gives more weight to 'neighbouring' observations.There are several types of Kriging; this paper discusses - besides Ordinary Kriging - a novel type, which 'detrends' data through the use of linear regression.Results are presented for two examples of input/output behaviour of the underlying random simulation model: A perfectly specified detrending function gives the best predictions, but Ordinary Kriging gives quite acceptable results; traditional linear regression gives the worst predictions.
---
paper_title: Efficient Global Optimization of Expensive Black-Box Functions

paper_content:

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.
---
paper_title: Approximation methods in multidisciplinary analysis and optimization: a panel discussion

paper_content:

This paper summarizes the discussion at the Approximation Methods Panel that was held at the 9 th AIAA/ISSMO Symposium on Multidisciplinary Analysis & Optimization in Atlanta, GA on September 2–4, 2002. The objective of the panel was to discuss the current state-of-the-art of approximation methods and identify future research directions important to the community. The panel consisted of five representatives from industry and government: (1) Andrew J. Booker from The Boeing Company, (2) Dipankar Ghosh from Vanderplaats Research & Development, (3) Anthony A. Giunta from Sandia National Laboratories, (4) Patrick N. Koch from Engineous Software, Inc., and (5) Ren-Jye Yang from Ford Motor Company. Each panelist was asked to (i) give one or two brief examples of typical uses of approximation methods by his company, (ii) describe the current state-of-the-art of these methods used by his company, (iii) describe the current challenges in the use and adoption of approximation methods within his company, and (iv) identify future research directions in approximation methods. Several common themes arose from the discussion, including differentiating between design of experiments and design and analysis of computer experiments, visualizing experimental results and data from approximation models, capturing uncertainty with approximation methods, and handling problems with large numbers of variables. These are discussed in turn along with the future directions identified by the panelists, which emphasized educating engineers in using approximation methods.
---
paper_title: Kriging models for global approximation in simulation-based multidisciplinary design optimization

paper_content:

Response surface methods have been used for a variety of applications in aerospace engineering, particularly in multidisciplinary design optimization. We investigate the use of kriging models as alternatives to traditional second-order polynomial response surfaces for constructing global approximations for use in a real aerospace engineering application, namely, the design of an aerospike nozzle. Our objective is to examine the difeculties in building and using kriging models to create accurate global approximations to facilitate multidisciplinary design optimization. Error analysis of the response surface and kriging models is performed, along with a graphical comparison of the approximations. Four optimization problems are also formulated and solved using both sets of approximation models to gain insight into their use for multidisciplinary design optimization. We end that the kriging models, which use only a constant “global” model and a Gaussian correlation function, yield global approximations that are slightly more accurate than the response surface models.
---
paper_title: Simulation Modeling and Analysis

paper_content:

From the Publisher: ::: This second edition of Simulation Modeling and Analysis includes a chapter on "Simulation in Manufacturing Systems" and examples. The text is designed for a one-term or two-quarter course in simulation offered in departments of industrial engineering,business,computer science and operations research.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---
paper_title: Statistical testing of optimality conditions in multiresponse simulation-based optimization

paper_content:

This article studies simulation-based optimization with multiple outputs. It assumes that the simulation model has one random objective function and must satisfy given constraints on the other random outputs. It presents a statistical procedure for testing whether a specific input combination (proposed by some optimization heuristic) satisfies the Karush-Kuhn-Tucker (KKT) first-order optimality conditions. The article focuses on "expensive" simulations, which have small sample sizes. The article applies the classic t test to check whether the specific input combination is feasible, and whether any constraints are binding; next, it applies bootstrapping (resampling) to test the estimated gradients in the KKT conditions. The new methodology is applied to three examples, which gives encouraging empirical results.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---
paper_title: An Introduction to the Bootstrap

paper_content:

Introduction The Accuracy of a Sample Mean Random Samples and Probabilities The Empirical Distribution Function and the Plug-In Principle Standard Errors and Estimated Standard Errors The Bootstrap Estimate of Standard Error Bootstrap Standard Errors: Some Examples More Complicated Data Structures Regression Models Estimates of Bias The Jackknife Confidence Intervals Based on Bootstrap "Tables" Confidence Intervals Based on Bootstrap Percentiles Better Bootstrap Confidence Intervals Permutation Tests Hypothesis Testing with the Bootstrap Cross-Validation and Other Estimates of Prediction Error Adaptive Estimation and Calibration Assessing the Error in Bootstrap Estimates A Geometrical Representation for the Bootstrap and Jackknife An Overview of Nonparametric and Parametric Inference Further Topics in Bootstrap Confidence Intervals Efficient Bootstrap Computations Approximate Likelihoods Bootstrap Bioequivalence Discussion and Further Topics Appendix: Software for Bootstrap Computations References
---
paper_title: 2 Robustness of Kriging when interpolating in 3 random simulation with heterogeneous variances: 4 Some experiments

paper_content:

This paper investigates the use of Kriging in random simulation when the simulation output variances are not constant. Kriging gives a response surface or metamodel that can be used for interpolation. Because Ordinary Kriging assumes constant variances, this paper also applies Detrended Kriging to estimate a non-constant signal function, and then standardizes the residual noise through the heterogeneous variances estimated from replicated simulation runs. Numerical examples, however, suggest that Ordinary Kriging is a robust interpolation method.
---
paper_title: Use of Kriging Models to Approximate Deterministic Computer Models

paper_content:

The use of kriging models for approximation and metamodel-based design and optimization has been steadily on the rise in the past decade. The widespread use of kriging models appears to be hampered by 1) computationally efficient algorithms for accurately estimating the model's parameters, 2) an effective method to assess the resulting model's quality, and 3) the lack of guidance in selecting the appropriate form of the kriging model. We attempt to address these issues by comparing 1) maximum likelihood estimation and cross validation parameter estimation methods for selecting a kriging model's parameters given its form and 2) an R 2 of prediction and the corrected Akaike information criterion assessment methods for quantifying the quality of the created kriging model. These methods are demonstrated with six test problems. Finally, different forms of kriging models are examined to determine if more complex forms are more accurate and easier to fit than simple forms of kriging models for approximating computer models.
---
paper_title: The Design and Analysis of Computer Experiments

paper_content:

Physical Experiments and Computer Experiments.- Basic Elements of Computer Experiments.- Analyzing Output from Computer Experiments-Predicting Output from Training Data.- Space Filling Designs for Computer Experiments.- Criteria Based Designs for Computer Experiments.- Other Issues.
---
paper_title: Kriging for interpolation in random simulation

paper_content:

Whenever simulation requires much computer time, interpolation is needed. There are several interpolation techniques in use (for example, linear regression), but this paper focuses on Kriging.This technique was originally developed in geostatistics by D.G.Krige, and has recently been widely applied in deterministic simulation.This paper, however, focuses on random or stochastic simulation.Essentially, Kriging gives more weight to 'neighbouring' observations.There are several types of Kriging; this paper discusses - besides Ordinary Kriging - a novel type, which 'detrends' data through the use of linear regression.Results are presented for two examples of input/output behaviour of the underlying random simulation model: A perfectly specified detrending function gives the best predictions, but Ordinary Kriging gives quite acceptable results; traditional linear regression gives the worst predictions.
---
paper_title: A Monte Carlo Simulation of the Kriging Model

paper_content:

In this paper, we investigate the resulting probability distribution of a kriging model when the values of the model parameters must be estimated from observations of the process being modeled. The output of a kriging model defines a Gaussian probability distribution when the values of the model parameters are given. In practice, these model parameters must be estimated from observations of the process being modeled (typically a computationally expensive computer model). We found that when the model parameters are treated as random variables instead of known values, the resulting probability distribution of the kriging model can be well approximated by a Student-t distribution, a distribution with fatter tails than the Gaussian or normal distribution. The Markov chain Monte Carlo (MCMC) method was used to determine the probability distributions of the model parameters and the output of the kriging model given the observations. The resulting model parameters were validated against the results of a Bayesian analysis of a simple one- dimensional test problem. The results were also compared to the standard method of Maximum Likelihood Estimation as an alternative method to estimate model parameters.
---
paper_title: Introducing BACCO, an R Bundle for Bayesian Analysis of Computer Code Output

paper_content:

This paper introduces the BACCO bundle of R routines for carrying out Bayesian analysis of computer code output. The bundle comprises packages emulator and calibrator, computerized implementations of the ideas of Oakley and O'Hagan (2002) and Kennedy and O'Hagan (2001a) respectively. The bundle is self-contained and fully documented R code, and includes a toy dataset that furnishes a working example of the functions. Package emulator carries out Bayesian emulation of computer code output; package calibrator allows the incorporation of observational data into model calibration using Bayesian techniques. The package is then applied to a dataset taken from climate science.
---
paper_title: A review of design and modeling in computer experiments

paper_content:

In this chapter, we provide a review of statistical methods that are useful in conducting computer experiments. Our focus is primarily on the task of metamodeling, which is driven by the goal of optimizing a complex system via a deterministic simulation model. However, we also mention the case of a stochastic simulation, and examples of both cases are discussed. The organization of our review separates the two primary tasks for metamodeling: (1) select an experimental design; (2) fit a statistical model. We provide an overview of the general strategy and discuss applications in electrical engineering, chemical engineering, mechanical engineering, and dynamic programming. Then, we dedicate a section to statistical modeling methods followed by a section on experimental designs. Designs are discussed in two paradigms, model-dependent and model-independent, to emphasize their different objectives. Both classical and modern methods are discussed.
---
paper_title: Theory of Modeling and Simulation

paper_content:

From the Publisher: ::: Although twenty-five years have passed since the first edition of this classical text, the world has seen many advances in modeling and simulation, the need for a widely accepted framework and theoretical foundation is even more necessary today. Methods of modeling and simulation are fragmented across disciplines making it difficult to re-use ideas from other disciplines and work collaboratively in multidisciplinary teams. Model building and simulation has been made easier and faster by riding piggyback on advances in software and hardware. However, difficult and fundamental issues such as model credibility and interoperation have received less attention. These issues are now front and center under the impetus of the High Level Architecture (HLA) standard mandated by the U.S. DoD for all contractors and agencies. ::: This book concentrates on integrating the continuous and discrete paradigms for modeling and simulation. A second major theme is that of distributed simulation and its potential to support the co-existence of multiple formalisms in multiple model components. Prominent throughout are the fundamental concepts of modular and hierarchical model composition. ::: This edition presents a rigorous mathematical foundation for modeling and simulation. Also, it now provides a comprehensive framework for integrating the various simulation approaches employed in practice. Including such popular modeling methods as cellular automata, chaotic systems, hierarchical block diagrams, and Petri nets. A unifying concept, called the DEVS Bus, enables models, as expressed in their native formalisms, to be transparently mapped into the Discrete Event System Specification (DEVS). The book shows how to construct computationally efficient, object-oriented simulations of DEVS models on parallel and distributed environments. If you are doing integrative simulations, whether or not they are HLA compliant, this is the only book available to provide the foundation to understand, simplify and successfully accomplish your task. ::: Herbert Praehofer is an Assistant Professor at the Johannes Kepler University in Linz, Austria. He has over 50 publications in international journals and conference proceedings on Modeling and Computer Simulation, Systems Theory, and Software Engineering. ::: Tag Gon Kim is a Professor of Electrical Engineering at the Korea Advanced Institutes of Science and Technology (KAIST), Taejon, Korea. His research interests include discrete event systems modeling/simulation, computer/communication systems analysis, and object-oriented simulation engineering. He is a senior member of IEEE and SCS, and a member of ACM. ::: * Provides a comprehensive framework for continuous and discrete event modeling and simulation ::: * Explores the mathematical foundation of simulation modeling ::: * Discusses system morphisms for model abstraction and simplification ::: * Presents a new approach to discrete event simulation of continuous processes ::: * Includes parallel and distributed simulation of discrete event models ::: * Presentation of a concept to achieve simulator interoperability in the form of the DEVS-Bus ::: * Complete coverage necessary for compliance with High Level Architecture (HLA) standards ::: ::: Bernard P Zeigler, is a Professor of Electrical & Computer Engineering at the University of Arizona and heads the Artificial Intelligence Simulation Research Group. He is the author of numerous books and publications, and he is the Editor-in-Chief of the Transactions of the Society for Computer Simulation International.
---
paper_title: Application-driven sequential designs for simulation experiments: Kriging metamodeling

paper_content:

This paper proposes a novel method to select an experimental design for interpolation in simulation. Although the paper focuses on Kriging in deterministic simulation, the method also applies to other types of metamodels (besides Kriging), and to stochastic simulation. The paper focuses on simulations that require much computer time, so it is important to select a design with a small number of observations. The proposed method is therefore sequential. The novelty of the method is that it accounts for the specific input/output function of the particular simulation model at hand; that is, the method is application-driven or customized. This customization is achieved through cross-validation and jackknifing. The new method is tested through two academic applications, which demonstrate that the method indeed gives better results than either sequential designs based on an approximate Kriging prediction variance formula or designs with prefixed sample sizes.
---
paper_title: On Using Kriging Models as Probabilistic Models in Design

paper_content:

Kriging models are frequently used as metamodels during system design optimization. In many applications, a kriging model is used as a deterministic model of a computationally expensive analysis or simulation. In this paper, a kriging model is employed as a probabilistic model on a one-dimensional and two two-dimensional test problems. A probabilistic model is a model in which the parameters are random variables resulting in a probability distribution of the output rather than a deterministic value. A probabilistic model can be used in design to quantify the knowledge designers have about a subsystem and the lack of knowledge or uncertainty in the model. Using a kriging model as a probabilistic model requires that the correlation of observations is only a function of the distance between the observations and that the observations have a Gaussian probability distribution. This paper will provide some methods to satisfy these requirements when using kriging models as probabilistic models.
---
paper_title: Kriging Models that are Robust with Respect to Simulation Errors

paper_content:

In the field of the Design and Analysis of Computer Experiments (DACE) meta-models are used to approximate time-consuming simulations. These simulations often contain simulation-model errors in the output variables. In the construction of meta-models, these errors are often ignored. Simulation-model errors may be magnified by the meta-model. Therefore, in this paper, we study the construction of Kriging models that are robust with respect to simulation-model errors. We introduce a robustness criterion, to quantify the robustness of a Kriging model. Based on this robustness criterion, two new methods to find robust Kriging models are introduced. We illustrate these methods with the approximation of the Six-hump camel back function and a real life example. Furthermore, we validate the two methods by simulating artificial perturbations. Finally, we consider the influence of the Design of Computer Experiments (DoCE) on the robustness of Kriging models.
---
paper_title: Sequential Metamodeling in Engineering Design

paper_content:

Statistical metamodels (surrogate models) are usually developed to replace complex, time-consuming simulation programs or expensive physical experiments and facilitate fast and accurate analysis. Various types of metamodeling methods have been used in the literature, each having its strong and weak points. The choice of the appropriate type of metamodel is important in design. To save time and money spent on expensive experiments, it is also very important to choose a method for sequential experimental design and metamodel development in which data points are identified along a design timeline to help obtain maximum information. Based on the observations in this paper and previous studies, a framework for sequential metamodeling is proposed, in which kriging and MARS metamodels are used to help designers develop appropriate metamodels with limited resources. The method of Sequential Exploratory Experimental Design (SEED) is improved through the utilization of kriging and MARS metamodels.
---
paper_title: 2 Robustness of Kriging when interpolating in 3 random simulation with heterogeneous variances: 4 Some experiments

paper_content:

This paper investigates the use of Kriging in random simulation when the simulation output variances are not constant. Kriging gives a response surface or metamodel that can be used for interpolation. Because Ordinary Kriging assumes constant variances, this paper also applies Detrended Kriging to estimate a non-constant signal function, and then standardizes the residual noise through the heterogeneous variances estimated from replicated simulation runs. Numerical examples, however, suggest that Ordinary Kriging is a robust interpolation method.
---
paper_title: Kriging for interpolation in random simulation

paper_content:

Whenever simulation requires much computer time, interpolation is needed. There are several interpolation techniques in use (for example, linear regression), but this paper focuses on Kriging.This technique was originally developed in geostatistics by D.G.Krige, and has recently been widely applied in deterministic simulation.This paper, however, focuses on random or stochastic simulation.Essentially, Kriging gives more weight to 'neighbouring' observations.There are several types of Kriging; this paper discusses - besides Ordinary Kriging - a novel type, which 'detrends' data through the use of linear regression.Results are presented for two examples of input/output behaviour of the underlying random simulation model: A perfectly specified detrending function gives the best predictions, but Ordinary Kriging gives quite acceptable results; traditional linear regression gives the worst predictions.
---
paper_title: Comparison of designs for computer experiments

paper_content:

Abstract For many complex processes laboratory experimentation is too expensive or too time-consuming to be carried out. A practical alternative is to simulate these phenomena by a computer code. This article considers the choice of an experimental design for computer experiments. We illustrate some drawbacks to criteria that have been proposed and suggest an alternative, based on the Bayesian interpretation of the alias matrix in Draper and Guttman (Ann. Inst. Statist. Math. 44 (1992) 659). Then we compare different design criteria by studying how they rate a variety of candidate designs for computer experiments such as Latin hypercube plans, U-designs, lattice designs and rotation designs.
---
paper_title: An Introduction to the Bootstrap

paper_content:

Introduction The Accuracy of a Sample Mean Random Samples and Probabilities The Empirical Distribution Function and the Plug-In Principle Standard Errors and Estimated Standard Errors The Bootstrap Estimate of Standard Error Bootstrap Standard Errors: Some Examples More Complicated Data Structures Regression Models Estimates of Bias The Jackknife Confidence Intervals Based on Bootstrap "Tables" Confidence Intervals Based on Bootstrap Percentiles Better Bootstrap Confidence Intervals Permutation Tests Hypothesis Testing with the Bootstrap Cross-Validation and Other Estimates of Prediction Error Adaptive Estimation and Calibration Assessing the Error in Bootstrap Estimates A Geometrical Representation for the Bootstrap and Jackknife An Overview of Nonparametric and Parametric Inference Further Topics in Bootstrap Confidence Intervals Efficient Bootstrap Computations Approximate Likelihoods Bootstrap Bioequivalence Discussion and Further Topics Appendix: Software for Bootstrap Computations References
---
paper_title: Use of Kriging Models to Approximate Deterministic Computer Models

paper_content:

The use of kriging models for approximation and metamodel-based design and optimization has been steadily on the rise in the past decade. The widespread use of kriging models appears to be hampered by 1) computationally efficient algorithms for accurately estimating the model's parameters, 2) an effective method to assess the resulting model's quality, and 3) the lack of guidance in selecting the appropriate form of the kriging model. We attempt to address these issues by comparing 1) maximum likelihood estimation and cross validation parameter estimation methods for selecting a kriging model's parameters given its form and 2) an R 2 of prediction and the corrected Akaike information criterion assessment methods for quantifying the quality of the created kriging model. These methods are demonstrated with six test problems. Finally, different forms of kriging models are examined to determine if more complex forms are more accurate and easier to fit than simple forms of kriging models for approximating computer models.
---
paper_title: The Correct Kriging Variance Estimated by Bootstrapping

paper_content:

The classic Kriging variance formula is widely used in geostatistics and in the design and analysis of computer experiments. This paper proves that this formula is wrong. Furthermore, it shows that the formula underestimates the Kriging variance in expectation. The paper develops parametric bootstrapping to estimate the Kriging variance. The new method is tested on several artificial examples and a real-life case study. These results demonstrate that the classic formula underestimates the true Kriging variance.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---
paper_title: A Monte Carlo Simulation of the Kriging Model

paper_content:

In this paper, we investigate the resulting probability distribution of a kriging model when the values of the model parameters must be estimated from observations of the process being modeled. The output of a kriging model defines a Gaussian probability distribution when the values of the model parameters are given. In practice, these model parameters must be estimated from observations of the process being modeled (typically a computationally expensive computer model). We found that when the model parameters are treated as random variables instead of known values, the resulting probability distribution of the kriging model can be well approximated by a Student-t distribution, a distribution with fatter tails than the Gaussian or normal distribution. The Markov chain Monte Carlo (MCMC) method was used to determine the probability distributions of the model parameters and the output of the kriging model given the observations. The resulting model parameters were validated against the results of a Bayesian analysis of a simple one- dimensional test problem. The results were also compared to the standard method of Maximum Likelihood Estimation as an alternative method to estimate model parameters.
---
paper_title: Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping

paper_content:

This paper proposes a novel method to select an experimental design for interpolation in random simulation, especially discrete event simulation.(Though the paper focuses on Kriging, this design approach may also apply to other types of metamodels such as linear regression models.)Assuming that simulation requires much computer time, it is important to select a design with a small number of observations (or simulation runs).The proposed method is therefore sequential.Its novelty is that it accounts for the specific input/output behavior (or response function) of the particular simulation at hand; i.e., the method is customized or application-driven.A tool for this customization is bootstrapping, which enables the estimation of the variances of predictions for inputs not yet simulated.The new method is tested through two classic simulation models: example 1 estimates the expected steady-state waiting time of the M/M/1 queueing model; example 2 estimates the mean costs of a terminating (s, S) inventory simulation.For these simulations the novel design indeed gives better results than Latin Hypercube Sampling (LHS) with a prefixed sample of the same size.
---
paper_title: OVERVIEW OF MODERN DESIGN OF EXPERIMENTS METHODS FOR COMPUTATIONAL SIMULATIONS

paper_content:

The intent of this paper is to provide an overview of modern design of experiments (DOE) techniques that can be applied in computational engineering design studies. The term modern refers to DOE techniques specifically designed for use with deterministic computer simulations. In addition, this term is used to contrast classical DOE techniques that were developed for laboratory and field experiments that possess random error sources. Several types of modern DOE methods are described including pseudo-Monte Carlo sampling, quasi-Monte Carlo sampling, Latin hypercube sampling, orthogonal array sampling, and Hammersley sequence sampling.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---
paper_title: D-Optimal Sequential Experiments for Generating a Simulation-Based Cycle Time-Throughput Curve

paper_content:

A cycle time-throughput curve quantifies the relationship of average cycle time to throughput rates in a manufacturing system. Moreover, it indicates the asymptotic capacity of a system. Such a curve is used to characterize system performance over a range of start rates. Simulation is a fundamental method for generating such curves since simulation can handle the complexity of real systems with acceptable precision and accuracy. A simulation-based cycle time-throughput curve requires a large amount of simulation output data; the precision and accuracy of a simulated curve may be poor if there is insufficient simulation data. To overcome these problems, sequential simulation experiments based on a nonlinear D-optimal design are suggested. Using the nonlinear shape of the curve, such a design pinpointsp starting design points, and then sequentially ranks the remainingn --p candidate design points, wheren is the total number of possible design points being considered. A model of a semiconductor wafer fabrication facility is used to validate the approach. The sequences of experimental runs generated can be used as references for simulation experimenters.
---
paper_title: 9 Computer experiments

paper_content:

Publisher Summary This chapter presents and compares two statistical approaches to computer experiments. The second approach does so by taking random input points. Randomness is required to generate probability or confidence intervals. The first approach introduces randomness by modeling the function, f, as a realization of a Gaussian process. Deterministic computer simulations of physical phenomena are becoming widely used in science and engineering. Some of the most widely used computer models arise in the design of the semiconductors used in the computers themselves. There are two main statistical approaches to computer experiments, one based on Bayesian statistics and a frequentist one based on sampling techniques. A Bayesian approach to modeling simulator output can be based on a spatial model adapted from the geo-statistical Kriging model. This approach treats the bias or systematic departure of the response surface from a linear model as the realization of a stationary random function. This model has exact predictions at the observed responses and predicts with increasing error variance as the prediction point moves away from all the design points.
---
paper_title: Comparison of designs for computer experiments

paper_content:

Abstract For many complex processes laboratory experimentation is too expensive or too time-consuming to be carried out. A practical alternative is to simulate these phenomena by a computer code. This article considers the choice of an experimental design for computer experiments. We illustrate some drawbacks to criteria that have been proposed and suggest an alternative, based on the Bayesian interpretation of the alias matrix in Draper and Guttman (Ann. Inst. Statist. Math. 44 (1992) 659). Then we compare different design criteria by studying how they rate a variety of candidate designs for computer experiments such as Latin hypercube plans, U-designs, lattice designs and rotation designs.
---
paper_title: Introducing BACCO, an R Bundle for Bayesian Analysis of Computer Code Output

paper_content:

This paper introduces the BACCO bundle of R routines for carrying out Bayesian analysis of computer code output. The bundle comprises packages emulator and calibrator, computerized implementations of the ideas of Oakley and O'Hagan (2002) and Kennedy and O'Hagan (2001a) respectively. The bundle is self-contained and fully documented R code, and includes a toy dataset that furnishes a working example of the functions. Package emulator carries out Bayesian emulation of computer code output; package calibrator allows the incorporation of observational data into model calibration using Bayesian techniques. The package is then applied to a dataset taken from climate science.
---
paper_title: Application-driven sequential designs for simulation experiments: Kriging metamodeling

paper_content:

This paper proposes a novel method to select an experimental design for interpolation in simulation. Although the paper focuses on Kriging in deterministic simulation, the method also applies to other types of metamodels (besides Kriging), and to stochastic simulation. The paper focuses on simulations that require much computer time, so it is important to select a design with a small number of observations. The proposed method is therefore sequential. The novelty of the method is that it accounts for the specific input/output function of the particular simulation model at hand; that is, the method is application-driven or customized. This customization is achieved through cross-validation and jackknifing. The new method is tested through two academic applications, which demonstrate that the method indeed gives better results than either sequential designs based on an approximate Kriging prediction variance formula or designs with prefixed sample sizes.
---
paper_title: Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping

paper_content:

This paper proposes a novel method to select an experimental design for interpolation in random simulation, especially discrete event simulation.(Though the paper focuses on Kriging, this design approach may also apply to other types of metamodels such as linear regression models.)Assuming that simulation requires much computer time, it is important to select a design with a small number of observations (or simulation runs).The proposed method is therefore sequential.Its novelty is that it accounts for the specific input/output behavior (or response function) of the particular simulation at hand; i.e., the method is customized or application-driven.A tool for this customization is bootstrapping, which enables the estimation of the variances of predictions for inputs not yet simulated.The new method is tested through two classic simulation models: example 1 estimates the expected steady-state waiting time of the M/M/1 queueing model; example 2 estimates the mean costs of a terminating (s, S) inventory simulation.For these simulations the novel design indeed gives better results than Latin Hypercube Sampling (LHS) with a prefixed sample of the same size.
---
paper_title: Simulation Modeling and Analysis

paper_content:

From the Publisher: ::: This second edition of Simulation Modeling and Analysis includes a chapter on "Simulation in Manufacturing Systems" and examples. The text is designed for a one-term or two-quarter course in simulation offered in departments of industrial engineering,business,computer science and operations research.
---
paper_title: Statistical testing of optimality conditions in multiresponse simulation-based optimization

paper_content:

This article studies simulation-based optimization with multiple outputs. It assumes that the simulation model has one random objective function and must satisfy given constraints on the other random outputs. It presents a statistical procedure for testing whether a specific input combination (proposed by some optimization heuristic) satisfies the Karush-Kuhn-Tucker (KKT) first-order optimality conditions. The article focuses on "expensive" simulations, which have small sample sizes. The article applies the classic t test to check whether the specific input combination is feasible, and whether any constraints are binding; next, it applies bootstrapping (resampling) to test the estimated gradients in the KKT conditions. The new methodology is applied to three examples, which gives encouraging empirical results.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---
paper_title: Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models

paper_content:

This paper proposes a new method that extends the efficient global optimization to address stochastic black-box systems. The method is based on a kriging meta-model that provides a global prediction of the objective values and a measure of prediction uncertainty at every point. The criterion for the infill sample selection is an augmented expected improvement function with desirable properties for stochastic responses. The method is empirically compared with the revised simplex search, the simultaneous perturbation stochastic approximation, and the DIRECT methods using six test problems from the literature. An application case study on an inventory system is also documented. The results suggest that the proposed method has excellent consistency and efficiency in finding global optimal solutions, and is particularly useful for expensive systems.
---
paper_title: Sequential kriging optimization using multiple-fidelity evaluations

paper_content:

When cost per evaluation on a system of interest is high, surrogate systems can provide cheaper but lower-fidelity information. In the proposed extension of the sequential kriging optimization method, surrogate systems are exploited to reduce the total evaluation cost. The method utilizes data on all systems to build a kriging metamodel that provides a global prediction of the objective function and a measure of prediction uncertainty. The location and fidelity level of the next evaluation are selected by maximizing an augmented expected improvement function, which is connected with the evaluation costs. The proposed method was applied to test functions from the literature and a metal-forming process design problem via finite element simulations. The method manifests sensible search patterns, robust performance, and appreciable reduction in total evaluation cost as compared to the original method.
---
paper_title: Efficient Global Optimization of Expensive Black-Box Functions

paper_content:

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.
---
paper_title: Introducing BACCO, an R Bundle for Bayesian Analysis of Computer Code Output

paper_content:

This paper introduces the BACCO bundle of R routines for carrying out Bayesian analysis of computer code output. The bundle comprises packages emulator and calibrator, computerized implementations of the ideas of Oakley and O'Hagan (2002) and Kennedy and O'Hagan (2001a) respectively. The bundle is self-contained and fully documented R code, and includes a toy dataset that furnishes a working example of the functions. Package emulator carries out Bayesian emulation of computer code output; package calibrator allows the incorporation of observational data into model calibration using Bayesian techniques. The package is then applied to a dataset taken from climate science.
---
paper_title: Approximation methods in multidisciplinary analysis and optimization: a panel discussion

paper_content:

This paper summarizes the discussion at the Approximation Methods Panel that was held at the 9 th AIAA/ISSMO Symposium on Multidisciplinary Analysis & Optimization in Atlanta, GA on September 2–4, 2002. The objective of the panel was to discuss the current state-of-the-art of approximation methods and identify future research directions important to the community. The panel consisted of five representatives from industry and government: (1) Andrew J. Booker from The Boeing Company, (2) Dipankar Ghosh from Vanderplaats Research & Development, (3) Anthony A. Giunta from Sandia National Laboratories, (4) Patrick N. Koch from Engineous Software, Inc., and (5) Ren-Jye Yang from Ford Motor Company. Each panelist was asked to (i) give one or two brief examples of typical uses of approximation methods by his company, (ii) describe the current state-of-the-art of these methods used by his company, (iii) describe the current challenges in the use and adoption of approximation methods within his company, and (iv) identify future research directions in approximation methods. Several common themes arose from the discussion, including differentiating between design of experiments and design and analysis of computer experiments, visualizing experimental results and data from approximation models, capturing uncertainty with approximation methods, and handling problems with large numbers of variables. These are discussed in turn along with the future directions identified by the panelists, which emphasized educating engineers in using approximation methods.
---
paper_title: Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models

paper_content:

This paper proposes a new method that extends the efficient global optimization to address stochastic black-box systems. The method is based on a kriging meta-model that provides a global prediction of the objective values and a measure of prediction uncertainty at every point. The criterion for the infill sample selection is an augmented expected improvement function with desirable properties for stochastic responses. The method is empirically compared with the revised simplex search, the simultaneous perturbation stochastic approximation, and the DIRECT methods using six test problems from the literature. An application case study on an inventory system is also documented. The results suggest that the proposed method has excellent consistency and efficiency in finding global optimal solutions, and is particularly useful for expensive systems.
---
paper_title: Sequential kriging optimization using multiple-fidelity evaluations

paper_content:

When cost per evaluation on a system of interest is high, surrogate systems can provide cheaper but lower-fidelity information. In the proposed extension of the sequential kriging optimization method, surrogate systems are exploited to reduce the total evaluation cost. The method utilizes data on all systems to build a kriging metamodel that provides a global prediction of the objective function and a measure of prediction uncertainty. The location and fidelity level of the next evaluation are selected by maximizing an augmented expected improvement function, which is connected with the evaluation costs. The proposed method was applied to test functions from the literature and a metal-forming process design problem via finite element simulations. The method manifests sensible search patterns, robust performance, and appreciable reduction in total evaluation cost as compared to the original method.
---
paper_title: Sequential design of computer experiments to minimize integrated response functions

paper_content:

In the last ten to fifteen years many phenomena that could be studied only using physical experiments can now be studied by computer experiments. Advances in the mathematical modeling of many physical processes, in algorithms for solving mathematical systems, and in computer speeds, have combined to make it possible to augment or replace physical experiments with computer experiments. In a computer experiment, a response z( x), usually deterministic, is computed for each set of input variables, x, according to an experimental design strategy. This strategy is determined by the goal of the experiment and depends, for example, on whether response prediction at unsampled input sites or response optimization is of primary interest. ::: We are concerned with the commonly occuring situation in which there are two types of input variables: suppose x = ( xc, x e) where xc is a set of “control” (manufacturing) variables and xe is a set of “environmental” (noise) variables. Manufacturing variables can be controlled while noise variables are not controllable but have values governed by some probability distribution. ::: For single response settings, we introduce a sequential experimental design for finding the optimum of e(x c) = E[z(x c, Xe)], where the expectation is taken over the distribution of the environmental variables. For bivariate response settings, we introduce a sequential experimental design for finding the constrained optimum of e1( xc)) = E[z( xc, X e)], subject to e2 (x c) = E[z2(x c, Xe)] ≤ U. The approach is Bayesian; the prior information is that the responses are a draw from a stationary Gaussian stochastic process with correlation function belonging to a parametric family with unknown parameters. The idea of the methods is to compute the posterior expected “improvement” over the current optimum for each untested site; the design selects the next site to maximize the expected improvement. Both procedures are illustrated by examples utilizing test functions from the numerical optimization literature.
---
paper_title: Sequential Metamodeling in Engineering Design

paper_content:

Statistical metamodels (surrogate models) are usually developed to replace complex, time-consuming simulation programs or expensive physical experiments and facilitate fast and accurate analysis. Various types of metamodeling methods have been used in the literature, each having its strong and weak points. The choice of the appropriate type of metamodel is important in design. To save time and money spent on expensive experiments, it is also very important to choose a method for sequential experimental design and metamodel development in which data points are identified along a design timeline to help obtain maximum information. Based on the observations in this paper and previous studies, a framework for sequential metamodeling is proposed, in which kriging and MARS metamodels are used to help designers develop appropriate metamodels with limited resources. The method of Sequential Exploratory Experimental Design (SEED) is improved through the utilization of kriging and MARS metamodels.
---
paper_title: Estimating percentiles of uncertain computer code outputs

paper_content:

Summary. A deterministic computer model is to be used in a situation where there is uncertainty about the values of some or all of the input parameters. This uncertainty induces uncertainty in the output of the model. We consider the problem of estimating a specific percentile of the distribution of this uncertain output. We also suppose that the computer code is computationally expensive, so we can run the model only at a small number of distinct inputs. This means that we must consider our uncertainty about the computer code itself at all untested inputs. We model the output, as a function of its inputs, as a Gaussian process, and after a few initial runs of the code use a simulation approach to choose further suitable design points and to make inferences about the percentile of interest itself. An example is given involving a model that is used in sewer design.
---
paper_title: Design and Analysis of Simulation Experiments

paper_content:

Design and Analysis of Simulation Experiments (DASE)focuses on statistical methods for discrete-event simulation (such as queuing and inventory simulations). In addition, the book discusses DASE for deterministic simulation (such as engineering and physics simulations). The text presents both classic and modern statistical designs. Classic designs (e.g., fractional factorials) assume only a few factors with a few values per factor. The resulting input/output data of the simulation experiment are analyzed through low-order polynomials, which are linear regression (meta) models. Modern designs allow many more factors, possible with many values per factor. These designs include group screening (e.g., Sequential Bifurcation, SB) and space filling designs (e.g., Latin Hypercube Sampling, LHS). The data resulting from these modern designs may be analyzed through low-order polynomials for group screening and various metamodel types (e.g., Kriging) for LHS. Design and Analysis of Simulation Experimentsis an authoritative textbook and reference work for researchers, graduate students, and technical practitioners in simulation. Basic knowledge of simulation and mathematical statistics are expected; however, the book does summarize these basics, for the readers' convenience. In addition, the book provides relatively simple solutions for (a) selecting problems to simulate, (b) how to analyze the resulting data from simulation, and (c) computationally challenging simulation problems.
---

Title: Kriging Metamodeling in Simulation: A Review

Section 1: Introduction
Description 1: Provide an overview of metamodels in simulation, explaining deterministic and random simulations, and introducing the focus on Kriging metamodels.

Section 2: Kriging versus linear regression
Description 2: Compare Kriging with classic linear regression, highlighting differences and focusing on single (univariate) simulation output.

Section 3: Linear regression
Description 3: Detail the first-order polynomial regression metamodel and its application in simulation.

Section 4: Kriging: Basics
Description 4: Explain the basic assumptions and formulas of Kriging, particularly Ordinary Kriging, including correlation functions and prediction methods.

Section 5: Kriging: New results
Description 5: Summarize new results and advancements in Kriging, especially its application in random simulation and the estimation of Kriging weights.

Section 6: Designs for Kriging
Description 6: Discuss the statistical designs for Kriging metamodels, both oneshot and sequential designs, and their application in sensitivity analysis and optimization.

Section 7: Designs for sensitivity analysis
Description 7: Highlight specific designs and procedures for sensitivity analysis in simulation, emphasizing the steps involved in sequential design.

Section 8: Designs for optimization
Description 8: Focus on constrained optimization in random simulation models and the steps involved in sequential design for optimization.

Section 9: Alternative designs
Description 9: Summarize alternative sequentialized and customized design approaches for sensitivity analysis and optimization in simulation experimentation.

Section 10: Conclusions and future research
Description 10: Conclude with a summary of the article’s key points and suggest areas for future research and improvements in Kriging software and methodologies.