---
abstract: |
    Dependability assurance of systems embedding components—so called ** ()—is a key step for their use in safety-critical applications. In emerging standardization and guidance efforts, there is a growing consensus in the value of using assurance cases for that purpose. This paper develops a quantitative notion of assurance that an is dependable, as a core component of its assurance case, also extending our prior work that applied to *components*. Specifically, we characterize assurance in the form of *assurance measures*: a probabilistic quantification of confidence that an possesses system-level properties associated with functional capabilities and dependability attributes. We illustrate the utility of assurance measures by application to a real world autonomous aviation system, also describing their role both in

    guiding high-level, runtime risk mitigation decisions and

    as a core component of the associated *dynamic assurance case*.

    \
    **Keywords**: Assurance, Autonomy, Confidence, Learning-enabled systems, Machine learning, Quantification
author:
- Erfan Asaadi
- Ewen Denney
- 'Ganesh Pai$^{(\textrm{\Letter})}$'
title: 'Quantifying Assurance in Learning-enabled Systems[^1]'
---

Introduction {#s:intro}
============

The pursuit of developing systems with increasingly autonomous capabilities is amongst the main reasons for the emergence of ** (), [i.e.,]{}systems embedding based software components. There is a growing consensus in autonomy standardization efforts [@ul4600std] on the value of using ** () as the mechanism by which to convince various stakeholders that an can be relied upon. have been successfully used for safety assurance of novel aviation applications where—like —regulations and standards continue to be under development [@cdp-atio-2017]. However, pose particular assurance challenges [@mcdermid2019] and existing technologies may not be sufficient, requiring a framework where the system and its evolve in tandem [@dhp-icse2015]. Here too, there are specific additional challenges: first, structured arguments[^2] in many are effectively *static*, [i.e.,]{}they are usually developed prior to system deployment under assumptions about the environment and intended system behavior. Evolution of the system or its components ([e.g.,]{}via online learning, or by adaptation in operation) can render invalid a previously accepted . In principle, although it is possible to dynamically evolve structured arguments [@dhp-icse2015], since their role is primarily to convince human stakeholders, it makes more sense for such updates to happen between missions at well-defined points.

Second, an operational evaluation of the extent of assurance in an (or its components, where appropriate) is a valuable system-level indicator of continued fitness for purpose. That, in turn, can facilitate potential intervention and counter-measures when assurance drops below an acceptable level during a mission. Indeed, *online assurance updates* that are aimed at machine consumption must necessarily be in a computable form, [e.g.,]{}using a formal language, such as a logic, or as a quantification. So far as we are aware, prevailing notions of do not yet admit such evaluation. Prior efforts at confidence assessment [@dhp-esem-11; @guiochet2019] have focused on the argument structure rather than the system itself, and face challenges in repeatable, objective validation due to their reliance on subjective data. They have also not been applied to . Thus, there is a general need to capture a computable form of assurance to bolster an otherwise qualitative . Note that although a qualitative may well refer to quantitative evidence items, here we are identifying the necessity to have quantified assurance as a core facet of .

This paper focuses on the problem of assurance quantification, deferring its use in dynamic updates to future work. The main contribution is an approach to characterize assurance in an through of system-level dependability attributes, demonstrated by application to an aviation domain .

Methodology {#s:approach}
===========

Previously [@adp-edcc2019], we have described how assurance of *components* in an can be characterized through of component-level properties associated with the corresponding (component-level) dependability attributes. Here, we extend our methodology to the system-level, relying on the following concepts: *assurance* is the provision of (justified) confidence that an *item*—[i.e.,]{}a (learning-enabled) component, system, or service—possesses the relevant assurance properties. An *assurance property* is a logical, possibly probabilistic characteristic associated with *dependability attributes* [@dependability-taxonomy] and functional capabilities. One or more assurance properties applied to a particular item give an *assurance claim*[^3]. An *assurance measure* characterizes the extent of confidence that an assurance property holds for an item through a probabilistic quantification of uncertainty. It can be seen as implementing a model on which to query the confidence in an assurance property.[^4]

In general, we can define multiple assurance properties (and assurance measures), based on the functionality and dependability attributes for which assurance is sought. For example, the proposition “*the aircraft location does not exceed a specified lateral offset from the runway centerline during taxiing*” is a system-level assurance claim associated with the attribute of *reliability*. Similarly, the assurance property “*the aircraft does not veer off the sides of the runway during taxiing*” is associated with the attribute of *system safety*. Such assurance properties directly map to the claims made in the structured arguments of an . Thus, we can leverage the methodology for creating structured arguments [@dp-jase2018] to also specify assurance properties.

For quantification, we mainly consider assurance measures for those system-level properties that can be reasonably and feasibly quantified. For example, assurance measures for the preceding example quantify the uncertainty that the aircraft location does not exceed, respectively, the specified lateral offset from the runway centerline (reliability), and half the width of the runway pavement (safety), over the duration of taxiing. used in safety-critical applications, especially aviation, are effectively stochastic dynamical systems. The insights from this observation are that we can:

capture behavior through model-based representations of the underlying stochastic process;

view system-level assurance properties as specific realizations of particular of that process; and

express confidence in the assurance properties—[i.e.,]{}the assurance measures—by propagating uncertainty through the model to determine the distributions over the corresponding .

One challenge is selecting an appropriate model and representation of the stochastic process to be used to model . Although there is not a generic answer for this, such a model could be built, for example, by eliciting the expected system behavior from domain experts, by transforming a formal system description, using model fitting and statistical optimization techniques applied to (pre-deployment) system simulation and execution traces, or through a combination of the three. For , a formal system description may be often unavailable. As such, we rely on elicitation and statistical techniques, using Bayesian models where possible, making allowance to admit and use other well-known, related stochastic process models—such as Markov chains—and leveraging data from analytical representations of system dynamics, simulations, and execution. The Bayesian concepts of *credible* intervals and regions—determined on the posterior distribution of the for assurance properties—give a formal footing to the intuitive, subjective notion of confidence that usually accompanies claims in assurance arguments, and in general [@Hawkins2011].

Illustrative Example – Runway Centerline Tracking {#s:example}
=================================================

System Description
------------------

To show our methodology is feasible, we now apply it to quantify assurance in an aviation domain supplied by our industrial collaborators: a embedding an component, trained offline using supervised learning, to support an autonomous taxiing capability. The broader goal is to enable safe aircraft movement on a runway without human pilot input. Fig. \[f:uas-example\] shows a simplified *pipeline architecture* used to realize this capability. A deep implements a perception function that ingests video images from a wing-mounted camera pointed to the nose of the aircraft. The input layer is $(360 \times 200)$ pixels $\times~3$ channels wide; the network size and complexity is of the order of $100$ layers with greater than two million tunable parameters. Effectively, this component performs regression under supervised learning producing estimates of ** ()[^5] and ** ()[^6] as output. These estimates are input to a classical controller that generates the appropriate steering and actuation signals.

![Pipeline architecture to implement an autonomous taxiing capability in a .[]{data-label="f:uas-example"}](uas-example){width="60.00000%"}

Assurance Properties {#sss:uas-assurance-properties}
--------------------

The main objective during taxiing (autonomously, or under pilot control) is to safely follow the runway (or taxiway) centerline. Safety during taxiing entails avoiding *lateral runway overrun*, [i.e.,]{}not veering off the sides of the runway pavement. Although avoiding obstacles on the runway is also a safety concern, it is a separate assurance property that we do not consider in this paper. Thus, safety can be achieved here, in part, by meeting a performance objective of maintaining an acceptable lateral offset (ideally zero) on either side of the runway centerline during a taxi *mission* from starting taxi to stopping (or taking off).[^7] In other words, the closer the aircraft is to the runway centerline during taxiing, the less likely it is to veer off the sides of the runway.

This performance objective relates to the attribute of reliability, where *taxi failure* is considered to be the violation of the specified lateral offset. Here, we focus on the corresponding assurance property, ${\mathtt{Assured Taxi}}: \abs{{\mathrm{CTE}_a}} < {\mathtt{offset}}$, where ${\mathtt{offset}}=2\meter$ is the maximum acceptable lateral offset on either side of the runway centerline for this application and aircraft type. ${\mathrm{CTE}_a}$, which is the true (or actual) for the , is a signed, real valued scalar; the absolute value gives the magnitude of the offset, and the sign indicates where the is located relative to the centerline, [i.e.,]{}to its left or its right.

Assurance Quantification {#sss:uas-uq}
------------------------

### Model Choice

The assurance measure corresponding to ${\mathtt{Assured Taxi}}$, establishes ${\mathrm{Pr}}\left(\abs{{\mathrm{CTE}_a}} < 2\meter\right)$, which characterizes the uncertainty (or conversely, confidence) in the true (or actual) (${\mathrm{CTE}_a}$) relative to the specified offset. ${\mathrm{CTE}_a}$ evolves in time as the controller responds to *estimates* of and , themselves the responses of the deep component, to runway images captured by the wing mounted camera (see Fig. \[f:uas-example\]). ${\mathrm{CTE}_a}$ is thus uncertain and depends on other variables, of which those that can be observed are the estimated (${\mathrm{CTE}_e}$), estimated (${\mathrm{HE}_e}$), and a sequence of images. We can also model the controller behavior in terms of a time series evolution of ${\mathrm{CTE}_a}$ since, during taxiing, the true at a given time is affected by the controller actuation signals at prior times. An abstracted model of behavior is reflected in the joint distribution of the relevant observed and uncertain variables. In fact, a ** () [@ml-book] is a convenient and compact representation of this joint distribution, as we will see subsequently in this section. It takes into account the temporal evolution of the variables and their (known or assumed) conditional independence relations. Thus, to determine the assurance measure, we effectively seek to quantify the (posterior) distribution over ${\mathrm{CTE}_a}$, given a sequence of runway images, the estimates of and produced by the component, and the controller behavior, as a query over the corresponding model.

![image](uas-model-variables){width="\columnwidth"}

### Model Variables

Model variables can be discrete or continuous, and there are tradeoffs between information loss and computational cost involved in the choice. Table \[t:uas-model-vars\] lists the discrete variables we have chosen, giving the interval boundaries for their states. The choice of the intervals that constitute the states of the variables has been based, in part, on:

domain knowledge,

an assessment of the data sampled from the environments used for training and testing the , and

the need to develop an executable model that was modest in its computational needs.

Here, $w$ is the width of the runway in meters, and negative values represent measured on the left of the runway centerline. The is given in degrees, while $D$ is dimensionless. An additional variable ($I$, not shown in Table \[t:uas-model-vars\]) models the runway image captured from the camera video feed as a vector of values in the range $[0\dots1]$ representing normalized pixel values. The Boolean variable $D$ represents the detection of outliers in camera image data. Such outliers may manifest due to various causes, including camera errors and *covariate shift*, [i.e.,]{}when the data input to the has a distribution different from that of its training data. Note that the shown in Fig. \[f:uas-example\] does not indicate whether or not it includes a mechanism to detect outliers or covariate shift. However, we include this variable here, motivated by our earlier work on component-level assurance quantification of the  [@adp-edcc2019], which revealed its susceptibility to outlier images. In fact, $D$ models a runtime monitor for detecting inputs to the .

![ structure for assurance quantification, showing two adjacent slices at times $t-1$, and $t$; shaded nodes represent observed variables, clear nodes are the uncertain, latent variables.[]{data-label="f:uas-dbn"}](uas-dbn){width="0.65\columnwidth"}

### Model Structure

Each variable in Table \[t:uas-model-vars\] is indexed over time: we will denote a variable $X$ at time $t$ as $X^{(t)}$. The causal ordering of the model variables (Fig. \[f:uas-dbn\]) informs the structure of the : the estimated and at time $t$ are inputs to the controller which, in turn, impacts the future location of the aircraft at time $t+\varepsilon$. The directed links between the corresponding variables in adjacent time slices capture this dependency. For example, in Fig. \[f:uas-dbn\], these are the directed links ${\mathrm{CTE}_e}^{(t-1)} \to {\mathrm{CTE}_a}^{(t)}$, and ${\mathrm{HE}_e}^{(t-1)} \to {\mathrm{CTE}_a}^{(t)}$ (and likewise for the preceding and subsequent time slices). The directed links ${\mathrm{CTE}_a}^{(t-1)} \to {\mathrm{CTE}_a}^{(t)}$ model the correlation between actual vehicle position over time, also capturing vehicle inertia.

At time $t$, the runway image $I^{(t)}$ influences the belief about the true aircraft location, [i.e.,]{}the states of ${\mathrm{CTE}_a}^{(t)}$, with the node $D$ modeling the associated structural uncertainty. This reflects the intuition that upon detecting an outlier image (more generally an input), we are no longer confident that the image seen is an indicator of the actual aircraft location. Fig. \[f:uas-dbn\] reflects these dependencies by the directed edges ${\mathrm{CTE}_a}^{(t)} \gets I^{(t)}$, and ${\mathrm{CTE}_a}^{(t)} \gets D^{(t)}$, respectively.

Fig. \[f:uas-dbn\] shows two adjacent time slices of the structure, although the actual structure is unrolled for $T$ time steps, the duration of taxiing, to compute the assurance measure over the taxi phase. At time $t$, this is, in fact, the sum of the probability mass over the seven states of ${\mathrm{CTE}_a}^{(t)}$ that lie within the interval $[-2, 2]$ (see Table \[t:uas-model-vars\]). By unrolling the for an additional $\varepsilon$ time steps and propagating the uncertainty through the model from the time of the last observations, the model can provide an assurance forecast.

### Probability Distributions

To complete the model specification, we need to specify the over the model variables, as encoded by its structure. One way to identify the is through of the physical system model [@najm2009uncertainty]. Practically, the latter may not be available, especially for .

Another alternative—the approach we take here—is to assume a functional form for the that is then tuned based on execution and simulation data. Specifically, to construct the represented by the transition edge between the time slices, [i.e.,]{}${\mathrm{Pr}}({\mathrm{CTE}_a}^{(t)}\,|\, {\mathrm{CTE}_a}^{(t-1)}, {\mathrm{CTE}_e}^{(t-1)}, {\mathrm{HE}_e}^{(t-1)})$, we chose a multinomial distribution with a uniform prior, tuned using the estimate on simulation data. This choice was advantageous in the sense that the produces a uniform posterior distribution over ${\mathrm{CTE}_a}$ when the observed variables take on values from a distribution different from that of the data used to build the . For this example, the simulation data comprised sequences of runway images, estimated and as produced by the , and true . Section \[s:uas-experimental-results\] gives more details on the simulation platform and data gathered.

To determine the emission probability ${\mathrm{Pr}}({\mathrm{CTE}_a}^{(t)}\,|\,I^{(t)})$, first we used the model underpinning our prior work on component-level assurance quantification [@adp-edcc2019]. In brief, the idea is to use a to model the error performance of the ([i.e.,]{}its accuracy) on its input ([i.e.,]{}runway images). Then, adding the error distribution to the estimate of gives the distribution over the true . However, for high dimensional data (such as images), this is computationally expensive. Instead, in this paper we used an ensemble of decision trees [@criminisi2012decision] as a classifier that ascribes a probability distribution over the states of ${\mathrm{CTE}_a}$, given a runway image, $I$. This approach builds uncorrelated decision trees such that their combined estimate is more accurate than that of any single decision tree. To identify the decision rules, we used supervised learning over the collection of runway images and corresponding true , sampled from the same environments used to train and test the (see Section \[s:uas-experimental-results\]). For this example, we built $280$ decision trees with terminal node size of at least $10$, by randomly sampling $100$ data points using the *Gini index* as a performance metric, selecting the model parameters to balance classification accuracy and computational resources.

Experimental Results {#s:uas-experimental-results}
====================

We now present some results of our experiments in quantifying assurance in terms of the assurance measure, ${\mathrm{Pr}}\left(|{\mathrm{CTE}_a}| < 
2\meter\right)$, based upon simulations of constant speed taxiing missions.

Simulation Setup {#ss:uas-simulation-setup}
----------------

We use a commercial-off-the-shelf flight simulator instrumented to reflect the pipeline architecture of Fig. \[f:uas-example\]. The simulation environment includes various airports and runways with centerlines of varying quality, [e.g.,]{}portions of the centerline may be obscured at various locations (see Fig. \[f:uas-example\]). We can create different training and test environments by changing various simulation settings, among which two that we have selected are:

weather induced visibility (*clear* and *overcast*), and

the time of day (*07:30am* to *2:00pm*).

Two such environments are, for example, “*Clear at 07:30am*”, and “*Overcast at 12:15pm*”. More generally, we can construct environments such as “*Clear Morning*”, “*Overcast Afternoon*”, and so on. The former refers to the collection of data sampled from the environment having clear weather, and the time of day incremented in steps of $15$ and $30$ minutes from *07:30am* until *noon*. A similar interpretation applies to other such environments.

From these environments, we gathered images via automated screen capture (simulating the camera output) whilst taxiing the aircraft on the airport runway, using different software controllers, as well as different for perception: [i.e.,]{}the same architecture described in Section \[s:example\], but trained by our industrial collaborators with data drawn from the various environments identified earlier. In tandem, for each image, we collected true (from internal simulation variables), along with estimates of and . We used several such data sets, one for each of the different environments identified above, from which data samples were drawn to build the of the model. Here, note that these data samples were *not* identical to those used to train and test the , even though the samples were drawn from the collection of environments common to both the and the .

![${\mathrm{Pr}}({\mathtt{Assured Taxi}})$ for ${\mathtt{offset}}=2\meter$ and ${\mathtt{offset}}= 1.43\meter$.[]{data-label="f:uas-assurancemeasure"}](uas-uncertainty){width="0.75\columnwidth"}

![${\mathrm{Pr}}({\mathtt{Assured Taxi}})$ for ${\mathtt{offset}}=2\meter$ and ${\mathtt{offset}}= 1.43\meter$.[]{data-label="f:uas-assurancemeasure"}](uas-assurancemeasure){width="0.725\columnwidth"}

Uncertainty Quantification
--------------------------

Fig. \[f:uas-uncertainty\] shows the results of assurance quantification for one test scenario, visualized as a probability surface overlaid on a stretch of the runway, itself shown as a grid. The horizontal axis—discretized using the interval boundaries for the states of ${\mathrm{CTE}_a}$ (see Table \[t:uas-model-vars\])—gives the true aircraft location, which is uncertain during taxiing. Thus, moving from left to right (or vice versa) constitutes lateral aircraft movement. The vertical axis (discretized into $6$ steps, each of duration $0.33\second$) represents the number of time slices for which the model is unrolled. We selected this based on the time taken for the to laterally depart the runway after violating the $2\meter$ bound, given: runway dimensions, maximum allowed taxiing speed, and other constraints on the dynamics, [e.g.,]{}non-accelerating taxiing.

At $t=0$, the horizontal axis gives the aircraft location at the current time. The time steps $t=1,\ldots, 6$ are *lookahead times* for which the horizontal axis gives the *predicted* location of the aircraft relative to the centerline, given the estimates of and at $t=0$. Thus, moving from the bottom to the top of Fig. \[f:uas-uncertainty\] represents forward taxiing, [i.e.,]{}the temporal evolution of aircraft position over the runway. Each cell of the grid formed by discretizing the two axes is, therefore, a state of ${\mathrm{CTE}_a}$ at a given time, shaded such that darker shades indicate lower uncertainty (or higher confidence) and lighter shades indicate higher uncertainty (or lower confidence). Thus, the row at $t=0$ shows the estimate of uncertainty over ${\mathrm{CTE}_a}$ at the current time. Similarly, each row for $t=1, \ldots, 5$ shows the *predicted* uncertainty over ${\mathrm{CTE}_a}$ for those lookahead times, given that the last known values for the observed variables are at $t=0$. The solid white line in Fig. \[f:uas-uncertainty\] at $t=0$ is *ground truth*, [i.e.,]{}the true at the current time based on internal simulation variables. Although this may not be otherwise available during taxiing, we show it here primarily for model validation, [i.e.,]{}to show that the interval (state of ${\mathrm{CTE}_a}$) estimated by the to be the least uncertain is also the one that includes the ground truth. The solid black line is as estimated by the ([i.e.,]{}${\mathrm{CTE}_e}$) at the current time.

Recall that assured taxiing involves maintaining ${\mathrm{CTE}_a}$ between a $2\meter$ lateral offset on either side of the centerline. To quantify assurance in this property, we sum up the probability mass in each cell between the two offsets. Fig. \[f:uas-assurancemeasure\] shows the assurance measure, ${\mathrm{Pr}}(|{\mathrm{CTE}_a}^{(t=0)}| < {\mathtt{offset}})$ computed for two different offset values: $2\meter$ and $1.43\meter$.[^8] The interval $[-2, 2]$ is a Bayesian *credible interval* within which the true lies with probability $\approx 95\%$, based on Fig. \[f:uas-assurancemeasure\]. In other words, *the model is $\approx 95\%$ confident that the aircraft is truly located within $2\meter$ of the runway centerline*. In general, the expected (and desired) behavior is to be more uncertain over longer term assurance forecasts, when there are no additional observations with which to update the posterior distributions on the assurance measures.

Sufficient Assurance
--------------------

We must select a threshold on the assurance measure to establish what sufficient assurance constitutes, based on which we can assert whether or not the assurance claim holds. The criterion we have selected here is: when the is $\geq 30\%$ confident that the true location exceeds the allowed lateral offset, the assurance claim does not hold, [i.e.,]{} $\Pr(|{\mathrm{CTE}_a}^{(t)}| \geq 2\meter) \geq 0.3 \Rightarrow \neg({\mathtt{Assured Taxi}})$. We determined this threshold under conservative assumptions about vehicle behavior, leveraging the engineering judgment of our industry collaborators, to balance the tradeoff between safety (avoiding runway overrun) and mission effectiveness (not stopping too often).

Discussion {#s:discussion}
==========

We now evaluate how the performs relative to the , in the context of ground truth. The intent is to show that it is a reasonable ([i.e.,]{}valid) *reference model* of the system suitable for runtime use ([i.e.,]{}simple and abstract), based on which to make certain decisions, [e.g.,]{}whether or not to stop taxiing. Moreover, we must also show that the software implementation of the can be relied upon. In this paper, we primarily address the former, leaving the latter for future work.

Validity
--------

We compare how well the and the can discriminate between *true positive* and *true negative* situations when their respective outputs are transformed into a classification on a plurality of image data drawn from multiple simulated taxiing scenarios for different test environments unseen by both the and the .

A true positive (negative) situation for the is one where it indicates that the assurance property is satisfied (not satisfied) based on the criterion for sufficient assurance (see Section \[s:uas-experimental-results\]), and ground truth data also indicates that it is truly the case that the location is within (exceeds) the allowed lateral offset from the runway centerline. Likewise for the , a true negative (positive) situation is one where the estimate of indicates (does not indicate) an offset violation [i.e.,]{}${\mathrm{CTE}_e}\geq 2\meter$ (equivalently, ${\mathrm{CTE}_e}< 
2\meter$), and so does ground truth data.

![image](uas-model-validation){width="0.9\columnwidth"}

Table \[t:uas-model-validation\] shows our evaluation results in terms of *sensitivity* (true positive rate) and the *specificity* (true negative rate) of both the model and the , varying the embedded used for perception. The variability arises from using trained under two different training environments. We also used these training environments to build the for both variants using $\approx 37000$ image samples. These samples were not the same as those that were used to train the variants: indeed, we did not have access to the actual training data for the different . Also, the test environments listed in the table (and, therefore, the resulting test data), are unseen during the development of both variants, and the models of the same.

Based on Table \[t:uas-model-validation\], in the context of the sensitivity and specificity metrics shown, as well as the criterion for sufficient assurance, we are cautiously optimistic in claiming that the models the reasonably well. For the test environments “*Clear at 11:45am*”, and “*Overcast at 12:15pm*”, the has a lower sensitivity than the , however its specificity is substantially better. This suggests that the may be biased in its estimates of for those operating conditions.

Suitability
-----------

The model structure—in particular, the conditional independence relations encoded by the structure—is informed by (our knowledge of) the causal impacts of the identified variables and the system dynamics, and the resulting assumptions. We note that it is always possible to relax these assumptions and learn the structure as well as its parameters. However, in most cases, especially when there is limited data available, structure learning can be an unidentifiable problem, or can produce a non-unique solution. In our case, the conditional independence assumptions used have turned out to be neither too strong to affect model performance nor too conservative to impose a problem in identifying the given limited data.

Our assessment in Table \[t:uas-model-validation\] does *not* compare the and the that estimates . Indeed, the latter is a learned, static regression function for a *component*, that associates a vector of real values with a real-valued scalar, whereas here we are assessing a stochastic process model of a (learning-eabled) *system* ([i.e.,]{}the ) against the system itself. When we use the for runtime assurance, we implement it as a software component integrated into the . This can be viewed as an item to which we can apply our own assurance methodology, [i.e.,]{}as in Section \[s:approach\], and [@adp-edcc2019]. Thus, although we have not formulated assurance properties for the , sensitivity and specificity are probabilistic performance metrics (albeit in a frequentist sense) that we can view as assurance measures in their own right, that we have now applied to our model.

The validation above is admittedly not exhaustive although the following observations are worth noting: the is a relatively simple and abstract model of the time-series evolution of the *system*, whose estimates can be updated through Bayesian inference given observed data. Thus, it is amenable to applying other verification techniques including inspection, and formal verification.

Moreover, the *does not* produce point estimates of ; rather, in quantifying confidence in a system-level assurance property, a by-product is the uncertainty in true given as a probability distribution over the range of admissible values of ${\mathrm{CTE}_a}$. Thus, in unseen situations where the can produce an inaccurate estimate of (see Fig. \[f:uas-uncertainty\]), the gives a distribution over possible values of true . As sucsh, it is more conservative in potentially unsafe scenarios. Based on this assessment, we submit that the is a reasonable and suitable runtime reference model of the for the autonomous taxiing application, when used for centerline tracking.

Utility
-------

A key advantage of an abstract assurance quantification model is a small implementation footprint for runtime integration into the . As indicated in Section \[s:intro\], one of the primary motivations for quantified assurance measures is to provide feedback signals (in a computable form) to the , that can be acted on, [e.g.,]{}by a ** (), in operation. In this work, the assurance measure values were translated into commands to either *stop*, *slow down*, or *continue* based on

the chosen decision thresholds (Section \[s:uas-experimental-results\]), and

a simple model of the system-level effect ([i.e.,]{}likelihood of lateral runway overrun) given the assurance measure and current system state. [^9]

In general, deciding between a series of options in the presence of conflicting and uncertain outcomes is a special case of *decision making under uncertainty* [@Kochenderfer2015]. We plan to investigate such techniques as future work to develop a principled approach to contingency management using assurance measures.

The aim of *run-time assurance*, also known as *run-time verification*, is to provide updates as to whether a system satisfies specified properties as it executes [@R2U2]. This is done using a run-time *monitor*, which evaluates the property using values extracted from the state of the system and its environment. In a sense, therefore, the notion of assurance measure we have described here is a kind of monitor. However, it is worth making several distinctions. A monitor relates directly to properties of the system, whereas an assurance measure characterizes *confidence* in our knowledge of such properties. Second, an assurance measure seeks to aggregate a range of sources of information, including monitors. Thus it can be seen as a form of *data fusion*. Third, monitors typically provide values that relate to the current state of the system, whereas the assurance measures we have defined are predictive, intended to give a probabilistic quantification on dependability attributes.

In general, our approach to assurance quantification admits other models including runtime monitors: recall that the node $D^{(t)}$ in Fig. \[f:uas-dbn\] is a runtime monitor detecting data distribution shift in the input image at time $t$. Indeed, our framework is not intended to replace runtime verification, and the assurance measures generated show the assurance contribution of the runtime monitors, additionally providing an assurance/uncertainty forecast. We are not aware of existing runtime verification techniques that do this.

Related Work {#s:related-work}
============

The work in this paper is closely related to our earlier research on assurance case confidence quantification [@dhp-esem-11]. There, although confidence estimation in an assurance claim also uses Bayesian techniques, it relies primarily on the argument structure to build the model. Similarly, based on the structure of an argument, the use of an evidential theory basis has been explored for confidence quantification in assurance claims [@guiochet2019]. However, neither work has been applied to assurance quantification. Moreover, in this paper the focus is on those properties where quantification is possible, relying upon models of the system that can be assessed against objective, measured data.

This paper is a natural extension of our prior work on quantifying assurance in components [@adp-edcc2019]: the assurance property we consider there is ${\mathrm{CTE}_e}$ *accuracy*. Assurance quantification then entails using to determine the uncertainty in the error of ${\mathrm{CTE}_e}$, which is inversely proportional to accuracy. However, the data used are not (and need not be) time dependent and the model used applies regardless of whether or not the aircraft position has violated ${\mathtt{Assured Taxi}}$. Indeed, despite a high assurance that accurately estimates , it is nevertheless possible to violate ${\mathtt{Assured Taxi}}$. However, in this paper we model the as a stochastic process, including any runtime mitigations, [e.g.,]{}a monitor for detection images. As such, the models used for are a generalization of that in [@adp-edcc2019] to time-series behavior.

As previously indicated (Section \[s:intro\]), one of the motivations is to support dynamic . Our prior work [@dhp-icse2015] first explored this concept, which has subsequently been tailored for so-called *self-adaptive software* [@calinescu2017]. Again, neither work has considered , although self-adaptation is one of the properties that can exhibit. In [@dhp-icse2015], confidence quantification has been situated as a core principle of dynamic assurance which has also motivated this paper to an appreciable degree. However, that work relies on the quantification methodology in [@dhp-esem-11]. In [@calinescu2017], assurance quantification employs probabilistic model checking, which can be leveraged for if they can be represented using state-space models, [e.g.,]{}as in [@verisig] which uses hybrid model checking instead. Neither technique is incompatible with the stochastic processes-based modeling approach that we have adopted. As such, they may be a candidate means to check properties of the stochastic models that we build as a means of (meta-)assurance.

*Dynamic safety management* as an assurance concept has also been proposed as a run-time assurance method [@trapp2018], but it is largely speculative about applicability for . The idea of *requirements-aware* runtime models [@bencomo2019] is very closely related to our notion of building a reference model. Quantified and probabilistic guarantees in reinforcement learning have been explored in developing assured components in [@kochenderfer2019-rl]. That work is also closely related to what we have presented here, though its focus is mainly on assurance of correctness properties that have a safety impact. Additionally, the assurance approach there is *intrusive* in the sense that the component being built is modified. In our case, assurance quantification does not modify the components. *Benchmarking* of uncertainty estimation techniques [@weiss-uncertainty-benchmarking] has also been investigated, although mainly in the context of image classification. It is unclear if the reported results translate to assurance quantification as applied in this paper. However, the benchmarking principles and metrics used could be candidates for evaluating various system models built using our approach.

Kalman filters have long been used to address uncertainty during state estimation, and have some similarities to our approach. A Kalman filters is a special case of a where amongst the main assumptions are that sensor errors are distributed as zero mean Gaussians, and that the uncertainty does not vary between sensing outputs. In contrast, our model uses discrete distributions, admitting varying sensor uncertainty for each image input, in a more general graphical model that has a different structure, whilst including detections of inputs.

Conclusion and Future Work {#s:conclusions}
==========================

We have described our approach to quantifiable assurance using assurance measures, run-time computations of uncertainty (conversely, confidence) in specified assurance properties, and their application to . Assurance measures complement design-time assurance activities, each of which forms part of an overall . In collaboration with system integrators from industry, we have applied our framework to an aviation platform that employed supervised learning using a deep . Collaboration was crucial to develop the contingency management capability, which relied on engineering judgment to tradeoff safety risk reduction and achieving performance objectives. Feedback from the end-users ([i.e.,]{}our industry collaborators) was also essential in refining the final visualizations of the assurance measure that we ultimately deployed in the system (based on Fig. \[f:uas-assurancemeasure\]). Those are intended to provide insight into the system assurance state for safety observer crew.

We have shown that our methodology can feasibly quantify assurance in system-level properties of an aviation domain , though we have used classical techniques. Our work in quantifying assurance in is ongoing, and we will be developing assurance measures for other autonomous platforms in the context of more complex mission objectives that require additional components and learning schemes. The work in this paper is one strand of our overall approach to assurance through . The diverse components of an assurance case, including structured arguments, safety architecture [@dpw-ress2019], as well as the assurance measures described here, each represent one facet of an integrated . There are close connections between the probabilistic models underlying assurance measures and the safety architecture, as well as between assurance properties and claims in an assurance arguments. Our future work will place these connections on a rigorous basis. In part, this can be achieved through use of a high-level ** () that will let us

abstract from the details of the individual probabilistic models, and

conversely, allow compilation into a range of different models, whilst making more explicit the connections to domain concepts used elsewhere in the assurance case.

A related avenue of future work is providing comprehensive assurance for our approach itself, and in turn, the assurance measures produced. From a verification standpoint, we can consider *correctness* properties entailing

consistency between the quantification model and the other components, [e.g.,]{}the risk scenarios captured by a safety architecture, and

correctness of the low-level implementation against the higher level specification embodied by the quantification model.

Additionally, assurance measure validity is related, in part, to the limits of the statistical techniques used to infer the underpinning stochastic models, and the data used to build them.

Indeed, one of the challenges we faced in this work was obtaining sufficient useful data. Moreover, the quality of the data gathered also plays a key role in corroborating that the assurance quantification models sufficiently represent the system behavior across its intended operational profile. We believe that a more principled approach to specifying a variety of training data should be possible ([e.g.,]{}to include various types of perturbed and adversarial inputs), and that such specifications could be derived from the used to specify the assurance measures themselves. The dynamic nature of will also bear further investigation, to see how real-time updates provided by assurance measures during a mission can inform updates between missions, to the qualitative arguments of .

[10]{}

: . (April 2020)

Clothier, R., Denney, E., Pai, G.: . In: 17th AIAA Aviation Technology, Integration, and Operations Conference (ATIO 2017), AIAA Aviation Forum. (June 2017)

McDermid, J., Jia, Y., Habli, I.: . In Espinoza, H., et al., eds.: 2019 AAAI Workshop on Artificial Intelligence Safety (SafeAI 2019), CEUR Workshop Proceedings (January 2019)

Denney, E., Habli, I., Pai, G.: . In: IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE 2015). Volume 2., (May 2015) 587–590

Denney, E., Pai, G., Habli, I.: Towards measurement of confidence in safety cases. In: 5th International Symposium on Empirical Software Engineering and Measurement (ESEM 2011). (Sept. 2011) 380–383

Wang, R., Guiochet, J., Motet, G., Schön, W.: . Intl. Journal of Approximate Reasoning **107** (2019) 46 – 64

Asaadi, E., Denney, E., Pai, G.: . In: 15th European Dependable Computing Conference (EDCC 2019), IEEE (September 2019) 55–62

Avižienis, A., Laprie, J.C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing **1**(1) (Jan 2004) 11–33

Denney, E., Pai, G.: . Journal of Automated Software Engineering **25**(3) (September 2018) 435–499

Hawkins, R., Kelly, T., Knight, J., Graydon, P.: A new approach to creating clear safety arguments. In Dale, C., Anderson, T., eds.: Advances in Systems Safety, (2011) 3–23

Murphy, K.P.: . MIT Press (2012)

, H.N.: . Annual Review of Fluid Mechanics **41**(1) (January 2009) 35–52

Criminisi, A., Shotton, J., Konukoglu, E.: . Foundations and Trends in Computer Graphics and Vision **7**(2–3) (2012) 81–227

Kochenderfer, M.J.: Decision Making Under Uncertainty: Theory and Application. MIT Press (2015)

Moosbrugger, P., Rozier, K.Y., Schumann, J.: . (April 2017) 1–31

, R., [Weyns]{}, D., [Gerasimou]{}, S., [Iftikhar]{}, M.U., [Habli]{}, I., [Kelly]{}, T.: . IEEE Transactions on Software Engineering **44**(11) (November 2018) 1039–1069

Ivanov, R., Weimer, J., Alur, R., Pappas, G.J., Lee, I.: . In: 22nd ACM Intl. Conference on Hybrid Systems: Computation and Control. (HSCC ’19) (2019) 169–178

, M., [Schneider]{}, D., [Weiss]{}, G.: . In: 14th European Dependable Computing Conference (EDCC 2018). (September 2018) 107–111

Bencomo, N., Garcia-Paucar, L.H.: . In: 22nd IEEE/ACM International Conference on Model Driven Engineering Languages and Systems (MODELS 2019). (September 2019)

Bouton, M., Karlsson, J., Nakhaei, A., Fujimura, K., Kochenderfer, M.J., Tumova, J.: . Computing Research Repository (CoRR) arXiv:1904.07189v2 \[cs.RO\] (May 2019)

Henne, M., Schwaiger, A., Roscher, K., Weiss, G.: Benchmarking uncertainty estimation methods for deep learning with safety-related metrics. In Espinoza, H., et al., eds.: 2020 AAAI Workshop on Artificial Intelligence Safety (SafeAI 2020), [CEUR]{} Workshop Proceedings, Volume 2560 (February 2020) 83–90

Denney, E., Pai, G., Whiteside, I.: . Reliability Engineering & System Safety **191** (2019)

[^1]: This work was supported by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL) under contract FA8750-18-C-0094 of the Assured Autonomy Program. The opinions, findings, recommendations or conclusions expressed are those of the authors and should not be interpreted as representing the official views or policies of DARPA, AFRL, the Department of Defense, or the United States Government.

[^2]: The systematic reasoning that captures the rationale why specific conclusions, [e.g.,]{}of system safety, can be drawn from the evidence supplied.

[^3]: Henceforth, we do not distinguish assurance properties from assurance claims.

[^4]: When the assurance property is itself probabilistic, the corresponding assurance measure is deterministic, [i.e.,]{}either 0 or 1.

[^5]: The horizontal distance between the aircraft nose wheel and the runway centerline.

[^6]: *Heading* refers to the compass direction in which an object is pointed; here, is thus the angular distance between the aircraft heading and the runway heading.

[^7]: Our industry collaborators elicited the exact performance objectives from current and proficient professional pilots.

[^8]: The introduction of a second offset was motivated by our industry collaborators to integrate the assurance measure on the platform.

[^9]: Although the content of integrating assurance measures with a is very closely related to the work here, it is not in scope for this paper, and will be the topic of a forthcoming article.
