---
author:
- |
    Mario Amrehn, Stefan Steidl, Reinier Kortekaas, Maddalena Strumia,\
    Markus Weingarten, Markus Kowarschik, Andreas Maier
bibliography:
- 'references.bib'
title: 'A Semi-Automated Usability Evaluation Framework for Interactive Image Segmentation Systems'
---

[Amrehn : Usability Evaluation of Interactive Image Segmentation Systems]{}

at (current page.north east)

(fmogh) at (0pt, 0pt) [ **[ [Fork me on GitHub](https://github.com/mamrehn/interactive_image_segmentation_evaluation) ]{}** ]{}; (-25em,1.2em) rectangle (25em,-1.2em);

;

the best of our knowledge, there is not one publication in which user based scribbles are combined with standardized questionnaires in order to assess an interactive image segmentation system’s quality. This type of synergetic usability measure is a contribution of this work. In order to provide a guideline for an objective comparison of interactive image segmentation approaches, a prototype providing a pictorial user input, introduced in **Sec.**\[sec:semi-manual\_prototype\], is compared to a prototype with a guiding menu-driven [UI]{}, described in **Sec.**\[sec:guided\_prototype\]. Both evaluation results are analyzed with respect to a joint prototype, defined in **Sec.**\[sec:joint\_prototype\], incorporating aspects of both interface techniques. All three prototypes are built utilizing modern web technologies. An evaluation of the interactive prototypes is performed utilizing pragmatic usability aspects described in **Sec.**\[sec:results\_pragmatic\], as well as hedonic usability aspects analyzed in **Sec.**\[sec:results\_hedonic\].

Image Segmentation Systems
--------------------------

Image segmentation can be defined as the partitioning of an image into a finite number of semantically non-overlapping regions. A semantic label can be assigned to each region. In medical imaging, each individual region of a patients’ abdominal tissue might be regarded as healthy or cancerous. Segmentation systems can be grouped into three principal categories, each differing in the degree of involvement of an operating person (user): manual, automatic, and interactive. (1) During manual tumor segmentation, a user provides all elements $i$ in the image grid which have neighboring elements $N(i)$ of different labels than $i$. The system then utilizes this closed curve contour line information to infer the labels for remaining image elements via simple region growing. This minimal assistance by the system causes the overall segmentation process of one lesion to take up to several minutes of user interaction time. However, reaching an appropriate or even perfect segmentation result (despite noteworthy difference [@becker2017increased]) is feasible [@kim2016interobserver; @hong2014interobserver]. In practice, few manual segmentations are performed by domain experts, in order to utilize the results as a reference standard in radiotherapy planning [@moltz2011analysis]. (2) A fully automated approach does not involve a user’s interference with the system. The introduced deficiency in domain knowledge for accurately labeling regions may be restored partially by automated segmentation approaches. The maximum accuracy of the segmentation result is therefore highly dependent on the individual set of rules or amount of training data available. If the segmentation task is sufficiently complex, a perfect result may not be reachable. (3) Interactive approaches aim at a fast and exact segmentation by combining substantial assistance by the system with knowledge about a very good estimate of the true tumor extent provided by trained physicians during the segmentation process [@olabarriaga1997setting]. In contrast to fully automated solutions, prior knowledge is (also) provided during the segmentation process. Although, interactive approaches are also costly in terms of manual labor to some extent, they can supersede fully automated techniques in terms of accuracy. Due to their exact segmentation capabilities, interactive segmentation techniques are frequently chosen to outline pathologies during imaging assisted medical procedures, like hepatocellular carcinomata during trans-catheter arterial chemoembolization (see **Sec.**\[sec:tace\]).

Evaluation of Image Segmentation Systems
----------------------------------------

Performance evaluation is one of the most important aspects during the continuous improvement of systems and methodologies. With non-interactive computer vision and machine learning systems for image segmentation, an objective comparison of systems can be achieved by evaluating data sets for training and testing. Similarity measures between segmentation outcome and ground truth images are utilized to quantify the quality of the segmentation result.

With (), a complete ground truth data set would also consist of the adaptive user interactions which advance the segmentation process. Therefore, when comparing , the user needs to be involved in the evaluation process. User interaction data however is highly dependent on (1) the users’ domain knowledge and the unique learning effect of the human throughout a period of exposure to the problem domain, (2) the system’s underlying segmentation method and the users’ preferences toward this technique, as well as (3) the design and usability (the user experience [@hassenzahl2006user; @law2009understanding]) of the interface which is presented to the user during the interactive segmentation procedure [@caro1979inter; @hong2014interobserver]. This includes users’ differing preferences towards diverse interaction systems and tolerances for unexpected system behavior. Considering , an analytically expressed objective function for an interactive system is hard to define. Intuitively, the user wants to achieve a satisfying result in a short amount of time with ease [@kohli2012user]. A direct assessment of a system’s usability is enabled via standardized questionnaires, as described in **Sec.**\[sec:questionnaires\]. Individual usage of can be evaluated via the segmentation result’s similarity to the ground truth labeling according to the S[ø]{}rensen-Dice coefficient () [@dice1945measures] after each interaction. The interaction data utilized for these segmentations has to be representative in order to generalize the evaluation results.

Types of User Interaction
-------------------------

As described by Olabarriaga et al. [@olabarriaga2001interaction] as well as Zhao and Xie [@zhao2012interactive], user interactions can be categorized with regards to the type of interface an provides. The following categories are emphasized. (1) A pictorial mask image is the most intuitive form of user input. Humans use this technique when transferring knowledge via a visual medium [@puranik2011scribbles]. The mask overlayed on the visualization of the image to segment consists of structures called scribbles, where $w$ is the width and $h$ is the height of the image $\mathbf{I}$ in pixels. Scribbles are seed points, lines, and complex shapes, each represented as a set of individual seed points. One seed point is a tuple , where describes the position of the seed in image space. The class label of a scribble in a binary segmentation system is represented by . Scribbles need to be defined by the user in order to act as a representative subset $\mathbf{S}$ of the ground truth segmentation .

\(2) A menu-driven user input scheme as in [@rupprecht2015image; @udupa1997multiple] limits the user’s scope of action. Users trade distinct control over the segmentation outcome for more guidance provided by the system. The locations or the shapes of newly created scribbles are fixed before presentation to the user. It is challenging to achieve an exact segmentation result using a method from this category. Rupprecht et al. [@rupprecht2015image] describe significant deficits in finding small objects and outline a tendency of the system to automatically choose seed point locations near the object border, which cannot be labeled by most users’ visual inspection and would therefore not have been selected by the users themselves. Advantages of user input are the high level of abstraction of the process, enabling efficient guidance for inexperienced users in their decision which action to perform for an optimal segmentation outcome (regarding accuracy over time or number of interactions) [@olabarriaga1999human; @olabarriaga2001interaction].

Generation of Representative User Input
---------------------------------------

Nickisch et al. [@nickisch2010learning] describe crowd sourcing and user studies as two methods to generate plausible user input data. The cost efficient crowd sourcing method often lacks control and knowledge of the users’ motivation. Missing context information for crucial aspects of the data acquisition procedure creates a challenging task objectifying the evaluation results. Specialized fraud detection methods are commonly used in an attempt to pre-filter the recorded corpus and extract a usable subset of data. McGuinness and O’Connor [@mcguinness2010comparative] proposed an evaluation of via extensive user experiments. In these experiments, users are shown images with descriptions of the objects they are required to extract. Then, users mark foreground and background pixels utilizing a platform designed for this purpose. These acquisitions are more time-consuming and cost intensive than , since they require a constant involvement of users. However, the study’s creators are able to control many aspects of the data recording process, which enables detailed observations of user reactions. The data samples recorded are a representative subset of the focus group of the finalized system. A user study aims at maximizing repeatability of its results. In order to increase the objectivity of the evaluation in this work, a user study is chosen to be conducted. The study is described in **Sec.**\[sec:usability\_test\_setup\].

State-of-the-art Evaluation of Interactive Segmentation Systems
---------------------------------------------------------------

### Segmentation Challenges

In segmentation challenges like  [@van20073d] (mainly) fully automated approaches are competing for the highest score regarding a predefined image quality metric. Semi-automatic methods are allowed for submission if the manual interaction with the test data is strictly limited to pre-processing and (single seed point) initialization of an otherwise fully automated process. may be included into the contests’ final ranking, but are regarded as non-competing, since the structure of the challenges is solely designed for automated approaches. The challenge [@litjens2014evaluation] had a separate category for proposed interactive approaches, where the user (in this case, the person also describing the algorithm) may add an unlimited number of hints during segmentation, without observing the experts’ ground truth for the test set. No group of experts was provided to operate the interactive method for comparative results. The submitted interactive methods’ scores in the challenge’s ranking are therefore highly dependent on the domain knowledge of single operating users and can not be regarded as an objective measure.

### Comparisons for Novel Segmentation Approaches

In principle, with every new proposal of an interactive segmentation algorithm or interface, the authors have to demonstrate the new method’s capabilities in an objective comparison with already established techniques. The effort spent for these comparisons by the original authors varies substantially. According to [@kohli2012user], many evaluation methods only consider a fixed input. This approach is especially unsuited for evaluation, without simultaneously defining an appropriate interface, which actually validates that a real person utilizing this [UI]{} is capable of generating similar input patterns to the ones provided. Although, there are some overview publications, which compare several approaches [@zhao2013overview; @olabarriaga2001interaction; @mcguinness2010comparative; @mcguinness2011toward; @amrehn2016comparative], the number of publications outlining new methods is disproportionately greater, leaving comparisons insufficiently covered. In **Tab.**\[tab:interactiveSegmentationEvaluationComparison\], a clustering of popular publications describing novel interactive segmentation techniques is depicted. The evaluation methods can be compared by the type of data utilized as user input. Note that there is a trend towards more elaborate evaluations in more recent publications.

Clinical Application for Interactive Segmentation {#sec:tace}
-------------------------------------------------

Hepatocellular carcinoma () is among the most prevalent malignant tumors worldwide [@chung2006transcatheter; @mcglynn2011global]. Only of cases are curable via surgery. Both, a patient’s and hepatic cirrhosis in advanced stages may lead on to the necessity of alternative treatment methods. For these inoperable cases, trans-catheter arterial chemoembolization () [@lewandowski2011transcatheter] is a promising and widely used minimally invasive intervention technique [@bruix2005management; @bruix2011management]. During , collateral vessels are occluded, which previously supplied the [HCC]{} with oxygenated blood. To locate these vessels, it is crucial to find the exact shape as well as the position of the tumor inside the liver. Interventional radiology is utilized to generate a volumetric cone-beam C-arm computed tomography () [@strobel20093d] image of the patient’s abdomen, which is processed to precisely outline and label the lesion. The toxicity of decreases, the less healthy tissue is labeled as pathologic. The efficacy of the therapy increases, the less cancerous tissue is falsely labeled as healthy [@lo2002randomized]. However, precisely outlining the tumor is challenging, especially due to its variations in size and shape, as well as a high diversity in X-ray attenuation coefficient values representing the lesion as illustrated in **Fig.**\[fig:hepatic\_tumor\_segmentation\_outcome\]. While fully automated systems may yield insufficiently accurate segmentation results, tend to be well suited for an application during .

Methods {#sec:methods}
=======

Segmentation Method {#sec:segmentation_method}
-------------------

 [@vezhnevets2005growcut] is a seeded image segmentation algorithm based on cellular automaton theory. The automaton is a tuple , where $\mathbf{G}_\mathbf{I}$ is the $\mathbf{I}$, where the pixels/voxels act as nodes $\mathbf{v}_e$. The nodes are connected by edges on a grid defined by the Moore neighborhood system. $$\mbox{$\mathbf{Q}\ni\mathbf{Q}_e^t=\left((\mathbf{p}_e, \,\mathbf{\ell}_e^t), \,\mathbf{\Theta}_e^t, \,\mathbf{c}_e, \,\mathbf{h}_e^t\right)$}
\label{eq:growcutgraph}$$ of node $e$ at iteration $t$. the node’s characteristics. Here, we additionally define $\mathbf{h}_e^t \in \mathbb{N}^{0}$ as a counter for accumulated label changes of $e$ during the iteration, as described in [@amrehn2018ideal], with . is initialized with $1$ for scribbles, i.e. , and $0$ otherwise.

Iterations are performed utilizing local state transition rule $\delta$: starting from initial seeds, labels are propagated based on local intensity features $\mathbf{c}$. At each discrete time step $t$, each node $f$ attempts to conquer its direct neighbors. A node $e$ is conquered if $$\begin{aligned}
    \mathbf{\Theta}_f^t\cdot\operatorname{g}(\mathbf{c}_e,\mathbf{c}_f)&>\mathbf{\Theta}_e^t\,,\ \text{where}\label{eq:growcutisconquered}\\
    \operatorname{g}(\mathbf{c}_e,\mathbf{c}_f) &= 1 - \frac{\Vert\mathbf{c}_e-\mathbf{c}_f\Vert_2}{\max_{j,k}\Vert\mathbf{c}_j-\mathbf{c}_k\Vert_2}\end{aligned}$$ If node $e$ is conquered, the automaton’s state set is updated $$\mbox{$\mathbf{Q}_e^{t+1}=((\mathbf{p}_e,\mathbf{\ell}_f^t),\mathbf{\Theta}_f^t\cdot\operatorname{g}(c_e,c_f),\mathbf{c}_e,\mathbf{h}_e^t+1)$},
\label{eq:growcutupdatestate}$$ The process is guaranteed to converge with positive and bounded node strengths monotonously decreasing

Interactive Segmentation Prototypes {#sec:sgmentation_prototypes}
-----------------------------------

Three interactive segmentation prototypes with different were implemented for usability testing. The segmentation technique applied in all prototypes is based on the approach as described in **Sec.**\[sec:segmentation\_method\]. allows for efficient and parallelizable computation of image segmentations while providing an acceptable accuracy from only few initial seed points. It is therefore well suited for an integration into a highly interactive system.

All three user interfaces provided include an *undo* button to reverse the effects of the user’s latest action. A *finish* button is used to define the stopping criterion for the interactive image partitioning. The transparency of both, the contour line and seed mask displayed, is adjustable to one of five fixed values via the *opacity* toggle button. The image contrast and brightness (windowing) can be adapted with standard control sliders for the window width and the window center operating on the image intensity value range [@jin2001contrast]. All protoypes incorporate a *help* button used to provide additional guidance for the prototype’s usage during the segmentation task. The segmentation process starts with a set of pre-defined background-labels $\mathbf{S}^0$ along the edges of the image, since an object is assumed to be located in its entirety inside the displayed region of the image.

###  Segmentation Prototype {#sec:semi-manual_prototype}

The of the prototype, depicted in **Fig.**\[fig:semi-manual\_prototype\], provides several interaction elements. A user can add seed points as an overlay mask displayed on top of the image. These seed points have a pre-defined label of either for the object or used for all other image elements. The label of the next brush strokes (scribbles) can be altered via the buttons named and . After each interaction , a new iteration of the seeded segmentation is started given the image $\mathbf{I}$ as well as the updated set of seeds as input.

![ segmentation prototype user interface. The current segmentation’s contour line (light blue) is [adjusted towards the user’s estimate of the ground truth segmentation]{} by manually adding foreground (blue) or background (red) seed points.[]{data-label="fig:semi-manual_prototype"}](images/semi-manual_segmentation_prototype_contrast.png){width="\columnwidth" height="0.61296534017\columnwidth"}

### Guided Segmentation Prototype {#sec:guided_prototype}

The system selects two seed point locations , each with the lowest label certainty values assigned by the previous segmentation process. The seed point locations are shown to the user in each iteration $n$, as depicted in **Fig.**\[fig:guided\_prototype\]. There are four possible labeling schemes for those points in the underlying classification problem, since each seed point has a label . The interface providing advanced user guidance displays the four alternative segmentation contour lines, which are a result of the four possible next steps during the iterative interactive segmentation with respect to the labeling of the new seed points $\mathbf{s}^n_1$ and $\mathbf{s}^n_2$. The user selects the only correct labeling, where all displayed object and background seeds are inside the object of interest and the image background, respectively. The alternative views on the right act as four buttons to define a selection. To further assist the user in their decision making, the region of interest, defined by $\mathbf{p}^n_1$ and $\mathbf{p}^n_2$, is zoomed in for the option view on the right and displayed as a cyan rectangle in the overview image on the left of the . The differences regarding the previous iteration’s contour line and one of the four new options each are highlighted by dotted areas in the four overlay mask images. After the user selects one of the labelings, the two new seed points are added to the current set of scribbles $\mathbf{S}^n$. The scribbles are utilized as input for the next iteration, on which basis two new locations are computed.

The system-defined locations of the additional seed points can be determined by , the location(s) with maximum number of label changes during segmentation. Frequent changes define specific image elements and areas in which the algorithm indicates uncertainty in finding the correct labels. Two locations in $\mathbf{h}^{t=\infty,{n-1}}$ are then selected as $\mathbf{p}^n_1$ and $\mathbf{p}^n_2$, which stated the most changes in labeling during the previous segmentation with input image $\mathbf{I}$ and seeds $\mathbf{S}^{n - 1}$.

![Guided segmentation prototype user interface. The current segmentation displayed on the upper left can be improved by choosing one of the four segmentation alternatives displayed on the right. The user is expected to choose the upper-right option in this configuration, [due to the two new seeds’ matching background and foreground labels]{}.[]{data-label="fig:guided_prototype"}](images/guided_segmentation_prototype_contrast.png){width="\columnwidth" height="0.61296534017\columnwidth"}

### Joint Segmentation Prototype {#sec:joint_prototype}

The joint prototype depicted in **Fig.**\[fig:joint\_prototype\] is a combination of a pictorial interaction scheme and a menu-driven approach. (1) A set of pre-selected new seeds is displayed in each iteration. The seeds’ initial labels are set automatically, based on whether their position is inside (foreground) or outside (background) the current segmentation mask. The user may toggle the label of each of the new seeds, which also provides an intuitive functionality. The automated suggestion process for new seed point locations is depicted in **Fig.**\[fig:joint\_prototype\_prob\_map\]. The seed points are suggested deterministically based on the indices of the maximum values in an element-wise sum of three approximated influence maps. These maps are the gradient magnitude image of $\mathbf{I}$, the previous label changes per element in $\mathbf{G}_\mathbf{I}$ weighted by an empirically determined factor of $17/12$, and an influence map based on the distance of each element in $\mathbf{I}$ to the current contour line. Note that for the guided prototype (see **Sec.**\[sec:guided\_prototype\]), only $\mathbf{h}$ was used for the selection of suggested seed point locations. This scheme was extended for the joint prototype, since extracting instead of only the top two points solely from $\mathbf{h}$ potentially introduces suggested point locations forming impractical local clusters instead of spreading out with higher variance in the image domain. This process approximates the true influence or entropy (information gain) of each possible location for a new seed.

When all seed points presented to the user are toggled to their correct label, the user may click on the *new points* button to initiate the next iteration with an updated set of seed points . Another set of seed points is generated and displayed.

\(2) In addition to pre-selected seeds, a single new seed point $\mathbf{s}^n_0$ can be added manually via a user’s long-press on any location in the image. A desired change in the current labeling of this region is interpreted given this user action. Therefore, the new seed point’s initial label is set by inverting the current label of the given location. A new segmentation is initiated by this interaction based on . Note that the labels of are still subject to change via toggle interactions until the button is pressed.

![Joint segmentation prototype user interface. The user toggles the labels of pre-positioned seed points[, which positions are displayed to them as colored circles,]{} to properly indicate their inclusion into the set of object or background representatives. New seeds can be added at the position of [current]{} interaction via a long-press on the overlay image. The segmentation result as well as [the]{} displayed contour line adapt accordingly after each interaction.[]{data-label="fig:joint_prototype"}](images/joint_segmentation_prototype_contrast.png){width="\columnwidth" height="0.61296534017\columnwidth"}

![The approximated influence map for new seed point locations [for the joint segmentation prototype]{}. The map is generated by a weighted sum of gradient magnitude image, number of cell changes [$h_e^{t=\infty}$ per cell $e$]{} obtained from [the]{} previous segmentation, [as well as the]{} distance to the contour line of the current segmentation. []{data-label="fig:joint_prototype_prob_map"}](images/prob_map_1.png){height="0.61296534017\columnwidth"}

Questionnaires {#sec:questionnaires}
--------------

### System Usability Scale () {#sec:questionnaires_sus}

The  [@brooke1996sus; @lewis2009factor] is a widely used, reliable, and low-cost survey to assess the overall usability of a prototype, product, or service [@kortum2013usability]. Its focus is on pragmatic quality evaluation [@ISO92411998; @ISO92412018]. The survey is technology agnostic, which enables a utilization of the usability of many types of user interfaces and  [@bangor2009determining]. The questionnaire consists of ten statements and an unipolar five-point Likert scale [@likert1932technique]. This allows for an assessment in a time span of about three minutes per participant. The statements are as follows:

1.  I think that I would like to use this system frequently.

2.  I found the system unnecessarily complex.

3.  I thought the system was easy to use.

4.  I think that I would need the support of a technical person to be able to use this system.

5.  I found the various functions in this system were well integrated.

6.  I thought there was too much inconsistency in this system.

7.  I would imagine that most people would learn to use this system very quickly.

8.  I found the system very cumbersome to use.

9.  I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

The Likert scale provides a fixed choice response format to these expressions. The choice in an Likert scale always is the neutral element. Using the scale, subjects are asked to define their degree of consent to each given statement. The fixed choices for the five-point scale are named *strongly disagree*, *disagree*, *undecided*, *agree*, and *strongly agree*. During the evaluation of the survey, these names are assigned values [$\mathbf{x}_i$ from zero to four]{} in the order presented, for statements with index . scores enable simple interpretation schemes, understandable also in multi-disciplinary project teams. The result of the survey is a single scalar value, in the range of zero to $100$ as a composite measure of the overall usability. The score is computed according to , given $S$ participants, where is the response to $i$ by subject $s$. $$\operatorname{sus}(\mathbf{x}) = \frac{2.5}{S} \sum_{s}\left[\, \sum_{\text{odd } i} \mathbf{x}^\text{SUS}_{s,i} + \sum_{\text{even } i} (4 - \mathbf{x}^\text{SUS}_{s,i})\, \right]
    \label{eq:sus_score}$$ Although the score allows for straightforward comparison of the usability throughout different systems, there is no simple intuition associated with the resulting scalar value. In practice, a of less than $80$ is often interpreted as an indicator of a substantial usability problem with the system. Bangor et al. [@bangor2008empirical; @bangor2009determining] proposed an interpretation of the score in a seven-point scale. : *worst imaginable*, *awful*, *poor*, *OK*, *good*, *excellent*, and *best imaginable*. This mapping also enables an absolute interpretation of a single score.

![Mapping from a score to an adjective rating scheme proposed by Bangor et al. [@bangor2009determining][. Given a rating, the relative height of the Gaussian distributions approximate the probabilities for each adjective. Distributions’ $\mu$ and $\sigma$ were extracted evaluating]{} $959$ surveys [with added adjective rating as an 11th question]{}. []{data-label="fig:sus_adjective"}](images/sus/sus_adjective_rating "fig:"){width="\linewidth"}\

System usability scale () rating

### Semantic Differential  {#sec:questionnaires_attrakdiff}

A semantic differential is a technique for the measurement of meaning as defined by Osgood et al. [@osgood1952nature; @osgood1957measurement]. Semantic differentials are based on the theory, that the implicit anticipatory response of a person to a stimulus object is regarded as the object’s meaning. Since these implicit responses themselves cannot be recorded directly, more apparent responses like verbal expressions have to be considered [@mehrabian1974approach; @fishbein1975belief]. These verbal responses have to be sensitive to and maximally dependent on meaningful states while independent from each other [@osgood1957measurement]. Hassenzahl et al. [@hassenzahl2003attrakdiff; @hassenzahl2000hedonic] defined a set of $28$ pairs of verbal expressions suitable to represent a subject’s opinion on the hedonic as well as pragmatic quality (both aspects of perception) and attractiveness (an aspect of assessment) of a given interactive system separately [@hassenzahl2001effect]. During evaluation, the pairs of complementary adjectives are clustered into four groups, each associated with a different aspect of quality. Pragmatic quality () is defined as the perceived usability of the interactive system, which is the ability to assist users to reach their goals by providing utile and usable functions [@hassenzahl2008user]. The attractiveness () quantizes the overall appeal of the system . The hedonic quality () [@diefenbach2008give] is separable into hedonic identity () and hedonic stimulus (). focuses on a user’s identification with the system and describes the ability of a product to communicate with other persons benefiting the user’s self-esteem [@hassenzahl2007hedonic]. describes the perceived novelty of the system. is associated with the desire to advance ones knowledge and proficiencies. The clustering into these four groups for the $28$ word pairs are defined as depicted in **Tab.**\[tab:attrakdiff\_statements\].

For each participant, the order of word pairs and order of the two elements of each pair are randomized prior to the survey’s execution. A bipolar [@mccroskey1989bipolar] seven-point Likert scale is presented to the subjects to express their relative tendencies toward one of the two opposing statements () of each expression pair, where index three denotes the neutral element. For the questionnaire’s evaluation for subject , each of the seven adjective pairs per group is assigned a score by each participant, reflecting their tendency towards the positive of the two adjectives. The overall ratings per group are the mean scores computed over all subjects $s$ and statements $i$, . Here, $S$ is the number of participants in the survey. $$\operatorname{attrakdiff}(\mathbf{x}, \,g) = \frac{1}{7 \cdot S} \sum_{s} \sum_{i} \mathbf{x}^g_{s,i}
    \label{eq:attrakdiff_score}$$ Therefore, a neutral participant would produce an score of four. The final averaged score of each group $g$ ranges from one (worst) to seven (best rating).

An overall evaluation of the results can be conducted in the form of a portfolio representation [@hassenzahl2008user]. is the mean of a system’s and scores. [PQ]{} and [HQ]{} scores of a specific system and user are visualized as a point in a two-dimensional graph. The $95$% confidence interval is an estimate of plausible values for rating scores from additional study participants, and determines the extension of the rectangle around the described data point in each dimension. A small rectangle area represents a more homogeneous rating among the participants than a larger area. If a rectangle completely lies inside one of the seven fields with associated adjectives defined in [@hassenzahl2008user], this adjective is regarded as the dominant descriptor of the system. Otherwise, systems can be particularized by overlapping fields’ adjectives. If the confidence rectangles of two systems overlap in their one-dimensional projection on either or , their difference in scores in regards to this dimension is not significant.

Qualitative Measures {#sec:qualitative_measures}
--------------------

In order to collect, normalize, and analyze visual and verbal feedback given by the participants, a summative qualitative content analysis is conducted via abstraction [@hsieh2005three; @elo2008qualitative]. The abstraction method reduces the overall transcript material while preserving its substantial contents by summarization. The corpus retains a valid mapping of the recording. An essential part of abstraction is the formulation of macro operators like elimination, generalization, construction, integration, selection and bundling. The abstraction of statements is increased iteratively by the use of macro operators, which map statements of the current level of abstraction to the next, while clustering items based on their similarity [@mayring2014qualitative].

HCI Evaluation {#sec:hci_evaluation}
--------------

A user study is the most precise method for the evaluation of the quality of different interactive segmentation approaches [@nickisch2010learning]. Analytical measures as well as subjective measures can be derived from standardized user tests [@gao2013mental]. From interaction data recorded during the study, the reproducibility of segmentation results as well as the achievable accuracy with a given system per time can be estimated. The complexity and novelty of the system can be expressed via the observed convergence to the ground truth over time spent by the participants segmenting multiple images each. The user’s satisfaction with the interactive approaches is expressed by the analysis of questionnaires, which the study participant fills out immediately after their tests are conducted and before any discussion or debriefing has started. The respondent is asked to fill in the questionnaire as spontaneously as possible. Intuitive answers are desired as user feedback instead of well-thought-out responses for each item in the questionnaire [@brooke1996sus].

For the randomized A/B study, individuals are selected to approximate a representative sample of the intended users of the final system [@siroker2013b]. During the study, subjects are given multiple interactive segmentation tasks to fulfill each in a limit time frame. The user segments all $m$ images provided with two different methods (A and B). All subjects are given $2 \cdot m$ tasks in a randomized order to prevent a learning effect bias, which would allow for higher quality outcomes for the later tasks. Video and audio data of the subjects are recorded. Every user interaction recognized by the system and its time of occurrence are logged.

Experiments {#sec:experiments}
===========

Data Set for the Segmentation Tasks {#sec:study_data_sets}
-----------------------------------

In **Fig.**\[fig:study\_data\_sets\] the data set used for the usability test is depicted. For this evaluation, the colored images are converted to grayscale in order to increase similarity to the segmentation process of medical images acquired from . The conversion is performed in accordance with the [](https://www.itu.int/rec/R-REC-BT.709/en) recommendation [@recommendation1990basic] for the extraction of true luminance defined by the International Commission on Illumination () from contemporary cathode ray tube () phosphors via are the linear red, green, and blue color channels respectively. $$\mathbf{I} = 0.2126 \cdot \mathbf{I}'_R + 0.7152 \cdot \mathbf{I}'_G + 0.0722 \cdot \mathbf{I}'_B     \label{eq:rgbtograyscale}$$ (b) is initially presented to the in order to familiarize themselves with the upcoming segmentation process. The segmentation tasks associated with images (a,c,d) are then displayed sequentially to the subjects in randomized order. The images are chosen to fulfill two goals of the study. (1) Ambiguity of the ground truth has to be minimized in order to suppress noise in the quantitative data. Each test person should have the same understanding and consent about the correct outline of the object to segment. Therefore, clinical images can only be utilized with groups of specialized domain experts. (2) The degree of complexity should vary between the images displayed to the users. Image (b), depicted in **Fig.**\[fig:study\_data\_sets\], of moderate complexity with regards to its disagreement coefficient [@hanneke2007bound], is displayed first to learn the process of segmentation with the given prototype. The complexity increases from (a) to (d), . The varying complexity enables a more objective and extended differentiation of subjects’ performances with given prototypes.

Usability Test Setup {#sec:usability_test_setup}
--------------------

Two separate user studies are conducted to test all prototypes described in **Sec.**\[sec:sgmentation\_prototypes\], in order to keep the time for each test short (less than ), thus retaining the focus of the participants, while minimizing the occurrence of learning effect artifacts in the acquired data. (1) The first user test is a randomized A/B test of the prototype (**Sec.**\[sec:semi-manual\_prototype\]) and the guided prototype (**Sec.**\[sec:guided\_prototype\]). Ten individuals are selected as test subjects due to their advanced domain knowledge in the fields of medical image processing and mobile input devices. The subjects are given the task to segment different images with varying complexity, which are described in **Sec.**\[sec:study\_data\_sets\], in random order. A fourth input image of medium complexity is provided for the users to familiarize themselves with the before the tests. As an interaction device, a mobile tablet computer is utilized, since the final segmentation method is intended for usage via such a medium. The small $10.1$ inch display and fingers utilized as a multi-touch pointing device further exacerbate the challenge to fabricate an exact segmentation for the participants [@norman2010gestural]. The user study environment is depicted in **Fig.**\[fig:study\_setup\]. Audio and video recordings are evaluated via a qualitative content analysis, described in **Sec.**\[sec:qualitative\_measures\], in order to detect possible improvements for the tested prototypes and their interfaces. After segmentation, each participant fills out the (**Sec.**\[sec:questionnaires\_sus\]) and (**Sec.**\[sec:questionnaires\_attrakdiff\]) questionnaires.

\(2) The second user test is conducted for the joint segmentation prototype (**Sec.**\[sec:joint\_prototype\]). The data set and test setup are the same as in the first user study and all test persons of study (1) also participated in study (2). One additional subject participated only in study (2). Two months passed between the conduction of the two studies, in which the former participants were not exposed to any of the prototypes. Therefore, the learning effect bias for the second test is neglectable.

![User testing setup for the usability evaluation of the prototypes. In this environment, a user performs an interactive segmentation on a mobile tablet computer while sitting. cameras record the hand motions on the input device and facial expressions of the participant. [In addition, each recognized input is recorded on the tablet device (the interaction log).]{} []{data-label="fig:study_setup"}](images/usability_test_setup_new_.pdf){width="\columnwidth"}

Prediction of Questionnaire Results {#sec:prediction_of_questionnaire_results}
-----------------------------------

The questionnaires’ , , , , , and results are predicted, based on features extracted from the interaction log data. For the prediction, a regression analysis is performed. Stochastic Gradient Boosting Regression Forests () are an additive model for regression analysis [@friedman2001greedy; @friedman2002stochastic; @hastie2009boosting]. In several stages, shallow regression trees are generated. Such a tree is a weak base learner each resulting in a prediction error , with high bias $b$ and low variance $v$. These regression trees are utilized to minimize an arbitrarily differentiable loss function each on the negative gradient of the previous stage’s outcome, thus reducing the overall bias via boosting [@breiman1999using]. The Huber loss function [@huber1964robust] is utilized for this evaluation due to its increased robustness to outliers in the data with respect to the squared error loss. The collected data set of user logs is split randomly in a ratio of for training and testing. An exhaustive grid search over $20,480$ parameter combinations is performed for each of the six estimators (one for each questionnaire result) with scorings based on an eight-fold cross-validation on the training set.

### Feature Definition {#sec:feature_definition}

The collected data contains $31$ samples with $216$ possible features each. The $31$ questionnaire results (, , , , , ), are predicted based on features extracted from the interaction log data of the four images segmented with the system. Four features are the relative median seed positions per user and their standard deviation in two dimensions. $22$ additional features, like the number of undo operations () and number of interactions (), the overall computation time (), overall interaction time (), elapsed real time (), , and are reduced to one scalar value each by the mean and median, over the four segmentations per prototype and user, to obtain $48$ base features. Since these features each only correlate weakly with the questionnaire results, composite features are added in order to assist the model’s learning process for feature relations. Added features are composed of one base feature value divided by (the mean or median of) computation time, interaction time, or elapsed real time. The relations between those time values themselves are also added. In total, $216$ features directly related to the interaction log data are used. In addition, a principal component analysis () is performed in order to add $10$% ($22$) features with maximized variance to the directly assessed ones to further assist the feature selection step via .

### Feature Selection for Prediction {#sec:sus_prediction}

For the approximation of results, a feature selection step is added to decrease the prediction error by an additional three percent points: here, after the described initial grid search, $1$% (205) of the estimators, with the lowest mean deviance from the ground truth, are selected to approximate the most important features. From those estimators, the most important features for the are extracted via a *$1/\text{loss}$*-weighted feature importance voting. This feature importance voting by $205$ estimators ensures a more robust selection than deciding the feature ranking from only a single trained . After the voting, a second grid search over the same $20,480$ parameter combinations, but with a reduction from $238$ to only $25$ of the most important features is performed.

Results {#sec:results}
=======

Overall Usability {#sec:results_overall_usability}
-----------------

The result of the score is depicted in **Fig.**\[fig:result\_sus\]. According to the mapping (**Fig.**\[fig:sus\_adjective\]) introduced in **Sec.**\[sec:questionnaires\_sus\], the adjective rating of the and joint prototypes are *excellent* ($88$ respective $82$), the adjective associated with the guided prototype is *good* ($67$).

A graph representation of the similarity of individual usability aspects, based on the acquired questionnaire data, is depicted in **Fig.**\[fig:result\_questionnaire\_results\_correlation\]. Based on the Pearson correlation coefficients utilized as a metric for similarity, the score has the most similarity to the pragmatic () and attractiveness () usability aspects provided by the questionnaire.

Pragmatic Quality {#sec:results_pragmatic}
-----------------

The results of the questionnaire are illustrated in **Fig.**\[fig:result\_attrakdiff\]. The scores for , guided, and joint prototypes are $88$%, $50$%, and $74$% of the maximum score, respectively. Since each of the $95$% confidence intervals are non-overlapping, the prototypes’ ranking regarding are significant.

The quantitative evaluation of recorded interaction data is depicted in **Fig.**\[fig:result\_logs\]. Dice scores before the first interaction are zero, except for the guided prototype ($0.82\pm0.02$), where few fixed seed points had to be provided to initialize the system. Utilizing the prototype and starting from zero, a similar Dice measure to the guided prototype’s initialization is reached after about seven interactions, which takes $13.06\pm2.05$ seconds on average. The median values of final Dice scores per prototype are $0.95$ (), $0.94$ (guided), and $0.82$ (joint). The mean overall elapsed wall time in seconds spent for interactive segmentations per prototype are $73\pm11$ (), $279\pm36$ (), and $214\pm24$ (). Since segmenting with the guided version takes the longest time and does not yield the highest final Dice scores, the initial advantage from pre-existing seed points does not bias the top ranking of a prototype in this evaluation.

Hedonic Quality {#sec:results_hedonic}
---------------

### Identity and Stimulus

The questionnaire provides a measure for the of identity and stimulus introduced in **Sec.**\[sec:questionnaires\_attrakdiff\]. The scores for , guided, and joint prototypes are $72$%, $70$%, and $77$% of the maximum score, respectively. Since the $95$% confidence intervals are overlapping for all three prototypes, no system ranks significantly higher than the others. An overall evaluation of the results is conducted in the form of a portfolio representation depicted in **Fig.**\[fig:result\_attrakdiff\_portfolio\].

at (axis cs:2,2) [super-fluous]{}; at (axis cs:2,6) [too self-oriented]{}; at (axis cs:4,4) [neutral]{}; at (axis cs:4,6) [self-oriented]{}; at (axis cs:6,2) [too task-oriented]{}; at (axis cs:6,4) [task-oriented]{}; at (axis cs:6,6) [desired]{}; (axis cs:5.64106645752803,4.798427683218617) rectangle (axis cs:6.158933542471971,5.301572316781383); coordinates [(5.9,5.05)]{}; (axis cs:3.143832023165031,4.6750586928307385) rectangle (axis cs:3.856167976834969,5.167798450026405); coordinates [(3.5,4.921428571428572)]{}; (axis cs:4.696608735705655,4.953083574359387) rectangle (axis cs:5.446248407151487,5.389773568497756); coordinates [(5.071428571428571,5.171428571428572)]{}; coordinates [(6.4285714285714288, 3.8571428571428572)(6.1428571428571432, 6.0)(5.5714285714285712, 4.6428571428571432)(6.4285714285714288, 6.2142857142857144)(5.7142857142857144, 5.2857142857142856)(6.8571428571428568, 5.2857142857142856)(5.1428571428571432, 4.4285714285714288)(6.4285714285714288, 5.1428571428571432)(5.0, 4.6428571428571432)(5.2857142857142856, 5.0)]{}; coordinates [(2.0, 4.2857142857142856)(3.2857142857142856, 5.9285714285714288)(3.8571428571428572, 5.1428571428571432)(4.2857142857142856, 5.5)(2.7142857142857144, 4.7142857142857144)(5.5714285714285712, 5.1428571428571432)(4.7142857142857144, 4.0)(3.2857142857142856, 5.2142857142857144)(3.4285714285714284, 4.9285714285714288)(1.8571428571428572, 4.3571428571428568)]{}; coordinates [(5.1428571428571432, 4.7142857142857144)(4.8571428571428568, 5.0714285714285712)(5.8571428571428568, 5.5714285714285712)(4.4285714285714288, 4.3571428571428568)(6.0, 6.0)(5.2857142857142856, 4.7857142857142856)(5.7142857142857144, 6.3571428571428568)(2.1428571428571428, 4.8571428571428568)(6.4285714285714288, 5.0)(4.8571428571428568, 5.0)]{};

=5.25pt

  ---------------- ------- ------ ------- ------ ------- -------
  Relative Error       ATT     HQ    HQ-I   HQ-S      PQ     SUS
  \[1pt\] Mean       11.5%   7.4%   10.5%   8.0%   15.7%   10.4%
  Median              8.9%   6.3%    9.4%   6.2%   13.7%    8.8%
  Std                 8.0%   5.5%    6.7%   6.9%   12.0%    7.1%
  ---------------- ------- ------ ------- ------ ------- -------

  : Relative absolute prediction errors for and test set samples. Predictions are computed by six separately trained [Stochastic Gradient Boosting Regression Forests ()]{}, one for each figure of merit. Note that each training process only utilizes the interaction log data. Results displayed are the median values of $10^4$ randomly initialized training processes.[]{data-label="tab:prediction_results_gbrf"}

### Qualitative Content Analysis

A summative qualitative content analysis as described in **Sec.**\[sec:qualitative\_measures\] is conducted on the audio and video data recorded during the study. After generalization and reduction of given statements, the following user feedback is extracted with respect to three problem statements: positive usability aspects, negative usability aspects, and user suggestions concerning existing functions or new functions.

**Feedback for multiple prototypes**

1.  Responsiveness: the most common statement concerning the and joint version is that the user expected the zoom function to be more responsive and thus more time efficient.

2.  Visibility: $20$% of the participants had difficulties distinguishing between the segmentation contour line and either the background image or the foreground scribbles in the overlay mask, due to the proximity of their assigned color values.

3.  Feature suggestion: deletion of individual seed points instead of all seeds from last interaction using *undo*.

**segmentation prototype**

1.  Mental model: $30$% of test persons suggested clearly visible indication whether the label for the scribble drawn next will be foreground or background.

2.  Visibility: hide previously drawn seed points, in order to prevent confusion with the current contour line and occultation of the underlying image.

**Guided segmentation prototype**

1.  Responsiveness: $50$% of test persons suggested an indicator for ongoing computations during their time of waiting.

2.  Control: users would like to influence the location of new seed points, support for manual image zoom, and fine grained control for the *undo* function.

**Joint prototype**

1.  Visibility: $64$% of users intuitively found the toggle functionality for seed labels without prior explanation.

2.  Visibility: $64$% of participants suggested visible instructions for manual seed generation.

Prediction of Questionnaire Results from Log Data {#sec:prediction_of_questionnaire_results_from_log_data}
-------------------------------------------------

The questionnaires’ results are predicted via a regression analysis, based on features extracted from the interaction log data. A visualization of the feature importances for the regression analysis with respect to the is depicted in **Fig.**\[fig:gbrf\_feature\_importance\]. An evaluation with the test set is conducted as depicted in **Tab.**\[tab:prediction\_results\_gbrf\]. The mean prediction errors for the questionnaires’ results are $15.7$% for and $7.4$% for . In both cases, the error of these (first) estimates is larger but close to the average $95$% confidence intervals of $5.5$% () and $4.0$% () for the overall questionnaire results in the portfolio representation.

The similarity graph for the acquired usability aspects introduced in **Fig.**\[fig:result\_questionnaire\_results\_correlation\] can be extended to outline the direct relationship between questionnaire results and recorded features. Such a graph is depicted in **Fig.**\[fig:feature\_correlations\_and\_feature\_importance\]. Notably, there is no individual feature, which strongly correlates with one of the questionnaire results. However, as the results of the regression analysis in **Tab.**\[tab:prediction\_results\_gbrf\] depict, there is a noteworthy dependence of the usability aspects measured by the and questionnaires and combinations of the recorded features. The most important features for the approximation of the questionnaire results are depicted in **Tab.**\[tab:most\_frequently\_used\_features\].

Discussion {#sec:discussion}
==========

Usability Aspects
-----------------

Altough the underlying segmentation algorithm is the interactive method for all three prototypes tested, the measured user experiences varied significantly. In terms of user stimulus a more innovative interaction system like the joint prototype is preferred to a traditional one. Pragmatic quality aspects, evaluated by as well as ’s , clearly outline that the approach has an advantage over the other two techniques. This conclusion also manifests in the [Dice]{} coefficient values’ fast convergence rate towards its maximum for this prototype. The normalized median spent for the overall segmentation of each image are $100$% (), $550$% (guided), and $380$% (joint). As a result, users prefer the simple, pragmatic interface as well as a substantial degree of freedom to control each iterative step of the segmentation. The less cognitively challenging approach is preferred [@ramkumar2016user]. The other methods provide more guidance for aspects which the user aims to control themselves. In order to improve the productivity of an , less guidance should be imposed in these cases, while providing more guidance on aspects of the process not apparent to the users’ focus of attention [@heron1957perception].

Usability Aspects Approximation
-------------------------------

For and , the most discriminative features selected by are the receiver operating characteristic area under the curve () of the final interactive segmentations over the elapsed real time which passed during segmentation (). The Jaccard index [@jaccard1912distribution] as well as the relative absolute area/volume difference () each divided by the computation time are most relevant for , respective . The pragmatic quality’s dominant features are composed of final Dice scores and time measurements per segmentation. The results, quantifying the overall usability of a prototype, is mainly predicted based on the features with the highest level of abstraction used. In the top $10$% ($22$) selected features, $45$% of top features are values, as indicated in **Tab.**\[tab:most\_frequently\_used\_features\] and **Fig.**\[fig:gbrf\_feature\_importance\](top). In comparison: $41$%, $36$%, $18$%, $14$%, and $9$%.

Conclusion {#sec:conclusion}
==========

For sufficiently complex tasks like the accurate segmentation of lesions during , fully automated systems are, by their lack of domain knowledge, inherently limited in the achievable quality of their segmentation results. may supersede fully automated systems in certain niches by cooperating with the human user in order to reach the common goal of an exact segmentation result in a short amount of time. The evaluation of interactive approaches is more demanding and less automated than the evaluation with other approaches, due to complex human behavior.

However, there are methods like extensive user studies to assess the quality of a given system. It was shown, that even a suitable approximation of a study’s results regarding pragmatic as well as hedonic usability aspects is achievable from a sole analysis of the users’ interaction recordings. Those records are straightforward to acquire during normal (digital) prototype usage and can lead to a good first estimate of the system’s usability aspects, without the need to significantly increase the temporal demands on each participant by a mandatory completion of questionnaires after each system usage.

This mapping of quantitative low-level features, which are exclusively based on measurable interactions with the system (like the final Dice score, computation times, or relative seed positions), may allow for a fully automated assessment of an interactive system’s quality.

[Outlook]{} {#sec:outlook}
===========

For proposed automation, a rule-based user model (robot user) like [@amrehn2017uinet; @amrehn2019interactive] or a learning-based user model could interact with the prototype system instead of a human user. This evaluation scheme may significantly reduce the amount of resources necessary to investigate each variation of a prototype’s features and segmentation methodologies.

Disclaimer {#disclaimer .unnumbered}
==========

The concept and software presented in this paper are based on research and are not commercially available. Due to regulatory reasons its future availability cannot be guaranteed.

Conflicts of Interest {#conflicts-of-interest .unnumbered}
=====================

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgment {#acknowledgment .unnumbered}
==============

Thanks to Christian Kisker and Carina Lehle for their hard work with the data collection.

Example for Evaluation (**Eq.** \[eq:sus\_score\]) {#example-for-evaluation-eq.eqsus_score .unnumbered}
==================================================

The result of the survey is a single scalar value, in the range of zero to $100$ as a composite measure of the overall usability. The score is computed according to **Eq.**\[eq:sus\_score\], as outlined in [@brooke1996sus], given $S$ participants, where $\mathbf{x}^\text{SUS}_{s,i}$ is the response to the statement $i$ by subject $s$. $$\operatorname{sus}(\mathbf{x}) = \frac{2.5}{S} \sum_{s}\left[\, \sum_{\text{odd } i} \mathbf{x}^\text{SUS}_{s,i} + \sum_{\text{even } i} (4 - \mathbf{x}^\text{SUS}_{s,i})\, \right]$$

Let $S=3$ participants answer the $10$ questions (listed in of the SUS questionnaire as follows: $$\mathbf{x}^\text{SUS} = \left| \begin{array}{c}
\mathbf{x}^\text{SUS}_0 \\
\mathbf{x}^\text{SUS}_1 \\
\mathbf{x}^\text{SUS}_2 \end{array} \right| = \left| \begin{array}{cccccccccc}
 0 & 1 & 2 & 3 & 4 & 0 & 1 & 2 & 3 & 4 \\
 1 & 2 & 3 & 4 & 0 & 1 & 2 & 3 & 4 & 0 \\
 2 & 3 & 4 & 0 & 1 & 2 & 3 & 4 & 0 & 1 \end{array} \right|,$$

where $\mathbf{x}^\text{SUS}_s$ are rows in matrix $\mathbf{x}^\text{SUS}$. Then: $$\begin{aligned}
\operatorname{sus}(\mathbf{x}) = \frac{2.5}{3} \cdot (&(0 + 3 + 2 + 1 + 4 + 4 + 1 + 2 + 3 + 0) + \\
 & (1 + 2 + 3 + 0 + 0 + 3 + 2 + 1 + 4 + 4) + \\
 & (2 + 1 + 4 + 4 + 1 + 2 + 3 + 0 + 0 + 3))\end{aligned}$$

Example for Evaluation (**Eq.** \[eq:attrakdiff\_score\]) {#example-for-evaluation-eq.eqattrakdiff_score .unnumbered}
=========================================================

For the questionnaire’s evaluation for subject , each of the seven adjective pairs per group is assigned a score by each participant, reflecting their tendency towards the positive of the two adjectives. The overall ratings per group are defined in [@hassenzahl2003attrakdiff] as the mean scores computed over all subjects $s$ and statements $i$, as depicted in **Eq.**\[eq:attrakdiff\_score\]. Here, $S$ is the number of participants in the survey. $$\operatorname{attrakdiff}(\mathbf{x}, \,g) = \frac{1}{7 \cdot S} \sum_{s} \sum_{i} \mathbf{x}^g_{s,i}$$ Let $S=3$ participants fill in the $28$ choices (listed in of the questionnaire as follows, where $\mathbf{x}^g_s$ are rows in matrix $\mathbf{x^g}$:

Group PQ: $$\mathbf{x}^\text{PQ} = \left| \begin{array}{c}
\mathbf{x}^\text{PQ}_0 \\
\mathbf{x}^\text{PQ}_1 \\
\mathbf{x}^\text{PQ}_2 \end{array} \right| = \left| \begin{array}{ccccccc}
1 & 2 & 3 & 4 & 5 & 6 & 7 \\
2 & 3 & 4 & 5 & 6 & 7 & 7 \\
3 & 4 & 5 & 6 & 7 & 7 & 7 \end{array} \right|$$

Group ATT: $$\mathbf{x}^\text{ATT} = \left| \begin{array}{c}
\mathbf{x}^\text{ATT}_0 \\
\mathbf{x}^\text{ATT}_1 \\
\mathbf{x}^\text{ATT}_2 \end{array} \right| = \left| \begin{array}{ccccccc}
2 & 3 & 4 & 5 & 6 & 7 & 7 \\
3 & 4 & 5 & 6 & 7 & 7 & 7 \\
4 & 5 & 6 & 7 & 7 & 7 & 7 \end{array} \right|$$

Group HQ-I: $$\mathbf{x}^\text{HQ-I} = \left| \begin{array}{c}
\mathbf{x}^\text{HQ-I}_0 \\
\mathbf{x}^\text{HQ-I}_1 \\
\mathbf{x}^\text{HQ-I}_2 \end{array} \right| = \left| \begin{array}{ccccccc}
3 & 4 & 5 & 6 & 7 & 7 & 7 \\
4 & 5 & 6 & 7 & 7 & 7 & 7 \\
5 & 6 & 7 & 7 & 7 & 7 & 7 \end{array} \right|$$

Group HQ-S: $$\mathbf{x}^\text{HQ-S} = \left| \begin{array}{c}
\mathbf{x}^\text{HQ-S}_0 \\
\mathbf{x}^\text{HQ-S}_1 \\
\mathbf{x}^\text{HQ-S}_2 \end{array} \right| = \left| \begin{array}{ccccccc}
4 & 5 & 6 & 7 & 7 & 7 & 7 \\
5 & 6 & 7 & 7 & 7 & 7 & 7 \\
6 & 7 & 7 & 7 & 7 & 7 & 7 \end{array} \right|$$

After evaluation via **Eq.**\[eq:attrakdiff\_score\]: $$\begin{aligned}
\operatorname{attrakdiff}(\mathbf{x}, \text{\makebox[2.6em][r]{PQ}}) = (&(1 + 2 + 3 + 4 + 5 + 6 + 7) + \\
& (2 + 3 + 4 + 5 + 6 + 2 \cdot 7) + \\
& (3 + 4 + 5 + 6 + 3 \cdot 7)) \,/\, 21 \\
\operatorname{attrakdiff}(\mathbf{x}, \text{\makebox[2.6em][r]{ATT}}) = (&(2 + 3 + 4 + 5 + 6 + 2 \cdot 7) + \\
& (3 + 4 + 5 + 6 + 3 \cdot 7) + \\
& (4 + 5 + 6 + 4 \cdot 7)) \,/\, 21 \\
\operatorname{attrakdiff}(\mathbf{x}, \text{\makebox[2.6em][r]{HQ-I}}) = (&(3 + 4 + 5 + 6 + 3 \cdot 7) + \\
& (4 + 5 + 6 + 4 \cdot 7) + \\
& (5 + 6 + 5 \cdot 7)) \,/\, 21 \\
\operatorname{attrakdiff}(\mathbf{x}, \text{\makebox[2.6em][r]{HQ-S}}) = (&(4 + 5 + 6 + 4 \cdot 7) + \\
& (5 + 6 + 5 \cdot 7) + \\
& (6 + 6 \cdot 7)) \,/\, 21 \\\end{aligned}$$ In this case, , , , and . The confidence intervals $\operatorname{conf}(.)$ can then be extracted via the percent point function $\operatorname{ppf}(.)$ (also called quantile function or inverse cumulative distribution function) for the selected $95$% confidence interval. $$\begin{aligned}
 z &= \operatorname{ppf}(0.95 \cdot 0.5) = 1.95996 \\
\operatorname{conf}(\mathbf{x}, \,g) &= \operatorname{mean}(\mathbf{x}^g) \pm z \cdot \frac{\operatorname{std}(\mathbf{x}^g)}{\sqrt{7 \cdot S}} \\\end{aligned}$$
