Quantitative imaging biomarkers: A review of statistical methods for computer algorithm comparisons
---
paper_title: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

paper_content:

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples...
---
paper_title: The FDA Critical Path Initiative and Its Influence on New Drug Development

paper_content:

Societal expectations about drug safety and efficacy are rising while productivity in the pharmaceutical industry is falling. In 2004, the US Food and Drug Administration introduced the Critical Path Initiative with the intent of modernizing drug development by incorporating recent scientific advances, such as genomics and advanced imaging technologies, into the process. An important part of the initiative is the use of public-private partnerships and consortia to accomplish the needed research. This article explicates the reasoning behind the Critical Path Initiative and discusses examples of successful consortia.
---
paper_title: A Collaborative Enterprise for Multi-Stakeholder Participation in the Advancement of Quantitative Imaging

paper_content:

We have formed the Quantitative Imaging Biomarker Alliance to enable cooperation and address issues in quantitative medical imaging by adapting the successful Integrating the Healthcare Enterprise precedent to the needs of imaging science.
---
paper_title: Reproducibility of Standardized Uptake Value Measurements Determined by 18F-FDG PET in Malignant Tumors

paper_content:

18 F-FDG PET is increasingly being used to monitor the early response of malignant tumors to chemotherapy. Understanding the reproducibility of standardized uptake values (SUVs) is an important prerequisite in estimating what constitutes a significant change. Methods: Twenty-six patients were studied on 2 separate occasions (mean interval ± SD, 3 ± 2 d; range, 1-5 d). A static PET/CT scan was performed 94 ± 9 min after the intravenous injection of 383 ± 15 MBq of 18 F-FDG. Mean and maximum SUVs (SUV mean and SUV max , respectively) were determined for regions of interest drawn around the tumor on the first study and for the same regions of interest transferred to the second study. Results: SUV mean in tumors ranged from 1.49 to 17.48 and SUV max ranged from 2.99 to 24.09. The correlation between SUV mean determined on the 2 separate visits was 0.99; the mean difference between the 2 measurements was 0.01 ± 0.27 SUV. The 95% confidence limits for the measurements were ±0.53. For SUV max , the mean difference was -0.05 ± 1.14 SUV. Conclusion: Our study demonstrates that repeated measurements of SUV mean performed a few days apart are highly reproducible. A decrease of 0.5 in the SUV is statistically significant.
---
paper_title: Reproducibility of Metabolic Measurements in Malignant Tumors Using FDG PET

paper_content:

PET using 18 F-fluorodeoxyglucose (FDG) is increasingly applied to monitor the response of malignant tumors to radiotherapy and chemotherapy. The aim of this study was to assess the reproducibility of serial FDG PET measurements to define objective criteria for the evaluation of treatment-induced changes. Methods : Sixteen patients participating in phase I studies of novel antineoplastic compounds were examined twice by FDG PET within 10 d while they were receiving no therapy. Standardized uptake values (SUVs), FDG net influx constants (K i ), glucose normalized SUVs (SUV gluc ) and influx constants (K i,gluc ) were determined for 50 separate lesions. The precision of repeated measurements was determined on a lesion-by-lesion and a patient-by-patient basis. Results: None of the parameters showed a significant increase or decrease at the two examinations. The differences of repeated measurements were approximately normally distributed for all parameters with an SD of the mean percentage difference of about 10%. The 95% normal ranges for spontaneous fluctuations of SUV, SUV gluc , K i and K i,gluc were determined to be ±0.91, ±1.14, ±0.52 mL/100 g/min and ±0.64 mU100 g/min, respectively. Analysis on a lesion-by-lesion basis yielded similar results. Conclusion: FDG PET provides several highly reproducible quantitative parameters of tumor glucose metabolism. Changes of a parameter that are outside the 95% normal range determined in this study may be used to define a metabolic response to therapy.
---
paper_title: The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans.

paper_content:

PURPOSE ::: The development of computer-aided diagnostic (CAD) methods for lung nodule detection, classification, and quantitative assessment can be facilitated through a well-characterized repository of computed tomography (CT) scans. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) completed such a database, establishing a publicly available reference for the medical imaging research community. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process. ::: ::: ::: METHODS ::: Seven academic centers and eight medical imaging companies collaborated to identify, address, and resolve challenging organizational, technical, and clinical issues to provide a solid foundation for a robust database. The LIDC/IDRI Database contains 1018 cases, each of which includes images from a clinical thoracic CT scan and an associated XML file that records the results of a two-phase image annotation process performed by four experienced thoracic radiologists. In the initial blinded-read phase, each radiologist independently reviewed each CT scan and marked lesions belonging to one of three categories ("nodule > or =3 mm," "nodule <3 mm," and "non-nodule > or =3 mm"). In the subsequent unblinded-read phase, each radiologist independently reviewed their own marks along with the anonymized marks of the three other radiologists to render a final opinion. The goal of this process was to identify as completely as possible all lung nodules in each CT scan without requiring forced consensus. ::: ::: ::: RESULTS ::: The Database contains 7371 lesions marked "nodule" by at least one radiologist. 2669 of these lesions were marked "nodule > or =3 mm" by at least one radiologist, of which 928 (34.7%) received such marks from all four radiologists. These 2669 lesions include nodule outlines and subjective nodule characteristic ratings. ::: ::: ::: CONCLUSIONS ::: The LIDC/IDRI Database is expected to provide an essential medical imaging research resource to spur CAD development, validation, and dissemination in clinical practice.
---
paper_title: Variance of SUVs for FDG-PET/CT is greater in clinical practice than under ideal study settings.

paper_content:

PURPOSE ::: Measurement variance affects the clinical effectiveness of PET-based measurement as a semiquantitative imaging biomarker for cancer response in individual patients and for planning clinical trials. In this study, we measured test-retest reproducibility of SUV measurements under clinical practice conditions and recorded recognized deviations from protocol compliance. ::: ::: ::: METHODS ::: Instrument performance calibration, display, and analyses conformed to manufacture recommendations. Baseline clinical (18)F-FDG PET/CT examinations were performed and then repeated at 1 to 7 days. Intended scan initiation uptake period was to repeat the examinations at the same time for each study after injection of 12 mCi FDG tracer. Avidity of uptake was measured in 62 tumors in 21 patients as SUV for maximum voxel (SUV(max)) and for a mean of sampled tumor voxels (SUV(mean)). ::: ::: ::: RESULTS ::: The range of SUV(max) and SUV(mean) was 1.07 to 21.47 and 0.91 to 14.69, respectively. Intraclass correlation coefficient between log of SUV(max) and log of SUV(mean) was 0.93 (95% confidence interval [CI], 0.88-0.95) and 0.92 (95% CI, 0.87-0.95), respectively.Correlation analysis failed to show an effect on uptake period variation on SUV measurements between the 2 examinations, suggesting additional sources of noise.The threshold criteria for relative difference from baseline for the 95% CI were ± 49% or ± 44% for SUV(max) or SUV(mean), respectively. ::: ::: ::: CONCLUSIONS ::: Variance of SUV for FDG-PET/CT in current clinical practice in a single institution was greater than expected when compared with benchmarks reported under stringent efficacy study settings. Under comparable clinical practice conditions, interpretation of changes in tumor avidity in individuals and assumptions in planning clinical trials may be affected.
---
paper_title: Lung cancer: reproducibility of quantitative measurements for evaluating 2-[F-18]-fluoro-2-deoxy-D-glucose uptake at PET.

paper_content:

PURPOSE ::: To study the precision of repeated 2-[fluorine-18]-fluoro-2-deoxy-D-glucose (FDG) uptake measurements at positron emission tomography (PET) in patients with primary lung cancer. ::: ::: ::: MATERIALS AND METHODS ::: Ten patients with untreated lung cancer underwent two dynamic FDG PET examinations after a 4-hour fast within 1 week. Kinetic modeling of tumor FDG uptake was performed on the basis of a three-compartment model. The tumor concentration of F-18 (standardized uptake value calculated on the basis of predicted lean body mass [SUV-lean]) was also measured 50-60 minutes after injection of a tracer. Blood glucose, insulin, and free fatty acid levels were monitored. ::: ::: ::: RESULTS ::: SUV-lean and the FDG influx constant Ki were measured with a mean +/- standard deviation difference of 10% +/- 7 and 10% +/- 8, respectively, over repeated PET scans. The mean difference was reduced to 6% +/- 6 and 6% +/- 5 by multiplying SUV-lean and Ki by plasma glucose concentration. ::: ::: ::: CONCLUSION ::: SUV-lean and graphical Ki can be measured reproducibly, supporting their use in quantitative FDG PET algorithms.
---
paper_title: Reproducibility of Common Semi-quantitative Parameters for Evaluating Lung Cancer Glucose Metabolism with Positron Emission Tomography using 2-Deoxy-2-[18F]Fluoro-D-Glucose

paper_content:

Abstract Purpose: Positron emission tomography (PET) with 2-deoxy-2-[ 18 F]fluoro-D-glucose (FDG) has been used for various cancers, but reproducibility of common utilized semi-quantitative parameters, such as the maximal single pixel standardized uptake value (SUV) and effective glycolytic volume (EGV), remains unknown. Knowledge of precision is essential for applying these parameters to treatment monitoring. The purpose of this investigation was to assess the precision of PET results obtained by repeated examinations of patients with untreated lung cancer. Patients and Methods: Ten patients with lung cancer underwent two PET examinations within a week with no intervening treatment. The reproducibility of three parameters:((1) maximal SUV of 1 × 1 pixel anywhere in the tumor, calculated on the basis of predicted lean body mass [SULmax]; (2) highest average SUV at 4 × 4 pixels in the tumor adjusted by predicted lean body mass [SULmean]; and (3) EGV calculated by multiplying SUL by tumor volume), using PET images obtained at 50–60 min post-injection, were examined. Plasma glucose, insulin and free fatty acid levels were also monitored. Results: The SULmax, SULmean, and EGV were measured with a mean ± S.D. difference of 11.3% ± 8.0, 10.1% ± 8.2, and 10.1% ± 8.0%, respectively. By multiplying SUL by plasma glucose concentration, the mean differences were slightly reduced to 7.2% ± 5.8, 6.7% ± 6.2, and 9.5% ± 8.2, respectively. Conclusion: These data indicate that commonly used semi-quantitative indices of glucose metabolism on PET show high reproducibly. This supports their use in sequential quantitative analysis in PET, such as in treatment response monitoring. (Mol Imag Biol 2002;4:171–178)
---
paper_title: Reproducibility of Semi-quantitative Parameters in FDG-PET Using Two Different PET Scanners: Influence of Attenuation Correction Method and Examination Interval

paper_content:

PURPOSE ::: The aim of this study is to evaluate the reproducibility of semi-quantitative parameters obtained from two 2-deoxy-2-[F-18]fluoro-D-glucose-positron emission tomography (FDG-PET) studies using two different PET scanners. ::: ::: ::: METHODS ::: Forty-five patients underwent FDG-PET examination with two different PET scanners on separate days. Two PET images with different attenuation correction method were generated in each patient, and three regions of interest (ROIs) were placed on the lung tumor and normal organs (mediastinum and liver) in each image. Mean and maximum standardized uptake values (SUVs), tumor-to-mediastinum and tumor-to-liver ratios (T/M and T/L), and the percentage difference in parameters between two PET images (% Diff.) were compared. ::: ::: ::: RESULTS ::: All measured values except maximum SUV in the liver and tumor-related parameters (SUV in lung tumor, T/M, T/L) showed no significant difference between two PET images. ::: ::: ::: CONCLUSION ::: The mean measured values showed high reproducibility and demonstrate that follow-up study or measurement of tumor response to anticancer drugs can be undertaken by FDG-PET examination without specifying the particular type of PET scanner.
---
paper_title: Reproducibility of 18F-FDG and 3'-Deoxy-3'-18F-Fluorothymidine PET Tumor Volume Measurements

paper_content:

The objective of this study was to establish the repeatability and reproducibility limits of several volume-related PET image-derived indices—namely tumor volume (TV), mean standardized uptake value, total glycolytic volume (TGV), and total proliferative volume (TPV)—relative to those of maximum standardized uptake value (SUV max ), commonly used in clinical practice. Methods: Fixed and adaptive thresholding, fuzzy C-means, and fuzzy locally adaptive Bayesian methodology were considered for TV delineation. Double-baseline 18 F-FDG (17 lesions, 14 esophageal cancer patients) and 3'-deoxy-3'- 18 F-fluorothymidine ( 18 F-FLT) (12 lesions, 9 breast cancer patients) PET scans, acquired at a mean interval of 4 d and before any treatment, were used for reproducibility evaluation. The repeatability of each method was evaluated for the same datasets and compared with manual delineation. Results: A negligible variability of less than 5% was measured for all segmentation approaches in comparison to manual delineation (5%-35%). SUV max reproducibility levels were similar to others previously reported, with a mean percentage difference of 1.8% ± 16.7% and -0.9% ± 14.9% for the 18 F-FDG and 18 F-FLT lesions, respectively. The best TV, TGV, and TPV reproducibility limits ranged from -21% to 31 % and -30% to 37% for 18 -FDG and 18 F-FLT images, respectively, whereas the worst reproducibility limits ranged from -90% to 73% and -68% to 52%, respectively. Conclusion: The reproducibility of estimating TV, mean standardized uptake value, and derived TGV and TPV was found to vary among segmentation algorithms. Some differences between 18 F-FDG and 18 F-FLT scans were observed, mainly because of differences in overall image quality. The smaller reproducibility limits for volume-derived image indices were similar to those for SUV maX , suggesting that the use of appropriate delineation tools should allow the determination of tumor functional volumes in PET images in a repeatable and reproducible fashion.
---
paper_title: Quantitative imaging for evaluation of response to cancer therapy.

paper_content:

Advances in molecular medicine offer the potential to move cancer therapy beyond traditional cytotoxic treatments to safer and more effective targeted therapies based on molecular characteristics of a patient's tumor. Within this context, the role of quantitative imaging as an in vivo biomarker has received considerable attention as a means to predict and measure the response to therapy. For example, the ability to predict the response to therapy quantitatively, early in the drug or radiation therapy regime, would facilitate adaptive therapy trial strategies, that is, that permit alternative treatment regimens in cases where initial therapy response was ineffective. Similarly, the ability to measure the response to therapy should provide a more robust means for both therapy dose management and correlation of imaging results with other laboratory biomarkers. The latter is required for clinical decision making in the clinical setting. The National Cancer Institute (NCI) in collaboration with the Food and Drug Administration (FDA) has therefore promoted a number of initiatives supporting the role of molecular imaging in drug trials. The major goal of these initiatives is the “qualification” of the proposed molecular imaging protocol(s) that can be incorporated into current or future drug trials submitted to the FDA. Clinical research strategies that will help achieve these goals are described in the published literature [1–5].
---
paper_title: Five-year lung cancer screening experience: CT appearance, growth rate, location, and histologic features of 61 lung cancers.

paper_content:

PURPOSE ::: To retrospectively evaluate the computed tomography (CT)-determined size, morphology, location, morphologic change, and growth rate of incidence and prevalence lung cancers detected in high-risk individuals who underwent annual chest CT screening for 5 years and to evaluate the histologic features and stages of these cancers. ::: ::: ::: MATERIALS AND METHODS ::: The study was institutional review board approved and HIPAA compliant. Informed consent was waived. CT scans of 61 cancers (24 in men, 37 in women; age range, 53-79 years; mean, 65 years) were retrospectively reviewed for cancer size, morphology, and location. Forty-eight cancers were assessed for morphologic change and volume doubling time (VDT), which was calculated by using a modified Schwartz equation. Histologic sections were retrospectively reviewed. ::: ::: ::: RESULTS ::: Mean tumor size was 16.4 mm (range, 5.5-52.5 mm). Most common CT morphologic features were as follows: for bronchioloalveolar carcinoma (BAC) (n = 9), ground-glass attenuation (n = 6, 67%) and smooth (n = 3, 33%), irregular (n = 3, 33%), or spiculated (n = 3, 33%) margin; for non-BAC adenocarcinomas (n = 25), semisolid (n = 11, 44%) or solid (n = 12, 48%) attenuation and irregular margin (n = 14, 56%); for squamous cell carcinoma (n = 14), solid attenuation (n = 12, 86%) and irregular margin (n = 10, 71%); for small cell or mixed small and large cell neuroendocrine carcinoma (n = 7), solid attenuation (n = 6, 86%) and irregular margin (n = 5, 71%); for non-small cell carcinoma not otherwise specified (n = 5), solid attenuation (n = 4, 80%) and irregular margin (n = 3, 60%); and for large cell carcinoma (n = 1), solid attenuation and spiculated shape (n = 1, 100%). Attenuation most often (in 12 of 21 cases) increased. Margins most often (in 16 of 20 cases) became more irregular or spiculated. Mean VDT was 518 days. Thirteen of 48 cancers had a VDT longer than 400 days; 11 of these 13 cancers were in women. ::: ::: ::: CONCLUSION ::: Overdiagnosis, especially in women, may be a substantial concern in lung cancer screening.
---
paper_title: Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment

paper_content:

Technological developments and greater rigor in the quantitative measurement of biological features in medical images have given rise to an increased interest in using quantitative imaging biomarkers to measure changes in these features. Critical to the performance of a quantitative imaging biomarker in preclinical or clinical settings are three primary metrology areas of interest: measurement linearity and bias, repeatability, and the ability to consistently reproduce equivalent results when conditions change, as would be expected in any clinical trial. Unfortunately, performance studies to date differ greatly in designs, analysis method, and metrics used to assess a quantitative imaging biomarker for clinical use. It is therefore difficult or not possible to integrate results from different studies or to use reported results to design studies. The Radiological Society of North America and the Quantitative Imaging Biomarker Alliance with technical, radiological, and statistical experts developed a set of...
---
paper_title: Repeatability of 18F-FDG PET in a Multicenter Phase I Study of Patients with Advanced Gastrointestinal Malignancies

paper_content:

18 F-FDG PET is often used to monitor tumor response in multicenter oncology clinical trials. This study assessed the repeatability of several semiquantitative standardized uptake values (mean SUV [SUV mean ], maximum SUV [SUV max ], peak SUV [SUV peak ], and the 3-dimensional isocontour at 70% of the maximum pixel value [SUV 70% ]) as measured by repeated baseline 18 F-FDG PET studies in a multicenter phase I oncology trial. Methods: Double-baseline 18 F-FDG PET studies were acquired for 62 sequentially enrolled patients. Tumor metabolic activity was assessed by SUV mean , SUV max , SUV peak , and SUV 70% . The effect on SUV repeatability of compliance with recommended image-acquisition guidelines and quality assurance (QA) standards was assessed. Summary statistics for absolute differences relative to the average of baseline values and repeatability analysis were performed for all patients and for a subgroup that passed QA, in both a multi- and a single-observer setting. Intrasubject precision of baseline measurements was assessed by repeatability coefficients, intrasubject coefficients of variation (CV), and confidence intervals on mean baseline differences for all SUV parameters. Results: The mean differences between the 2 SUV baseline measurements were small, varying from -2.1 % to 1.9%, and the 95% confidence intervals for these mean differences had a maximum half-width of about 5.6% across the SUV parameters assessed. For SUV max , the intrasubject CV varied from 10.7% to 12.8% for the QA multi-and single-observer datasets and was 16% for the full dataset. The 95% repeatability coefficients ranged from -28.4% to 39.6% for the QA datasets and up to -34.3% to 52.3% for the full dataset. Conclusion: Repeatability results of double-baseline 18 F-FDG PET scans were similar for all SUV parameters assessed, for both the full and the QA datasets, in both the multi- and the single-observer settings. Centralized quality assurance and analysis of data improved intrasubject CV from 15.9% to 10.7% for averaged SUV max . Thresholds for metabolic response in the multicenter multiobserver non-QA settings were -34% and 52% and in the range of -26% to 39% with centralized QA. These results support the use of 18 F-FDG PET for tumor assessment in multicenter oncology clinical trials.
---
paper_title: The Lung Image Database Consortium (LIDC): A comparison of different size metrics for pulmonary nodule measurements

paper_content:

RATIONALE AND OBJECTIVES ::: The goal was to investigate the effects of choosing between different metrics in estimating the size of pulmonary nodules as a factor both of nodule characterization and of performance of computer aided detection systems, because the latter are always qualified with respect to a given size range of nodules. ::: ::: ::: MATERIALS AND METHODS ::: This study used 265 whole-lung CT scans documented by the Lung Image Database Consortium (LIDC) using their protocol for nodule evaluation. Each inspected lesion was reviewed independently by four experienced radiologists who provided boundary markings for nodules larger than 3 mm. Four size metrics, based on the boundary markings, were considered: a unidimensional and two bidimensional measures on a single image slice and a volumetric measurement based on all the image slices. The radiologist boundaries were processed and those with four markings were analyzed to characterize the interradiologist variation, while those with at least one marking were used to examine the difference between the metrics. ::: ::: ::: RESULTS ::: The processing of the annotations found 127 nodules marked by all of the four radiologists and an extended set of 518 nodules each having at least one observation with three-dimensional sizes ranging from 2.03 to 29.4 mm (average 7.05 mm, median 5.71 mm). A very high interobserver variation was observed for all these metrics: 95% of estimated standard deviations were in the following ranges for the three-dimensional, unidimensional, and two bidimensional size metrics, respectively (in mm): 0.49-1.25, 0.67-2.55, 0.78-2.11, and 0.96-2.69. Also, a very large difference among the metrics was observed: 0.95 probability-coverage region widths for the volume estimation conditional on unidimensional, and the two bidimensional size measurements of 10 mm were 7.32, 7.72, and 6.29 mm, respectively. ::: ::: ::: CONCLUSIONS ::: The selection of data subsets for performance evaluation is highly impacted by the size metric choice. The LIDC plans to include a single size measure for each nodule in its database. This metric is not intended as a gold standard for nodule size; rather, it is intended to facilitate the selection of unique repeatable size limited nodule subsets.
---
paper_title: An Introduction to the Bootstrap

paper_content:

Introduction The Accuracy of a Sample Mean Random Samples and Probabilities The Empirical Distribution Function and the Plug-In Principle Standard Errors and Estimated Standard Errors The Bootstrap Estimate of Standard Error Bootstrap Standard Errors: Some Examples More Complicated Data Structures Regression Models Estimates of Bias The Jackknife Confidence Intervals Based on Bootstrap "Tables" Confidence Intervals Based on Bootstrap Percentiles Better Bootstrap Confidence Intervals Permutation Tests Hypothesis Testing with the Bootstrap Cross-Validation and Other Estimates of Prediction Error Adaptive Estimation and Calibration Assessing the Error in Bootstrap Estimates A Geometrical Representation for the Bootstrap and Jackknife An Overview of Nonparametric and Parametric Inference Further Topics in Bootstrap Confidence Intervals Efficient Bootstrap Computations Approximate Likelihoods Bootstrap Bioequivalence Discussion and Further Topics Appendix: Software for Bootstrap Computations References
---
paper_title: A comparison of the Two One-Sided Tests Procedure and the Power Approach for assessing the equivalence of average bioavailability

paper_content:

The statistical test of hypothesis of no difference between the average bioavailabilities of two drug formulations, usually supplemented by an assessment of what the power of the statistical test would have been if the true averages had been inequivalent, continues to be used in the statistical analysis of bioavailability/bioequivalence studies. In the present article, this Power Approach (which in practice usually consists of testing the hypothesis of no difference at level 0.05 and requiring an estimated power of 0.80) is compared to another statistical approach, the Two One-Sided Tests Procedure, which leads to the same conclusion as the approach proposed by Westlake based on the usual (shortest) 1-2 alpha confidence interval for the true average difference. It is found that for the specific choice of alpha = 0.05 as the nominal level of the one-sided tests, the two one-sided tests procedure has uniformly superior properties to the power approach in most cases. The only cases where the power approach has superior properties when the true averages are equivalent correspond to cases where the chance of concluding equivalence with the power approach when the true averages are not equivalent exceeds 0.05. With appropriate choice of the nominal level of significance of the one-sided tests, the two one-sided tests procedure always has uniformly superior properties to the power approach. The two one-sided tests procedure is compared to the procedure proposed by Hauck and Anderson.
---
paper_title: Testing Statistical Hypotheses of Equivalence

paper_content:

Introduction. Methods for One-Sided Equivalence Problems. General Approaches to the Construction of Tests for Equivalence in the Strict Sense. Equivalence Tests for Selected One-Parameter Problems. Equivalence Tests for Designs with Paired Observation. Equivalence Tests for Two Unrelated Samples. Multi-sample Tests for Equivalence. Tests for Establishing Goodness of Fit. The Assessment of Bio-Equivalence. Appendix. References. Index
---
paper_title: Comparison of thallium-201 SPECT and planar imaging methods for quantification of experimental myocardial infarct size

paper_content:

Abstract To compare single photon emission computed tomography (SPECT) and planar thallium-201 (TI-201) myocardial perfusion imaging methods for quantification of left ventricular infarct size, 12 dogs with 6 to 8 hours of closed-chest coronary occlusion and 5 normal dogs were studied. After intravenous administration of TI-201, SPECT and three-view planar images were obtained. After the animals were put to death, hearts were sliced and stained with triphenyltetrazolium chloride (TTC) for planimetric determination of left ventricular infarct size. Infarct size on each SPECT slice and planar image was defined as the percentage of circumferential count profiles falling below the limits derived from normal dogs. Infarct size as a percentage of left ventricular mass was determined from SPECT and planar images before and after correcting for differences in myocardial slice mass from apex to base. The correlation coefficients, the concordance correlation coefficients (reflecting closeness to the line of identity), and the mean absolute deviations of the four methods versus TTC staining were 0.83, 0.77, and 5.1% (SPECT, no correction); 0.85, 0.84, and 3.7% (SPECT with correction); 0.81, 0.42, and 12.9% (planar, no correction); and 0.75, 0.49, and 10.4% (planar with correction). The regression lines did not differ from the line of identity for SPECT, whereas they differed significantly for planar imaging. Thus both SPECT and planar imaging are well suited for quantification of left ventricular infarct size. SPECT, however, appears to be superior to planar imaging, since its regression line more closely approximates the line of identity.
---
paper_title: A quantitative comparison of motion detection algorithms in fMRI

paper_content:

An important step in the analysis of fMRI time-series data is to detect, and as much as possible, correct for subject motion during the course of the scanning session. Several public domain algorithms are currently available for motion detection in fMRI. This paper compares the performance of four commonly used programs: AIR 3.08, SPM99, AFNI98, and the pyramid method of Thevenaz, Ruttimann, and Unser (TRU). The comparison is based on the performance of the algorithms in correcting a range of simulated known motions in the presence of various degrees of noise. SPM99 provided the most accurate motion detection amongst the algorithms studied. AFNI98 provided only slightly less accurate results than SPM99, however, it was several times faster than the other programs. This algorithm represents a good compromise between speed and accuracy. AFNI98 was also the most robust program in presence of noise. It yielded reasonable results for very low signal to noise levels. For small initial misalignments, TRU's performance was similar to SPM99 and AFNI98. However, its accuracy diminished rapidly for larger misalignments. AIR was found to be the least accurate program studied. © 2001 Elsevier Science Inc. All rights reserved.
---
paper_title: Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment

paper_content:

Technological developments and greater rigor in the quantitative measurement of biological features in medical images have given rise to an increased interest in using quantitative imaging biomarkers to measure changes in these features. Critical to the performance of a quantitative imaging biomarker in preclinical or clinical settings are three primary metrology areas of interest: measurement linearity and bias, repeatability, and the ability to consistently reproduce equivalent results when conditions change, as would be expected in any clinical trial. Unfortunately, performance studies to date differ greatly in designs, analysis method, and metrics used to assess a quantitative imaging biomarker for clinical use. It is therefore difficult or not possible to integrate results from different studies or to use reported results to design studies. The Radiological Society of North America and the Quantitative Imaging Biomarker Alliance with technical, radiological, and statistical experts developed a set of...
---
paper_title: The FDA Critical Path Initiative and Its Influence on New Drug Development

paper_content:

Societal expectations about drug safety and efficacy are rising while productivity in the pharmaceutical industry is falling. In 2004, the US Food and Drug Administration introduced the Critical Path Initiative with the intent of modernizing drug development by incorporating recent scientific advances, such as genomics and advanced imaging technologies, into the process. An important part of the initiative is the use of public-private partnerships and consortia to accomplish the needed research. This article explicates the reasoning behind the Critical Path Initiative and discusses examples of successful consortia.
---
paper_title: Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

paper_content:

Quantitative imaging biomarkers are being used increasingly in medicine to diagnose and monitor patients’ disease. The computer algorithms that measure quantitative imaging biomarkers have different technical performance characteristics. In this paper we illustrate the appropriate statistical methods for assessing and comparing the bias, precision, and agreement of computer algorithms. We use data from three studies of pulmonary nodules. The first study is a small phantom study used to illustrate metrics for assessing repeatability. The second study is a large phantom study allowing assessment of four algorithms’ bias and reproducibility for measuring tumor volume and the change in tumor volume. The third study is a small clinical study of patients whose tumors were measured on two occasions. This study allows a direct assessment of six algorithms’ performance for measuring tumor change. With these three examples we compare and contrast study designs and performance metrics, and we illustrate the advantag...
---
paper_title: Measuring agreement in method comparison studies

paper_content:

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference +/- 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.
---
paper_title: Properties of Sufficiency and Statistical Tests

paper_content:

1—In a previous paper*, dealing with the importance of properties of sufficiency in the statistical theory of small samples, attention was mainly confined to the theory of estimation. In the present paper the structure of small sample tests, whether these are related to problems of estimation and fiducial distributions, or are of the nature of tests of goodness of fit, is considered further.
---
paper_title: Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment

paper_content:

Technological developments and greater rigor in the quantitative measurement of biological features in medical images have given rise to an increased interest in using quantitative imaging biomarkers to measure changes in these features. Critical to the performance of a quantitative imaging biomarker in preclinical or clinical settings are three primary metrology areas of interest: measurement linearity and bias, repeatability, and the ability to consistently reproduce equivalent results when conditions change, as would be expected in any clinical trial. Unfortunately, performance studies to date differ greatly in designs, analysis method, and metrics used to assess a quantitative imaging biomarker for clinical use. It is therefore difficult or not possible to integrate results from different studies or to use reported results to design studies. The Radiological Society of North America and the Quantitative Imaging Biomarker Alliance with technical, radiological, and statistical experts developed a set of...
---
paper_title: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

paper_content:

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples...
---
paper_title: Statistical methods in assessing agreement: Models, issues, and tools

paper_content:

Measurements of agreement are needed to assess the acceptability of a new or generic process, methodology, and formulation in areas of laboratory performance, instrument or assay validation, method comparisons, statistical process control, goodness of fit, and individual bioequivalence. In all of these areas, one needs measurements that capture a large proportion of data that are within a meaningful boundary from target values. Target values can be considered random (measured with error) or fixed (known), depending on the situation. Various meaningful measures to cope with such diverse and complex situations have become available only in the last decade. These measures often assume that the target values are random. This article reviews the literature and presents methodologies in terms of “coverage probability.” In addition, analytical expressions are introduced for all of the aforementioned measurements when the target values are fixed and when the error structure is homogenous or heterogeneous (proport...
---
paper_title: Statistical methods in diagnostic medicine

paper_content:

Preface. Acknowledgments. 1. Introduction. 1.1 Why This Book? 1.2 What Is Diagnostic Accuracy? 1.3 Landmarks in Statistical Methods for Diagnostic Medicine. 1.4 Software. 1.5 Topics not Covered in This Book. 1.6 Summary. I BASIC CONCEPTS AND METHODS. 2. Measures of Diagnostic Accuracy. 2.1 Sensitivity and Specificity. 2.2 The Combined Measures of Sensitivity and Specificity. 2.3 The ROC Curve. 2.4 The Area Under the ROC Curve. 2.5 The Sensitivity at a Fixed FPR. 2.6 The Partial Area Under the ROC Curve. 2.7 Likelihood Ratios. 2.8 Other ROC Curve Indices. 2.9 The Localization and Detection of Multiple Abnormalities. 2.10 Interpretation of Diagnostic Tests. 2.11 Optimal Decision Threshold on the ROC Curve. 2.12 Multiple Tests. 3. The Design of Diagnostic Accuracy Studies. 3.1 Determining the Objective of the Study. 3.2 Identifying the Target Patient Population. 3.3 Selecting a Sampling Plan for Patients. 3.3.1 Phase I: Exploratory Studies. 3.3.2 Phase II: Challenge Studies. 3.3.3 Phase III: Clinical Studies. 3.4 Selecting the Gold Standard. 3.5 Choosing a Measure of Accuracy. 3.6 Identifying the Target Reader Population. 3.7 Selecting a Sampling Plan for Readers. 3.8 Planning the Data Collection. 3.8.1 Format for the Test Results. 3.8.2 Data Collection for the Reader Studies. 3.8.3 Reader Training. 3.9 Planning the Data Analyses. 3.9.1 Statistical Hypotheses. 3.9.2 Reporting the Test Results. 3.10 Determining the Sample Size. 4. Estimation and Hypothesis Testing in a Single Sample. 4.1 Binary Scale Data. 4.1.1 Sensitivity and Specificity. 4.1.2 The Sensitivity and Specificity of Clustered Binary Data. 4.1.3 The Likelihood Ratio (LR). 4.1.4 The Odds Ratio. 4.2 Ordinal Scale Data. 4.2.1 The Empirical ROC Curve. 4.2.2 Fitting a Smooth Curve (Parametric Model). 4.2.3 Estimation of Sensitivity at a Particular FPR. 4.2.4 The Area and Partial Area Under the ROC Curve (Parametric Model). 4.2.5 The Area Under the Curve (Nonparametric Method). 4.2.6 Nonparametric Analysis of Clustered Data. 4.2.7 The Degenerate Data. 4.2.8 Choosing Between Parametric and Nonparametric Methods. 4.3 Continuous Scale Data. 4.3.1 The Empirical ROC Curve. 4.3.2 Fitting a Smooth ROC Curve (Parametric and Nonparametric Methods). 4.3.3 Area Under the ROC Curve (Parametric and Nonparametric). 4.3.4 Fixed FPR The Sensitivity and Decision Threshold. 4.3.5 Choosing the Optimal Operating Point. 4.3.6 Choosing Between Parametric and Nonparametric Techniques. 4.4 Hypothesis Testing About the ROC Area. 5. Comparing the Accuracy of Two Diagnostic Tests. 5.1 Binary Scale Data. 5.1.1 Sensitivity and Specificity. 5.1.2 Sensitivity and Specificity of Clustered Binary Data. 5.2 Ordinal and Continuous Scale Data. 5.2.1 Determining the Equality of Two ROC Curves. 5.2.2 Comparing ROC Curves at a Particular Point. 5.2.3 Determining the Range of FPR for Which TPR Differ. 5.2.4 A Comparison of the Area or Partial Area. 5.3 Tests of Equivalence. 6. Sample Size Calculation. 6.1 The Sample Size for Accuracy Studies of a Single Test. 6.1.1 Sensitivity and Specificity. 6.1.2 The Area Under the ROC Curve. 6.1.3 The Sensitivity at a Fixed FPR. 6.1.4 The Partial Area Under the ROC Curve. 6.2 The Sample Size for the Accuracy of Two Tests. 6.2.1 Sensitivity and Specificity. 6.2.2 The Area Under the ROC Curve. 6.2.3 The Sensitivity at a Fixed FPR. 6.2.4 The Partial Area Under the ROC Curve. 6.3 The Sample Size for Equivalent Studies of Two Tests. 6.4 The Sample Size for Determining a Suitable Cutoff Value. 7. Issues in Meta Analysis for Diagnostic Tests. 7.1 Objectives. 7.2 Retrieval of the Literature. 7.3 Inclusion Exclusion Criteria. 7.4 Extracting Information From the Literature. 7.5 Statistical Analysis. 7.6 Public Presentation. II ADVANCED METHODS. 8. Regression Analysis for Independent ROC Data. 8.1 Four Clinical Studies. 8.1.1 Surgical Lesion in a Carotid Vessel Example. 8.1.2 Pancreatic Cancer Exampl. 8.1.3 Adult Obesity Example. 8.1.4 Staging of Prostate Cancer Example. 8.2 Regression Models for Continuous Scale Tests. 8.2.1 Indirect Regression Models for Smooth ROC Curves. 8.2.2 Direct Regression Models for Smooth ROC Curves. 8.2.3 MRA Use for Surgical Lesion Detection in the Carotid Vessel. 8.2.4 Biomarkers for the Detection of Pancreatic Cancer. 8.2.5 Prediction of Adult Obesity by Using Childhood BMI Measurements. 8.3 Regression Models for Ordinal Scale Tests. 8.3.1 Indirect Regression Models for Latent Smooth ROC Curves. 8.3.2 Direct Regression Model for Latent Smooth ROC Curves. 8.3.3 Detection of Periprostatic Invasion With US. 9. Analysis of Correlated ROC Data. 9.1 Studies With Multiple Test Measurements of the Same Patient. 9.1.1 Indirect Regression Models for Ordinal Scale Tests. 9.1.2 Neonatal Examination Example. 9.1.3 Direct Regression Models for Continuous Scale Tests. 9.2 Studies With Multiple Readers and Tests. 9.2.1 A Mixed Effects ANOVA Model for Summary Measures of Diagnostic Accuracy. 9.2.2 Detection of TAD Example. 9.2.3 The Mixed Effects ANOVA Model for Jackknife Pseudovalues. 9.2.4 Neonatal Examination Example. 9.2.5 A Bootstrap Method. 9.3 Sample Size Calculation for Multireader Studies. 10. Methods for Correcting Verification Bias. 10.1 A Single Binary Scale Test. 10.1.1 Correction Methods With the MAR Assumption. 10.1.2 Correction Methods Without the MAR Assumption. 10.1.3 Hepatic Scintigraph Example. 10.2 Correlated Binary Scale Tests. 10.2.1 An ML Approach Without Covariates. 10.2.2 An ML Approach With Covariates. 10.2.3 Screening Tests for Dementia Disorder Example. 10.3 A Single Ordinal Scale Test. 10.3.1 An ML Approach Without Covariates. 10.3.2 Fever of Uncertain Origin Example. 10.3.3 An ML Approach With Covariates. 10.3.4 Screening Test for Dementia Disorder Example. 10.4 Correlated Ordinal Scale Tests. 10.4.1 The Weighted GEE Approach for Latent Smooth ROC Curves. 10.4.2 A Likelihood Based Approach for ROC Areas. 10.4.3 Use of CT and MRI for Staging Pancreatic Cancer Example. 11. Methods for Correcting Imperfect Standard Bias. 11.1 One Single Test in a Single Population. 11.1.1 Hypothetical and Strongyloides Infection Examples. 11.2 One Single Test in G Populations. 11.2.1 Tuberculosis Example. 11.3 Multiple Tests in One Single Population. 11.3.1 MLEs Under the CIA. 11.3.2 Assessment of Pleural Thickening Example. 11.3.3 ML Approaches Without the CIA. 11.3.4 Bioassays for HIV Example. 11.4 Multiple Binary Tests in G Populations. 11.4.1 ML Approaches Under the CIA. 11.4.2 ML Approaches Without the CIA. 12. Statistical Methods for Meta Analysis. 12.1 Sensitivity and Specificity Pairs. 12.1.1 One Common SROC Curve. 12.1.2 Study Specific SROC Curve. 12.1.3 Evaluation of Duplex Ultrasonography, With and Without Color Guidance. 12.2 ROC Curve Areas. 12.2.1 Fixed Effects Models. 12.2.2 Random Effects Models. 12.2.3 Evaluation of the Dexamethasone Suppression.Test. Index.
---
paper_title: Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence.

paper_content:

In areas of inter-laboratory quality control, method comparisons, assay validation and individual bioequivalence, etc., the agreement between observations and target (reference) values is of interest. The mean of the squared difference between observations and target values (MSD) is a good measure of the total deviation. A new user-friendly statistic, the total deviation index (TDI(1-p)), is introduced that translates the MSD into an index that can be directly compared to a predetermined criterion. The TDI(1-p) describes a boundary such that a majority, 100(1-p) per cent, of the observations are within the boundary (measurement unit and/or per cent) from their target values. Statistical inference using the sample counter part (estimate) is presented. A Monte Carlo experiment with 5000 runs was performed to confirm the estimate's validity. Applications in laboratory performance and validation, as well as individual bioequivalence, are presented.
---
paper_title: A Collaborative Enterprise for Multi-Stakeholder Participation in the Advancement of Quantitative Imaging

paper_content:

We have formed the Quantitative Imaging Biomarker Alliance to enable cooperation and address issues in quantitative medical imaging by adapting the successful Integrating the Healthcare Enterprise precedent to the needs of imaging science.
---
paper_title: Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

paper_content:

Quantitative imaging biomarkers are being used increasingly in medicine to diagnose and monitor patients’ disease. The computer algorithms that measure quantitative imaging biomarkers have different technical performance characteristics. In this paper we illustrate the appropriate statistical methods for assessing and comparing the bias, precision, and agreement of computer algorithms. We use data from three studies of pulmonary nodules. The first study is a small phantom study used to illustrate metrics for assessing repeatability. The second study is a large phantom study allowing assessment of four algorithms’ bias and reproducibility for measuring tumor volume and the change in tumor volume. The third study is a small clinical study of patients whose tumors were measured on two occasions. This study allows a direct assessment of six algorithms’ performance for measuring tumor change. With these three examples we compare and contrast study designs and performance metrics, and we illustrate the advantag...
---
paper_title: Statistical Tools for Measuring Agreement

paper_content:

Introduction. - Basic approach for paired continuous data when target values are random or fixed. - Sample size and power. - Unified approach for continuous and categorical data.
---
paper_title: An ROC-type measure of diagnostic accuracy when the gold standard is continuous-scale.

paper_content:

ROC curves and summary measures of accuracy derived from them, such as the area under the ROC curve, have become the standard for describing and comparing the accuracy of diagnostic tests. Methods for estimating ROC curves rely on the existence of a gold standard which dichotomizes patients into disease present or absent. There are, however, many examples of diagnostic tests whose gold standards are not binary-scale, but rather continuous-scale. Unnatural dichotomization of these gold standards leads to bias and inconsistency in estimates of diagnostic accuracy. In this paper, we propose a non-parametric estimator of diagnostic test accuracy which does not require dichotomization of the gold standard. This estimator has an interpretation analogous to the area under the ROC curve. We propose a confidence interval for test accuracy and a statistical test for comparing accuracies of tests from paired designs. We compare the performance (i.e. CI coverage, type I error rate, power) of the proposed methods with several alternatives. An example is presented where the accuracies of two quick blood tests for measuring serum iron concentrations are estimated and compared.
---
paper_title: Modeling Concordance Correlation via GEE to Evaluate Reproducibility

paper_content:

Summary. Clinical studies are often concerned with assessing whether different raters/methods produce similar values for measuring a quantitative variable. Use of the concordance correlation coefficient as a measure of reproducibility has gained popularity in practice since its introduction by Lin (1989, Biometrics45, 255–268). Lin's method is applicable for studies evaluating two raters/two methods without replications. Chinchilli et al. (1996, Biometrics52, 341–353) extended Lin's approach to repeated measures designs by using a weighted concordance correlation coefficient. However, the existing methods cannot easily accommodate covariate adjustment, especially when one needs to model agreement. In this article, we propose a generalized estimating equations (GEE) approach to model the concordance correlation coefficient via three sets of estimating equations. The proposed approach is flexible in that (1) it can accommodate more than two correlated readings and test for the equality of dependent concordant correlation estimates; (2) it can incorporate covariates predictive of the marginal distribution; (3) it can be used to identify covariates predictive of concordance correlation; and (4) it requires minimal distribution assumptions. A simulation study is conducted to evaluate the asymptotic properties of the proposed approach. The method is illustrated with data from two biomedical studies.
---
paper_title: A Concordance Correlation Coefficient to Evaluate Reproducibility

paper_content:

A new reproducibility index is developed and studied. This index is the correlation between the two readings that fall on the 45 degree line through the origin. It is simple to use and possesses desirable properties. The statistical properties of this estimate can be satisfactorily evaluated using an inverse hyperbolic tangent transformation. A Monte Carlo experiment with 5,000 runs was performed to confirm the estimate's validity. An application using actual data is given.
---
paper_title: Measuring agreement in method comparison studies

paper_content:

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference +/- 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.
---
paper_title: Evaluating agreement with a gold standard in method comparison studies.

paper_content:

We develop a statistical model for method comparison studies where a gold standard is present and propose a measure of agreement. This measure can be interpreted as a population correlation coefficient in a constrained bivariate model. An estimator of this coefficient is proposed and its statistical properties explored. Applications of the new methodology to data from the medical literature are presented.
---
paper_title: Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment

paper_content:

Technological developments and greater rigor in the quantitative measurement of biological features in medical images have given rise to an increased interest in using quantitative imaging biomarkers to measure changes in these features. Critical to the performance of a quantitative imaging biomarker in preclinical or clinical settings are three primary metrology areas of interest: measurement linearity and bias, repeatability, and the ability to consistently reproduce equivalent results when conditions change, as would be expected in any clinical trial. Unfortunately, performance studies to date differ greatly in designs, analysis method, and metrics used to assess a quantitative imaging biomarker for clinical use. It is therefore difficult or not possible to integrate results from different studies or to use reported results to design studies. The Radiological Society of North America and the Quantitative Imaging Biomarker Alliance with technical, radiological, and statistical experts developed a set of...
---
paper_title: An Overview on Assessing Agreement with Continuous Measurements

paper_content:

Reliable and accurate measurements serve as the basis for evaluation in many scientific disciplines. Issues related to reliable and accurate measurement have evolved over many decades, dating back to the nineteenth century and the pioneering work of Galton (1886), Pearson (1896, 1899, 1901), and Fisher (1925). Requiring a new measurement to be identical to the truth is often impractical, either because (1) we are willing to accept a measurement up to some tolerable (or acceptable) error, or (2) the truth is simply not available to us, either because it is not measurable or is only measurable with some degree of error. To deal with issues related to both (1) and (2), a number of concepts, methods, and theories have been developed in various disciplines. Some of these concepts have been used across disciplines, while others have been limited to a particular field but may have potential uses in other disciplines. In this paper, we elucidate and contrast fundamental concepts employed in different disciplines and unite these concepts into one common theme: assessing closeness (agreement) of observations. We focus on assessing agreement with continuous measurements and classify different statistical approaches as (1) descriptive tools; (2) unscaled summary indices based on absolute differences of measurements; and (3) scaled summary indices attaining values between -1 and 1 for various data structures, and for cases with and without a reference. We also identify gaps that require further research and discuss future directions in assessing agreement.
---
paper_title: On population and individual bioequivalence.

paper_content:

In a traditional assessment of the bioequivalence of two formulations of a drug one compares the average bioavailability from the two formulations. Anderson and Hauck argued that in some situations it is not sufficient to demonstrate average bioequivalence, and they proposed a method for the assessment of what they called individual bioequivalence, which essentially is the comparison of the individual responses to the two drug formulations within subjects. In this paper we propose a unified strategy for the assessment of bioequivalence that encompasses new approaches to the assessment of both population bioequivalence, which is the comparison of the marginal or population distributions of bioavailabilities, and individual bioequivalence, which is the comparison of the conditional or within-subject distributions of bioavailabilities. The general idea is to use a comparison of the reference formulation to itself as the basis for the comparison of the test with the reference formulation. The new approaches overcome the main weakness of the current methods for the assessment of bioequivalence by considering the variability of bioavailabilities in addition to their means. The current methods for the assessment of bioequivalence, namely the conventional assessment of average bioequivalence and the proposal by Anderson and Hauck for the assessment of individual bioequivalence, emerge as special cases. One can evaluate the new bioequivalence criteria statistically by use of bootstrap confidence intervals.
---
paper_title: Assessment of agreement using intersection-union principle.

paper_content:

We consider the problem of assessing agreement between two instruments and test whether the normally distributed bivariate data have evidence to claim satisfactory agreement. We focus on a comprehensive intersection-union formulation of the hypotheses of agreement. Confidence intervals associated with this approach provide information regarding the extent of agreement and nature of disagreement. We illustrate the suggested methodology using a dataset from the literature.
---
paper_title: Assessing individual agreement.

paper_content:

Evaluating agreement between measurement methods or between observers is important in method comparison studies and in reliability studies. Often we are interested in whether a new method can replace an existing invasive or expensive method, or whether multiple methods or multiple observers can be used interchangeably. Ideally, interchangeability is established only if individual measurements from different methods are similar to replicated measurements from the same method. This is the concept of individual equivalence. Interchangeability between methods is similar to bioequivalence between drugs in bioequivalence studies. Following the FDA guidelines on individual bioequivalence, we propose to assess individual agreement among multiple methods via individual equivalence using the moment criteria. In the case where there is a reference method, we extend the individual bioequivalence criteria to individual equivalence criteria and propose to use individual equivalence coefficient (IEC) to compare multiple methods to one or multiple references. In the case where there is no reference method available, we propose a new IEC to assess individual agreement between multiple methods. Furthermore, we propose a coefficient of individual agreement (CIA) that links the IEC with two recent agreement indices. A method of moments is used for estimation, where one can utilize output from ANOVA models. The nonparametric and bootstrap approaches are used for inference. Five examples are used for illustration.
---
paper_title: Tests for assessment of agreement using probability criteria

paper_content:

Abstract For the assessment of agreement using probability criteria, we obtain an exact test, and for sample sizes exceeding 30, we give a bootstrap- t test that is remarkably accurate. We show that for assessing agreement, the total deviation index approach of Lin [2000. Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence. Statist. Med. 19, 255–270] is not consistent and may not preserve its asymptotic nominal level, and that the coverage probability approach of Lin et al. [2002. Statistical methods in assessing agreement: models, issues and tools. J. Amer. Statist. Assoc. 97, 257–270] is overly conservative for moderate sample sizes. We also show that the nearly unbiased test of Wang and Hwang [2001. A nearly unbiased test for individual bioequivalence problems using probability criteria. J. Statist. Plann. Inference 99, 41–58] may be liberal for large sample sizes, and suggest a minor modification that gives numerically equivalent approximation to the exact test for sample sizes 30 or less. We present a simple and accurate sample size formula for planning studies on assessing agreement, and illustrate our methodology with a real data set from the literature.
---
paper_title: An Introduction to the Bootstrap

paper_content:

Introduction The Accuracy of a Sample Mean Random Samples and Probabilities The Empirical Distribution Function and the Plug-In Principle Standard Errors and Estimated Standard Errors The Bootstrap Estimate of Standard Error Bootstrap Standard Errors: Some Examples More Complicated Data Structures Regression Models Estimates of Bias The Jackknife Confidence Intervals Based on Bootstrap "Tables" Confidence Intervals Based on Bootstrap Percentiles Better Bootstrap Confidence Intervals Permutation Tests Hypothesis Testing with the Bootstrap Cross-Validation and Other Estimates of Prediction Error Adaptive Estimation and Calibration Assessing the Error in Bootstrap Estimates A Geometrical Representation for the Bootstrap and Jackknife An Overview of Nonparametric and Parametric Inference Further Topics in Bootstrap Confidence Intervals Efficient Bootstrap Computations Approximate Likelihoods Bootstrap Bioequivalence Discussion and Further Topics Appendix: Software for Bootstrap Computations References
---
paper_title: Evaluating estimation techniques in medical imaging without a gold standard: experimental validation

paper_content:

Imaging is often used for the purpose of estimating the value of some parameter of interest. For example, a cardiologist may measure the ejection fraction (EF) of the heart to quantify how much blood is being pumped out of the heart on each stroke. In clinical practice, however, it is difficult to evaluate an estimation method because the gold standard is not known, e.g., a cardiologist does not know the true EF of a patient. An estimation method is typically evaluated by plotting its results against the results of another (more accepted) estimation method. This approach results in the use of one set of estimates as the pseudo-gold standard. We have developed a maximum-likelihood approach for comparing different estimation methods to the gold standard without the use of the gold standard. In previous works we have displayed the results of numerous simulation studies indicating the method can precisely and accurately estimate the parameters of a regression line without a gold standard, i.e., without the x-axis. In an attempt to further validate our method we have designed an experiment performing volume estimation using a physical phantom and two imaging systems (SPECT< CT).
---
paper_title: Estimation in Medical Imaging without a Gold Standard 1

paper_content:

Rationale and Objectives ::: In medical imaging, physicians often estimate a parameter of interest (eg, cardiac ejection fraction) for a patient to assist in establishing a diagnosis. Many different estimation methods may exist, but rarely can one be considered a gold standard. Therefore, evaluation and comparison of different estimation methods are difficult. The purpose of this study was to examine a method of evaluating different estimation methods without use of a gold standard.
---
paper_title: The Use and Misuse of Orthogonal Regression in Linear Errors-in-Variables Models

paper_content:

Abstract Orthogonal regression is one of the standard linear regression methods to correct for the effects of measurement error in predictors. We argue that orthogonal regression is often misused in errors-in-variables linear regression because of a failure to account for equation errors. The typical result is to overcorrect for measurement error, that is, overestimate the slope, because equation error is ignored. The use of orthogonal regression must include a careful assessment of equation error, and not merely the usual (often informal) estimation of the ratio of measurement error variances. There are rarer instances, for example, an example from geology discussed here, where the use of orthogonal regression without proper attention to modeling may lead to either overcorrection or undercorrection, depending on the relative sizes of the variances involved. Thus our main point, which does not seem to be widely appreciated, is that orthogonal regression, just like any measurement error analysis, requires ...
---
paper_title: Modelling method comparison data

paper_content:

We explore a range of linear regression models that might be useful for either: (a) the relative calibration of two or more methods or (b) to evaluate their precisions relative to each other. Ideally, one should be able to use a single data set to carry out the jobs (a) and (b) together. Throughout this review we consider the constraints (assumptions) needed to attain identifiability of the models and the possible pitfalls to the unwary in having to introduce them. We also pay particular attention to the possible problems arising from the presence of random matrix effects (reproducible random measurement 'errors' that are characteristic of a given method when being used on a given specimen or sample, i.e. specimen specific biases or subject by method interactions). Finally, we stress the importance of a fully-informative design (using replicate measurements on each subject using at least three independent methods) and large sample sizes.
---
paper_title: Objective Comparison of Quantitative Imaging Modalities Without the Use of a Gold Standard

paper_content:

Imaging is often used for the purpose of estimating the value of some parameter of interest. For example, a cardiologist may measure the ejection fraction (EF) of the heart in order to know how much blood is being pumped out of the heart on each stroke. In clinical practice, however, it is difficult to evaluate an estimation method because the gold standard is not known, e.g., a cardiologist does not know the true EF of a patient. Thus, researchers have often evaluated an estimation method by plotting its results against the results of another (more accepted) estimation method, which amounts to using one set of estimates as the pseudogold standard. In this paper, we present a maximum-likelihood approach for evaluating and comparing different estimation methods without the use of a gold standard with specific emphasis on the problem of evaluating EF estimation methods. Results of numerous simulation studies will be presented and indicate that the method can precisely and accurately estimate the parameters of a regression line without a gold standard, i.e., without the x axis.
---
paper_title: Validation of image segmentation by estimating rater bias and variance

paper_content:

The accuracy and precision of segmentations of medical images has been difficult to quantify in the absence of a 'ground truth' or reference standard segmentation for clinical data. Although physical or digital phantoms can help by providing a reference standard, they do not allow the reproduction of the full range of imaging and anatomical characteristics observed in clinical data. An alternative assessment approach is to compare with segmentations generated by domain experts. Segmentations may be generated by raters who are trained experts or by automated image analysis algorithms. Typically, these segmentations differ due to intra-rater and inter-rater variability. The most appropriate way to compare such segmentations has been unclear. We present here a new algorithm to enable the estimation of performance characteristics, and a true labelling, from observations of segmentations of imaging data where segmentation labels may be ordered or continuous measures. This approach may be used with, among others, surface, distance transform or level-set representations of segmentations, and can be used to assess whether or not a rater consistently overestimates or underestimates the position of a boundary.
---
paper_title: Estimation in Medical Imaging without a Gold Standard 1

paper_content:

Rationale and Objectives ::: In medical imaging, physicians often estimate a parameter of interest (eg, cardiac ejection fraction) for a patient to assist in establishing a diagnosis. Many different estimation methods may exist, but rarely can one be considered a gold standard. Therefore, evaluation and comparison of different estimation methods are difficult. The purpose of this study was to examine a method of evaluating different estimation methods without use of a gold standard.
---
paper_title: Objective Comparison of Quantitative Imaging Modalities Without the Use of a Gold Standard

paper_content:

Imaging is often used for the purpose of estimating the value of some parameter of interest. For example, a cardiologist may measure the ejection fraction (EF) of the heart in order to know how much blood is being pumped out of the heart on each stroke. In clinical practice, however, it is difficult to evaluate an estimation method because the gold standard is not known, e.g., a cardiologist does not know the true EF of a patient. Thus, researchers have often evaluated an estimation method by plotting its results against the results of another (more accepted) estimation method, which amounts to using one set of estimates as the pseudogold standard. In this paper, we present a maximum-likelihood approach for evaluating and comparing different estimation methods without the use of a gold standard with specific emphasis on the problem of evaluating EF estimation methods. Results of numerous simulation studies will be presented and indicate that the method can precisely and accurately estimate the parameters of a regression line without a gold standard, i.e., without the x axis.
---
paper_title: A New Biometrical Procedure for Testing the Equality of Measurements from Two Different Analytical Methods. Application of linear regression procedures for method comparison studies in Clinical Chemistry, Part I

paper_content:

Procedures for the statistical evaluation of method comparisons and instrument tests often have a requirement for distributional properties of the experimental data, but this requirement is frequently not met. In our paper we propose a new linear regression procedure with no special assumptions regarding the distribution of the samples and the measurement errors. The result does not depend on the assignment of the methods (instruments) to X and Y. After testing a linear relationship between X and Y confidence limits are given for the slope beta and the intercept alpha; they are used to determine whether there is only a chance difference between beta and 1 and between alpha and 0. The mathematical background is amplified separately in an appendix.
---
paper_title: Modelling method comparison data

paper_content:

We explore a range of linear regression models that might be useful for either: (a) the relative calibration of two or more methods or (b) to evaluate their precisions relative to each other. Ideally, one should be able to use a single data set to carry out the jobs (a) and (b) together. Throughout this review we consider the constraints (assumptions) needed to attain identifiability of the models and the possible pitfalls to the unwary in having to introduce them. We also pay particular attention to the possible problems arising from the presence of random matrix effects (reproducible random measurement 'errors' that are characteristic of a given method when being used on a given specimen or sample, i.e. specimen specific biases or subject by method interactions). Finally, we stress the importance of a fully-informative design (using replicate measurements on each subject using at least three independent methods) and large sample sizes.
---
paper_title: Regression Models for Method Comparison Data

paper_content:

Regression methods for the analysis of paired measurements produced by two fallible assay methods are described and their advantages and pitfalls discussed. The difficulties for the analysis, as in any errors-in-variables problem lies in the lack of identifiability of the model and the need to introduce questionable and often naive assumptions in order to gain identifiability. Although not a panacea, the use of instrumental variables and associated instrumental variable (IV) regression methods in this area of application has great potential to improve the situation. Large samples are frequently needed and two-phase sampling methods are introduced to improve the efficiency of the IV estimators.
---
paper_title: A missing information principle: theory and applications

paper_content:

Abstract : The problem that a relatively simple analysis is changed into a complex one just because some of the information is missing, is one which faces most practicing statisticians at some point in their career. Obviously the best way to treat missing information problems is not to have them. Unfortunately circumstances arise in which information is missing and nothing can be done to replace it for one reason or another.
---
paper_title: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

paper_content:

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples...
---
paper_title: Statistical methods in assessing agreement: Models, issues, and tools

paper_content:

Measurements of agreement are needed to assess the acceptability of a new or generic process, methodology, and formulation in areas of laboratory performance, instrument or assay validation, method comparisons, statistical process control, goodness of fit, and individual bioequivalence. In all of these areas, one needs measurements that capture a large proportion of data that are within a meaningful boundary from target values. Target values can be considered random (measured with error) or fixed (known), depending on the situation. Various meaningful measures to cope with such diverse and complex situations have become available only in the last decade. These measures often assume that the target values are random. This article reviews the literature and presents methodologies in terms of “coverage probability.” In addition, analytical expressions are introduced for all of the aforementioned measurements when the target values are fixed and when the error structure is homogenous or heterogeneous (proport...
---
paper_title: Forming inferences about some intraclass correlation coefficients.

paper_content:

Although intraclass correlation coefficients (ICCs) are commonly used in behavioral measurement, psychometrics, and behavioral genetics, procedures available for forming inferences about ICCs are not widely known. Following a review of the distinction between various forms of the ICC, this article presents procedures available for calculating confidence intervals and conducting tests on ICCs developed using data from one-way and two-way random and mixed-effect analysis of variance models.
---
paper_title: Measuring agreement in method comparison studies

paper_content:

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference +/- 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.
---
paper_title: Comparison of ICC and CCC for assessing agreement for data without and with replications

paper_content:

The intraclass correlation coefficient (ICC) has been traditionally used for assessing reliability between multiple observers for data with or without replications. Definitions of different versions of ICCs depend on the assumptions of specific ANOVA models. The parameter estimator for the ICC is usually based on the method of moments with the underlying assumed ANOVA model. This estimator is consistent only if the ANOVA model assumptions hold. Often these ANOVA assumptions are not met in practice and researchers may compute these estimates without verifying the assumptions. ICC is biased if the ANOVA assumptions are not met. We compute the expected value of the ICC estimator under a very general model to get a sense of the population parameter that the ICC estimator provides. We compare this expected value to another popular agreement index, concordance correlation coefficient (CCC), which is defined without ANOVA assumptions. The main findings are reported for data without replication and with replications for three types of ICCs defined by one-way ANOVA model, two-way ANOVA model without interaction and two-way ANOVA model with interaction. A blood pressure example is used for illustration. If the ICC is the choice of agreement index, we recommend to use ICC"3 over other ICCs as its estimate is similar to the estimate of CCC regardless whether the ANOVA assumptions are met or not.
---
paper_title: Statistical methods in assessing agreement: Models, issues, and tools

paper_content:

Measurements of agreement are needed to assess the acceptability of a new or generic process, methodology, and formulation in areas of laboratory performance, instrument or assay validation, method comparisons, statistical process control, goodness of fit, and individual bioequivalence. In all of these areas, one needs measurements that capture a large proportion of data that are within a meaningful boundary from target values. Target values can be considered random (measured with error) or fixed (known), depending on the situation. Various meaningful measures to cope with such diverse and complex situations have become available only in the last decade. These measures often assume that the target values are random. This article reviews the literature and presents methodologies in terms of “coverage probability.” In addition, analytical expressions are introduced for all of the aforementioned measurements when the target values are fixed and when the error structure is homogenous or heterogeneous (proport...
---
paper_title: STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT

paper_content:

In clinical measurement comparison of a new measurement technique with an established one is often needed to see whether they agree sufficiently for the new to replace the old. Such investigations are often analysed inappropriately, notably by using correlation coefficients. The use of correlation is misleading. An alternative approach, based on graphical techniques and simple calculations, is described, together with the relation between this analysis and the assessment of repeatability.
---
paper_title: Five-year lung cancer screening experience: CT appearance, growth rate, location, and histologic features of 61 lung cancers.

paper_content:

PURPOSE ::: To retrospectively evaluate the computed tomography (CT)-determined size, morphology, location, morphologic change, and growth rate of incidence and prevalence lung cancers detected in high-risk individuals who underwent annual chest CT screening for 5 years and to evaluate the histologic features and stages of these cancers. ::: ::: ::: MATERIALS AND METHODS ::: The study was institutional review board approved and HIPAA compliant. Informed consent was waived. CT scans of 61 cancers (24 in men, 37 in women; age range, 53-79 years; mean, 65 years) were retrospectively reviewed for cancer size, morphology, and location. Forty-eight cancers were assessed for morphologic change and volume doubling time (VDT), which was calculated by using a modified Schwartz equation. Histologic sections were retrospectively reviewed. ::: ::: ::: RESULTS ::: Mean tumor size was 16.4 mm (range, 5.5-52.5 mm). Most common CT morphologic features were as follows: for bronchioloalveolar carcinoma (BAC) (n = 9), ground-glass attenuation (n = 6, 67%) and smooth (n = 3, 33%), irregular (n = 3, 33%), or spiculated (n = 3, 33%) margin; for non-BAC adenocarcinomas (n = 25), semisolid (n = 11, 44%) or solid (n = 12, 48%) attenuation and irregular margin (n = 14, 56%); for squamous cell carcinoma (n = 14), solid attenuation (n = 12, 86%) and irregular margin (n = 10, 71%); for small cell or mixed small and large cell neuroendocrine carcinoma (n = 7), solid attenuation (n = 6, 86%) and irregular margin (n = 5, 71%); for non-small cell carcinoma not otherwise specified (n = 5), solid attenuation (n = 4, 80%) and irregular margin (n = 3, 60%); and for large cell carcinoma (n = 1), solid attenuation and spiculated shape (n = 1, 100%). Attenuation most often (in 12 of 21 cases) increased. Margins most often (in 16 of 20 cases) became more irregular or spiculated. Mean VDT was 518 days. Thirteen of 48 cancers had a VDT longer than 400 days; 11 of these 13 cancers were in women. ::: ::: ::: CONCLUSION ::: Overdiagnosis, especially in women, may be a substantial concern in lung cancer screening.
---
paper_title: Forming inferences about some intraclass correlation coefficients.

paper_content:

Although intraclass correlation coefficients (ICCs) are commonly used in behavioral measurement, psychometrics, and behavioral genetics, procedures available for forming inferences about ICCs are not widely known. Following a review of the distinction between various forms of the ICC, this article presents procedures available for calculating confidence intervals and conducting tests on ICCs developed using data from one-way and two-way random and mixed-effect analysis of variance models.
---
paper_title: On Estimating Precision of Measuring Instruments and Product Variability

paper_content:

Abstract A measurement or observed value is discussed as the sum of two components—one the absolute value of the characteristic measured and the other an error of measurement. The variation in absolute values of the characteristic or items measured is termed product variability, whereas the variation in errors of measurement of an instrument is called the precision or reproducibility of measurement. Techniques are given for separating and estimating product variability and precision of measurement. Comparisons of the various techniques are also discussed for cases involving two or more instruments.
---
paper_title: Measuring agreement in method comparison studies

paper_content:

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference +/- 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.
---
paper_title: Comparison of ICC and CCC for assessing agreement for data without and with replications

paper_content:

The intraclass correlation coefficient (ICC) has been traditionally used for assessing reliability between multiple observers for data with or without replications. Definitions of different versions of ICCs depend on the assumptions of specific ANOVA models. The parameter estimator for the ICC is usually based on the method of moments with the underlying assumed ANOVA model. This estimator is consistent only if the ANOVA model assumptions hold. Often these ANOVA assumptions are not met in practice and researchers may compute these estimates without verifying the assumptions. ICC is biased if the ANOVA assumptions are not met. We compute the expected value of the ICC estimator under a very general model to get a sense of the population parameter that the ICC estimator provides. We compare this expected value to another popular agreement index, concordance correlation coefficient (CCC), which is defined without ANOVA assumptions. The main findings are reported for data without replication and with replications for three types of ICCs defined by one-way ANOVA model, two-way ANOVA model without interaction and two-way ANOVA model with interaction. A blood pressure example is used for illustration. If the ICC is the choice of agreement index, we recommend to use ICC"3 over other ICCs as its estimate is similar to the estimate of CCC regardless whether the ANOVA assumptions are met or not.
---
paper_title: Assessing reproducibility by the within-subject coefficient of variation with random effects models.

paper_content:

In this paper we consider the use of within-subject coefficient of variation (WCV) for assessing the reproducibility or reliability of a measurement. Application to assessing reproducibility of biochemical markers for measuring bone turnover is described and the comparison with intraclass correlation is discussed. Both maximum likelihood and moment confidence intervals of WCV are obtained through their corresponding asymptotic distributions. Normal and log-normal cases are considered. In general, WCV is preferred when the measurement scale bears intrinsic meaning and is not subject to arbitrary shifting. The intraclass correlation may be preferred when a fixed population of subjects can be well identified.
---
paper_title: Testing Statistical Hypotheses of Equivalence

paper_content:

Introduction. Methods for One-Sided Equivalence Problems. General Approaches to the Construction of Tests for Equivalence in the Strict Sense. Equivalence Tests for Selected One-Parameter Problems. Equivalence Tests for Designs with Paired Observation. Equivalence Tests for Two Unrelated Samples. Multi-sample Tests for Equivalence. Tests for Establishing Goodness of Fit. The Assessment of Bio-Equivalence. Appendix. References. Index
---
paper_title: Measuring agreement in method comparison studies

paper_content:

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference +/- 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.
---
paper_title: Regression Models for Method Comparison Data

paper_content:

Regression methods for the analysis of paired measurements produced by two fallible assay methods are described and their advantages and pitfalls discussed. The difficulties for the analysis, as in any errors-in-variables problem lies in the lack of identifiability of the model and the need to introduce questionable and often naive assumptions in order to gain identifiability. Although not a panacea, the use of instrumental variables and associated instrumental variable (IV) regression methods in this area of application has great potential to improve the situation. Large samples are frequently needed and two-phase sampling methods are introduced to improve the efficiency of the IV estimators.
---
paper_title: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

paper_content:

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples...
---
paper_title: The Problem of Conversion in Method Comparison Studies

paper_content:

SUMMARY This paper introduces an extension of the well-known method comparison problem previously studied by Altman and Bland and others. The problem concerns the comparison of two approximate methods of measuring a quantity which return results expressed in different units. Such situations can arise where the methods in question proceed by measuring proxy quantities. Thus the data sets need to be converted to the same units before comparisons may take place. The paper presents an approach to this problem based on the theory of structural relationship models. This approach is applied to data gathered by two approximate methods of measuring the fuel consumption of motor vehicles.
---
paper_title: Confidence Intervals on Nonnegative Linear Combinations of Variances

paper_content:

Abstract Smith (1936) suggested a method that can be used for setting confidence limits on linear combinations of variances. This method was studied and expanded by Satterthwaite (1941, 1946) and has become known as Satterthwaite's procedure. The procedure has been widely used for the past 40 years. In this article a new procedure is proposed for this problem that is better than Satterthwaite's procedure and very easy to compute from existing tables.
---
paper_title: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions

paper_content:

The development and implementation of quantitative imaging biomarkers has been hampered by the inconsistent and often incorrect use of terminology related to these markers. Sponsored by the Radiological Society of North America, an interdisciplinary group of radiologists, statisticians, physicists, and other researchers worked to develop a comprehensive terminology to serve as a foundation for quantitative imaging biomarker claims. Where possible, this working group adapted existing definitions derived from national or international standards bodies rather than invent new definitions for these terms. This terminology also serves as a foundation for the design of studies that evaluate the technical performance of quantitative imaging biomarkers and for studies of algorithms that generate the quantitative imaging biomarkers from clinical scans. This paper provides examples of research studies and quantitative imaging biomarker claims that use terminology consistent with these definitions as well as examples...
---
paper_title: Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets.

paper_content:

Repeat imaging data sets performed on patients with cancer are becoming publicly available. The potential utility of these data sets for addressing important questions in imaging biomarker development is vast. In particular, these data sets may be useful to help characterize the variability of quantitative parameters derived from imaging. This article reviews statistical analysis that may be performed to use results of repeat imaging to 1) calculate the level of change in parameter value that may be seen in individual patients to confidently characterize that patient as showing true parameter change, 2) calculate the level of change in parameters value that may be seen in individual patients to confidently categorize that patient as showing true lack of parameter change, 3) determine if different imaging devices are interchangeable from the standpoint of repeatability, and 4) estimate the numbers of patients needed to precisely calculate repeatability. In addition, we recommend a set of statistical parameters that should be reported when the repeatability of continuous parameters is studied.
---
paper_title: Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science.

paper_content:

OBJECTIVE ::: The consensus of opinion suggests that when assessing measurement agreement, the most appropriate statistic to report is the "95% limits of agreement". The precise form that this interval takes depends on whether a positive relation exists between the differences in measurement methods (errors) and the size of the measurements--that is, heteroscedastic errors. If a positive and significant relation exists, the recommended procedure is to report "the ratio limits of agreement" using log transformed measurements. This study assessed the prevalence of heteroscedastic errors when investigating measurement agreement of variables recorded on a ratio scale in sports medicine and sports science. ::: ::: ::: METHODS ::: Measurement agreement (or repeatability) was assessed in 13 studies (providing 23 examples) conducted in the Centre for Sport and Exercise Sciences at Liverpool John Moores University over the past five years. ::: ::: ::: RESULTS ::: The correlation between the absolute differences and the mean was positive in all 23 examples (median r = 0.37), eight being significant (P < 0.05). In 21 of 23 examples analysed, the correlation was greater than the equivalent correlation using log transformed measurements (median r = 0.01). Based on a simple meta-analysis, the assumption that no relation exists between the measurement differences and the size of measurement must be rejected (P < 0.001). ::: ::: ::: CONCLUSIONS ::: When assessing measurement agreement of variables recorded on a ratio scale in sports medicine and sports science, this study (23 examples) provides strong evidence that heteroscedastic errors are the norm. If the correlation between the absolute measurement differences and the means is positive (but not necessarily significant) and greater than the equivalent correlation using log transformed measurements, the authors recommend reporting the "ratio limits of agreement".
---

Title: Quantitative Imaging Biomarkers: A Review of Statistical Methods for Computer Algorithm Comparisons

Section 1: Background and problem statement
Description 1: This section introduces the utility of medical imaging in clinical settings and defines quantitative imaging biomarkers, outlining the focus of the paper on computer algorithms used to generate QIBs and the challenges associated with their evaluation and comparison.

Section 2: Study design issues for QIB algorithm comparisons
Description 2: This section discusses the various design considerations for comparing QIB algorithms, including the types of studies, the importance of selecting appropriate measurement types, and the challenges related to human involvement and lack of true reference values.

Section 3: Testing superiority
Description 3: This section details the statistical hypothesis testing framework for demonstrating the superiority of a new or upgraded QIB algorithm over a standard algorithm, including one-sided hypothesis tests and confidence interval approaches.

Section 4: Testing equivalence
Description 4: This section explains the procedures for conducting equivalence tests, where researchers define appropriate equivalence limits to compare the performance of QIB algorithms against predefined thresholds.

Section 5: Testing non-inferiority (NI)
Description 5: This section describes the methodology for demonstrating that a QIB algorithm is not inferior to a standard method by a predefined margin, often involving a stepwise approach to assess both non-inferiority and superiority.

Section 6: Evaluating performance when the true value or reference standard is present
Description 6: This section discusses methods for assessing the closeness between QIB algorithm results and true values or reference standards, covering both disaggregated and aggregated approaches to evaluate bias and precision.

Section 7: Evaluating performance in the absence of the true value
Description 7: This section reviews alternative approaches for assessing algorithm performance when true values are unavailable, including error-in-variable models and techniques that relax certain assumptions.

Section 8: Assessing bias with no clear reference standard
Description 8: This section considers how to evaluate algorithm performance when no clear reference standard exists, focusing on symmetric consideration of all QIB algorithms under study.

Section 9: Nonlinearity and heteroscedasticity
Description 9: This section addresses issues of heteroscedasticity and nonlinearity in the relationship between QIB measurements and the true values, and discusses nonparametric methods for dealing with these issues.

Section 10: Assessing the agreement among algorithms
Description 10: This section explains the methods for assessing whether measurements from different QIB algorithms are sufficiently close to be considered interchangeable, emphasizing both scaled and unscaled measures of agreement.

Section 11: Evaluating algorithm precision
Description 11: This section discusses methods for comparing the precision of QIB algorithms, including specific indices for repeatability and reproducibility and the appropriate statistical tests for these comparisons.

Section 12: Process for establishing the performance of QIB algorithms
Description 12: This section outlines a seven-step process for validating the technical performance of QIB algorithms for clinical use or regulatory approval, covering aspects from definition and variability assessment to statistical testing.

Section 13: Discussion and recommendations
Description 13: This section summarizes the key findings and provides recommendations for future research and development in the field of quantitative imaging biomarkers, emphasizing the importance of rigorous evaluation and validation methods.