A Review of Standards and Statistics Used to Describe Blood Glucose Monitor Performance
---
paper_title: Criteria for Judging Precision and Accuracy in Method Development and Evaluation

paper_content:

We describe an approach for formulating criteria that can be used to judge whether an analytical method has acceptable precision and accuracy. We derive criteria for several experiments that are commonly used in method-evaluation studies: precision or replicates, recovery, interference, and comparison of patient values between the new method and a proven method. These criteria are based on the medical usefulness of the test results, thus the acceptability of the method is judged with respect to the clinical requirements.
---
paper_title: Recommendation to treat continuous variable errors like attribute errors

paper_content:

Clinical laboratory errors can be considered as either belonging to attribute or continuous variables. Attribute errors are usually considered to be pre- or post-analytical errors, whereas continuous variable errors are analytical. Goals for each error type are different. Error goals for continuous variables are often specified as limits that contain 95% of the results, whereas attribute error goals are specified as allowed error rates for serious events. This leads to a discrepancy, because for a million results, there can be up to 50,000 medically unacceptable analytical errors, but allowable pre- and post-analytical error rates are much lower than 5%. Steps to remedy this are to classify analytical error rates into severity categories, exemplified by existing glucose error grids. The results in each error grid zone are then counted, as has been recommended by the Food and Drug Administration (FDA). This in effect transforms the continuous variable errors into attribute errors. This is an improvement over current practices for analytical errors, whereby the use of uncertainty intervals is recommended that include only 95% of the results (i.e., leaves out the worst 5%), and it is precisely this 5% of results that are likely to be in the most severe zones of an error grid.
---
paper_title: Criteria for Judging Precision and Accuracy in Method Development and Evaluation

paper_content:

We describe an approach for formulating criteria that can be used to judge whether an analytical method has acceptable precision and accuracy. We derive criteria for several experiments that are commonly used in method-evaluation studies: precision or replicates, recovery, interference, and comparison of patient values between the new method and a proven method. These criteria are based on the medical usefulness of the test results, thus the acceptability of the method is judged with respect to the clinical requirements.
---
paper_title: Instruments for Self-Monitoring of Blood Glucose: Comparisons of Testing Quality Achieved by Patients and a Technician

paper_content:

Background: Instruments for self-monitoring of blood glucose (SMBG) are increasingly used by patients with diabetes. The analytical quality of meters in routine use is poorly characterized. ::: ::: Methods: We compared SMBG performance achieved by patients and by a medical laboratory technician. Imprecision was calculated from duplicate measurements, and deviation as the difference between the first measurement and the mean of duplicate laboratory-method results (calibrated with NIST material). Analytical quality for five groups of SMBG instruments was compared with quality specifications for BG measurements. All participants completed a questionnaire assessing both SMBG training and use of the meters. ::: ::: Results: We recruited 159 SMBG users from a hospital outpatient clinic and 263 others from 65 randomly selected general practices (total of 422). Most (two thirds) used insulin. CVs for the five meter types were 7%, 11%, 18%, 18%, and 20% in the hands of patients and 2.5–5.9% for the technician. For three of five meter types, patients’ BG measurements had larger deviations from the laboratory results than did the technician’s results. The technician’s performance could not predict the patients’. No instrument when used by patients (but two operated by the technician) met published quality specifications. The analytical quality of patients’ results was not related to whether they had chosen the instruments on advice from healthcare personnel (one-third of patients), were only self-educated in SMBG (50%), or performed SMBG fewer than seven times/week (62%). ::: ::: Conclusions: The analytical quality of SMBG among patients was poorer than, and could not be predicted from, the performance of the meters in the hands of a technician. We suggest that new instruments be tested in the hands of patients who are trained on meter use in a routine way.
---
paper_title: Assuring the accuracy of home glucose monitoring.

paper_content:

BACKGROUND An estimated 2.5 million diabetic patients in the United States practice self-monitoring of blood glucose (SMBG). The validity of the glucose values they obtain is in doubt. An American Diabetes Association consensus panel reported that up to 50% of SMBG determinations might vary more than 20% from their true value. Accurate glucose values are an integral part of intensive treatment and reduction of long-term complications. The objective of this study was to determine the technical skill and accuracy of SMBG in an outpatient population. METHODS This study was conducted in two family practice residency sites where 111 patients with type 1 and type 2 adult diabetes were observed testing their blood glucose values on their own glucose monitors. Patient-measured glucose levels were immediately compared with a laboratory value obtained from a calibrated hand-held glucose monitor. RESULTS Fifty-three percent of patient glucose values were within 10% of the control value, 84% were within 20% of the control value, and 16% varied 20% or more from the control value. Two patients had dangerously inaccurate glucose determinations. Four glucose monitors required replacement. The patients were observed using a 13-point checklist of critical steps in calibration and operation of their glucose monitor. Only 1 patient made no errors in testing. CONCLUSIONS Despite multiple technical errors when using SMBG, most patients obtained clinically useful values. This project can be easily introduced into a medical office.
---
paper_title: A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose

paper_content:

OBJECTIVE ::: The objectives of this study were 1) to construct new error grids (EGs) for blood glucose (BG) self-monitoring by using the expertise of a large panel of clinicians and 2) to use the new EGs to evaluate the accuracy of BG measurements made by patients. ::: ::: ::: RESEARCH DESIGN AND METHODS ::: To construct new EGs for type 1 and type 2 diabetic patients, a total of 100 experts of diabetes were asked to assign any error in BG measurement to 1 of 5 risk categories. We used these EGs to evaluate the accuracy of self-monitoring of blood glucose (SMBG) levels in 152 diabetic patients. The SMBG data were used to compare the new type 1 diabetes EG with a traditional EG. ::: ::: ::: RESULTS ::: Both the type 1 and type 2 diabetes EGs divide the risk plane into 8 concentric zones with no discontinuities. The new EGs are similar to each other, but they differ from the traditional EG in several significant ways. When used to evaluate a data set of measurements made by a sample of patients experienced in SMBG, the new type 1 diabetes EG rated 98.6% of their measurements as clinically acceptable, compared with 95% for the traditional EG. ::: ::: ::: CONCLUSIONS ::: The consensus EGs furnish a new tool for evaluating errors in the measurement of BG for patients with type 1 and type 2 diabetes.
---
paper_title: Toward metrological traceability in the determination of prostate-specific antigen (PSA): calibrating Beckman Coulter Hybritech Access PSA assays to WHO standards compared with the traditional Hybritech standards

paper_content:

Background: The metrological traceability of prostate-specific antigen (PSA) assay calibration to WHO standards is desirable to potentially improve the comparability between PSA assays. A method comparison was performed between the traditionally standardized Beckman Coulter Hybritech Access PSA and free PSA (fPSA) assays and a new alternate calibration of assays aligned to the WHO standards 96/670 and 96/668, respectively. Methods: Sera from 641 men with and without prostate cancer, various control materials and mixtures of different proportions of the WHO standards were measured with both assay calibrations. Results: Excellent comparability between the corresponding assay calibrations was observed, with correlation coefficients of at least 0.996. The Passing-Bablok slopes were 0.747 for total PSA (tPSA), 0.776 for fPSA and 1.02 for the percentage ratio of fPSA to tPSA (%fPSA), while the corresponding percentages of the new WHO-aligned assay results related to the traditional assays were 76.2%, 77% and 102.2%. Receiver operating characteristics revealed no differences between the two PSA assay calibrations. Conclusions: The WHO calibration yields results approximately 25% lower for tPSA and fPSA values when compared with the conventional Hybritech calibration. Using the WHO-aligned PSA assay, a tPSA cut-off of 3 μg/L should be considered in clinical practice, while %fPSA cut-offs could be retained.
---
paper_title: Multi-factor designs. IV. How multi-factor designs improve the estimate of total error by accounting for protocol-specific biases

paper_content:

Total error is often calculated as a combination of random error and fixed bias. However, the specific protocols used to estimate random error and fixed bias are themselves variable factors that can affect the estimate of total error. We refer to biases such as assay drift, sample-to-sample carryover, and reagent carryover as examples of fixed biases that are protocol-specific and distinguish them from other fixed biases. Failing to account for protocol-specific biases that are present will lead to incorrect estimates of total error when routine use of the assay involves a protocol different from that used to estimate total error. Multi-factor protocols are recommended to determine protocol-specific biases, which, if present, should be included in the estimate of total error.
---
paper_title: Estimating total analytical error and its sources. Techniques to improve method evaluation.

paper_content:

The process of method evaluation starts with identifying goals either to demonstrate the clinical validity of an assay or to identify assay error sources that require improvement. Taguchi's idea of continual quality improvement vs the notion of meeting or failing specification has been applied to clinical chemistry. In this article, I propose a model of assay performance that includes the terms random interferences and protocol-specific biases (a series of systematic errors). I explain these terms, as well as the consequences of failing to consider them. To validate an assay clinically, I recommend direct estimation of total analytical error from a method comparison. To identify assay error sources that require improvement, I recommend a multifactor protocol (in addition to a method comparison). Individual error sources are related to total analytical error with the use of an error propagation technique. Much of the proposed data analysis techniques are straightforward but not routinely practiced. I demonstrate principles with the use of a cholesterol assay.
---
paper_title: Glucose: A Simple Molecule That Is Not Simple to Quantify

paper_content:

Small increments in blood glucose substantially increase the risk of developing diabetes mellitus; but preanalytical and analytical variables, such as the absence of harmonization for glucose assays, make it difficult to correctly apply these epidemiological insights to individual patients. Harmonization can be improved if 3 variables are addressed: changing proficiency test grading from consensus based to accuracy based, effectively controlling glycolysis, and taking into account the time of day blood was collected. ::: ::: The continuous and graded quantitative relationship of fasting glucose measurements to the risk of developing diabetes was well documented recently by Tirosh et al.(1). They found an increased risk of type 2 diabetes across quintiles of fasting plasma glucose (FPG) concentrations within the newly defined reference range, <5.55 mmol/L (<100 mg/dL). For example, a person with an FPG between 4.83 and 5.00 mmol/L (87 and 90 mg/dL) has an age-adjusted risk of developing diabetes that is 1.81 times that of a person with an FPG <4.55 mmol/L (82 mg/dL; 95% CI 1.16–2.83). Thus, a difference as small as 0.28 mmol/L (5 mg/dL) nearly doubles the risk. Higher concentrations of FPG …
---
paper_title: Performance of Four Homogeneous Direct Methods for LDL-Cholesterol

paper_content:

Background: Homogeneous LDL-cholesterol methods from Genzyme, Reference Diagnostics, Roche, and Sigma were evaluated for precision, accuracy, and specificity for LDL in the presence of abnormal lipoproteins. ::: ::: Methods: Each homogeneous method was performed by a Roche/Hitachi 911 according to the vendors’ instructions, and the results were compared with the β-quantification reference method. We measured precision over 20 days using quality-control and frozen serum specimens. Sera from 100 study participants, including 60 with hyperlipidemias, were assayed by each method. Accuracy was evaluated from regression and total error analysis. Specificity was evaluated from the bias (as a percentage) vs concentration of triglycerides. ::: ::: Results: The total CV was 12% bias were as follows: Genzyme, 8.0%; Reference Diagnostics, 11.0%; Roche, 10.0%; and Sigma, 30.0%. Total error calculated from mean systematic bias and all-sources random bias was as follows: Genzyme, 12.6%; Reference Diagnostics, 16.5%; Roche, 41.6%; and Sigma, 38.3%. Slopes of bias (as a percentage) vs triglycerides were P <0.001 for all methods except the Roche method, which was P = 0.094. ::: ::: Conclusions: The evaluated methods show nonspecificity toward abnormal lipoproteins, thus compromising their ability to satisfy the National Cholesterol Education Program goal for a total error of <12%. These homogeneous LDL-cholesterol results do not improve on the performance of LDL-cholesterol calculated by the Friedewald equation at triglyceride concentrations <4000 mg/L.
---
paper_title: How to Improve Total Error Modeling by Accounting for Error Sources Beyond Imprecision and Bias

paper_content:

Boyd and Bruns (1) have used Monte Carlo simulations to assess glucose meter specifications. This letter suggests that their modeling methods do not account for all possible error types and thus their conclusions may not follow. A more realistic modeling method is reviewed as well as an alternative to modeling. ::: ::: The error simulation method chosen for glucose by Boyd and Bruns (1) was also used in principle to generate the National Cholesterol Education Program goals for cholesterol analytical performance (2). Boyd and Bruns (1) generate glucose error by adding various levels of assay imprecision to various levels of assay bias. This method is intuitively appealing as a way of simulating total analytical error. Although the method is a simulation, it is helpful to consider how the data relate to an actual experiment. Thus, if one measures glucose in a set of patient specimens in both a field and a reference method, one can imagine obtaining a bias as …
---
paper_title: Guidelines and recommendations for laboratory analysis in the diagnosis and management of diabetes mellitus.

paper_content:

BACKGROUND ::: Multiple laboratory tests are used in the diagnosis and management of patients with diabetes mellitus. The quality of the scientific evidence supporting the use of these assays varies substantially. ::: ::: ::: APPROACH ::: An expert committee drafted evidence-based recommendations for the use of laboratory analysis in patients with diabetes. An external panel of experts reviewed a draft of the guidelines, which were modified in response to the reviewers' suggestions. A revised draft was posted on the Internet and was presented at the AACC Annual Meeting in July, 2000. The recommendations were modified again in response to oral and written comments. The guidelines were reviewed by the Professional Practice Committee of the American Diabetes Association. ::: ::: ::: CONTENT ::: Measurement of plasma glucose remains the sole diagnostic criterion for diabetes. Monitoring of glycemic control is performed by the patients, who measure their own plasma or blood glucose with meters, and by laboratory analysis of glycated hemoglobin. The potential roles of noninvasive glucose monitoring, genetic testing, autoantibodies, microalbumin, proinsulin, C-peptide, and other analytes are addressed. ::: ::: ::: SUMMARY ::: The guidelines provide specific recommendations based on published data or derived from expert consensus. Several analytes are of minimal clinical value at the present time, and measurement of them is not recommended.
---
paper_title: Statistical Comparison of Multiple Analytic Procedures: Application to Clinical Chemistry

paper_content:

The basic sciences all require an ability to measure the amounts of substances under study. With new methods of measurement constantly being proposed there is a need for techniques for comparing these methods in terms of their precision and accuracy. Of particular interest is the case in which none of the individual methods are known to measure “truth”. A multiple methods comparison technique for this case is proposed in this paper, and is illustrated by an example from the field of clinical chemistry. Estimates of the components of variance for each method are developed, and some of their properties explored.
---
paper_title: Performance of Four Homogeneous Direct Methods for LDL-Cholesterol

paper_content:

Background: Homogeneous LDL-cholesterol methods from Genzyme, Reference Diagnostics, Roche, and Sigma were evaluated for precision, accuracy, and specificity for LDL in the presence of abnormal lipoproteins. ::: ::: Methods: Each homogeneous method was performed by a Roche/Hitachi 911 according to the vendors’ instructions, and the results were compared with the β-quantification reference method. We measured precision over 20 days using quality-control and frozen serum specimens. Sera from 100 study participants, including 60 with hyperlipidemias, were assayed by each method. Accuracy was evaluated from regression and total error analysis. Specificity was evaluated from the bias (as a percentage) vs concentration of triglycerides. ::: ::: Results: The total CV was 12% bias were as follows: Genzyme, 8.0%; Reference Diagnostics, 11.0%; Roche, 10.0%; and Sigma, 30.0%. Total error calculated from mean systematic bias and all-sources random bias was as follows: Genzyme, 12.6%; Reference Diagnostics, 16.5%; Roche, 41.6%; and Sigma, 38.3%. Slopes of bias (as a percentage) vs triglycerides were P <0.001 for all methods except the Roche method, which was P = 0.094. ::: ::: Conclusions: The evaluated methods show nonspecificity toward abnormal lipoproteins, thus compromising their ability to satisfy the National Cholesterol Education Program goal for a total error of <12%. These homogeneous LDL-cholesterol results do not improve on the performance of LDL-cholesterol calculated by the Friedewald equation at triglyceride concentrations <4000 mg/L.
---
paper_title: Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program

paper_content:

We have assessed the laboratory specifications necessary for ensuring that cholesterol testing processes satisfy the quality required by the U.S. National Cholesterol Education Program (NCEP). A model for setting process specifications has been developed to relate the NCEP guidelines for medical interpretation of a cholesterol test to the pre-analytical and analytical variables that can affect a test result. Using this model, we derived specifications for the imprecision (coefficient of variation, CV, or standard deviation, s) and inaccuracy (bias) that are allowable under stable operation, as well as the quality-control procedures (control rules and number of control measurements) that are necessary to detect unstable operation. The NCEP goals of an allowable CV less than or equal to 3% and an allowable bias no greater than +/- 3% are inadequate for assuring the quality of an individual or single cholesterol test when monitoring performance with many of the statistical quality-control procedures currently used in the U.S. With quality-control procedures having two control measurements per run, a CV of 3% is allowable only when bias is zero; a CV less than or equal to 2% is necessary if bias is +/- 3%. With quality-control procedures having four control measurements per run, a CV of 3% is allowable when bias is +/- 1.5%; a CV less than or equal to 2.5% is required if bias is as large as +/- 3%. For two serial tests, the NCEP 3% goals are adequate for current quality-control procedures having four control measurements per run.
---
paper_title: Estimating total analytical error and its sources. Techniques to improve method evaluation.

paper_content:

The process of method evaluation starts with identifying goals either to demonstrate the clinical validity of an assay or to identify assay error sources that require improvement. Taguchi's idea of continual quality improvement vs the notion of meeting or failing specification has been applied to clinical chemistry. In this article, I propose a model of assay performance that includes the terms random interferences and protocol-specific biases (a series of systematic errors). I explain these terms, as well as the consequences of failing to consider them. To validate an assay clinically, I recommend direct estimation of total analytical error from a method comparison. To identify assay error sources that require improvement, I recommend a multifactor protocol (in addition to a method comparison). Individual error sources are related to total analytical error with the use of an error propagation technique. Much of the proposed data analysis techniques are straightforward but not routinely practiced. I demonstrate principles with the use of a cholesterol assay.
---
paper_title: STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT

paper_content:

In clinical measurement comparison of a new measurement technique with an established one is often needed to see whether they agree sufficiently for the new to replace the old. Such investigations are often analysed inappropriately, notably by using correlation coefficients. The use of correlation is misleading. An alternative approach, based on graphical techniques and simple calculations, is described, together with the relation between this analysis and the assessment of repeatability.
---
paper_title: A Simple, Graphical Method to Evaluate Laboratory Assays

paper_content:

Evaluation methods of laboratory assays often fail to predict the large, infrequent errors that are a major source of clinician complaints. We present a simple, graphical method to evaluate laboratory assays, which focuses on detecting large, infrequent errors. Our method, the folded empirical cumulative distribution plot or, more simply, mountain plot, is prepared by computing a percentile for each ranked difference between the new and reference method. To get a folded plot, one performs the following subtraction for all percentiles over 50 : percentile = 100 - percentile. Percentiles (y axis) are then plotted against differences or percent differences (x axis). The calculations and plots are simple enough to perform in a spreadsheet. We also offer Windows based software to perform all calculations and plots. The mountain plot compared to the difference plot focuses attention on two features of the data : the center and the tails. We prefer the mountain plot over other graphical techniques because : 1. It is easier to find the central 95% of the data. 2. It is easier to estimate percentiles for large differences (e.g., percentiles greater than 95%). 3. Unlike a histogram, the plot shape is not a function of the intervals. 4. Comparing different distributions is easier. 5. The plot is easier to interpret than a standard empirical cumulative distribution plot. Difference and moutain plots each provide complementary perspectives on the data. We recommend both plots. This method can also be used with data from a wide variety of other applications, such as clinical trials and quality control.
---
paper_title: Misuse of correlation and regression in three medical journals

paper_content:

Errors relating to the use of the correlation coefficient and bivariate linear regression are often to be found in medical publications. This paper reports a literature search to define the problems. All the papers and letters published in the British Medical Journal, The Lancet and the New England Journal of Medicine during 1997 were screened for examples. Fifteen categories of errors were identified of which eight were important or common. These included: failure to define clearly the relevant sample number; the display of potentially misleading scatterplots; attachment of unwarranted importance to significance levels; and the omission of confidence intervals for correlation coefficients and around regression lines.
---
paper_title: Evaluation of point-of-care glucose testing accuracy using locally-smoothed median absolute difference curves.

paper_content:

BACKGROUND ::: We introduce locally-smoothed (LS) median absolute difference (MAD) curves for the evaluation of hospital point-of-care (POC) glucose testing accuracy. ::: ::: ::: METHODS ::: Arterial blood samples (613) were obtained from a university hospital blood gas laboratory. Four hospital glucose meter systems (GMS) were tested against the YSI 2300 glucose analyzer for paired reference observations. We made statistical comparisons using conventional methods (e.g., linear regression, mean absolute differences). ::: ::: ::: RESULTS ::: Difference plots with superimposed ISO 15197 tolerance bands showed bias, scatter, heteroscedasticity, and erroneous results well. LS MAD curves readily revealed GMS accuracy patterns. Performance in hypoglycemic and hyperglycemic ranges erratically exceeded the recommended LS MAD error tolerance limit (5 mg/dl). Some systems showed acceptable (within LS MAD tolerance) or nearly acceptable performance in and around a tight glycemic control (TGC) interval of 80-110 mg/dl. Performance patterns varied in this interval, creating potential for discrepant therapeutic decisions. ::: ::: ::: CONCLUSIONS ::: Erroneous results demonstrated by ISO 15197-difference plots must be carefully considered. LS MAD curves draw on the unique human ability to recognize patterns quickly and discriminate accuracy visually. Performance standards should incorporate LS MAD curves and the recommended error tolerance limit of 5 mg/dl for hospital bedside glucose testing. Each GMS must be considered individually when assessing overall performance for therapeutic decision making in TGC.
---

Title: A Review of Standards and Statistics Used to Describe Blood Glucose Monitor Performance

Section 1: Introduction
Description 1: Introduce the importance of blood glucose testing in diabetes management and provide an overview of the different types and sources of errors associated with glucose measurements.

Section 2: Published Glucose Standards
Description 2: Detail the different standards published regarding glucose monitoring, including discussions on ISO limits and the implications of these standards.

Section 3: Standards That Specify 100% of Data
Description 3: Discuss standards from organizations like ADA that specify stringent error limits for glucose measurements and analyze the implications of these standards.

Section 4: Error Grid Details
Description 4: Explore the Clarke and Parkes error grids and other grids used for assessing glucose monitor accuracy, including their distinctions and applications.

Section 5: Analytical Error Sources
Description 5: Describe different sources of analytical error, including imprecision, patient interferences, and biases, and their contributions to total error.

Section 6: Methods Used to Estimate Total Error
Description 6: Review various methods for estimating total error in glucose measurements, including modeling, parametric, and nonparametric approaches.

Section 7: Westgard
Description 7: Describe the Westgard model for assessing glucose measurement errors and discuss its use and limitations.

Section 8: Boyd and Bruns
Description 8: Review the proposals by Boyd and Bruns for glucose requirements and the criticisms of their model.

Section 9: Lawton
Description 9: Explain the Lawton model for accounting for non-specificity and interferences in total error estimation.

Section 10: Bland-Altman
Description 10: Detail the Bland-Altman approach to comparing glucose measurement methods and its usability.

Section 11: Mountain Plots
Description 11: Describe mountain plots and their use in analyzing glucose measurement differences.

Section 12: Error Grids (Revisited)
Description 12: Revisit error grids as a method for estimating total error, emphasizing their practical applications and limitations.

Section 13: Methods That Estimate Total Error Components
Description 13: Discuss Clinical and Laboratory Standards Institute protocols developed for various analytical performance parameters related to glucose.

Section 14: Correlation Coefficient
Description 14: Explain the use and limitations of the correlation coefficient in the context of glucose measurement accuracy.

Section 15: Locally Smoothed Median Absolute Differences (LS MAD)
Description 15: Describe LS MAD curves and their application in evaluating performance over different glucose ranges.

Section 16: Total Error Evaluation Protocol
Description 16: Outline the protocols for estimating total error in routine use, emphasizing the need for comprehensive error assessments.

Section 17: Conclusions
Description 17: Summarize the paper's findings and argue for the need for comprehensive glucose specifications that encompass total error limits and detailed testing protocols.