This document provides guidance for studies intended to establish the within-site precision performance characteristics of quantitative measurement procedures used in clinical laboratories, and also for studies addressing site-to-site variability. This third edition of the standard narrows the scope of EP05 by limiting its discussion of single-site experimental designs to procedures suitable for establishing or validating precision performance characteristics. Therefore, EP05 is now addressed primarily to manufacturers and developers. The single-site protocol familiar from previous editions of EP05—calling for measurements on 20 days, with two runs per day and two replicates per run for a given sample, reagent lot, etc.
|Published (Last):||3 January 2004|
|PDF File Size:||15.32 Mb|
|ePub File Size:||12.46 Mb|
|Price:||Free* [*Free Regsitration Required]|
For labs seeking a quick check to insure their methods are meeting manufacturer specifications, EP15 may be the right choice. This is its fourth iteration, and although it retains much of its original approach, there were some significant changes in the A3 version. The most significant change is the creation of a relatively simple experiment that gives reliable estimates of a measurement procedure's imprecision and its bias. The essentials to accomplish this were present in EP15 through all of its previous versions, but they are refined and combined in EPA3 to make a single experiment.
Here's a brief description of the protocol. The user should ascertain that the imprecision of the candidate measurement procedure meets the criterion for allowable imprecision before beginning the evaluation.
If the measurement procedure's imprecision reported in publications, such as the manufacturer's stated imprecision, does not meet the criterion, the precision verification procedure described in EPA3 is not appropriate. EP15 first describes a precision verification experiment. If the user is evaluating a procedure for which there are manufacturer's precision claims, or published precision results, that were developed using CLSI EP5, the user can verify the published precision in an experiment lasting as few as five days.
Patient samples, reference materials, proficiency testing samples, or control materials may be used as the test samples, provided there is sufficient sample material for testing each sample five times per run for five to seven runs. Precision should be tested with two or more sample materials at different medical decision point concentrations.
The experiment produces at least 25 replicates collected over at least 5 days for each sample material. The repeatability previously termed "within-run" and the within-laboratory previously termed "total" standard deviations are calculated by an analysis of variance technique ANOVA that properly accounts for the within-run and between-run contributions to the overall imprecision of the measurement procedure.
The user needs access to software to do the ANOVA calculations, but they are available in Excel, Minitab, Analyze-it, and other software packages that do statistical calculations. The repeatability and within-laboratory standard deviations are then compared to the claimed or published standard deviations.
If the calculated standard deviations are less than the published values, the user has verified the claim. Sometimes the calculated standard deviations may exceed the published values, and yet the true standard deviations are less than the published values. For example, if the true standard deviations were actually exactly equal to their claimed counterparts, the calculated standard deviations would exceed their published counterparts fifty percent of the time in verification experiments.
To allow for this possibility, the user calculates a "verification limit" based on the published standard deviation and the size of the user's experiment.
If the calculated standard deviation is less than the verification limit, it is not statistically significantly larger than the published standard deviation, and the user has verified the published precision. If the calculated precision exceeds the verification limit, the calculated standard deviation is statistically significantly larger than the published standard deviation, and the user has failed to verify the published imprecision.
The document includes tables to simplify the calculation of the verification limit. Because the precision experiment has so many replicate measurements, collected over several days, results from the precision experiment may be used to make a reliable estimate of the bias of the measurement procedure relative to the assigned target values of the sample materials used in the experiment. The only requirement is that the assigned value must be available. The choice of material depends on the purpose of the user in estimating the bias.
Two or more appropriate materials should be tested in the precision experiment. If the user is interested in estimating bias relative to the peer group for proficiency testing, and wants to estimate how the measurement procedure will perform well on proficiency testing, proficiency testing materials with peer group values for the measurement procedure being evaluated are appropriate.
For bias relative to the quality control peer group, quality control materials with peer group values for the measurement procedure are appropriate. Typically, there is no way to estimate the uncertainty of the "assayed" values, which is needed to determine if the calculated bias is statistically significant.
Internationally recognized high order reference materials, such as a material from the U. National Institute of Standards and Technology, or from the Joint Committee for Traceability in Laboratory Medicine, or from similar organizations may be appropriate if the user wishes to estimate the bias relative to the assigned concentrations of such materials.
Use of these materials is important in establishing the traceability of measurement procedures. Patient samples or control materials which have been repeatedly assayed with a measurement procedure felt to be substantially equivalent to the measurement procedure being evaluated may be appropriate if the user is interested in estimating bias relative to that measurement procedure.
This could be useful, for example, if the intent of the experiment was to estimate the bias of one laboratory in a system relative to another, or to the mean of the laboratories in a system. If the sample materials are appropriate, and target concentrations are available, the user can estimate the bias between the mean concentration calculated in the precision experiment relative to the target concentration of each of the materials.
The width of the verification interval depends on the uncertainty of the target value of the reference material and the standard error of the calculated mean concentration from the experiment. Calculation of the verification interval would be complicated, but the committee simplified it greatly by providing tables for the difficult-to-calculate quantities based on the number of replicate measurements per run, the number of runs, and the uncertainty of the target value.
If the mean concentration from the user's experiment is within the verification interval, there is no statistically significant bias. If the mean concentration from the user's experiment is beyond the verification interval, statistically significant bias exists. The user must evaluate the estimated bias versus allowable bias. If the estimated bias is less than allowable bias, the bias is acceptable. If the estimated bias exceeds allowable bias, it is not acceptable.
Previous versions of EP15 included a small comparison experiment, involving 20 patient samples, which was to be used to verify a manufacturer's claimed bias. There were two problems with this approach. First, users rarely have access to the measurement procedure used by the manufacturer or authors of a publication as the comparative method for the published bias.
Sometimes the manufacturer identifies the comparative measurement procedure only generically. Second, most manufacturers provide only regression statistics as the results of comparison experiments, and do not provide bias claims, so the user has to calculate the bias to be expected from the regression statistics provided and has little idea of the uncertainty of this estimated bias.
The EPA3 committee felt that the patient comparison experiment had little value as it was, and that users who needed to perform a patient comparison experiment should consult CLSI EP9-A3 "Measurement procedure comparison and bias estimation using patient samples.
This is valuable when the user wishes to verify precision and to estimate bias relative to a peer group or target concentration.
It may be especially useful when patient samples are difficult to obtain for a traditional comparison of methods experiment. The EPA3 document development committee was team of experts who worked together well. Tools, Technologies and Training for Healthcare Laboratories. Feedback Form. Neill Carey, Ph. Verification of Precision EP15 first describes a precision verification experiment.
Estimation of Bias Because the precision experiment has so many replicate measurements, collected over several days, results from the precision experiment may be used to make a reliable estimate of the bias of the measurement procedure relative to the assigned target values of the sample materials used in the experiment. Acknowledge Committee Members The EPA3 document development committee was team of experts who worked together well.
CLSI Released an Updated Standard on Evaluation of Precision of Quantitative Measurement Procedures
When evaluating the precision of a method it is necessary to assess the repeatability within-run and the total or within-laboratory precision. CLSI document EPA2 describes the protocols that should be undertaken by the user to verify precision claims by a manufacturer. Precision claims by a manufacturer should be tested at at-least two levels, by running three replicates over five days. A spreadsheet for assisting with the calculations described in this article is available from the AACB web-site. Part of the process of verifying or validating a method to confirm that it is suitable for use is an assessment of precision.
Evaluating Assay Precision
NCCLS document EP5-A2, Evaluation of Precision Performance of Quantitative Measurement Methods; Approved Guideline - Second Edition provides guidance and procedures for evaluating the precision of in vitro diagnostic devices and includes recommendations for manufacturers in evaluating their devices and methods when establishing performance claims. Included are guidelines for the duration, procedures, materials, data summaries, and interpretation techniques that are adaptable for the widest possible range of analytes and device complexity. The procedures are designed for manufacturers or developers of clinical laboratory measurement methods, and for users of those methods who wish to determine their own performance capabilities or to verify claims from a manufacturer. A balance is created in the document between complexity of design and formulae, and simplicity of operation. Your Alert Profile lists the documents that will be monitored. If the document is revised or amended, you will be notified by email.