NQR SAMPLING BASICS
A random sample indicates that any one item within the population has an equal opportunity of being selected. Random selection is critical to ensure statistical validity of the results. Over time, this approach will ensure that the mix of work reviewed by NQR will mirror the actual mix of work being processed by the transfer agency.
SAMPLE QUALITY—TRANSACTION PROCESSING ANALYSIS
National Quality Review's transaction processing analysis is based on a statistically valid, random sample of customer transactions. We review approximately 800 to 1,000 transactions per quarter for each client, which allows us to report overall results at 95% confidence levels (±1.5). This means that if the overall accuracy rate for the items NQR reviews is 94.5%, we know that if we performed the same test 100 times, the average would be between 93.0% and 96.0% in 95 cases. As the sample size declines, the precision of the findings also declines.
Sampling Methodology:
Samples are collected on a daily basis. Our workflow team receives a daily list of all items scanned into the image system the prior day. Each transaction is identified by a number and each day a known range of transactions occur. A random number generator that uses the range of transaction numbers identifies the sample. A random sample indicates that any one item within the population has an equal opportunity of being selected. Random selection is critical to ensure statistical validity of the results.
Over time, this approach will ensure that the mix of work reviewed by NQR will mirror the actual mix of work being processed by the transfer agency. For example, if 50% of the requests being processed are Account Maintenance items, roughly half of the NQR sample for the quarter will also be Account Maintenance. Similarly, if the percentage of Exchanges being processed is low, NQR will see a relatively small number of those transactions in the sample. Because NQR does not select by transaction type when determining the random sample, we will also see our sample adjust over time to reflect changes in the mix of work being handled by the transfer agent. If there is an increase in the volume of New Account applications during tax season, that will be reflected in an increase in the number of New Accounts reviewed as part of NQR's random sample.
In some situations, clients also ask that we select a separate, additional sample of a specific transaction type. These items are coded as additional transactions and are excluded from our overall accuracy and timeliness findings. They are included, however, in any analyses conducted by transaction type. Additional samples are selected in the same manner as the overall random sample, but only the specified transaction type can be analyzed.
Statistical Model:
The model we use to determine confidence levels and sample sizes is described in the following section. This model was not created specifically for NQR; it is the same model used by researchers in all areas to measure reliability in this type of environment.
Reliability
Reliability equates to consistency of measurement. A measurement is highly reliable when the observed or sampled proportion is highly correlated (for example, .95) with the true value.
A sample mean will not be exactly equal to the true population mean because of sampling error, which occurs any time a sample rather than a census is used in measurement. To qualify the sample mean in a way that indicates the general magnitude of the sampling error, a confidence interval is applied. A confidence interval is an estimated range of values with a given high probability of covering the true population value.
Precision
Precision is a test of how exact a measurement is. A result will vary in precision depending on sample size and the reliability of the measurement. For example, if a sample mean is 92% with ±1% precision, the true mean lies between 91% and 93% at some level of confidence. Precision is calculated by multiplying the reliability of a measurement by the measurement's standard error. Related formulas are shown below.
Use in Research
When very precise measurements (within ±1%) and high reliability (99% confidence) are needed, sample size increases substantially. Smaller samples yield lower precision and confidence. A standard level for a wide variety of research interpretations and decisions is 95% confidence.
As an example, consider NQR's standard of 800 transactions. This sample size allows us to report our findings at 95% confidence and ±1.5% precision. To increase the confidence level to 99%, without changing the precision, we would need to double the sample size. Alternatively, to increase the precision to ±1% at 95% confidence, it would be necessary to increase the sample size even more.
Measurement
The basic relationship between precision, reliability, and standard error is:
precision = (reliability) x (standard error)
When considering the precision of a population proportion, such as 95% accuracy, the following is used as the standard error:
p = Population Proportion Estimate
q = 1-Population Proportion Estimate
N = Population Size
n = Sample Size
For an infinitely large population, the standard error of a proportion is:
The precision (d2) is estimated by:
where z = standard score for area in the normal distribution of interest. A z transformation changes a normal random variable mean to zero and the standard deviation to one. It is a way to standardize all normal distributions.
Sample size needed for a specific precision and confidence level that accounts for sampling without replacement is calculated by:
