Comparison of Different Calculation Approaches for Defining Microbiological Control Levels Based on Historical Data

Discussion

In the present work we analyzed different calculation approaches for their ability to accurately define microbiological control levels. To that end, simulation experiments with real microbiological data were performed. A random selection of 100 values from a data set was used to calculate control levels, and the explanatory power of these control levels for the remaining values was assessed. Although it is clear that statistically calculated control levels would be updated more frequently during routine application, we believe that these simulations provide valuable information regarding the predictive power of the different calculation approaches examined in this study. It must also be clearly understood that quantitative conclusions of our work are restricted to the data used in this study and may not be generalized for all microbiological count data from the environment (for this, much larger amounts of data need to be evaluated). However, given that the data used were from routine EM, it is reasonable to assume that similar trends may also exist with other data and that the basic conclusions of this article may have broad applicability.

The results of our study confirmed that assuming a normal distribution is typically not appropriate for microbiological data. Importantly, not a single data set we examined in the course of our work could be well fitted with a normal distribution. On that basis, we consider this model as inappropriate for determination of microbiological control levels and recommend against using it for that purpose. The main reason for this apparent lack of fit is that microbiological data generally have a higher variability than could be explained by a normal distribution. For lower percentiles, the inability to explain the full variability of the data is less problematic, as the highest values are not considered part of the usual count data anyway. This explains why prediction at a 95^th percentile was among the best, but prediction got clearly worse for higher percentiles (Figure 4).

Assumption of a Poisson distribution worked to a certain extent for some data with low arithmetic mean (Figure 2A). However, the simulation experiments indicated that even for data with low arithmetic mean the microbiological control level was systematically underestimated in the majority of cases (Tables I, II, III and Figure 4). For data sets with higher arithmetic mean (Figures 2B, C and 3A, B, and C), the Poisson distribution distinctly underestimated the control level and the predictive power for future values was highly insufficient (Tables I, II, III and Figure 4). The Poisson distribution assumes that the variance/mean ratio is close to 1; if this ratio is significantly larger, the control level is set too low. During our evaluation, a variance/mean ratio as expected for Poisson-distributed data was observed in rare instances for data from high-grade cleanrooms and non-sterile product testing (in these cases the observed counts were basically confined to 0 or 1 CFU) (data not shown). In these special cases, the Poisson distribution could theoretically be applied. However, for such data all models typically indicated the same control level (i.e., 1 CFU). In order to evaluate whether the variance/mean ratio may be consistent with a Poisson distribution, the Fisher dispersion test (13) could be applied.

For the formula proposed by Hussong and Madsen, which represents a mixture of normal and Poisson distribution characteristics, a similar picture emerged as under the assumption of a Poisson distribution. It should be considered, however, that Hussong and Madsen designed their formula for microbiology data from higher-grade cleanrooms (ISO 4.8/5) and that the formula worked relatively well for data structures that would be expected from such areas. For instance, the predictive power for data with low means was superior to the Poisson distribution and normal distribution (Table III).

The non-parametric Excel percentile predicted future values well in our simulation experiments (Tables I and III). However, particularly with the PERCENTILE.EXC function, the calculated control levels were on average higher compared to the parametric models, and also the range of levels calculated for the different simulation runs was broader (Table II). With the 100 values used for the simulation, the highest observed value (or the highest observed value minus 1 CFU) was considered the control level by this Excel function when working at a 99^th percentile. This strong dependency on the largest observed value explains the broad range of calculated control levels in our simulation experiments. If the percentile of interest would have been 95% instead of 99%, then the largest observed value would have had a smaller impact at a sample size of 100 values (but the same impact at a sample size of 20 values). In principle, the closer the number of values available for calculation of the control level is to the minimal sample size requirement of the PERCENTILE.EXC function, the stronger the impact of the highest observed value is. Accordingly, if the available sample size is close to the minimal requirement of the percentile function, whether the highest observed value could eventually represent an outlier causing an unrealistically high control level should be very carefully assessed. However, on which basis such a decision should be made is difficult to generalize and may even be harder to justify, representing a potential drawback of the Excel percentile function. When the available sample size is clearly larger than the minimal requirement of the PERCENTILE.EXC function, then the calculated control levels are less affected by the largest observed value. Therefore, the use of the PERCENTILE.EXC function may reasonably be expected to represent an appropriate solution if sufficient data is available to mitigate the difficulties outlined above. Of note, the PERCENTILE.INC function tended to underestimate microbiological control levels (data not shown) and accordingly made predictions at higher percentiles worse as compared to the PERCENTILE.EXC function (Figure 4).

Both the negative binomial distribution as well as the ZINB distribution fitted the microbiological data well and had good predictive power of future values (Tables I, II, III and Figure 4). However, on basis of our simulation experiments, we saw no evidence to generally prefer the ZINB over the corresponding non-inflated model, as it consistently made worse predictions. In addition, the algorithm employed for calculating the ZINB distribution did not find evidence for excess zeroes with 43 % of the data sets used, and accordingly applied the non-inflated negative binomial model. Thus, we saw no evidence to generally prefer the zero-inflated model as proposed by Yang et al. (10).

Finally, the gamma distribution on average made the best predictions in our simulation experiments, although it is a continuous distribution. Surprisingly, this was even the case for data sets with low arithmetic mean (i.e., with raw data dominated by low counts and many zeroes) (Table III), where Chi-squared goodness-of-fit values for the gamma distribution were worse compared to discrete models like the negative binomial or ZINB (data not shown). As discussed in the Methods section, the gamma distribution cannot reflect the count result “zero” and therefore has shortcomings in modeling the lower end of the data. However, because this lower end of the data is actually not very relevant when calculating a percentile >95%, it is indeed possible to achieve very good results with a model that only accurately predicts and fits the right tail of the data (i.e., the high count results) (Figure 5). This explains why the predictive power of the gamma distribution was not so good at the 95^th percentile, but became increasingly better for higher percentiles (Figure 4). Of note, we used a method of moments approach to calculate the gamma distribution parameters α and β in order to deal with the zeroes in the data. The use of a maximum likelihood estimation, which is often applied to that end by statistical standard software, would not have been possible with such data because it applies logarithms.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Exemplary data set fitted with the negative binomial distribution (discrete) and the gamma distribution (continuous). The right-hand figure represents a magnification of the fits for the highest percentiles.

From a statistical point of view, a common approach would be to run goodness-of-fit tests for each of the models and then use the one that best explains the data (as was done in the “Best fit” approach in Figure 4). On the basis of our simulation results, it may be reasonable to confine such calculations on the negative binomial distribution, ZINB, and gamma distribution (although the necessity to include the ZINB may indeed be debatable, as we have not seen any clear benefit compared to the non-zero-inflated negative binomial model). However, goodness-of-fit-tests typically integrate all observed count results and provide a rating of how well a particular model explains all the data. But with our particular type of question (calculation of a control level at a percentile >95%), the goodness-of-fit for the low count numbers is less relevant. In other words, the conclusions from unmodified goodness-of-fit-tests may be misleading in the light of our question of interest, as these are influenced by the fitting accuracy for the (less relevant) low count results. As shown in Figure 5, the gamma distribution cannot fit the low count results well (up to 8 CFU) compared to the negative binomial distribution. Nonetheless, the high count results (larger than 8 CFU) are well fitted by the gamma distribution. Therefore, predictions for such large values (corresponding to high percentiles) may be good, although the overall fit is not the best. As shown in our simulations with real microbiological monitoring data, assuming a gamma distribution by default led on average to at least as good prediction of percentiles ≥98% when compared to a best-fit procedure (Figure 4). Accordingly, from a pragmatic point of view, microbiologists may also exclusively rely on the gamma distribution for calculation of their control levels. This approach is attractive because it can be performed entirely in standard office calculation software with the formulae provided in the Methods section and does not require use of specialized statistical software. Particularly for smaller companies, which may not be able to invest resources in establishing more sophisticated statistical frameworks and best-fit algorithms, application of the gamma distribution may be a superior alternative to the widely used Excel percentile functions if the amount of available data for calculation of microbiological control levels is limited.

In our study we relied on EM data. Importantly, the proposed approaches are applicable for any microbiological data that follow similar patterns that we observed in our study. During our internal data review, we have so far not encountered different patterns in data from other microbiological tests (e.g., microbial enumeration tests of non-sterile products, water, bioburden in intermediate products or bulk solutions), suggesting that the above outlined approaches may be broadly applicable for control level calculation. It should be noted that a minimal number of historical values must be defined for which statistical modeling is considered reasonable. We propose that not less than 50 values should be available for fitting of parametric models. However, in cases where the data can be expected to be very stable, a smaller number may be sufficient. For non-parametric Excel percentile functions, in general more data should be used as explained above. If not enough data is available for fitting of a statistical model, the control level may, for instance, be arbitrarily defined as a percentage of the specification limit or recommended action level as an interim solution. Also, if there are sufficient values but all of these are below the detection limit of the method used, every result above the detection limit may be considered an excursion from the control level.

Despite mathematical definition of control levels, there are still areas where decision making by the microbiologist must occur. First, data used for calculation of control levels from historical data should not include outlier values that are due to special causes. Excursions from a compendial limit, for instance, do not represent the normal state of control of a product or process. On that basis, such values should not be included in the body of data used for calculation of control levels. Similarly, data associated with major deviations should be critically reviewed. Further, the microbiologist must decide which percentile should be applied for the statistical calculation. The percentile directly affects the threshold of acceptance below which a microbiological count result is considered part of the expected variation. Therefore, the choice of the percentile has consequences on the number of excursions that occur, even if the process or product of interest is actually under control. For instance, if 1000 analyses per year are performed in EM, a 95^th percentile or 99^th percentile would mean that 50 or 10 values, respectively, are theoretically expected to exceed the control level even if the environment is completely stable and the statistical models fit the data perfectly. Therefore, working with too low of a percentile would waste resources by investigating non-issues or normal/controlled conditions. Even more importantly, in such a situation the exceeding of a control level would probably lose its function as a warning signal and no longer be taken seriously (imagine a fire alarm that goes off every other night—after two weeks, no one will leave the building anymore). On the other hand, setting the percentile extremely high is also no solution, because then unusually high counts that may indeed signal a potential drift from normal operation would not be detected. In any case, use of microbiological control levels should also be combined with an appropriate trending system to determine if clusters of counts indicate an adverse trend (19⇓⇓–22). If such a trending system is well designed, it also supports the detection of drifts from normal operation.

There are different ways to integrate control levels calculated from historical data in the microbiological monitoring program. For instance, an action level can be set based on regulatory or industry guidelines, and an additional alert can be set level based on historical data (7). Alternatively, a specification level from regulatory documents could also be supplemented with action and alert levels that are both based on historical data. With such an approach, a higher percentile should be chosen for the action level compared to the alert level. However, whether use of three levels confers advantages over the use of two levels may be debatable. The authors, for instance, do not see significant additional value for most applications if the historically based control level has been appropriately defined. Finally, if no specification or action level exists in regulatory and guidance documents, it would also be conceivable that all levels are calculated based on historical data. The latter approach may also be indicated when defining control levels for non-growth-based alternative microbiological methods. Because such methods allow for direct measurement of microorganisms, uncritical adoption of compendial levels expressed in CFU (which implies growth of microorganisms under the conditions of the test) would most probably lead to difficulties, as only few microorganisms from our environment can actually be cultivated (23). In all of the above cases, a clear distinction in the investigation procedure should be made between the alert and action level to avoid confusion.

Calculated microbiological control levels should be re-assessed after a defined time period or if sufficient additional results can be integrated in the calculation. In the view of the authors, if the re-calculated control level is equivalent or lower than the previous one, it may be implemented as such. However, if the re-calculated control level is significantly higher than the previous one, it may only be implemented if a risk assessment concludes that product quality is not worsening or that the increase will not affect product quality.