Demonstrating Pharmaceutical Quality System Effectiveness and Driving Continual Improvement: Evidence-Based Risk Reduction

Practical Implications

While all of the above served as useful guidance, the practical link between the evolving and dynamic product and process knowledge, risk-based control strategies, and continual improvement and innovation has not been well-established, or sometimes not been even apparent. Furthermore, the effectiveness of control strategies in adequately mitigating and managing risks is not always evident. This is illustrated by the continuing high number of product quality issues and defective batches that continue to be manufactured and released by GMP-approved facilities every year, and by the product recalls that are required to manage risks to patients that are presented by those defective medicines (7, 8).

So, it is worth considering how can the above link be made, and then, how can it be made stronger? How can the effectiveness of the link between product/process knowledge, risk-based control strategies, and continual improvement/innovation be demonstrated?

This article is Part 1 of a two-part series. In Part 1 we lay out our conceptual thinking on evidence-based risk reduction and how it is an essential element for driving continual improvement and demonstrating PQS effectiveness. Part 2 of the series will provide examples to show practical application of this concept.

Risk Management and Knowledge Management

There is a correlation between risk and knowledge: The lesser the knowledge → the higher is the level of uncertainty/unknowns → the lower the ability to identify, assess, and control risks → resulting in higher potential risks to product quality and/or patient safety. This provides a correlation between the level of knowledge and the level of risk. During the early lifecycle of a product, knowledge is limited, and therefore the extent of unknowns and the level of risk can be relatively high. However, as product knowledge grows during the lifecycle of the product, the number of unknowns should reduce, and coupled with improved GMP and risk controls, there should be a reduction in the level of risk. This is why continual improvement efforts taking into account new knowledge are so important—when used correctly, they can be used to increase product and process understanding, and improve the ability to estimate and reduce risk.

Role of the Pharmaceutical Quality System (PQS)

Connecting the dots between knowledge, GMP controls, and risk, and ensuring the ability to effectively manage all of this dynamically within the structured framework of a PQS, becomes essential. In this regard, the PQS should provide:

1. A structured way to document

   1. the growing product and process knowledge throughout the product lifecycle

   2. evidence that demonstrates effectiveness of risk control measures in mitigating risks (e.g., validation/verification/qualification of controls, periodic reviews of trends, and GMP controls)

2. The ability to use both lagging (e.g., batch rejection, deviations, complaints, adverse events, out-of-specifications, issues, and recalls) and leading (e.g., product and process monitoring, annual product quality reviews, out-of-trends, near-misses) signals to effectively manage risks. A PDA Points to Consider paper describes key performance indicators that can be used to measure PQS effectiveness (9).

The knowledge gained from these leading and lagging indicators and knowledge documented in the PQS should be used to assess/reassess the level of residual risk in the process and to reduce it further, as appropriate. A shift in reliance from lagging to leading indicators improves the ability to proactively identify and reduce risks, and to prevent them from manifesting as real issues.

Constantly striving towards risk reduction based on evolving knowledge will lead to continual improvement and innovation. But in reality, this tends not to always be the case. Why is this?

Estimation of Risk Reduction

The objective estimation of risk reduction and residual risk has remained under-developed in the GMP environment for decades. This has led to a limited ability to demonstrate evidence of risk mitigation and ensure effectiveness of QRM activities. An evidence-based approach to risk reduction would deliver data to support meaningful risk estimates and risk control decisions. While relevant research on estimation of risks has been extensive in other fields (e.g., in relation to probability of occurrence estimation) (10⇓⇓⇓⇓⇓–16), the learnings from such research have not been generally integrated to any meaningful extent for evidence-based risk reduction within the GMP environment.

Evidence of risk reduction in the GMP environment can be demonstrated by assessing and challenging robustness of controls through, for example, worst-case validation testing, assessing equipment or process controls at their edge of failure, acceptable quality limit (AQL) testing, media fills, etc. It is more challenging to demonstrate evidence of risk reduction through controls such as a procedural changes to a standard operating procedure, or re-training operators, thereby reducing the assurance of risk mitigation through such controls.

Understanding risk reduction is important in the context of ICH Q8, Q9, Q10, and Q11 to promote continual improvement and innovation, more effective control strategies, greater process reliability, and an increased ability to monitor product and manufacturing processes. This is because one can reasonably expect continual improvement and innovation to drive a reduction in the residual risks that may relate to a variety of issues—such as producing defective units within a batch, having uncontrolled or unexpected process variability, recurring deviations and complaints, and performing validation programs and laborious product quality review activities that yield little process knowledge or process understanding.

Change Management and Risk Reduction

“To improve is to change.”

—Sir Winston Churchill

Every company strives for risk reduction, whether it explicitly documents it in their PQS or not. But how many companies truly understand, and use, evidence to estimate how much risk reduction their current change management activities are delivering?

We refer to change management here because it is fundamental to managing continual improvement and innovation within the framework of the PQS. Change management is a key quality system element that governs the review, assessment, implementation, and monitoring of improvement and innovative activities (i.e., changes). Changes to improve typically enable a reduction in risks, but how can companies actually obtain reliable risk reduction estimates with low levels of uncertainty and subjectivity during (or following) their change control activities?

It could be argued that a failure modes and effects analysis (FMEA)-type risk assessment for a proposed change provides a ready means of obtaining reliable risk reduction estimates—by simply calculating the reduction in the original risk priority number (RPN) for a particular failure mode (on the basis of planned or implemented risk control actions). While this approach may appear reasonable and practical, it is not scientifically sound, for various reasons:

• RPNs are not absolute numbers; they are the product of ordinal scale numbers (such as those on a 1 through 10 probability of occurrence scale), and their multiplication, addition, and subtraction have questionable mathematical validity (17, 18).

• There is often a high degree of subjectivity associated with RPN values (18⇓–20)—and this is compounded by the design of commonly used FMEA and other risk assessment worksheets. More often than not, these worksheets do not require the identification of the specific GMP controls that influence the probability of occurrence, the severity, and/or detectability of the failure mode in question. When GMP controls are identified, this may be after the probability of occurrence and severity scores have been decided on, not before. In many instances, a distinction is not made between prevention and detection controls and their associated distinct impact on severity, probability of occurrence, and detection scores. Furthermore, the design of most risk assessment worksheets does not require a data-driven assessment of the effectiveness of the GMP controls and how they relate to the probability, severity, and detectability scores making up the RPN. All of this can lead to an inadequate or false sense of security in the GMP controls that are identified in those risk assessment worksheets. Reliance on defect or complaint rates has limited usefulness and may also result in a false sense of security; it does not provide adequate evidence of risk reduction. It is important to focus on demonstrating effectiveness of the prevention and detection controls associated with mitigating a risk.

For a more detailed discussion on this topic, see O'Donnell et al. (21).

Enabling Regulatory Flexibility through an Effective PQS

Understanding risk reduction and how to reliably estimate and provide evidence to demonstrate it following risk control activities remains a significant challenge for the pharmaceutical industry. It probably also has had a direct impact upon the lack of success in realizing the true promise of ICH Q10. In the little-discussed Annex 1 of ICH Q10, guidance is presented on the various opportunities that exist for companies to obtain relief and flexibility from regulators, through the application of ICH Q8, Q9, and/or Q10. As Annex 1 sets out, such regulatory relief and flexibility is contingent upon the application of the science- and risk-based concepts in those tripartite guidelines, as well as upon the demonstrated effectiveness of the PQS that is in place (1).

But one key element of achieving this relief and flexibility has not been well addressed in any current guidance to date—how to actually demonstrate the effectiveness of the PQS. While it is hoped that upcoming ICH Q12, Technical and Regulatory Considerations for Pharmaceutical Product Lifecycle Management, will address this issue with clear and useful guidance for post-approval changes and product lifecycle management, the industry and regulators are probably still quite a way off from having a widely agreed view on what constitutes an effective PQS and how to demonstrate that.

We think that efforts by industry and regulators alike to come to a common understanding on how the effectiveness of the PQS can be demonstrated are vital, if the true benefits as envisaged by ICH Q8, Q9, Q10, and Q11 are ever to be realized. We suggest that efforts that lead to more scientific and evidence-based estimates of risk reduction (in relation to product quality issues) not only serve patients well; they can also serve as the foundation upon which the effectiveness of the PQS can be demonstrated.

Though there are challenges in reaching consensus on PQS effectiveness, one can at least assume that the benefits achieved via process improvements, innovation, and new ways of doing things should be capable of being directly and reliably expressed in terms of the level of risk reduction that those activities achieve. (This might be where an output of a change control is a reliable estimate of the reduction, if any, in the residual risk of producing poor quality and/or non-compliant batches of the relevant active pharmaceutical ingredient or drug product.)

In addition, a truly effective PQS will strive to allow only the implementation of those changes that will result in risk reduction (or in at least no increase in residual risk). In this way, the ability to reliably estimate reductions (or increases) in risk levels should assist companies in demonstrating the effectiveness of their change management and their continual improvement activities within the PQS.

One might argue that regulators and companies are too divergent in their work objectives to realistically reach a common understanding on how the effectiveness of the PQS can be demonstrated. There are important differences in the knowledge base that exist between industry and regulators when risks are being considered and assessed. Regulators draw upon their broad experiences from across many companies and products to support their assessment of risks, their views on risk reduction, and their risk tolerance levels generally. Companies, on the other hand, draw upon their deep product and process knowledge and experience. While these differences may seem large, both groups share an important common goal—the availability of safe, efficacious, and high-quality medicines for patients. This can easily serve as the basis for working together to reach a common understanding on how the effectiveness of the PQS can be demonstrated.

Risk Management Learnings from Other Industries

Putting the above thinking into practice may not seem easy at this time; the GMP environment currently lacks the tools (and perhaps the competencies also) to scientifically produce reliable estimates of risk reductions (or risk increases) following change implementation activities. Current approaches often over-rely on assumptions based on crude subjective RPN reductions, or in low deviation recurrence rates, both of which are of limited reliability. But there are three practical areas in which efforts could be usefully started. These are the following:

• Reviewing and learning from the wealth of peer-reviewed literature published by other fields and disciplines on the factors that influence probability of occurrence estimates during risk assessment work. Research in the fields of experimental and cognitive psychology, including human heuristics, as well as in mathematics and probability theory, has demonstrated that qualitative probability of occurrence estimates can be a significant contributor to higher subjectivity in the outputs of risk assessments (22⇓⇓⇓⇓⇓⇓⇓⇓–31). It is not possible to eliminate subjectivity entirely, but evidence-based estimation of risk/residual risks can reduce the level of subjectivity.

• Working to understand how probability of occurrence and risk estimation are performed in other industries, such as nuclear power generation, aeronautics, and semi-conductor manufacturing. These industries had a head start on the pharmaceutical industry in working to understand probability of occurrence and risk estimation, as well as in the use of quantitative probabilistic risk assessment tools and methodologies (13⇓–15, 32⇓⇓⇓⇓⇓–38). Working with these industries should enable the pharmaceutical industry to capture some of the learnings and experiences these industries have gained in these areas over at least four decades.

• Developing risk assessment tools that are specifically tailored for the GMP environment. The tools should have in-built design features that serve (a) to link all risk estimates with GMP controls of known effectiveness, and (b) to deliver reliable evidence-based risk estimates that do not have high levels of subjectivity and uncertainty.

Continual improvement and innovation are not a “nice to have”; they are essential activities within the PQS. If we as an industry can improve our ability to reliably estimate risk and demonstrate the reductions in risk that should arise as a result of continual improvement and innovation, the benefits could be far-reaching.

Consider a situation in which the level of risk reduction that is delivered by each control in a unit operation is objectively understood, documented, and adjusted, as needed, when warranted by new knowledge. This is process understanding at its best! This kind of knowledge will enable one to construct a spectrum of importance (or a continuum) of risk with respect to those GMP controls (39). This would help move away from the overly simplistic critical/non-critical binary approach to process parameters that is currently in widespread use. With such a spectrum in place, true data-driven and risk-based control strategies could be established, reflecting those controls that have been verified as being important in reducing risks or in keeping risks low. This could also lead to true risk-based validation and verification, where the type and extent of validation and/or verification performed on a process is directly related to the relative importance (from a risk to product quality perspective) of the various controls in the process that are being validated. The end result—lower levels of process variability, fewer recurring deviations, fewer defects, and higher levels of assurance in product quality and ultimately patient safety.

Conflict of Interest Statement

Authors declare no conflicts of interest. The views expressed in this paper are those of the authors, are aligned with the positions of the PDA Post-Approval Changes for Innovation and Availability of Medicines Task Force (PAC iAM) and should not be taken to represent the views of the Health Products Regulatory Authority (HPRA) or Roche/Genentech.