What Companies Need to Know About FDA’s Quality Metrics Proposal

On March 9, 2022, FDA requested comments on how it is considering approaching quality metrics reporting from drug manufacturers. FDA’s proposed outline for this reporting program raises a number of important considerations for drug manufacturers who will want to assess their current quality metrics and may want to provide input to the Agency to inform the development of this program.

Brief Overview of the Developing Quality Metrics Reporting Program

FDA’s approach to quality metrics reporting has evolved significantly over time. In 2015, the Agency issued a draft guidance document that described a potential mandatory program for quality metrics reporting on a product basis. After significant input from industry, FDA issued a revised draft guidance in 2016 that outlined an initially voluntary program with participants reporting data on three specified metrics, either by product or establishment. FDA subsequently undertook two pilot programs in 2018 in which it conducted visits at 14 different manufacturing sites and received feedback from 13 sites about how these sites implemented their quality metrics programs.

FDA took a number of learnings from these pilot programs including:

• Different industry sectors prefer different quality metrics;
• In some cases, a combination of metrics may be more desirable than a single metric; and
• Metrics related to the Pharmaceutical Quality System (“PQS”), such as CAPA effectiveness or repeat deviations, are key to many companies’ metrics.

Outline of Currently Proposed Quality Metrics Reporting Program

In its latest outline for the proposed program, FDA would seek metrics from the same “covered establishments” it described in its 2016 draft guidance document: manufacturers of products with NDAs, BLAs, and OTC monograph products, as well as Active Pharmaceutical Ingredients (“APIs”) used in the manufacture of these products. This would include contract manufacturers and testing laboratories.

As part of its current outline, FDA has identified four (4) different “practice areas,” or broad categories for the metrics it seeks to collect. Manufacturers would select metric(s) from each of the practice areas and would report those metrics to FDA. The different practice areas and initial proposed metrics are described below:

• Manufacturing Process Performance: Process capability performance indices, lot acceptance rate, Right First Time Rate, and Lot Release Cycle Time.
• PQS Effectiveness: CAPA effectiveness, repeat deviation rate, change control effectiveness, overall equipment effectiveness, and unplanned maintenance.
• Laboratory Performance: Adherence to lead time, Right First Time Rate, invalidated / overturned out-of-specification (“OOS”) rate, and calibration timeliness.
• Supply Chain Robustness: On time in full (measure of shipments delivered on-time and in correct quantities), fill rate (order shipped as a percentage of total demand), disposition on time (proportion of lots where disposition carried out on-time); and days of inventory on hand.

Importantly, FDA currently is not proposing any standardized definition of these metrics. Manufacturers would report the selected metrics and inform FDA how they calculated them. FDA continues to evaluate collecting data at the establishment level with the option to segment by manufacturing train, product type, or product level.

Important Considerations for Manufacturers

• FDA’s Use of the Data: FDA has explained that it may use the metrics data to inform its risk-based inspection scheduling. Notably, an FDA representative speaking about the proposed outline indicated that FDA did not see the collection and analysis of the data as an “enforcement tool.”^[1] But it is possible that the Agency may use the data to not only inform how frequently sites receive inspections, but also to follow up with questions when presented with particular metrics or significant changes. The FDA representative noted that the Agency is contemplating a “comment” field as part of the data submission portal where manufacturers could include contextual information about the metrics they submit.
• FDA’s Purpose: It is also important to understand FDA’s purpose in collecting the metrics. FDA views the metrics as an opportunity to encourage industry to develop more mature quality systems. FDA views compliance with current Good Manufacturing Practice (“cGMP”) requirements as a minimum baseline, but such cGMP compliance “does not necessarily indicate whether a manufacturer is investing in improvements and striving for sustainable compliance.”^[2] An FDA representative explained that the Agency hopes to leverage machine learning, artificial intelligence, and other tools to analyze the quality metrics data it receives, together with other information available to the Agency, to develop more predictive models about potential issues, such as drug shortages or recalls.

In addition to such internal analysis, FDA’s approach to the quality metrics data also suggests that it might serve as a step towards a quality “rating system.” At this point, FDA has not directly suggested that this data would be used for this purpose, but FDA recommended such a rating system in its 2019 report on drug shortages as a way to incentivize drug manufacturers to invest in achieving quality management system maturity.^[3] Certainly if the quality system metric data were effective at predicting product quality issues, it could eventually serve as an objective measure in developing rating criteria for quality systems.

• Assessment of Current Metrics: Against this backdrop, manufacturers should assess their current use of quality system metrics. Although FDA is not proposing immediate or mandatory collection of this data, it remains committed to further developing this initiative. In the future, companies might need to report such data to FDA, and that data may inform their inspection schedules and compliance posture. As a result, companies should review the metrics they use and their effectiveness in helping develop mature systems and in measuring that success. Our prior work reviewing these metrics for companies suggests continuing opportunities to implement and develop metrics that will reflect the strong quality systems that companies have worked hard to achieve, and to avoid mistaken impressions from metrics that have not been fully refined.
• Opportunity to Comment: FDA has established a docket to solicit comments on its outlined approach, and companies should actively consider submitting feedback based on their own experience with using quality metrics in these selected practice areas or others of relevance. The FDA’s approach raises a number of questions that are important to manufacturers including: Who should be responsible for reporting data related to contract facilities? What metrics can help objectively measure quality culture? What limitations are there in reporting statistical process control and process capability data at an establishment level across multiple processes? What criteria should be used to calculate metrics reported to FDA? How often will data be reported to FDA?

FDA’s approach to collecting and using quality metrics remains at an early stage, but the Agency has begun to better define how it will approach this important topic. It is important that manufacturers begin now to anticipate FDA’s future direction for this reporting, to think about how they currently use these metrics, to evaluate where they can further develop their own programs, and to provide input to FDA on the direction of this program.