Abstract
Purpose: A ranibizumab prefilled syringe (PFS) has been approved by the U.S. Food and Drug Administration. Here we evaluate the use of the ranibizumab PFS for intravitreal injection by assessing whether the PFS enables healthcare providers to successfully prepare and administer an injection without prior training.
Design: Simulated-use and actual-use human factors usability studies.
Participants: Retina specialists and ophthalmic medical personnel.
Methods: In a simulated-use summative usability study, retina specialists (n = 15) and ophthalmic medical personnel (n = 15) prepared the ranibizumab PFS and performed injections into a model eye. In an actual-use formative usability study (ClinicalTrials.gov identifier: NCT02698566), three assistants and three retina specialists prepared the PFS and performed intravitreal injections, respectively, in study eyes of patients with retinal diseases (n = 35).
Main Outcome Measures: Twelve tasks specific to the unpacking, preparing, and properly administering the PFS for intravitreal injection were evaluated by a study assessor. Task performances were evaluated for use errors, close calls, and operational difficulties. Post-injection subjective user evaluations were performed to assess ease of use.
Results: All participants successfully performed all essential and safety-critical tasks without use error in both the simulated-use and actual-use human factors usability studies. The majority of participants rated the tasks required to use the ranibizumab PFS as “Easy” or “Very Easy.”
Conclusions: Both the simulated-use and actual-use usability studies yielded consistent data, showing that healthcare professionals are able to use the ranibizumab PFS by successfully performing all critical tasks involved in preparing and delivering an intravitreal injection. The simulated-use usability testing was sufficiently realistic and representative of real-world use, and was appropriate and preferred over actual-use usability testing for proper evaluation of the product user interface.
LAY ABSTRACT: Ranibizumab is approved in the United States to treat various eye conditions, including neovascular (wet) age-related macular degeneration, macular edema following retinal vein occlusion, diabetic macular edema, diabetic retinopathy, and myopic choroidal neovascularization. It is administered as an injection into the eye once a month, and is available in a vial from which medication needs to be withdrawn using a standard syringe with a 19-gauge filter needle. The filter needle is then replaced by a smaller gauge needle for the intravitreal injection. The recent U.S. Food and Drug Administration approval of a 0.5 mg ranibizumab prefilled syringe eliminates the need for withdrawing medication from a vial and changing needles prior to use. The studies described in this report assessed the usability of the ranibizumab prefilled syringe by retina specialists and ophthalmic medical personnel in simulated- and actual-use settings. Twelve tasks that included unpacking, preparing, and properly administering the prefilled syringe for intravitreal injection were evaluated by a study assessor. Task performances were evaluated for use errors, close calls, and operational difficulties. Participants successfully performed all the tasks without any critical errors in both simulated-use and actual-use human factors usability studies, and most participants found the syringe to be “Easy” or “Very Easy” to use.
- Intravitreal
- Ophthalmic specialist/personnel
- Prefilled syringe
- Ranibizumab
- Retinal disease
- Usability study
- Human factors
Introduction
The efficacy and safety profile of ranibizumab has been well characterized across a number of retinal disease indications in large phase 3 and 4 randomized clinical trials (1⇓⇓–4). The favorable overall benefit-risk profile of ranibizumab across the approved disease indications demonstrated in these clinical trials has been consistently supported by postmarketing pharmacovigilance. In the United States, ranibizumab has historically been available in a vial configuration. This configuration requires withdrawal of ranibizumab solution from a vial using a generic, 1-cc, sterilized, single-use plastic syringe with a 19-gauge filter needle, which then must be replaced with a smaller gauge needle suitable for intravitreal injection.
Because patients and doctors have a need for maximal safety and more efficient syringe preparation, a ranibizumab prefilled syringe (PFS) was developed as an alternative to the ranibizumab vial configuration. A 0.5 mg ranibizumab PFS has been approved in the United States and in 49 countries outside the United States. In the United States, the 0.5 mg ranibizumab PFS is approved for treatment of neovascular (wet) age-related macular degeneration (nAMD), macular edema (ME) secondary to branch retinal vein occlusion (BRVO) or central retinal vein occlusion (CRVO), and choroidal neovascularization (CNV) secondary to pathologic myopia (5)*. Outside the United States, the 0.5 mg ranibizumab PFS is approved for the treatment of nAMD, visual impairment due to CNV, visual impairment due to diabetic macular edema (DME), and visual impairment due to ME secondary to retinal vein occlusion (RVO; BRVO or CRVO) (6).
The U.S. Food and Drug Administration (FDA) recommends human factors evaluations for medical devices and drug–device combination products in order to demonstrate that the device can be used by the intended users for the intended purpose and under the expected conditions of use without serious errors or problems (7). Collection of human factors usability data during development of a new device such as a PFS is a key component to ensure that potential risks related to use of the device have been identified and addressed. Formative usability evaluations that assess user interaction with the product user interface are conducted during product development to identify potential use errors that could result in harm to the patient or user, and to inform product development. At the end of the device development process, summative usability testing is conducted to demonstrate that the final product user interface supports safe and effective use by the intended users. Together with risk management, human factors evaluations are used to demonstrate the usability and use safety of products.
Genentech, Inc. has conducted several human factors usability studies as part of ranibizumab PFS development, two of which are reported here. One was a summative usability study conducted per FDA guidance to demonstrate that the ranibizumab PFS has met user needs and can be used safely and effectively. This simulated-use usability study was conducted to reflect realistic use of the product, using intended users in a representative use environment. Injections were administered to a model eye. In addition to the simulated-use usability study and at the recommendation of the FDA, Genentech, Inc. also conducted an actual-use formative usability study in which injections were administered to patients. In this report we describe the results of these two studies conducted to assess the usability of the ranibizumab PFS: simulated-use administered to an eye model, and actual-use administered to patients with retinal disease.
Methods
Description of Device–User Interface
The 0.5 mg ranibizumab PFS consists of a sterile, single-use PFS with the following features (Figure 1): a syringe cap, which provides tamper-evident protection of the primary drug product container; a Luer lock, which allows secure attachment of the needle to the syringe; a rubber stopper, which provides closure of the primary drug product container; a finger grip, which enables the user to support the PFS during use; and a nonretractable plunger rod, which is detached from the rubber stopper to prevent pull-back and aspiration of air or fluids during use. The PFS is packaged sterile in a sealed blister pack inside a carton.
The ranibizumab prefilled syringe configuration
Simulated-Use Summative Human Factors Usability Study
The objective of the summative human factors study was to assess whether the PFS could be used safely and effectively by evaluating the performance of essential tasks (tasks essential to the execution of the injection) and safety-critical tasks (tasks in which use errors could result in unacceptable clinical impact or harm) involved in preparing and administering a ranibizumab PFS intravitreal injection dose.
A total of 15 practicing retina specialists (RSs) and 15 ophthalmic medical personnel (ophthalmic assistants or technicians) participated in the study. The study was conducted at three U.S. research facilities configured to simulate a retina clinic. Participants were provided with the ranibizumab PFS product [including the U.S. prescribing information (USPI) and instructions for use (IFU)], a manikin head with a model eye, and all other supplies required to perform an intravitreal injection. Participants were asked to imagine that the manikin was their patient who was to be treated using the ranibizumab PFS. RSs were asked to prepare and administer an intravitreal injection into the model eye. In a separate evaluation, assistants were asked to prepare the PFS. Each participant prepared and/or administered the ranibizumab PFS two times. No demonstration or training was provided to participants.
Participants were evaluated by direct observation for use errors [user action or lack of user action while using the medical device that leads to a different result than intended by the manufacturer or expected by the user (8)], operational difficulties (instances in which a participant used the device safely and in accordance with the IFU but experienced some difficulty that did not affect safe use), and close calls [instances in which the participant self-corrected without intervention (7)]. The observational data were combined with subjective assessments and interviews conducted after the use assessments to determine the root causes of any errors, difficulties, or confusion with product use. Participants were also asked for subjective feedback in which they rated the ease of use of various aspects of the ranibizumab PFS. At the end of the assessment session, the IFU was reviewed for clarity and feedback.
Actual-Use Human Factors Usability Study
The actual-use usability study was a single-arm, open-label, phase 3b study in patients with nAMD or ME secondary to RVO (ClinicalTrials.gov identifier: NCT02698566). The primary objectives of the study were to assess the ability of healthcare professionals (HCPs) to follow the IFU to prepare and administer a 0.5 mg ranibizumab PFS intravitreal injection to patients while maintaining aseptic conditions in the intended use environment, and to document any use errors on essential tasks and safety-critical tasks. The study was conducted in ophthalmology outpatient clinics in three U.S. locations.
Injections were administered by three RSs who were assisted with the PFS preparation procedures by three assistants. RSs were retina subspecialty-trained ophthalmologists who were required to have a valid license to practice in the United States and previous experience performing ranibizumab intravitreal injections using the vial.
A total of 35 patients were enrolled. Eligible patients had to have a confirmed diagnosis by the study site investigator of nAMD (any subtype) or ME secondary to BRVO or CRVO in one or both eyes. Only one eye was selected as the study eye as determined by the RS. Both previously ranibizumab-treated and treatment-naive patients were eligible for enrollment.
Prior to the day of the injection procedure, each clinical study site received the USPI/IFU to allow HCPs time to read them. No demonstration or training was provided to the participating HCPs. Each RS performed one intravitreal injection per patient in approximately 10 patients. Assistants participated in the PFS preparation procedure.
Two human factors professionals observed and videotaped the procedures for use errors, operational difficulties, and close calls on essential tasks and safety-critical tasks. Once the participants completed all patient injection procedures, each participant attended a follow-up session with the study assessor to perform a root cause analysis of any observed use issues.
Adverse events (AEs), AEs of special interest (defined as AEs resulting from medication errors, sight-threatening AEs, and suspected transmission of an infectious agent by the study drug), and serious AEs that occurred during a 7-day study-reporting period beginning on the day of the injection procedure were monitored and recorded.
Essential and Safety-Critical Tasks
Essential and safety-critical tasks for both studies are listed in Table I. In some instances, task performance was not assessed because the task was not part of the participant's normal practice.
Task Assessment Results: Simulated-Use Usability Study
Ethical Conduct of Studies
The clinical/actual-use study was conducted in accordance with the principles of good clinical practice. Investigational review board approval was obtained for both studies before the studies began. Written informed consent was obtained from all HCPs and patients participating in the studies.
Results
Simulated-Use Usability Study
All participants successfully completed all essential and safety-critical tasks. No use errors or close calls were observed. Task assessment results are summarized in Table I. A total of five operational difficulties were observed during use (Table II). None of these instances prevented safe or effective device use.
Summary of Operational Difficulties in the Simulated-Use Usability Study
Average participant ratings of the ease of completing preparation and dosing tasks, on a scale of 1 (“Very Easy”) to 5 (“Very Difficult”), are summarized in Table III. The majority of participant responses (196/216, 91%) rated the tasks required to use the ranibizumab PFS as “Easy” or “Very Easy.”
Subjective Questioning Responses: Ease of Completing Tasks (Simulated-Use Study)
During IFU review, all participants in the study were able to read, interpret, and comprehend all instructions related to the safe and effective performance of essential and safety-critical tasks. No participants reported any missing information, and the vast majority perceived the IFU to be clear and straightforward to follow.
Actual-Use Usability Study
A total of 35 patients with either nAMD (n = 31) or ME secondary to RVO (n = 4) were enrolled in the study to undergo one 0.5 mg ranibizumab PFS intravitreal injection (10 mg/mL drug product concentration).
All participants successfully performed all essential and safety-critical tasks during the study. Task assessment results are summarized in Table IV. Two close calls were observed. One RS's glove touched the injection needle on the third administration due to simultaneous handling of the PFS and lid speculum, which was corrected properly by disposal of the PFS and starting anew. One assistant attached a filter needle to the PFS instead of a 30-G injection needle; because the needle had not yet been primed and sterility was maintained, this was corrected properly by replacing the filter needle with the correct injection needle. Root cause analysis revealed that this was not a misunderstanding of the IFU with regard to recommended needle type. It was discovered that the study site provided the incorrect needle in the study supplies due to the similar packaging of the filter needle and injection needle used by the site, and that the assistant subsequently placed the incorrect needle from the provided study supplies onto the syringe. Both close call instances were identified and corrected by the RS immediately without intervention from study personnel or subsequent use error. Neither close call affected successful use of the product. No operational difficulties were observed during the study or reported by the HCPs in the follow-up sessions.
Task Assessment Results: Actual-Use Usability Study
A single AE of mild subconjunctival hemorrhage was observed that resolved by end of study and was considered unrelated to study treatment.
Discussion
Human Factors Study Design Considerations
The results of both the simulated-use usability study administering injections to eye models and the actual-use usability study administering injections to patients with nAMD and RVO demonstrate that the intended user population is able to use the ranibizumab PFS without use errors that could result in unacceptable clinical impact or harm. No demonstration or training was provided to participants in either study, corresponding to intended use and real-world training expectations for the 0.5 mg ranibizumab PFS. All study injections (simulated and actual) were performed successfully and no use errors were recorded. Subjective feedback indicates that the device is easy to use, with >90% of participant responses rating the tasks required to use the ranibizumab PFS as “Easy” or “Very Easy.” The IFU was perceived to be clear and straightforward to follow, with no missing information.
These studies provide a unique opportunity to compare the differences between conducting simulated-use and actual-use human factors studies, as well as the benefits and disadvantages of each. When a summative evaluation is conducted as a simulated-use study, it is important that the study design be representative of realistic product usage. Factors to consider in the design include test participants representative of the intended users, test environments representative of the actual use environments, and representative use scenarios. The ranibizumab PFS simulated-use usability study included test participants from the intended user groups: practicing RSs and ophthalmic assistants and technicians. The test facility was set up to simulate a retina clinic, including a manikin head representative of the patient. All essential and safety-critical tasks were assessed, with assistants evaluated on preparation tasks and RSs evaluated on both preparation and administration tasks. Ethnographic research has shown that in some retina practices assistants help prepare syringes for intravitreal injections, while in other practices the RS performs all preparation and administration tasks. The ranibizumab PFS simulated-use usability study evaluated RSs and assistants individually in single user group scenarios (as opposed to evaluating both assistants and RSs in the same sessions) to assess whether each individual user group could independently perform their relevant tasks without influence or intervention from other participants. Including these factors in the simulated-use summative study design afforded a study sufficiently representative of realistic use.
The actual-use study identified two close calls, neither of which were observed in the simulated-use study. Specifically, one RS accidentally touched the injection needle with a gloved hand, and one assistant attached an incorrect needle to the syringe. In both instances, the RS immediately identified and corrected the mistake prior to continuing with the procedure. The first close call could happen in either a simulated-use or actual-use environment. The second was a result of the assistant being provided incorrect study materials by the clinical study site during the trial. Although root cause analysis showed that the issue was not due to a misunderstanding of which needle should have been used, it does show the possibility of this happening in a clinical setting.
It is worth noting that although both the simulated-use and actual-use usability studies provided evidence that the ranibizumab PFS can be used safely and effectively, the simulated-use testing enabled a study design that most closely meets FDA guidance and recognized standards for assessing usability and use safety under intended use. Simulated use enabled the HCP to interact with the product uninterrupted, allowing potential use errors, difficulties, and close calls to play out in a way that could highlight potential issues with a product but with no risk to patient safety. While the actual-use usability study enabled the HCP to directly acquire real-life device handling experience, the study presented challenges for collection of robust human factors data. It was often difficult to observe all product use tasks in the busy and crowded clinical environment. Additionally, it was sometimes necessary to pause product use to ensure patient safety. This distraction has the potential to cause study artifacts that can complicate results. For these reasons, actual-use studies are typically recommended only when use-related risks have been identified that cannot be assessed by simulated use.
Assessment of Value to Users
PFS configurations offer a number of potential advantages over the conventional syringe and vial method, such as surface sterilization that eliminates the risk of contamination with airborne pathogens, and fewer preparation steps, resulting in shorter syringe preparation time and reduced opportunity for contamination and infection (9⇓⇓–12). Souied et al. showed that the average time savings over 122 syringe preparations at two clinical centers resulted in up to 39% reduction in total preparation time (11). In addition to the added convenience for the HCP, these factors may translate into improved clinic efficiency, more discussion time with patients, or more time to treat additional patients.
The appearance of intravitreal silicone oil droplets has been reported to occur after intravitreal injections (13⇓–15), and recently, the American Society for Retina Specialists issued notifications regarding reports of intravitreal silicone oil droplets associated with off-label intravitreal injection of bevacizumab (Avastin®, Genentech, Inc., South San Francisco, CA) (16⇓–18). The ranibizumab PFS is manufactured using an optimized baked-on siliconization process that ensures lower free silicone oil levels compared with syringes made using the traditional sprayed-on siliconization method (19).
Conclusion
HCPs were able to successfully use the ranibizumab PFS in both a simulated-use usability study administering injections to model eyes, and in an actual-use usability study administering injections to patients. Ninety-one percent of HCP responses rated the ranibizumab PFS preparation and injection procedures as “Easy” or “Very Easy.” Both the simulated-use and actual-use usability studies yielded consistent data, showing that HCPs are able to safely and effectively use the ranibizumab PFS to perform all critical tasks involved in preparing and delivering an intravitreal injection. The simulated-use usability testing was sufficiently realistic and representative of real-world use, and it was appropriate and preferred over actual-use usability testing for proper evaluation of the product user interface. Actual-use usability testing did not provide additional benefit over simulated-use usability testing.
Conflict of Interest Declaration
ANA: consultant, DRCR network, Genentech, Inc., Opthea; CB: consultant, Allergan, Genentech, Inc.; JC: consultant, AbbVie, Alcon, Genentech, Inc., Regeneron, RegenexBio; HC: none; JS: employee, Genentech, Inc.; CQ-R: employee, Genentech, Inc.; ZH: employee, Genentech, Inc.
Financial Support: Genentech, Inc., South San Francisco, CA, participated in the design and conduct of the studies; data collection, analysis, and interpretation of results; and preparation, review, and approval of the manuscript.
Acknowledgments
Funding was provided by Genentech, Inc., a member of the Roche Group, for the study and third-party writing assistance, which was provided by Michael Bennett, PhD, CMPP, of Envision Pharma Group. Usability studies were conducted by Thomas Grant, PhD, Diane Aston-James, MA, and Jonathan Bainbridge, MSc, CErgHF of Team Consulting Ltd.
Footnotes
↵* During preparation and acceptance of this manuscript, the FDA has also approved a ranibizumab 0.3 mg PFS (March 2018). The ranibizumab 0.3 mg dose is indicated for the treatment of patients with DME and diabetic retinopathy in the United States.
- © PDA, Inc. 2018
