Tangible return-on-engagement tools poised for pharma validation

Recent years have seen great strides in the availability of tools for measuring returns on patient engagement – promising to benefit all with an interest in patient-centric medicines development. Leaders in pharma should now pick up the baton and demonstrate the utility of these tools in real-world industry settings. 

Problems with measuring value from patient engagement  

“Despite [the] perceived importance of assessing the return on patient engagement, little is known about ‘what’ to evaluate, and even less about ‘how.’” In just one sentence, a peer-reviewed research paper summed up a problem that has perplexed the pharmaceutical industry ever since patient-centricity became a trend. 

The many reasons this thorny issue persists to this day are widely reported, but we’ll mention a few. There is a natural delay between any engagement program and the impact. What constitutes value or success is highly subjective. There has historically been a lack of reliable measurement tools and well-defined endpoints. Qualitative measures are much easier to identify than quantitative, tangible ones. And pharma companies don’t apply existing metrics consistently. 

Over the years, many frameworks have sprung up from different sources, guiding industry on how to establish patient involvement in pharmaceutical R&D. This has been hugely beneficial. And yet a criticism often leveled against these instruments is that they are conceptual rather than useful from an operational perspective. For example, they fail to detail how to actually implement patient-centric work in a pharma company, or how to measure impact. 

As a result, while all pharma companies would benefit from reliable, industry-wide data on the prevalence and impact of patient engagement, nobody has stepped up to gather this information in a systematic way. 

Informal data sources can nevertheless give an indication of the way the wind is blowing. The Drug Information Association (DIA) 2023 Global Annual Meeting included a session titled “Measuring Impact of Patient Engagement Across Research and Development“, where a poll of the audience showed that “68% of attendees’ companies are currently engaging patients systematically across their development programs, yet nearly two-thirds of that percentage aren’t measuring the impact”. 

It’s unsurprising, then, that much of the current evidence showing the value of patient-centricity is made up of “case studies, anecdotes, and cautionary tales”. 

Deep dive into the literature

It might be tempting to take a dim view of the current landscape. But dig deeper and it quickly becomes clear that much promising work has taken place in just the last few years. The sentence opening this report came from a 2019 research article published by a team working under the umbrella of PARADIGM, a public‐private partnership that wants to see meaningful patient involvement in medicines development. 

At the time of their 2019 article, the researchers had carried out a comprehensive literature review into the monitoring and evaluation of patient engagement. Specifically they were trying to identify indicators of impact, and methods for measuring this. 

Wading through a stack of papers published between 2000 and 2018, PARADIGM found some potential indicators for evaluating the benefits and costs of patient engagement. But they concluded that the indicators weren’t up to scratch for illuminating pathways to impact, or possible changes in the external environment. 

Novel monitoring and evaluation framework  

The work just described wasn’t merely an academic exercise but rather an initial scoping of the environment as a precursor to a much more pivotal PARADIGM project. Having concluded that there was no framework capable of showing the return on patient engagement in a way that would satisfy all stakeholders, the group set out to fill that gap.  

The necessary brain power came from a collective of 15 pharma companies, five patient associations, and two academic institutions. They were organized into a multi-stakeholder ‘working group’, a common practice for pooling expertise in a non-competitive, collaborative setting. As well as insights, working group participants contributed case studies of patient engagement to inform the new framework. 

The PARADIGM team published their “Patient Engagement Metrics Framework” in 2022. With it, they wanted to “facilitate understanding of how patient engagement translates into impact and why initiatives fail or succeed”. PARADIGM had also gone to great lengths to address the weaknesses of previous (concept-led) frameworks. In all, their offering comprised 87 metrics “to operationalize measurement of the different components”. 

Those metrics came in four flavors (see Table 1). ‘Input’ metrics show whether the conditions for meaningful and sustainable patient engagement exist. ‘Activities/process’ metrics focus on progress in the implementation of patient engagement. ‘Learning and change’ metrics measure the short-term, direct results of patient engagement. And ‘impact’ metrics shed light on long-term outcomes. 

Mindful that not all 87 metrics will be relevant in every patient engagement scenario, the PARADIGM framework includes a “Metrics Selector” tool allowing users to choose the ones most applicable to them. 

Table 1: Snapshot of PARADIGM Patient Engagement Metrics Framework* 

*Selected extracts of framework shown for illustrative purposes 

Toolkit with “standardized metrics” 

Interestingly, around the same time the PARADIGM team were going about their work, a parallel and highly complementary effort was taking place elsewhere. This one, co-led by the DIA and the Tufts Center for the Study of Drug Development, involved 17 industry stakeholders made up of pharma companies, contract research organizations, and other vendors providing services in patient engagement or patient-centric medicines development. 

Organized in a similar working group arrangement, this collective’s analysis surfaced hundreds of metrics covering costs, quantitative and qualitative impact measures, as well as lessons learned and challenges.  

Again keeping in mind what would be useful operationally, the team used their discoveries to co-design “a toolkit that industry and patient advocacy groups can use to assess the impact of patient engagement and [patient-centric initiatives] PCIs” (see Table 2). Their hope is that management decision-making will become more informed and that industry will adopt the “standardized metrics” in the toolkit to assess return on engagement. 

To give a flavor of what the DIA/Tufts project created, one group of metrics, organized into a category headed “Long-term drug development portfolio and enterprise”, looks at the economics of an initiative, its impact on a drug program and product portfolio, and whether it has influenced business decisions and strategy at corporate level. 

Table 2: Snapshot of DIA/Tufts metrics toolkit* 

*Selected extracts shown for illustrative purposes 

Now for pharma’s part 

All credit to PARADIGM and DIA/Tufts for their efforts. They’ve taken the initiative to create novel tools promising to benefit all with an interest in broader adoption of patient engagement in pharma. But both teams have made it clear it’s not yet a case of job done. 

PARADIGM researchers are keenly aware that their metrics have “hardly been tested in practice”. So beyond publishing them, they haven’t been able to say which methods work best in a given scenario. The DIA/Tufts group called their offering “an initial set of standardized return-on-engagement metrics”: a clear indication they expect to see further iterations and refinements over time. Crucially, these new tools have a distinct emphasis on measuring return on engagement in R&D. In other words, they leave an expanse of white space for complementary future work in the setting of commercialized therapies. 

While public-private collaborations were first off the mark, pharma companies should now be the ones to pick up the baton. Encouragingly, our separate report “Evaluation of return on patient engagement: How pharma leadership can create a rising tide to lift all boats”, based on unique insights from recent client conversations, points to a growing appetite within pharma to take up the challenge. The report also outlines A Life in a Day’s thoughts on what needs to happen next. Among our recommendations, we would like to see pharma validation of the new measurement tools described above – ideally longitudinally in real-world settings. Pharma should also take the lead in measuring the creation of value from patient engagement after R&D, when products are on the market and reaching patients. 

As in a race, getting a good start can be the hardest part. But leveraging the current momentum in measuring returns on patient engagement, bold leadership in pharma can surely get these business-critical efforts over the finish line. 

FURTHER READING 

Gorbenko, O. et al. (2022) ‘Co-creating with patients an impact framework across the medicine’s life cycle: A qualitative study exploring patients’ experiences of involvement in and perceptions of impact measures’, Research Involvement and Engagement, 8(1). doi:10.1186/s40900-022-00334-0.  

Kim, J.Y. et al. (2023) ‘Benchmarking patient engagement capabilities and preparedness of drug development sponsors’, Therapeutic Innovation & Regulatory Science, 57(5), pp. 1040–1049. doi:10.1007/s43441-023-00545-x. 

Stergiopoulos, S. et al. (2020) ‘Measuring the impact of patient engagement and patient Centricity in clinical research and development’, Therapeutic Innovation & Regulatory Science, 54(1), pp. 103–116. doi:10.1007/s43441-019-00034-0.  

Vat, L.E. et al. (2019) ‘Evaluating the “Return on patient engagement initiatives” in Medicines Research and Development: A Literature Review’, Health Expectations, 23(1), pp. 5–18. doi:10.1111/hex.12951. 

Vat, L.E. et al. (2021) ‘Evaluation of patient engagement in Medicine Development: A multi‐stakeholder framework with metrics’, Health Expectations, 24(2), pp. 491–506. doi:10.1111/hex.13191.