BEHIND THE METRICS — HOW AGATHOS SELECTS AND DEVELOPS NEW INSIGHTS
- Dorothy Whelan
- 06/17/2024
If you are well-versed in the art of reducing unwarranted clinical variation or the behavioral economic study of how peer comparison impacts change, you are likely in effect familiar with the work we do at Agathos. In short, Agathos empowers clinicians across the organization with individualized data on their practice patterns in a lightweight mobile experience that improves clinical practice, patient outcomes, and operational excellence.
So how does Agathos go about developing metrics to send to physicians and advanced practice providers? Let’s dive in and answer that question.
How do you identify potential new metrics?
First, we have to choose which metrics we want to track and send to clinicians. Typically we chose based on one of the following three criteria:
1. Reporting requirements
Hospitals and healthcare systems have metrics that they are required to report to payers — insurance companies, government, employers. They often need to improve on those metrics and struggle to figure out how to best give clinicians feedback on where they are and how they can improve.
A frequent challenge with these “population health” metrics is that typically only a piece of that metric is within the clinician’s control, so just flashing the scoreboard with the required metric is unlikely to help the clinician. What are they supposed to do about it?
In these cases, we figure out the influence on that metric that IS in the clinician's control, and then we seek to measure that part and give feedback on just that part. That is, we design each metric around a specific thing the clinician can do.
For example, take breast cancer screening. The official metric looks at how many people between this 50-74 have had their screening completed in a particular time window. A lot of scoring well on the metric is setting up population-based systems for reminding patients to get their mammogram. So what is the clinician’s role in that? After interviewing physicians and APPs, the thing that seems to make the difference is taking advantage of every patient interaction to remind them AND giving them a compelling reason to go. Even if it is a sick visit and even if clinicians are seeing a patient that is not their own or not otherwise assigned to them. Providing excellent care is a team effort, and if they have a patient in front of them who is overdue for their mammogram, we want to take that opportunity to remind them! There are some clinicians who are better at this than others.
We designed a metric around this. When a patient comes into the office who is overdue for screening, what percent of the time did they go get their screening after seeing you? That is how we make it actionable. Did that visit make a difference?
We exclude people who got their screenings all on their own. Maybe they saw an ad on TV or maybe the health system had some outreach. We design the metric to focus on the role of the clinician.
2. Literature review
Some of our metrics result from thorough review of medical literature that identifies areas where there may be unwarranted variation in a certain practice that could lead to either improvements in quality or economic benefits for patients or health systems. As you might imagine, care improvement is a big topic in medical research — not just guidelines and best practices, yet also practice measures toward outcomes. Agathos keeps track of cutting-edge medical research to provide clinicians with the most up-to-date metrics possible.
For example, discharges on patients who need physical therapy consults are often delayed because clinicians wait until the patient is ready for the physical or occupational therapy. These units, however, are often overtaxed, and there may be long delays before a referral leads to an actual consult. Putting in physical therapy orders or occupational therapy orders early during a hospital visit when a patient is likely to need it can meaningfully reduce length of stay.
Similarly, the use of common labs on a daily “set-and-forget-it” schedule leads to unnecessary labs, increased risk of infection, and decreased patient satisfaction. Decreasing the use of these daily scheduled labs can thus lead to improvements in patient care on a few fronts.
3. User or Agathos observation
Sometimes, the data itself speaks. When our partnered users or Agathos data scientists identify areas of care where there may be significant variation that may be associated with quality or value, we investigate further.
One such example is the utilization of respiratory panels. In a post-COVID world, Dr. Shephali Wulff (from SSM Health) came to us and asked us to look into ordering patterns for respiratory panels. Respiratory pathogen panels aid in the diagnosis of common respiratory pathogens including influenza, RSV, and COVID-19. Their ease of use makes them a tempting test for patients presenting with influenza-like illness, but their cost and often unactionable results prompt a need for diagnostic stewardship. An analysis showed wide variation in ordering patterns and utilization was increasing over time. This was a great candidate for an Agathos Insight.
The critical characteristic of any metric is that the physician or APP must be able to directly do something that will improve that metric. They must have some agency. It has to be within their control, and the metric must reflect that, or you will not see the types of change you seek.
Once we choose a metric that is actionable, how do we get the necessary data?
Once we have determined that the metric defined is actionable by the physician or APP, we must figure out how to get that data. Normally the data will be coming from the electronic health record (EHR), but raw data alone is not useful.
EHRs such as Epic contain thousands of tables with specific information. It is important to get everything needed, as accurately as possible, to maintain the trust of the clinician.
Even for a relatively simple breast cancer CMS measure, the data acquisition can be complicated. You need enrollment data to get age and sex and then historical claims data to find out if the patient has any conditions that would exclude them from the measure and current claims data to find out if the mammogram was completed.
To calculate the actionable version of this insight (i.e., whether a clinician influenced patients toward getting screened following a given sample encounters), we have to pull Epic patient data tables to find age and sex; Epic encounter tables to see if the patient was recently seen; Epic provider tables to see determine who the patient saw; Epic health maintenance tables to see if the patient was overdue at the time of their visit; Epic encounter, order, and diagnosis tables to see if the patient had any conditions or services that indicates they should be excluded from the measure; and finally Epic order and health maintenance data to see if the mammogram was completed.
How can you be sure that physicians trust the data? (aka, how do you get attribution right?)
The first time clinicians see themselves in comparison to their peers, they often raise questions about the accuracy of the data. With that, it is critical to be sure that attribution is absolutely accurate and clear.
One scenario where attribution has been a longstanding issue relates to hospital medicine and the metrics surrounding length of stay (LOS). Hospitals often track LOS and attribute that metric to the discharging physician. That patient, though, has likely been cared for by a number of hospitalists, not to mention other providers’ influence, and typically the others had more to do with the LOS than the discharging hospitalist. Sometimes the discharging hospitalist may only be in charge of the patient for a few hours! Other times, the discharging physician may be a hero for extra-long stays that would count against them if discharged. So how do you come up with a fair way of attributing LOS?
At Agathos, we discern who was actively caring for that patient each day. We analyze all patient “touches”: orders, notes, and prescriptions that physicians placed. Then, we figure out which days a hospitalist was responsible for that patient and only hold them responsible for the day or days (and more ideally their actions, inactions, and decisions those days) that that hospitalist truly cared for that patient. That is, if we had to make an individualized LOS metric, we would split up the percentage of LOS between the appropriate hospitalists for that patient.
This is a unique approach and far from easy. But it is an illustration of the type of care and depth needed to earn the trust of physicians and APPs.
Once we have the metric, then what?
Quality assurance. With accuracy being of utmost importance, we have a series of processes by which we ensure that the data we have processed matches with the data and experience the clinician is seeing or intuiting in their other contexts. So we ensure a range of testing occurs and use this as an opportunity to flag any issues in our data pipeline or processing. In addition to automated testing and internal data checks, we work with team leads who have a sense of their team’s performance to validate our analyses before any other clinicians see them.
Once we have verified that our data is cleanly working through the pipeline, we are ready to launch the metric.
What happens when the clinician disagrees with the data?
Believe it or not, some physicians will question your data. (That was a joke, btw.) What’s the best response?
Real case examples.
As mentioned earlier, we often have questions when clinicians first encounter their data in comparison to their peers. The most effective way to start is to explain that the intent of each metric is to illuminate general patterns and insights, that it inevitably will be imperfect (often intentionally, so as to capture recency and sample, and to not make it a debate about individual cases), and in general to affirm the skepticism and curiosity of the clinicians. The best way to validate how the raw data and attribution is reliable is by using real examples of patients they have recently treated. This is why we include a list of the last ten encounters (related to a metric) in our mobile experience, so they can see whether those encounters reflect their care. And guess what? The patient examples do not lie; they were (and are) there long before Agathos aggregated them.
While it can be a tough pill to swallow, most clinicians accept the data for what it is and adjust how they perceive themselves as well as how they handle processes around that metric moving forward. The clinicians also serve as a final measure of quality: they know the patients better than anyone else! We ensure that all clinicians have the ability to give Agathos feedback on the actionability and accuracy of our metrics, and we work with clinicians to build improvements to our metrics to make them even more useful.
Often the clinicians who are the loudest about the data are the ones that care the most and end up championing for change (or, perhaps, those who silently change the most :).
Language matters
One important step in creating a metric involves all the copy (i.e., the text) used in discussing the metric. Everything matters — from the name of the metric to the description to the text message that is sent to the physician or APP. We even change the notification copy depending on how the physician or APP is performing on that given metric! To be sure the clinician sees the data, we need to inspire them to click into the mobile experience. Through our research we have found how important that text message can be (just like a subject line of an email).
For example, if we say “Congratulations, you are a top-10 performer on discharge orders, click to see your scores,” we know that increases engagement with the data. So this part of the process is carefully considered, with multiple rounds of edits and discussion.
Some clinicians just want to review their data and how it compares to their peers. But some have questions. Why are you measuring this? Why is it important? Those questions need to be readily answered; otherwise you lose physicians. Directly below the data visualizations, clinicians can click on a section called Why this Matters. It will include a brief description of the importance and implications of the metric, including quality of care, financial implications, efficiency, job satisfaction, and other reasons they (or their group, facility) ultimately cares. The reasons for focusing on a particular metric can also be unique to a particular hospital or health system, so this section is customizable.
Similarly, some clinicians will ask: “Well, what can I do? I’m not satisfied with where I am compared to my colleagues, but the change needed is not definitionally obvious.” The best thing for clinicians to do is to talk with their colleagues, but for quick advice Agathos includes a Suggestions for Action section with tips for changing their practice, including resources sourced from scholarly articles, professional guidelines, and fellow physicians. This can also be customizable to include specific suggestions from physicians and protocols at each organization.
Summary
That is a high-level overview of what goes into selecting and creating a new metric at Agathos. As you can see, a lot goes into it! Our goal is always to provide the most actionable information to the “ground-level” clinicians to ensure patients are receiving the highest standard of care.
If you would like to see the full list of our metrics that we track and deliver to physicians and APPs, take a look at our Agathos Enterprise overview, below; the last few pages include what we are currently delivering.
Let us know if you have any questions or would like to talk to us about what delivering these metrics might mean to improving care at your healthcare organization.