One of the longest standing in-house debates at Agathos is around engagement. What is it? Is it an end in itself? If so, how does it compare to more ultimate ends such as practice improvement, care improvement, and — most ultimately — improved patient outcomes?
There are two major camps. I am going to “out” my team (and myself), bring you into the debate (all referenced parties have reviewed and consented to the below characterizations of their views 🙂), share some data on our success with physician engagement and practice change, and invite YOU to share what you think.
1. Camp “Change” — the only thing that matters is change (practice improvement).
This means that engagement (again, however defined), does not matter. Or at least it is not the best/only means to change. In this camp I put our medical lead, Michael van Duren, MD, and EHR integration advisor and brainchild behind “ULBC” (see below), Paul Hooven.
How could one ever justify engagement at the expense of change? How could one ever justify any tactic except from its expected yield on things that ultimately matter. Physicians should engage if it helps them improve, and they should skip engagement if they can improve without it (perhaps yet more with conscious goal ownership, or the same with less distraction).
Moreover, engagement can mean so many things. Does it have to be logging into an app? Are negative use experiences discounted? What about just reading an Agathos text? What about the Hawthorne effect? What about organic discussions that ensue with others about the practice goal (whether or not others “engaged”)? At very least managers, engineers, and scientists should not imprecisely define engagement as some arbitrary action, manage their teams and systems toward it, or otherwise talk about it as if it matters.
Engagement is one category of input among many, not an output, much less an essential goal.
2. Camp “Engage” — physician engagement is an essential end worth prioritizing.
This means, even without practice improvement, or perhaps even at the expense of practice improvement (if such a possibility could be imagined), physician engagement is valuable. In this camp I put our Chief Strategy Officer and Head of Agathos Engage, Stephanie Trees, and countless Agathos users (even when pressed under the logic of Camp “Change”).
There is an easy defense for Camp “Engage” that I sympathize with yet ultimately push back against, that being that it is easy to measure at least some forms of engagement, whereas it is extraordinarily difficult to attribute change to anything (whether individual agents, tactics like engagement cues, anything whatsoever), and especially in complex settings such as medicine. I hope analyses like that below, with ongoing data scientific innovation (e.g., Agathos’ User-Level Behavior Change model — ULBC), can better attribute practice change.
There is a practical defense for Camp “Engage” that I both sympathize with and support. Namely, physician engagement with their own data is so rare and suboptimal where present, any warm-blooded human instinctively knows this is not good for improving care. Until we managerially crack that nut, especially given both evidence and common understanding that engagement bolsters change, why move on to other tactics, or get ahead of ourselves in rhetoric or mindset? I think this is where a majority of our users are coming from when they take engagement with Agathos as a primary goal.
Moreover, there are essential reasons for prioritizing engagement, and it starts with how irreducible and chaotic managerial and human systems are. In theory it sounds great to just “follow the data” (i.e., change), keep optimizing everything based on predicted impact, and repeat ad infinitum. But what if your team is burnt out? What if your team is mostly different folks than they were a year ago? What if negative externalities creep in (e.g., lower patient satisfaction)? Keep adding to the inputs, Camp “Change” says, “Keep reindexing the endpoint(s).” But how expensive, dizzying, falsely precise does that become? When are we reducing or effectively eliminating agency? When does the heart and soul of medicine get lost? When are we stuck scratching our heads, thinking it may have been better to have started toward qualitative physician buy-in, organizational alignment, and human staying power, before just feeding our algorithms and interventions more data? Might engagement have been a useful surrogate outcome and primary goal toward that squishy stuff — before change?
Yes, I am Camp Engage
As you may have already sensed, I tend to lean Camp “Engage” 🙂. That said, all practical realities, variable time horizons, and semantic debates aside, I have utmost appreciation that Agathos products are driven by Camp “Change,” which is the more natural home for our mission: organize health data, generate care insights, and improve patient outcomes.
We are all on the same team and of the same conviction that physician engagement with data is a gaping unmet need and a likely (if not essential) path toward improving care — hence our investment in developing both product features and analytical methods that tighten associations between engagement and change, and in particular supporting the hypothesis that more engagement [with recurrent individualized data on practice variation] produces more change.
Example of higher engagement associated with more practice change
Data was collected following Agathos’ use by 200 hospitalists at a large midwestern community health system. There was a phased onboarding of regions and users within regions, although insights (i.e., metrics) were sent on a single cadence, so given users received different numbers of the same insight. We defined a unit of engagement as a user loading an insight (i.e., clicking without abandonment until login), and we cataloged engagement levels as “High” (i.e., >100% of insights loads relative to receipts, meaning the user loaded an insight multiple times for at least some receipts), “Medium” (i.e., 34-100% loads relative to receipts), and “Low” (i.e., 0-33% loads relative to receipts).
Relative percent change is defined as 100 x (post-Agathos value - baseline value) / baseline score. Changes were set to a positive directionality (meaning, if the goal was to reduce the practice, a negative relative change would be represented numerically positively). Users without baseline or post-Agathos scores, null scores, or outlier scores (i.e., relative percent change greater or less than 1.5x the 75th and 25th percentile) were excluded for given insights. Insights were included that had systemwide significant positive change and that were sent at least five times by August 1, 2023. These were the insights that qualified:
- Rate of common labs (i.e., CBC, BMP, CMP) ordered on a daily schedule
- Rate of CMPs (relative to BMPs) ordered
- Rate of discharge orders placed before 10 a.m.
- Rate of PT/OT orders placed within the first 48 hours of the stay
- Rate of weekend discharges relative to weekend encounters
There are some obvious limitations to this analysis and multiple defensible interpretations. Correlations may have more to do with physicians more likely to change also being more likely to engage (vs. the engagement causing the change). Due to the staggered onboarding and multiple regions, with the reality that users received different numbers of insights, it is not clear how much the count of insights engaged with (vs. rate of engagement) is more relevant (regional factors would likely confound the former, since earlier users were all in Wisconsin).
Nonetheless, this is powerful support across a robust sample of users and insight types that not only might engagement produce change, yet it may be worth investing in higher engagement even when the baseline for physician engagement [with particular metrics] is greater than zero.
Agathos is midway through updating its ULBC to treat each discrete login (i.e., engagement) as a feature, which can then be used to isolate both individual and compounding impacts of individual sends. This is a huge step toward enabling Agathos to optimally sequence insights from a set, trigger automated re-notifications to select users, and target recipients based on predicted change.
What do you think?
Well, if you have made it this far, what do you think? Regardless of your role, stance on the matter of physician engagement with data, and progress along the path of using physician data for practice improvement, we would love to hear from you. These matters are nuanced and often highly case dependent. If we can help, let us know. If we can learn from your experience and expertise, please let us know.
About Author
