On metrics and power structures
Recently I was at the 2024 Make Data Count Summit, at Wellcome Trust HQ in London. It's a follow-on from a project of the same name which has now become a more sustained long-term effort with a growing community, focused on metrics to measure the impact of data sharing and reuse.
There was a lot for me to absorb, process and reflect on, but one thing I noticed was that some examples gave me a really icky feeling while others had me mentally cheering them on, and I've been reflecting a little on why that is. My first analysis is that it boils down to this: is this metric being used to exert control over others, or otherwise perpetuate unjust power structures?
So for example, a few of the panellists were senior academic leaders with hiring & promotion responsibility who very matter-of-factly said that, while they understood the higher goals, the fact was that they had a limited pot of money and could only fund either A or B but not both, so using "objective" metrics is both more fair and more efficient. That one had the ick factor.
On the other hand, one of the panellists (representing a research funder) was very clear that they were not using any measures of data sharing & reuse to evaluate grantholders or allocate future funding, but only to evaluate whether their own policies and initiatives to encourage good data practice were working as intended. No ick for me there.
What's the difference? In the first case, we see an example of a group with significant power making use of metrics to exercise power over others. A generous interpretation is that they are themselves subject to the power of those who assign their departmental budgets, and using an "objective", "data-driven" process at least allows them to be as fair as possible. A less generous interpretation is that use of metrics saves them time at the expense of those further down the power gradient, while soothing their own conscience and forestalling any argument with a veneer of objectivity lent by the use of data.
I don't believe anyone at the event is deserving of that second, rather cynical (who, me?) interpretation. My point is that the power differential exists and, intentional or not, this way of using metrics both obscures and perpetuates that, which is undemocratic.
It also falls into the system trap of Success to the Successful, since it awards money and power on the basis of previous success. The more funding an academic receives and the higher their rank within their institution, the easier for them to ensure the next research activity meets whatever criteria are set, whether by having more capacity (their own, or that of research assistants and students) to do the necessary work or simply having influence over the criteria themselves. Reinforcing feedback loops are incredibly powerful, and the only remedies are to break the loop or introduce an opposing balancing loop.
There were some really interesting suggestions that the way to solve this is to accept that people will try to game the system and on that basis introduce systems where gaming them still results in desired behaviour. This is interesting enough to be worth trying, but I'm skeptical of its utility in practice. Trying to predict and fix all the different unintended consequences of such an intervention requires you to out-think human nature itself. On top of that, I'm increasingly uncomfortable with the framing about individuals "gaming the system": since it's the system that permits and rewards undesirable behaviour, it's inevitable that at some point that behaviour will take place.
It seems to me that it's preferable to change the system to be able to absorb a reasonable amount of disruption while still delivering the desired outcomes. Maybe that's just a different framing of the same idea, but it seems important to be looking at the system as a whole along with its goals and those of its constituent parts. Just as important, many of those constituent parts are people towards whom we should be directing some compassion.
I've wandered off-point now (quelle surprise) so let's sum up: I think quantitative metrics are an important source of information. It's important to be aware of the shortcomings of any given metric and to only use it as one signal in a broader analysis of both quantitative and qualitative evidence. Metrics are best used by the people whose policy they are measuring, as an important feedback loop in the development of that policy. I strongly believe they should never be used to control the behaviour of others where the measurer has power over the measured, although measuring up the power gradient can be a way of holding power to account.
Economist Charles Goodhart summed it up nicely with what is commonly referred to as Goodhart's Law, commonly stated as
"When a measure becomes a target, it ceases to be a good measure"1
I actually quite like Goodhart's own phrasing, in a nerdy way: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes"
Webmentions
You can respond to this post, "On metrics and power structures", by:
liking, boosting or replying to a tweet or toot that mentions it; or
sending a webmention from your own site to https://erambler.co.uk/blog/metrics-power-structures/
Comments
Powered by Cactus Comments 🌵