A fitness ring takes a nap
A few years ago I bought a wearable fitness ring. It measures movement, heart rate, temperature gradient, and various other things. When I looked at the app one day, it informed me I had taken a nap at 2pm. A nap?
It took me a while to realise I had been to the dentist, and had spent eighteen minutes lying in the dentist's chair, which the ring had mistaken for a nap.
Welcome to the world of precision and recall. An impressive deduction by the hard- and software – but not the right one. And one I would not want feeding directly into a system that lets my employer check on what I am up to. For that use case, no degree of accuracy under 100% is enough.
The two dimensions
This illustrates what I call the two dimensions of the content.
The data dimension applies when we extract entities and use them for analytics use cases. These include completeness checks, pre-population of fields, recommendations, and topic classification. If we inadvertently misinterpret an author's input, there is still a good chance that we assign the correct topic classification; and if we don't, it factors into the classifier's own precision and recall. All these use cases are in a way "backstage".
We enter a completely different world the moment we go on stage and present the article as a faithful representation of what the authors wrote – this is the presentation dimension. Then we need to get it absolutely right. It is the same as the difference between smiling at a ring detecting a nap (wow, that it can detect this!) and being furious that the world has been told a lie.
The variability of journal article submissions is so enormous that automatically converting an article to a more structured form is very complex. It is hard but doable for analytics use cases. So let's start there.
Ingredients of trust
This leads to a further observation. A knowledge platform creates a representation of the world of research – that is its objective. It does this by constructing entities based on accumulated evidence. That world is an imaginary, probabilistic construct. Everyone knows that a model of the world is necessarily a simplification of reality. But let's look more closely.
The representation is for the most part based on what we find in the scientific literature, because that is the richest corpus we have. But the scientific literature itself is a massive distortion of what really happens in the world of research. It completely escapes us that a researcher was awarded an important prize, chaired an influential committee, or was unmasked as a fraud. And even then, we don't capture the scientific literature very well: we have no record of editorial board memberships of all journals at any one time, so we cannot detect whether editors are helping their friends.
This is dimensionality reduction: a multi-dimensional reality is written up in a one-dimensional way, and that single dimension is then used to reconstruct reality.
Zoom into the corpus. Not all content is equal. Articles that appeared in journals selected by an independent board of scientists and researchers form the most valuable subset – the highest-standards content on which users can base important decisions with confidence. A preprint server contains many poor-quality papers; that collection exists for discovery use cases, not for high-end evaluation. The distinction must be crystal clear to every user.
When we want to improve the completeness of older records with bibliographic references, we must be careful. Adding references for one publisher's journals alone would introduce a bias, because citation counts would go up more for authors who happened to publish in that publisher's journals. The content should be as rich as possible; however, older content should deliberately be excluded from metric computation if there are such problems with comprehensiveness, and this exclusion should be announced as a clear policy.
Making principled decisions like this – and demonstrably adhering to them – is a key factor in building trust with the research community.