A visit to a research library
A few years ago, together with two colleagues, I visited a major French university's research library. We have a longstanding relationship with them – their data experts make heavy use of research databases and we meet regularly to discuss what they find.
During the meeting they showed the data on screen. It didn't require a trained eye to see that almost everything that was shown contained some kind of data error. Although the errors were rarely the topic of the conversation, some were very visible, others somewhat less. Let me describe a couple.
The missing accent
The university's own name was spelled without the accent – Universite rather than Université. Our policy is that we either use the English name or a properly spelled French name. The accent had somehow been forgotten or got lost. Not the end of the world, one might say, but unnecessary.
MathML removed
At some point an article came up whose abstract in the secondary database began with the word [removed] in the first line. A quick search for the original on the primary publishing platform confirmed the suspicion: the word [removed] appeared in place of a small piece of MathML. MathML, an integral part of HTML5, is simply not implemented in our secondary database. So a formula that renders perfectly in the primary platform becomes [removed] in the secondary one.
Occurrences live in the "there and then"; entities live in the "here and now"
This connects to a deeper issue in the data model. The way the data is organised is optimised for the graph of occurrences and links between occurrences. Occurrences are the works, researchers, and organisations detected in our content. They are grounded in time: an article written in the Soviet Union by someone using their maiden name will always be from the Soviet Union, written by that person. However, they now need to be attributed to a university in Uzbekistan and to a researcher using their married name. The occurrences talk about a topic that meant something in 1994 that we know means something different now.
This is why we must carefully separate the planes of existence of occurrences and entities. Our job is to map all the evidence to today's world. In the consumption layer, the translation between occurrences and entities must be made explicit. It is no longer reasonable simply to link the organisations we happened to be able to match. All the organisations an affiliation belongs to must be readily available, so that querying by organisation ID gives the results that everyone expects – all the documents that should be attributed to that organisation.
The mother of all metrics
Seeing all those data errors on a big screen was unsatisfying. And these are just a subset of a long list of problems.
In the early days of building the primary publishing platform, when there were no scientific metrics to guide us, we used a simple guiding principle: how long does it take for someone to go in and find an error? I consider this the mother of all quality metrics, trumping any precision and recall score. What good does it do to have high precision and recall values if one can find errors in seconds?
At that time, inspired by a Dutch supermarket chain's "hamster week" (a week focused obsessively on stocking up), we organised regular data-quality hamster weeks in which the whole team would spend the first ten minutes of each day looking at the product and noting down anything that seemed wrong. In close collaboration with content suppliers, operations systems, and the end products, we had vowed to tackle these things end to end. Quickly, we managed to move errors away from highly visible places. A side effect was that many more colleagues got a proper understanding of the data – and this led to a great deal of excitement.
The biggest takeaway from the Paris visit: solving content and data problems is never what it seems at first glance. To fix a problem at one end, it is sometimes better to solve it in a completely different place. This is often counterintuitive, and must therefore be orchestrated centrally.