Mottos are a powerful thing, provided they are repeated often. In my department we always had a set of short working principles – mottos – that people cited frequently. Over time I developed a specific set for data and content modelling. They are short enough to put on a Post-it, and when followed, they prevent an astonishing number of expensive mistakes.
The data mottos
- Always state provenance. Every assertion must say who made it and when.
- Respect zero. There is a difference between not recording anything and recording that something is not present. These are very different states, and conflating them causes persistent, hard-to-debug errors downstream.
- What is grounded in time should be grounded in time. Many properties change over time – countries cease to exist, organisations merge, people change names. If a datum was true at a specific moment, that moment must be part of the datum.
- Content is inherently error-prone. Not because of incompetence, but because the world is complex and authors write under time pressure. The system must be built to accommodate this, not to pretend it away.
Why ISO standards often violate these rules
One common best practice in data work is to adhere to international standards – ISO standards in particular. I do not think that is a best practice at all, because many such standards violate the data mottos.
Take ISO country codes. These aim to describe countries now, whereas a content database needs to record the country at the time of publication. An article written in the Soviet Union remains forever an article from the Soviet Union. The two-letter codes are notoriously unsuited: CS was used for Czechoslovakia and later for Serbia and Montenegro. Country code plus version number might be an option.
Date and time standards are also not fit for purpose. Publication dates can be expressed as a whole year, as a season (Spring 2022), as a period (July–August 2022), as a partial date. Whatever ISO standard is available, it doesn't cover all these needs. Two better options:
- Use the Library of Congress Extended Date/Time Format (EDTF). Dates can be stored in an intuitive way, though there is limited support in standard software libraries.
- Use time ranges consistently. Even a regular date would be stored as a time range from 00:00 to 23:59. This makes sorting and comparison natural and handles all edge cases gracefully.
Corrections as a special case
One further modelling point: how to deal with corrections and feedback. The question is whether a correction is a special kind of exhibit.
The answer is no. Corrections record different information than exhibits. An unusual but defensible approach is not to relate them to exhibits at all, but to the algorithm that produced the original assertion. Human input is human intelligence (HI) sitting alongside the artificial intelligence of our processing modules. What we need is to store the HI suggestions and then weigh them as we would for a committee of services.
This keeps the exhibit store clean – evidence is evidence, and a correction is not evidence – and allows the knowledge layer to hold both the original assertion and its challenge simultaneously, with full provenance.