While waiting for a takeaway Chinese order, I watched a few large orders being prepared for other customers. All the boxes were inspected – some quickly, some more thoroughly – and then packaged up. A basic principle at work: validation must be done by the producer. The consumer only needs to verify that validation has been done. Only the things absolutely vital to the consumer have to be re-validated.
This applies to data deliveries too.
A twenty-year-old lesson
It will be twenty years ago that the Electronic Warehouse II went live – Elsevier's system that has underpinned the delivery of scientific publishing content to downstream products ever since. Based on some years of earlier experience with a QC tool, ideas around validation at the source, verification at the target were formalised in what became known as Vtool (validation tool) and built into the warehouse.
Checking against a DTD or schema alone had long been seen as inadequate. A document can be schema-valid and still be substantively wrong. The validation tool added many additional checks, going well beyond structural correctness to things like semantic consistency, completeness of required fields, and adherence to agreed conventions.
The rule was: the sender validates the dataset with the tool before dispatch, and passes the results alongside the dataset to the warehouse in the form of a "fingerprint" – a digest that today we might call a checksum or a manifest. The warehouse can verify that it was indeed this dataset that was validated, and import the package. Crucially, the validation tool was provided by the platform, developed by a different team than the main recipients, so as to avoid baking in data-interpretation mistakes.
On output we introduced an equivalent "output validation tool", because the same principles we impose on suppliers must be imposed on ourselves when we deliver data. This was harder, because the whole corpus has a mixture of qualities owing to content created very early on when things weren't as rigid.
What validation describes
Vtool describes what is wrong, not what is right. When a dataset passes it, it is no guarantee that the dataset is correct; but if it does not pass, it is certain that something is wrong. The tool does not take away the supplier's own responsibility to deliver correct data.
This is an important distinction. It is not fair if someone inside the platform organisation can point to anything they see in the data and declare it wrong – how could anyone have known? At the same time, we don't want to prescribe every move a supplier must make, because there will always be a case not yet described.
The answer is education. The effort we put into thorough documentation – the Tag by Tag, careful guides for complex cases, worked examples of potential pitfalls – was deliberately different from giving suppliers a rules-and-mapping-tables approach. The latter reduces the supplier to someone who carries out rules mechanically. The former has the intention to educate people to understand the why, so they can handle novel cases correctly.
Mechanisms for managing change
Implementing a validation tool at both ends of a data delivery is only possible if there is a mechanism for managing change without bringing everything to a halt. We implemented a rule accepting two versions of the tool simultaneously, with the older version being retired after a period to allow suppliers time to switch over, and with the ability to quickly roll back if a problem emerged.
For data or workflows that didn't yet meet all the checks – older content, recently acquired material, content from special projects – a "skip checks" mechanism allowed specific checks to be waived. Waiving is a deliberate decision that does not remove responsibility. The act of waiving is itself recorded.
The most important cultural point: when something new came up, the starting point was that data design would precede anything else. Everyone knew about the checks on the data and could run them themselves. This meant that validation was always in the forefront of everyone's mind very early in every project – not an afterthought applied at the end.
That is still the right approach. When the principle is violated – when validation becomes someone else's problem, applied late, or negotiated away – the consequences accumulate in ways that are expensive to fix.