Abstract

Every organisation running PLM backs it up and most can show you a dashboard of green ticks. Database backup succeeded. Vault backup succeeded. Retention policy met.

Those ticks answer a question nobody will ask during a disaster. The question that matters is not whether the backups completed. It is whether restoring them together produces a system that works — and for PLM the answer is frequently no, for a reason that has nothing to do with backup quality.

A PLM is not one system. It is at minimum a database holding items, structures and the pointers to files, one or more file vaults holding the bytes and a configuration defining how the whole thing behaves. These are different technologies, backed up by different tools, on different schedules, by different teams. Restore them and you do not get the system as it was at a moment in time. You get a composite of three different moments — a state that never existed in production and that nobody has ever tested.

This paper is about that seam.

1. The split-brain restore

Picture the ordinary arrangement. The database is backed up by the DBA team at 23:00 with transaction logs every fifteen minutes. The vault is a file share backed up by the infrastructure team's storage snapshot at 02:00. Both jobs are reliable and both report success for months on end.

Now restore after a failure at 09:00. You recover the database to 08:45 using the logs, because you can — and that is the best available recovery point. You recover the vault from the 02:00 snapshot, because that is what exists.

The result is a database that knows about every file created between 02:00 and 08:45 and a vault that contains none of them. The records are there. The drawings they point to are not. The system starts, users log in, everything looks normal — until somebody opens one of those items and the file is missing.

The reverse happens too and is quieter. If the vault snapshot is newer than the database restore point, the vault holds files that no record references. They are invisible, they consume storage and nobody will ever find them without a reconciliation.

Neither failure is a backup failure. Both jobs did exactly what they were configured to do. The failure is that nothing in the arrangement enforces a common point in time across the components and nothing in the monitoring describes that gap. The green ticks are truthful and irrelevant.

2. Three seams, not one

The database-versus-vault seam is the obvious one. There are others and they fail in less obvious ways.

Vault to vault. In a replicated estate the replicas are themselves at different points. Restore the primary and the replica is now ahead of it, which means the replication relationship has to be rebuilt rather than resumed — and the usual instinct of letting it resynchronise can propagate the wrong direction. The cross-geography aspects of that are a separate problem discussed in vault replication across borders.

Configuration to data. The configuration — item types, lifecycles, workflows, permissions, server logic — is a third thing. Where it lives in the database it travels with the database restore, which is the fortunate case. Where it lives partly in files on the application server, partly in properties and partly in deployed artefacts, it is backed up by yet another mechanism and restoring a database from last week against an application server configured this week produces a system where the data and the rules disagree. This is the practical restore consequence of the configuration-location question raised in the code boundary: a configuration that lives entirely as data in the database restores atomically with the data. One spread across tiers does not.

Data to external systems. Restore PLM to 02:00 while ERP remains current and the two are now out of step on every change in that window. Nothing is corrupt in either system. The relationship between them is wrong — and it will be discovered by a planner, in production, days later.

3. Why backup monitoring cannot see any of this

Backup tooling reports on jobs. A job has a scope, a duration and an outcome. Nothing in that model has an opinion about whether two different jobs, run by two different products, represent a coherent instant.

More specifically:

  • Nothing tests the join. The database backup verifies the database. The vault backup verifies files. No tool verifies that the pointers in one resolve to objects in the other, because no tool owns both.
  • Recovery point objectives are set per component. A four-hour RPO on the database and a 24-hour RPO on the vault is a perfectly ordinary pair of numbers and an immediately inconsistent restore.
  • Restore is rarely rehearsed end to end. A DBA restores the database to verify the backup, which is good practice and tests one component. Full rehearsals involving the vault, the configuration and an actual user opening an actual file are rare, because they need several teams and an environment.
  • The failure is silent on restore. The system comes up. Nothing errors at startup. The damage surfaces later, file by file, which is the worst possible way to discover it because it looks like an application fault rather than a restore fault.

4. What a consistent restore requires

Four properties, in order of how much they are worth.

A common recovery point. The components must be recoverable to the same instant. The cheapest way is to make the slowest component's capability the target for all of them, which usually means increasing vault backup frequency rather than weakening the database RPO. Where the vault lives on storage that supports consistent snapshots across volumes, that is the clean answer.

A quiesce or a documented tolerance. Perfect consistency requires briefly pausing writes across components, which most organisations will not accept. The realistic alternative is to define the tolerance — "the vault may be up to N minutes behind the database" — and then design for the direction of the skew. Which brings us to the single most useful decision in this paper.

Deliberately choose the direction of inconsistency. If a gap is unavoidable, you want the vault ahead of the database, never behind. A vault containing files no record points to is wasted space and a reconciliation job. A database containing records whose files are missing is user-visible data loss. Same gap, two directions, wildly different consequences — and the direction is chosen by the order in which you snapshot, which is free.

A reconciliation step as part of the restore runbook. After any restore, before users return: list the file references in the database, check each resolves in the vault and produce the exception list. Either that list is empty and you proceed with confidence. Or it is not — and you know exactly which items are affected before a user finds out. This is a query and a file check. It is not sophisticated and almost nobody does it.

5. The rehearsal, which is the only real evidence

Everything above is design. This is the part that tells you whether it works, and it is the part that gets deferred.

A PLM restore rehearsal that proves anything looks like this:

  1. Restore all components to a target point, into an isolated environment, using only the documented runbook and only the backups — not a copy of production, not a helpful shortcut from someone who knows the system.
  2. Run the reconciliation. How many database file references do not resolve? This number is your actual inconsistency and it is usually the first time anyone has seen it.
  3. Open ten real items and their files. Including a large CAD assembly with its references, because an assembly that opens with missing children is the characteristic symptom.
  4. Check the configuration matches the data. Are the lifecycles, workflows and permissions the ones that were in force at the recovery point? Promote something and confirm it behaves as expected.
  5. Time it. The elapsed wall-clock from decision to usable system is your real recovery time objective. Compare it with the number in the business continuity plan, which was almost certainly written without a rehearsal behind it.
  6. Write down what you had to improvise. Every undocumented step discovered during a calm rehearsal is a step somebody would have had to invent during an actual outage.

Teams that run this for the first time typically find three things: the restore takes substantially longer than the plan claims, the runbook has gaps that only one person can fill and there is a non-zero count of unresolvable file references that nobody knew about.

6. Counter-arguments

"Our storage snapshots are crash-consistent across volumes." If the database and the vault genuinely sit on the same storage system and are snapshotted in one consistency group, this problem largely disappears and you are in good shape. Verify that this is actually how it is configured rather than how it was designed, because these arrangements drift. And note it does not cover an application-tier configuration that lives outside that storage.

"We are on SaaS, the vendor handles it." The vendor handles their components. Ask them two specific questions: what is the recovery point for file content relative to metadata and what does their restore runbook do about reconciliation. Also ask what happens to your integrations, which they do not own. A vendor answering "we take backups" has not answered either question.

"We have never needed a restore." The relevant probability is not the one you have observed. And the more common trigger in practice is not a disaster at all — it is a bad data load or an erroneous bulk operation, where somebody asks to restore a subset to yesterday and the subset restore has all the same consistency problems in a more fiddly form.

"This is an infrastructure concern." It is a seam between infrastructure and application, which is why it is nobody's by default. The infrastructure team owns the components and cannot test the join. The PLM team can test the join and does not own the components.

7. What to do this week

  1. Write down the recovery point for each component. Database, each vault, configuration. If the numbers differ, the difference is your inconsistency window and you have just quantified a risk that was previously invisible.
  2. Check the snapshot order. Ensure the vault is captured after the database so the skew runs in the survivable direction.
  3. Write the reconciliation query. References in, resolution check, exception list out. Keep it in the runbook.
  4. Book a rehearsal. Half a day, an isolated environment, the runbook only.
  5. Compare the measured recovery time to the stated one and correct whichever is wrong.

The underlying point is simple enough to state in a sentence — and it is worth saying to whoever owns business continuity: a backup is not a restore and a set of successful backups of separate components is not a restorable system. The only evidence that you can recover a PLM is having recovered one.

White paper. PLMClub implements on Aras Innovator and works on multi-vault, multi-site estates. This paper describes a class of failure and a test for it; the specifics of your recovery design depend on your storage platform and your continuity obligations. If you would like a restore rehearsal designed and run before you need one, get in touch.