Abstract
Ask a PLM team for their regression test suite and you will usually get a spreadsheet. Someone wrote it during the original implementation, from memory and from the requirements document of the day. It has been edited sporadically since. Nobody is confident it covers the system as it exists now. During the last upgrade the team ran the half of it that seemed most relevant, because the cutover window was fixed and the full set would not fit.
That is the normal state of PLM validation and it is strange, because the information needed to write the suite properly is sitting in the system, machine-readable, exactly current and nobody is using it.
A PLM configuration is not merely settings. It is a formal specification of system behaviour. Item types declare what data must exist. Lifecycles declare which state transitions are legal. Workflows declare who must approve what. Permissions declare who may see and do what in each state. Every one of those is a testable assertion that the system is making about itself — and unlike the spreadsheet, it cannot be out of date, because it is the system.
This paper argues the regression suite should be generated from the configuration rather than written against it, describes what each configuration element yields and is honest about the half of testing this approach cannot touch.
1. Why hand-written PLM test suites fail
Four failure modes, all familiar, all with the same root cause.
They are written once and the system keeps moving. Every configuration change after go-live — a new state, an extra approval, a tightened permission — is a change to what the system promises. The spreadsheet does not know. Within two years it describes a product that no longer exists and the gap is invisible because a stale test that still passes looks exactly like a good test.
They test what someone remembered. Coverage reflects what was salient to the author, which is the main flow of the main item type. The obscure lifecycle branch, the permission that applies only in one state, the workflow path taken twice a year — these are precisely the things that break on upgrade and precisely the things nobody thought to write down.
Nobody can say what the coverage actually is. "We have 200 test cases" is not coverage. Coverage is the proportion of declared behaviours exercised and without an enumeration of declared behaviours that number is unknowable. So testing is budgeted by time available rather than by risk.
They are the first thing cut. Because their value is invisible until something escapes and the cutover weekend is finite.
The common cause is that the suite and the system are two separate artefacts maintained by different means. Anything maintained separately from the thing it describes will drift. The fix is not more discipline. It is to stop keeping two copies.
2. The configuration is the specification
Walk through what a PLM configuration already declares and what each declaration implies as a test. This is the core of the idea and it is more productive than it first sounds.
| Configuration element | What it declares | The test it implies |
|---|---|---|
| Item type and its properties | This data exists, with these types, these mandatory fields | Create with mandatory fields empty — it must be rejected. Create valid — it must persist |
| Relationship type | These two types may be related, with this cardinality | Relate a valid pair; attempt an invalid pairing and confirm refusal |
| Lifecycle map | These states exist and only these transitions are legal | For every state, attempt every transition: the declared ones succeed, the undeclared ones fail |
| Transition conditions | This promotion requires these conditions | Attempt promotion with the condition unmet — it must be blocked |
| Workflow map | This approval path, with these activities and roles | Run the path; confirm each activity assigns to the declared role |
| Permission per state | This identity may do this in this state | For each state and identity, assert the allowed actions succeed and the disallowed fail |
| Server method bindings | This logic fires on this event | Trigger the event and confirm the effect |
Read down the right-hand column and notice two things.
First, this is a large number of tests — the lifecycle row alone generates states × transitions assertions per item type and in a real estate that runs to thousands. No human writes that suite, which is exactly why no human has.
Second, every one of them is derivable by query. The states are rows. The transitions are rows. The permissions are rows. The workflow activities and their roles are rows. A configuration probe can enumerate the whole declared behaviour of the system in a single pass and emit the assertions mechanically.
3. What this gives you that a written suite cannot
Coverage becomes measurable. You know the denominator, because you enumerated it. "We exercise 1,400 of the 1,860 declared transitions and here are the 460 we do not" is a sentence no hand-written suite can produce and it converts testing from a time budget into a risk decision.
The suite cannot go stale. Regenerate it and it matches the system by construction. A configuration change does not require anyone to remember to update the tests; it changes the tests.
The diff is the regression plan. Generate the specification before an upgrade and after it and compare. Transitions that disappeared, permissions that changed, workflow activities that moved — the delta is both the list of what to test and, frequently, the first notice anyone has that the upgrade altered a behaviour. This is the most valuable output of the whole approach and it addresses the exact risk described in running an Aras upgrade: the platform vendor carries your configuration forward, but you own verification and this tells you precisely where to look.
It finds configuration defects, not just regressions. Enumerating every declared behaviour routinely surfaces states with no exit transition, workflow activities assigned to an identity with nobody in it, permissions granted in a state the item can never reach. These are latent faults that no functional test would find because no user ever gets there — until one does.
It is honest about the obscure paths. The rarely-used branch gets exactly the same treatment as the main flow, because the generator has no sense of salience.
4. Doing it and what it costs
The mechanism is unglamorous: query the configuration, emit assertions, run them, diff the results.
The probe. A read-only pass over the configuration — types, properties, relationships, lifecycle states and transitions, workflow activities and their assignments, permissions by state and identity. Every PLM holds this in queryable form, because the platform itself must read it to run.
The emission. Turn each declared behaviour into an assertion in whatever form your team will actually execute: a structured document for a manual validation cycle or executable calls against the API for an automated one. Both are useful; the document is often the right first step because it needs no infrastructure and can be handed to a validation team as-is.
The run. Against a copy of production configuration, ideally with representative data.
The diff. Keep each generated specification. The comparison between two dates is the artefact with the longest-lasting value.
The honest cost: the probe is the work, it is specific to your platform's metadata model and it is a few days of someone who understands that model rather than an afternoon. After that it is a button. This is also where the argument in the code boundary applies directly — on a platform where the configuration is itself data in a queryable model, the probe is a query. Where configuration is spread across a database, property files and compiled artefacts, assembling the specification is materially harder because there is no single place that holds it.
We have built this: a complete business-test-case set derived purely from a configuration probe, with the query recipe retained so the set regenerates rather than ages.
5. What this cannot do, stated plainly
A paper proposing generated tests owes you the boundary, because the approach is genuinely blind to several important things.
It tests what the system declares, not what the business needs. The configuration says a part may move from In Review to Released. It does not say whether that is the right process for your organisation. Generated tests verify the system does what it was configured to do and will pass perfectly on a configuration that implements the wrong process correctly.
It cannot test usability. Nothing here detects that the form has forty fields in an order nobody can follow.
It cannot test integration semantics. That the ERP call fires is testable. That the value it sends means the right thing in ERP is not derivable from configuration on either side.
Server logic is only partly covered. You can assert that a method is bound to an event and that the event produces some effect. What the code does internally, with its branches and its edge cases, needs tests written against the logic itself.
Performance is out of scope entirely.
So this replaces the part of validation that is mechanical, exhaustive and currently done badly. It does not replace judgement and a team that generates 2,000 assertions and concludes it has finished testing has misread the paper. The right framing: the generated suite is the floor and human-written tests cover the things only a human knows to ask.
6. Where to start
- Enumerate one item type. Pick the one that matters most. List its states and every transition between them. Count them. That number is almost always larger than the number of tests you currently have for it.
- Generate the assertions for that one type and run them. You will find something — typically an unreachable state or a permission nobody intended.
- Snapshot the specification and keep it. Its value starts at the next configuration change.
- Before the next upgrade, generate again and diff. The delta is your regression plan and it took no planning meeting to produce.
The premise worth sitting with: the specification already exists, it is exactly current and the only reason it is not being used as one is that nobody has read it out of the system. It is not that PLM testing is hard. It is that the industry writes tests against a system that is perfectly capable of describing itself.
White paper. PLMClub implements on Aras Innovator and has built configuration-derived test suites in practice, including the probe-to-document pipeline described in section 4. If you are planning an upgrade and want the before-and-after specification produced before you commit to a cutover window, get in touch.