A vendor demonstration proves that a competent presenter, using prepared data, on a configured environment, can perform a chosen task. That is all it proves. Every vendor can clear that bar, which is why every demo goes well and why demos almost never change anyone's mind.

A proof of concept is different in one specific way that makes it the only evaluation instrument that resists gaming: it uses your data, your process and a pass mark you wrote down before you saw the result. Remove any one of those three and you are back to watching a demo with extra steps.

Bring your worst data, not your cleanest

The instinct is to prepare a representative dataset. Resist it. Prepared data tests the platform against a problem you have already solved.

Give every vendor the same three product structures, and make one of them the assembly everybody apologises for — the one with the inconsistent naming, the part that appears at two levels, the revision history with a gap in it, the supplier item nobody can trace. That structure is where your real cost lives, and how a platform copes with it tells you more than a hundred clean records.

Two things to watch, neither of which appears on a scorecard:

How does it fail? Does it reject the bad data with a message that identifies the problem, or does it accept it and produce something subtly wrong? A system that loudly refuses ambiguous input is safer to live with than one that guesses.

What does the vendor do when they hit it? A team that says "this record is ambiguous, here are the three ways we could model it and here is what each costs you later" is showing you what the implementation will feel like. A team that quietly cleans the data overnight has just shown you something too.

Bring a real change, end to end

Take one engineering change that actually happened recently and is fully closed, so you know the true answer. Walk it through each candidate platform from request through to release, with your real approvers doing the approving.

Not a change designed for the test. A real one, including whatever made it awkward — the approver who was on leave, the two parts that had to move together, the supplier who needed notifying.

This is where the differences show up, because a change process is where a PLM either matches how decisions are actually made in your organisation or forces you to pretend otherwise.

Write the pass mark before you start

This is the step that gets skipped, and skipping it invalidates everything else.

Before any vendor touches anything, write down what success means. Concretely:

  • The messy assembly loads with its revision history intact and traceable
  • The real change completes with the real approvers, without anyone editing the data model to make it work
  • A named person from your team, not the vendor, performs a specified configuration change — add an attribute, alter a workflow step — inside a fixed time box
  • The where-used query on a specified part returns the correct answer
  • The whole thing runs on infrastructure resembling what you would actually buy

Agree those in advance and circulate them. Without a written pass mark, the evaluation resolves to whichever demo felt best, which means it resolves to presentation quality — and presentation quality is the one attribute that has no correlation whatsoever with how the system will behave in year three.

The test almost nobody runs, and the one that matters most

Have your own administrator make a change during the PoC, with the vendor watching but not touching.

Not a developer. The person who will actually own this system after everyone goes home. Give them a small, realistic task: add a field to an item type and surface it on a form and in search.

Time it. Note how much help they needed and whether the change required a deployment, a restart or a code build.

That single measurement predicts your ongoing cost of ownership better than any other thing you can observe in an evaluation, because it tells you what proportion of future change requests your own team can absorb. A platform where your admin does this in twenty minutes and a platform where it needs a developer and a release cycle will produce very different five-year costs, no matter how similar the feature lists look.

Three ways PoCs go wrong

Letting the vendor run it. If the vendor drives the keyboard, you are evaluating the vendor's consultants, who will not be there in year two. Insist your people drive, with the vendor advising.

Scoping it too wide. A PoC attempting to cover every requirement takes months, exhausts everyone and still proves nothing decisively. Three structures, one change, one configuration task. A fortnight, not a quarter.

Judging on the wrong axes afterwards. Score against the pass mark you wrote, and nothing else. Not "which felt more modern". Not "which had the nicer interface". Those are real considerations for user adoption but they are not what a PoC measures, and letting them in through the back door defeats the purpose of having written the criteria down.

When the answer is "stay"

Run the same tests against the system you already have.

This is the step people forget, and it changes the outcome more often than any other. If the incumbent passes your own pass mark, the evaluation is over and you have saved yourself a migration — which, as the economics of replacing a working PLM make clear, is a large saving. If it fails a specific test, you now have exactly what you need: a written, demonstrated requirement your current platform cannot meet.

Either result is worth the fortnight. Only one of them is worth a migration.

Working paper. If you are planning an evaluation and want the pass mark pressure-tested before vendors see it, get in touch.