Dead on Arrival: Killed by What Wasn't Written

At OT SCADA CON this year, I gave a talk called "Dead on Arrival," built around a pattern I've watched play out again and again in over 20 years working in controls and automation: projects that fail at FAT, SAT, or both, not because of bad code, but because of what was never written down in the first place.

The controversial statement

Most automation projects fail before the PO is even signed. When something blows up during FAT or SAT, we tend to blame PLC bugs, network drops, or miscalibrated instruments. Those are real problems sometimes, but more often, what's actually missing is defined system behavior, clear acceptance criteria, and an agreed-upon condition for success. If a proposal's deliverable is "a complete, working system," the obvious question is: who's defining complete? Who's defining working?

A lesson from high-consequence industries

Industries where failure isn't an option (I spent the last year and a half in nuclear power) answer three questions before startup: What should happen? How do we know? What proves it? Those map to expected behavior, acceptance criteria, and a validation record. Commercial automation rarely requires this discipline. But the consequences show up anyway.

A war story

I shared a real project: a controls upgrade with no functional spec and no control narrative, because the contract never required one. The lead engineer requested a change order to write one before development started. It was declined. FAT passed, but it only verified wiring, I/O, and comms, not behavior. At SAT, the system didn't behave as expected, because there was never a shared baseline for "correct." One engineer and one week of planned SAT turned into up to six engineers over two and a half weeks. Change orders covered the cost. Undefined expectations didn't.

Verification vs. validation

These get conflated constantly. Verification asks: did we build it correctly? (I/O mapped right, HMI matches the P&IDs.) Validation asks: did we build the correct thing? (Does the process handle a real upset? Are operational objectives met?) A system can be perfectly verified and still be invalid, because it was built to the wrong spec, or no spec at all. You can't validate behavior that was never defined.

What good looks like

A requirement like "start the pump on low level" sounds sufficient but isn't testable. A real acceptance criterion looks like: Pump P-101 starts within 2.0 seconds when LT-101 falls below 25% for a 3-second deadband, in Auto Mode, with no active fault. Anyone — a junior tech, a third-party auditor, the client's plant manager, can independently verify pass or fail. No debate, no interpretation.

Applying it Monday morning

1. Define expected system behavior comprehensively, for every meaningful scenario — including failures.

2. Write definitive, validatable acceptance criteria.

3. Build FAT and SAT procedures directly from those criteria.

4. Use FAT and SAT to validate behavior, not just check I/O.

5. Write the test before you write the code.

Being Dead on Arrival is completely avoidable. It happens when nobody wrote down what success looked like. Define the behavior. Define the acceptance. Then write the code.

See the post here:

https://lnkd.in/p/dkhn3c4X