Inference · Field note 02

On-device inference as a product constraint

Moving inference closer to the sensor changes more than latency. It changes the shape of the entire system.

The budget is multidimensional

Model size is only one constraint. Peak memory, sustained thermals, startup latency, battery draw, and update bandwidth can dominate the design. Measure the complete path from input acquisition to usable output, including preprocessing and postprocessing.

Confidence needs an action

A confidence score becomes useful only after the product defines what happens next. The system might wait for another observation, choose a conservative action, request assistance, or fall back to a deterministic rule. That policy deserves the same testing as the model.

Updates are part of inference

Keep model artifacts versioned and independently replaceable. Make activation atomic, retain a known-good version, and record which version produced each decision. A model that cannot be rolled back safely is difficult to operate at scale.