From Prototype to Production AI: Data, Validation and Monitoring

A proof of concept shows that a model can work. A product needs it to keep working on inputs nobody collected during development: new lighting, new sites, ageing sensors and changes in what is being observed. Getting there is mostly engineering around the model rather than the model itself. This article covers the data, validation and monitoring practices that turn a prototype into something you can ship and support.

At a glance

3 splits
Training, validation and test sets, kept apart so the test score means something.
Leakage
Test data that resembles training data too closely, inflating the score.
Drift
Gradual change in real inputs that erodes accuracy without any error message.
Versioned
Each deployed model traceable to its code, data and settings.

The problem

Prototype datasets are usually small and gathered in one place under convenient conditions. Scores measured on them say little about a factory floor at night or a camera mounted at a different angle.

Leakage makes this worse. When frames from the same video or the same physical object appear in both training and test sets, the test score rewards memorisation, and the field result is lower than anyone expected.

A test score only predicts the field when the test data was collected the way the field will be.

Validation that predicts the field

Split data by site, session, device or object rather than by individual frame, so the test set contains situations the model has never seen. Report results per condition as well as overall, because an average can hide a condition where the model fails.

Set acceptance criteria from the cost of each error type before testing begins. In production, true labels are rarely available, so monitor input statistics and confidence distributions, and sample predictions for human review.

What good looks like

Write down the conditions the product will meet and check the dataset covers each one with enough examples.

Freeze a grouped test set, agree pass criteria, and record the data and code version behind every candidate model.

Deploy with monitoring and a rollback path, and feed reviewed field samples back into the next training round.

The production model life cycle

  1. Collect: Cover every operating condition
  2. Label: Written guidelines and review
  3. Train: Versioned code and data
  4. Validate: Grouped test set, agreed criteria
  5. Deploy: Staged, with rollback
  6. Monitor: Inputs, confidence, sampled review
Monitoring closes the loop back to data collection.
Random and grouped data splits
Random frame splitSplit by site or session
Effort to set upMinimalNeeds metadata
Leakage riskHigh with video or repeatsLow
Score realismOften optimisticCloser to field results
Data requiredLessMore distinct sources
Grouping by site, session or object takes more data but gives a more honest score.

What it changes

Realistic Scores

Validation results that hold up after deployment.

Early Warning

Drift is detected from monitoring rather than from customer complaints.

Traceability

Any model in the field can be reproduced and explained.

Safe Updates

New models can be trialled and withdrawn without disruption.

Where to go next

Consider shadow deployment, where a new model runs alongside the current one and its outputs are logged but not acted on, before switching over.

Budget for labelling field data over the product life. Retraining depends on it, and so does any evidence that a new model is better than the old one.

Problems this prevents

  • Test scores inflated by leaked data
  • An overall average hiding a failing condition
  • Accuracy drift nobody is measuring
  • A deployed model nobody can reproduce

Discuss your project

Email support@aavico.com or use Get in Touch when the website finishes loading.