From Prototype to Production AI: Data, Validation and Monitoring
A proof of concept shows that a model can work. A product needs it to keep working on inputs nobody collected during development: new lighting, new sites, ageing sensors and changes in what is being observed. Getting there is mostly engineering around the model rather than the model itself. This article covers the data, validation and monitoring practices that turn a prototype into something you can ship and support.
At a glance
- 3 splits
- Training, validation and test sets, kept apart so the test score means something.
- Leakage
- Test data that resembles training data too closely, inflating the score.
- Drift
- Gradual change in real inputs that erodes accuracy without any error message.
- Versioned
- Each deployed model traceable to its code, data and settings.
The problem
Prototype datasets are usually small and gathered in one place under convenient conditions. Scores measured on them say little about a factory floor at night or a camera mounted at a different angle.
Leakage makes this worse. When frames from the same video or the same physical object appear in both training and test sets, the test score rewards memorisation, and the field result is lower than anyone expected.
A test score only predicts the field when the test data was collected the way the field will be.
Validation that predicts the field
Split data by site, session, device or object rather than by individual frame, so the test set contains situations the model has never seen. Report results per condition as well as overall, because an average can hide a condition where the model fails.
Set acceptance criteria from the cost of each error type before testing begins. In production, true labels are rarely available, so monitor input statistics and confidence distributions, and sample predictions for human review.
What good looks like
Write down the conditions the product will meet and check the dataset covers each one with enough examples.
Freeze a grouped test set, agree pass criteria, and record the data and code version behind every candidate model.
Deploy with monitoring and a rollback path, and feed reviewed field samples back into the next training round.
The production model life cycle
- Collect: Cover every operating condition
- Label: Written guidelines and review
- Train: Versioned code and data
- Validate: Grouped test set, agreed criteria
- Deploy: Staged, with rollback
- Monitor: Inputs, confidence, sampled review
| Random frame split | Split by site or session | |
|---|---|---|
| Effort to set up | Minimal | Needs metadata |
| Leakage risk | High with video or repeats | Low |
| Score realism | Often optimistic | Closer to field results |
| Data required | Less | More distinct sources |
What it changes
Realistic Scores
Validation results that hold up after deployment.
Early Warning
Drift is detected from monitoring rather than from customer complaints.
Traceability
Any model in the field can be reproduced and explained.
Safe Updates
New models can be trialled and withdrawn without disruption.
Where to go next
Consider shadow deployment, where a new model runs alongside the current one and its outputs are logged but not acted on, before switching over.
Budget for labelling field data over the product life. Retraining depends on it, and so does any evidence that a new model is better than the old one.
Problems this prevents
- Test scores inflated by leaked data
- An overall average hiding a failing condition
- Accuracy drift nobody is measuring
- A deployed model nobody can reproduce
Discuss your project
Email support@aavico.com or use Get in Touch when the website finishes loading.