Building a Training Dataset: Collection, Labelling and Bias
For most applied machine learning projects the dataset matters more than the choice of network. A model learns the patterns present in its examples, so missing conditions, inconsistent labels or accidental correlations all end up in its behaviour. Building a good dataset is a planned engineering activity with its own specification and quality checks. This article covers how to collect data, how to label it consistently and where bias comes from.
At a glance
- Coverage
- Examples of every class and condition the product will meet in use.
- Guidelines
- Written labelling rules that make two annotators draw the same box.
- Agreement
- Measured consistency between annotators, which tests the rules themselves.
- Held out
- A test set collected and frozen before any tuning begins.
The problem
Data is usually collected wherever it is easiest: one site, one camera, daytime, a few operators. A model trained on it can latch onto the background, the camera or the lighting instead of the object, and fail as soon as any of those change.
Labels carry errors too. Without clear rules, annotators disagree on where a box ends, whether a partly hidden object counts, or which class an unusual item belongs to, and the model learns that inconsistency.
Write the labelling guidelines before the first label, and expect to revise them after the first hundred.
Where bias comes from
Sampling bias arises when the collected data does not reflect the conditions of use. Class imbalance means rare but important cases have too few examples to learn from. Labelling bias appears when annotators interpret ambiguous cases differently or follow their own assumptions.
Counter each one deliberately: plan collection as a matrix of conditions, record metadata for every sample, label a shared subset twice to measure agreement, and review the disagreements to tighten the guidelines.
What good looks like
List the classes, conditions and edge cases the product must handle, and turn them into a collection plan with target counts.
Write labelling guidelines with examples of difficult cases, run a pilot batch and measure agreement between annotators.
Freeze a test set from separate sessions or sites, then label the remaining data with regular quality review.
| Task | Label | Relative effort |
|---|---|---|
| Classification | One class per image or clip | Lowest |
| Object detection | Box and class for each object | Moderate |
| Keypoints | Named points on each object | Moderate to high |
| Semantic segmentation | A class for every pixel | High |
| Instance segmentation | An outline for each separate object | Highest |
From capture to usable dataset
- Plan: Conditions and target counts
- Capture: Record metadata with every sample
- Guidelines: Rules and difficult examples
- Pilot: Measure annotator agreement
- Label: With sampled review
- Freeze test set: From separate sessions
What it changes
Better Models
Consistent, representative data improves results more reliably than tuning.
Known Gaps
Metadata shows exactly which conditions are under-represented.
Fairer Behaviour
Performance is checked across groups and conditions, not just on average.
Reusable Asset
A documented dataset supports every later model version.
Where to go next
Where images or recordings include people, check consent and data protection requirements for each jurisdiction before collection starts.
Consider synthetic or augmented data to fill specific gaps, but always test on real data. A model can learn the artefacts of a simulator as easily as the objects in it.
Problems this prevents
- A model that learned the background
- Annotators following different rules
- Rare cases with too few examples
- Test data drawn from the training sessions
Discuss your project
Email support@aavico.com or use Get in Touch when the website finishes loading.