Building a Training Dataset: Collection, Labelling and Bias

For most applied machine learning projects the dataset matters more than the choice of network. A model learns the patterns present in its examples, so missing conditions, inconsistent labels or accidental correlations all end up in its behaviour. Building a good dataset is a planned engineering activity with its own specification and quality checks. This article covers how to collect data, how to label it consistently and where bias comes from.

At a glance

Coverage
Examples of every class and condition the product will meet in use.
Guidelines
Written labelling rules that make two annotators draw the same box.
Agreement
Measured consistency between annotators, which tests the rules themselves.
Held out
A test set collected and frozen before any tuning begins.

The problem

Data is usually collected wherever it is easiest: one site, one camera, daytime, a few operators. A model trained on it can latch onto the background, the camera or the lighting instead of the object, and fail as soon as any of those change.

Labels carry errors too. Without clear rules, annotators disagree on where a box ends, whether a partly hidden object counts, or which class an unusual item belongs to, and the model learns that inconsistency.

Write the labelling guidelines before the first label, and expect to revise them after the first hundred.

Where bias comes from

Sampling bias arises when the collected data does not reflect the conditions of use. Class imbalance means rare but important cases have too few examples to learn from. Labelling bias appears when annotators interpret ambiguous cases differently or follow their own assumptions.

Counter each one deliberately: plan collection as a matrix of conditions, record metadata for every sample, label a shared subset twice to measure agreement, and review the disagreements to tighten the guidelines.

What good looks like

List the classes, conditions and edge cases the product must handle, and turn them into a collection plan with target counts.

Write labelling guidelines with examples of difficult cases, run a pilot batch and measure agreement between annotators.

Freeze a test set from separate sessions or sites, then label the remaining data with regular quality review.

Common label types
TaskLabelRelative effort
ClassificationOne class per image or clipLowest
Object detectionBox and class for each objectModerate
KeypointsNamed points on each objectModerate to high
Semantic segmentationA class for every pixelHigh
Instance segmentationAn outline for each separate objectHighest
Relative effort per image is a general guide and depends on object count and tooling.

From capture to usable dataset

  1. Plan: Conditions and target counts
  2. Capture: Record metadata with every sample
  3. Guidelines: Rules and difficult examples
  4. Pilot: Measure annotator agreement
  5. Label: With sampled review
  6. Freeze test set: From separate sessions
Quality checks sit inside the process, not at the end.

What it changes

Better Models

Consistent, representative data improves results more reliably than tuning.

Known Gaps

Metadata shows exactly which conditions are under-represented.

Fairer Behaviour

Performance is checked across groups and conditions, not just on average.

Reusable Asset

A documented dataset supports every later model version.

Where to go next

Where images or recordings include people, check consent and data protection requirements for each jurisdiction before collection starts.

Consider synthetic or augmented data to fill specific gaps, but always test on real data. A model can learn the artefacts of a simulator as easily as the objects in it.

Problems this prevents

  • A model that learned the background
  • Annotators following different rules
  • Rare cases with too few examples
  • Test data drawn from the training sessions

Discuss your project

Email support@aavico.com or use Get in Touch when the website finishes loading.