Bounding Boxes, NER, and Text Classification: A Plain-English Guide to Annotation Types

If you're building a machine learning project and need training data, one of your first decisions is which annotation type to use. It's a decision that's easy to get wrong — either by choosing a method that's more expensive than your use case requires, or by choosing a simpler method that doesn't capture what your model needs to learn. Here's a practical guide.

Image annotation types

Bounding boxes

A rectangle drawn around each object of interest in an image. This is the most common form of object detection annotation and the right choice for most detection tasks.

Use when: You want a model that can locate and identify objects — pedestrians, vehicles, products on shelves, faces in photos.

Limitations: Bounding boxes are imprecise around irregular shapes. A bounding box around a standing person also includes background pixels on either side of their body. For tasks where exact object boundaries matter, you'll need polygons or segmentation.

Polygon annotation

A multi-point outline that traces the exact boundary of an irregular object. More expensive to produce than bounding boxes — annotators have to carefully place 8–20+ points per object — but much more accurate for oddly shaped targets.

Use when: Shape matters. Annotating aerial imagery for building footprints, medical images for organ outlines, or manufacturing images for defect shapes.

Semantic segmentation

Every pixel in the image gets a class label. The result is a colour-coded mask where each colour represents a category. Used heavily in autonomous driving (road, pavement, sky, car, pedestrian, sign) and medical imaging.

Use when: You need pixel-level understanding, not just object locations. This is the most expensive annotation type — expect it to cost 3–5x more than bounding boxes per image.

Image classification

A single label for the entire image. Is this a defective product or not? Is this X-ray showing a fracture? Is this satellite tile forested or cleared?

Use when: You only need to know what an image is, not where specific things are within it. Fastest and cheapest to produce.

Text annotation types

Text classification

Assign a label to an entire piece of text. Sentiment (positive/negative/neutral), topic (finance/sport/tech), intent (complaint/enquiry/praise), or urgency (high/medium/low).

Use when: You want to route, filter, or aggregate text by category. Great for customer feedback pipelines, support ticket triage, and content moderation.

Named Entity Recognition (NER)

Highlight spans of text and tag them with an entity type: PERSON, ORGANISATION, LOCATION, DATE, PRODUCT, and so on. Annotators mark up the text like this: "[Kampala LOCATION] [Health Ministry ORGANISATION] has approved [Artemether DRUG]."

Use when: You need a model to extract specific types of information from free text — extracting company names from news articles, finding drug mentions in clinical notes, pulling dates from contracts.

Relation extraction

Beyond just identifying entities, mark the relationships between them. "Scriptring [FOUNDED_IN] 2023." This is more complex and expensive to annotate but enables richer information extraction.

Coreference resolution

Link pronouns and noun phrases to the entity they refer to. "The CEO announced results. She said revenue grew 20%." Annotating that "she" refers to "the CEO" trains models to track entity references across a document.

Audio annotation types

Transcription

Convert speech to text, optionally with speaker labels and word-level timestamps. Foundation for speech recognition training.

Audio event detection

Tag time segments with what type of sound they contain: speech, music, background noise, laughter, machinery. Used in smart home devices, security systems, and broadcast monitoring.

Choosing between annotation types: a decision framework

What does your model need to output? If it needs to locate objects, you need bounding boxes minimum. If it needs pixel masks, you need segmentation. If it needs to extract entity types, you need NER.
What's your accuracy requirement? Higher accuracy requirements push you towards more expensive annotation types and double-key verification.
What's your budget? Classification is cheapest. Segmentation and relation extraction are most expensive. Map your model requirements against your annotation budget before you start — it's hard to switch mid-project.
What's your data volume? Some annotation types that are expensive at 1,000 samples become cost-effective with good tooling at 100,000 samples. Plan for scale.