A machine reads a pose. A human taught it.

Four ways to label an object. They answer four different questions. Pick the one that matches what the model needs to learn.

A box tells a model a runner is here. A mask tells it the exact pixels the runner occupies. Keypoints tell it the knee is bent, the arm is driving back, the trunk is leaning. ⁹³⁹

That's the distinction that decides your label budget. If you only need to find and count objects, boxes are cheaper. If you need to read posture, motion, or expression — a stride, a pinch, a smile — keypoints are the only type that carries it.

This is the cleanest line in computer-vision labeling: location vs. shape vs. structure.

Most guides tell you COCO stores keypoints as "a 3×17 array of x, y, v." Few show you the actual artifact. Here it is — the exact JSON a developer can copy.

{
  "categories": [
    {
      "id": 1,
      "name": "person",
      "keypoints": [
        "nose", "left_eye", "right_eye", "left_ear", "right_ear",
        "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
        "left_wrist", "right_wrist", "left_hip", "right_hip",
        "left_knee", "right_knee", "left_ankle", "right_ankle"
      ],
      "skeleton": [
        [16,14],[14,12],[17,15],[15,13],[12,13],[6,12],[7,13],
        [6,7],[6,8],[7,9],[8,10],[9,11],[2,3],[1,2],[1,3],
        [2,4],[3,5],[4,6],[5,7]
      ]
    }
  ],
  "annotations": [
    {
      "id": 1001,
      "image_id": 42,
      "category_id": 1,
      "bbox": [212, 88, 130, 410],
      "num_keypoints": 15,
      "keypoints": [
        267,120,2,  280,110,2,  255,110,2,  300,118,1,  240,118,1,
        330,200,2,  210,205,2,  360,290,2,  185,295,2,  375,370,2,
        170,378,2,  315,330,2,  235,332,2,  320,440,2,  232,442,2,
        0,0,0,      0,0,0
      ]
    }
  ]
}

Three things to read off this:

The keypoints array is flat. It's a length-3k list — for the 17 person keypoints that's 51 numbers: x1,y1,v1, x2,y2,v2, …. The order is fixed by the categories[].keypoints name list above it. ³⁷

The visibility flag is the third number in each triplet. v=2 visible, v=1 labeled-but-occluded, v=0 not labeled (and then x=y=0). In the example, both ankles are 0,0,0 — off-frame, so num_keypoints is 15, not 17. ²²³⁷

The skeleton is an edge list. Each pair is two 1-indexed keypoints to connect — [16,14] joins left_ankle to left_knee. The skeleton is for structure and visualization; it doesn't add data, it connects it. ³⁷

That's the whole schema. Points, visibility, edges. Drop it into any COCO-compatible pipeline and it trains.

"Keypoint annotation" is not one job. The dimensionality and the point count change with the question you're answering. Three flavors, cleanly:

2D keypoints — (x, y). A point on the image plane. This is the default and covers most of the field: sports pose, gesture, facial landmarks, animal tracking. COCO's 17-point person set is 2D. Use it when a flat camera view carries the signal you need. ³

3D keypoints — (x, y, z). Adds depth, so the pose reconstructs in space, not just on the picture. MediaPipe Pose and MediaPipe Hands both infer 3D landmarks; the z is a relative depth, not a true metric distance. Use it when the angle out of the plane matters — a hand reaching toward a VR controller, a body twisting away from the camera. ⁶⁷

Whole-body — COCO-WholeBody, 133 keypoints. Body + face + hands + feet in one annotation: 17 body, 68 face, 42 hand (21 per hand), 6 foot. Use it when one model has to read posture and expression and finger detail at once — sign language, fine-grained interaction, avatar capture. It's the most expensive to label, so reach for it only when you genuinely need all four regions. ¹²

The rule of thumb: start 2D. Add z only when depth changes the answer. Go whole-body only when the body, face, and hands all carry the lesson.

More points and more dimensions mean more cost and more places to be inconsistent. Every extra landmark is one more thing a human has to place the same way, every time. The schema you pick is a budget decision as much as a technical one.

Watch a sprinter at full speed. The eye sees a blur; a keypoint model sees structure — a point on each joint, connected into a skeleton, and now the motion has math: joint angles, stride length, cadence, left-right symmetry, where posture breaks down as fatigue hits.

This used to need a lab full of reflective markers. Now a model places the points straight from video — markerless sprint trackers report joint-angle errors of just 3.2–5.5° against the marker-based reference. ²⁸ It's gone prime time, too: Hawk-Eye tracks 29 skeletal points per player and has called semi-automated offside since 2022. ³⁰

But none of it runs without a human first. A model only knows a “knee” because annotators placed thousands of knees, precisely, at the joint center — the ground truth it learns the stride from. Humans teach the machine. Your AI is only as good as the people who taught it.

Two use cases. Same starting move: place a point.

Driver monitoring: the car learns to watch the driver

A driver monitoring system (DMS) watches one thing. The person behind the wheel.

It tracks the head. The eyes. Keypoints on the face that say where you are looking and how awake you are.

Drowsy? The eyelids close longer. Eye aspect ratio drops. Distracted? The head turns off the road. Researchers build these systems on facial-landmark and head-pose keypoints, computing eye-closure and yawn metrics frame by frame. ³²³³

None of that works without ground truth.

Someone has to label the eye corners. The nose. The head angle. Frame after frame, across drivers, lighting, glasses, night. The model only sees what a human taught it to see.

This is no longer a nice-to-have. Euro NCAP scores direct driver monitoring as part of its safety rating. Cars are expected to track eye gaze and head pose in real time, and to flag drowsiness at highway speeds. Inferring attention from steering input alone is no longer enough. ³⁴³⁵³⁶

That pressure is everywhere driver monitoring is headed — and it lands hardest where the roads are deadliest. Road crashes are the leading cause of death in Saudi Arabia, where Vision 2030 targets fewer than 10 road fatalities per 100,000 people by 2030. A car that can tell when a driver is falling asleep isn't a feature there — it's a life saved. ⁴⁰⁴¹

And every one of those cars ships a model trained on labeled keypoints.

Hands: the interface with no buttons

Now move to the hands.

A hand has 21 keypoints. Wrist, knuckles, finger joints, fingertips. MediaPipe Hands infers all 21 as 3D landmarks from a single camera frame. That is the map of a hand. ⁷

With that map, the machine reads what your hands are doing:

• AR and VR input. A pinch becomes a click. A grab becomes a hold. No controller. ³⁸
• Touchless interfaces. Wave, swipe, point. Control a screen without touching it. ⁴²
• Sign-language recognition. Track the hands, read the sign, turn it into text. Keypoint tracking sits at the core of these systems. ³⁷

But the model that found those 21 points? It was trained on hands a human labeled first. By Google's own account, MediaPipe's hand landmark model was built on roughly 30,000 real-world images, each manually annotated with 21 3D coordinates. ⁷

That is the whole point. The machine reads a hand because a person taught it where a hand's joints are.

We build the exact same keypoint datasets in the Annota8 Annotation Engine. Color-coded points on a face. Joints on a hand. A hand-authored gold standard as the correct answer. Export as standard JSON. One of 180 annotation UIs across 7 modalities, all live.

From a sleepy driver to a pinching finger, it starts with a point. And a human who placed it.

Book a demo →

Put enough points on a face, and the machine stops seeing pixels. It starts seeing a face.

That is the whole game. The same keypoints control that snaps a skeleton onto a sprinter snaps a mesh onto a human face. Same idea. Different landmarks.

Faces: filters, expressions, and proving you're real

A face filter has to know where your face is before it can put sunglasses on it. So you label the landmarks: eye corners, nose tip, lip edges, jawline. The classic dlib model uses 68 of them, trained on hand-annotated faces. Google's MediaPipe Face Mesh estimates 468 3D points on a single face from one ordinary camera — no depth sensor. ⁴³⁴⁴⁴⁵

Those points are how a filter tracks your nose when you turn your head.

Push further and you get expression analysis. Researchers code faces with the Facial Action Coding System (FACS) — adapted by Paul Ekman and Wallace Friesen and published in 1978 — which breaks expressions into "Action Units," the small facial-muscle movements that make a smile or a frown. Map the landmarks, track how they move, and a model can read the action units underneath. ⁴⁶⁴⁷

Then there's liveness. Is this a real face, or a photo held up to the camera? That's presentation attack detection, and it has a standard: ISO/IEC 30107-3, which defines how labs test a system's ability to tell a live person from a spoof. The training data behind it is annotated faces — real ones and fakes — labeled so the machine learns the difference. ⁴⁸⁴⁹

Our founders built this kind of perception at Affectiva and Smart Eye. The work always came back to the same thing: a model reads faces only as well as the people who labeled the faces it learned from.

Healthcare: precision where it counts

Medicine is where keypoints get serious.

In surgery, researchers track points on instruments — the tool tip, the tool base — so the system knows where a tool is in the frame. It's hard. The tips are small, the tools articulate, they get occluded, the lighting shifts, the video blurs. Every one of those points starts as a human annotation on a frame of surgical video. ⁵⁰⁵¹

In dentistry and orthodontics, the workhorse is cephalometric analysis: marking landmarks on a lateral skull X-ray to plan treatment. Doing it by hand is slow and inconsistent. Deep-learning models now place those landmarks automatically — but they were trained on radiographs a clinician labeled first. ⁵²⁵³

In rehab, there's gait analysis. Markerless pose estimation tracks body keypoints from ordinary video to measure how someone walks — validated in people post-stroke and with Parkinson's against reference motion-capture systems. ⁵⁴⁵⁵⁵⁶

One caution we hold to: these are research and assistive tools, not diagnoses. The model proposes. A clinician decides. We build the annotation; we don't make clinical claims.

One control, both worlds

A face mesh and a surgical X-ray look nothing alike. To the keypoints control, they're the same job: put a labeled point exactly where it belongs.

That's what the keypoints control in our Annotation Engine does — color-coded, labeled points placed at landmark precision against a hand-authored gold standard, exported as COCO-compatible JSON. One of 180 UIs across 7 modalities, all live.

From a selfie filter to a root canal, it starts with a point. And a human puts it there.

Book a demo →

A point on a sprinter's knee and a point on a salmon's tail are the same idea. Mark the landmark. Connect the structure. Teach the machine what matters.

Pose annotation left the gym a long time ago. It now runs in fields, in barns, and in sea cages.

Here is what that looks like in the real world.

Crops: where to cut, when to pick.

A strawberry isn't just "ripe or not." A harvest robot needs to know the exact spot to grasp and the exact spot to cut.

So researchers label keypoints on the fruit itself. One public dataset tags five points per strawberry: the picking point, the top and bottom of the fruit, and the left and right grasping points for the gripper. Ripeness gets labeled too — by color, from unripe (green/white) to half-ripe to fully ripe (red). ⁵⁷⁵⁸⁵⁹

Those labels are the ground truth. The robot learns where to reach by copying thousands of human-placed points.

Livestock: the limp you can't see by eye.

A lame cow costs money and signals pain. The earliest sign is a change in gait — subtle, easy to miss on a busy farm.

Keypoints catch it. Annotators mark anatomical points on the cow across video frames. One study used five keypoints along the back to measure arching, plus two head keypoints to catch head position. From those tracked points the model reads posture and gait, then flags the limp. ⁶⁰

Published systems report lameness accuracy in the high 80s to high 90s — one DeepLabCut-based tracker around 87–90%, another posture system 94–100% — all built on human-labeled animal pose. ⁶⁰⁶¹

Aquaculture: counting and measuring fish you can't touch.

You can't put a salmon on a scale every day. So farms measure it with cameras instead.

Annotators place keypoints on the fish — head, tail, fins — and the model learns to read size from those points. One offshore framework on cobia uses four keypoints: head, tail, left fin, right fin. A separate keypoint study reports fish-length estimates around 97% accurate, within roughly a centimeter of ground truth. ⁶²⁶³

Same workflow, different animal: structure marked by a human, measured by a machine.

Wildlife: behavior in the wild, no collar required.

Conservation teams now track animals by pose alone — wild chimpanzees and bonobos, whole elephant herds from drone footage, salmon in research tanks. Tools like DeepLabCut let researchers label keypoints on a few hundred frames, then track posture across hours of video. ^64656667

No tags. No collars. Just landmarks a human taught the model to find.

One method. A sprinter to a salmon.

Every one of these starts the same way: a person placing a point at a joint center, a fruit stem, a fin.

That's keypoint annotation in our Annotation Engine — labeled, color-coded points on an image, a human-authored gold standard as the correct answer, exported as standard JSON. One of 180 UIs across 7 modalities, all live.

The animal changes. The method doesn't. Humans teach the machine — and your AI is only as good as the people who taught it.

Ready to teach it to see? Book a demo at annota8.ai.

Occlusion is where most keypoint datasets quietly go wrong. A leg disappears behind the other leg. A hand goes behind a back. An annotator who guesses — or who skips inconsistently — teaches the model noise.

The fix is a written rule, applied the same way every time. COCO gives you the flag; you give it the policy.

Two rules make this airtight:

Estimate occluded, never invent absent. If a knee is behind a leg, you know roughly where it is — flag it 1 and place your estimate. If a foot is off the bottom of the frame, you don't — flag it 0 and leave it at the origin. ²²²³

Decide before you label, then enforce. Write the policy into the guideline with a picture for each row. Inconsistent occlusion handling is one of the quietest ways to poison a dataset — and one of the hardest to debug after training.

This is quality in. The model can't tell "gone" from "behind something" unless your annotators told it the difference, the same way, every time.

Quality keypoint annotation comes down to one rule: every annotator places the same point in the same spot, every time. Get that, and a model learns clean structure. Miss it, and you teach the machine your noise.

Here is the checklist we run.

1. Lock the ontology first. Decide exactly which points exist and what each one means before anyone labels. The body-pose standard is COCO's 17 keypoints — nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles. Hands, faces, and objects need their own ontology. Pick one and write it down. ⁶⁹

2. Write the guideline, with pictures. "Shoulder" is not an instruction. "Center of the gleno-humeral joint, even when the arm is raised" is. Define each landmark anatomically. Show a correct example and a wrong one for every point.

3. Demand joint-center precision. Keypoints are not "near the elbow." They sit on the joint center. A point that drifts a few pixels each time becomes a model that guesses. In our Annotation Engine, annotators place color-coded points — Shoulder, Elbow, Wrist, Knee — at joint-center precision on the image canvas.

4. Set an explicit occlusion policy. Real images hide things. COCO encodes this with a per-point visibility flag: v=0 not labeled, v=1 labeled but not visible (occluded), v=2 labeled and visible. Decide your rule up front — do annotators estimate a hidden knee, or skip it? Then enforce it. Inconsistent occlusion handling is one of the quietest ways to poison a dataset. ⁶⁸

5. Handle truncation and multiple subjects. A leg cut off by the frame edge. Two players overlapping. Your guideline must say what to do: which subject gets labeled, how points off-frame are treated, how to keep skeletons from crossing wires. Ambiguity here multiplies fast.

6. Measure inter-annotator agreement — then adjudicate. Have several people label the same images. Compare. The standard tolerance is spatial: how far apart, in pixels, do two annotators land on the same joint. When agreement drops on a specific joint, your guideline is ambiguous, not your annotators. Fix the guideline, retrain, re-measure. Send genuine disagreements to an adjudicator for a gold answer. ⁷⁰

7. Build a gold standard. A hand-authored "correct answer" set is how you score everyone against truth, not against each other. In our Engine it doubles as the benchmark for pre-annotation: a top-down model like ViTPose, paired with a person detector like YOLO, proposes points, a human corrects them, and the gold set catches drift. ⁷¹

Here is the part most teams underrate. COCO's own accuracy metric, OKS (Object Keypoint Similarity), scales the allowed error on each keypoint by how much human annotators disagreed on that point in the first place. COCO measured that spread across 5,000 redundantly labeled images — and a well-defined point like the eye gets a tighter tolerance than a fuzzy one like the hip. The metric is literally built from human consistency. Sloppy points do not just lower one label — they widen the bar the whole field is measured against. ⁶⁸⁶⁹

That's why annotation discipline shows up directly in the metric your model reports. How you place and flag a point decides the OKS it can score.

A pose model learns the skeleton you draw. A few points placed badly, repeated across a dataset, and the model learns the wrong joint. Quality in, quality out.

Humans teach the machine. Your AI is only as good as the people who taught it.

Keypoint annotation is expensive because a person places every point, on every instance, on every frame. Video makes it worse — a 30-second clip at 30 fps is 900 frames. Here's how teams actually keep the bill down without poisoning the data.

Model-assisted pre-labeling. A pose model (YOLO-pose, ViTPose, OpenPose, MediaPipe) drops a first-pass skeleton, and the human corrects instead of placing from scratch. Correcting is faster than authoring. The catch: a confident-but-wrong machine point is easy to rubber-stamp, so the human review stage is not optional. ²⁷⁷¹

Frame interpolation for video. Label keypoints on keyframes, let the tool interpolate the joints across the frames between, then spot-check and fix where motion breaks the interpolation. You pay for a fraction of the frames and review the rest.

Multi-view consistency. When you have synchronized cameras, a joint labeled in one view constrains where it sits in the others. One careful pass propagates.

Staged review. Annotate → peer review → adjudicate disagreements → gold-standard audit. Each stage costs time but catches the errors that cost far more after training.

Here's the number that frames all of it: across the data-annotation industry, overhead and rework eat roughly 60% of the budget. Wrong person labels it, it comes back, you redo it — and most of the spend never touched a correct label. That is the tax of bad process, not bad people.

That 60% is exactly the waste we built the Annota8 Annotation Engine to cut: pre-annotation to shrink the first pass, a gold standard to catch drift early, and agreement checks so rework happens before training, not after. The machine proposes, the human decides, and the gold set keeps both honest.

Quality in, quality out. Book a demo at annota8.ai.