---
title: "Testing Pose Estimation Accuracy with a Regression Corpus"
canonical: "https://aifitnessapi.com/test/pose-detection-accuracy"
cluster: "Testing"
primary_query: "how to test pose estimation accuracy"
last_reviewed: "2026-07-27"
description: "Build a labelled video regression suite with per-keypoint tolerances and a pass/fail budget, so a pose model upgrade cannot quietly cost you reps."
publisher: "AIFitnessAPI — independent, not sponsored"
cite_as: "\"Testing Pose Estimation Accuracy with a Regression Corpus\", AIFitnessAPI, https://aifitnessapi.com/test/pose-detection-accuracy"
---

# Testing Pose Estimation Accuracy with a Regression Corpus

> Write a test that replays a fixed set of labelled clips through an explicitly pinned model revision and fails when any keypoint your product actually reads drifts outside a per-keypoint tolerance. The constraint is that no published benchmark figure tells you anything about your camera, your exercises or your users, so every threshold has to be derived from footage you labelled yourself and from the displacement at which your own rep verdict flips. Score per keypoint rather than as one aggregate, because a mean over nineteen joints hides the ankle regression that breaks your squat counter. And label occlusion as a state rather than scoring it as a miss, or the suite will punish the model for correctly admitting it cannot see a hidden joint.

- Canonical: https://aifitnessapi.com/test/pose-detection-accuracy
- Last reviewed: 2026-07-27
- Publisher: AIFitnessAPI (https://aifitnessapi.com) — independent, not sponsored
- Cite as: "Testing Pose Estimation Accuracy with a Regression Corpus", AIFitnessAPI, https://aifitnessapi.com/test/pose-detection-accuracy

---

You bump a pose SDK by one minor version, or ship against a new OS that quietly rolls the vision framework's model forward. Nothing crashes. No test goes red, because you have no test. The ankle keypoint at the bottom of a squat now lands a little lower than it used to, your knee-angle threshold stops getting crossed on the shallower reps, and the counter starts dropping roughly one rep in a dozen — but only for the users who film from a three-quarter angle in a dim living room, which is most of them. Three weeks later support has a pile of tickets that all say "it stopped counting properly." Nobody connects them to the upgrade, because the upgrade was a one-line diff in a lockfile and it went green.

That is the failure this page exists to catch, and it is catchable — but almost none of the work is in the test runner. Everything hard sits in two places: the corpus you label and the tolerance you derive. The harness is a weekend, and the rest of this page is the two hard parts.

## The corpus is the artefact; the harness is trivial

A small, well-chosen corpus beats a large sloppy one, and it is not close. Labelling is the only expensive part of this, so every clip has to earn its place by being able to distinguish a working model from a broken one. The rule we use: **each clip varies one thing.** A clip that is simultaneously dim, oblique, and shot in a baggy hoodie tells you nothing when it fails, because you cannot attribute the failure. Three clips that each vary one of those tell you which one the new model version broke.

The axes worth spanning, in rough order of how often they change a verdict in practice:

- **Camera angle** relative to the plane of motion — front-on, side-on, and the three-quarter angle that real users default to because that is where the tripod fits.
- **Framing and distance.** Apple's own guidance for its body-pose request, documented on the *Detecting human body poses in images* page and current as of July 2026, is that "the subject's height should ideally be at least a third of the overall image height" and that "a large portion of the subject's key body regions and points should be present in the image." Those two sentences define the boundary between a clip that is diagnostic and a clip that is simply unfair.
- **Occlusion sources** — a couch arm across the shins, a barbell across the shoulders, one limb behind the torso at the bottom of a lunge.
- **Clothing.** Apple documents that "a subject wearing flowing or robe-like clothing reduces the detection accuracy." That is a known-hard condition, not a bug, and it should be labelled as such rather than allowed to fail your gate.
- **Body size and skin tone**, because a corpus that is four colleagues in a well-lit office is a corpus that will pass forever while the product degrades for everyone else.
- **Tempo**, including the deliberately sloppy fast set that smears the extremities.

Two categories, scored differently. **Fair clips** sit inside the documented operating envelope; a regression here is a hard failure. **Known-hard clips** sit outside it deliberately; they are tracked as a trend, and they exist so that the day a model version makes them dramatically worse you find out, without them holding your release hostage every week.

On how much to label: do not label every frame, and do not pick a target count out of the air. Label the **decision frames** — the frames where your feature actually forms a verdict. For a rep counter that is the top and bottom of each repetition plus two or three mid-phase frames, because those are the frames where a few normalized units of drift flips an outcome. Frames in the middle of a smooth concentric phase cost the same to label and prove almost nothing. Grow the corpus by adding a clip every time production produces a failure you could not reproduce; that is the only sizing heuristic that survives contact with reality.

## Per keypoint, never aggregate

Your rep logic reads maybe four joints. Apple's Vision body-pose observation exposes up to 19; other model families expose more. A mean error over all of them is a number in which your ankle regression is being averaged against fifteen keypoints you never read, most of which sit on the torso and are the easiest to predict. You can degrade the joint your product depends on and watch the aggregate improve.

The standard metrics concede this themselves. PCKh normalizes its threshold by head size; the COCO OKS convention scales each keypoint's error by person size *and* a per-keypoint constant, precisely because a wrist and a hip do not deserve the same allowance. [What those metrics measure and where each applies](/motion/pose-estimation-accuracy) is a separate discussion; the point here is narrower. For regression detection you want the per-keypoint residual, not the rolled-up score, because the rolled-up score is designed for ranking models against each other and you are not ranking models. You are asking one question: did this joint get worse.

### Deriving the tolerance, since you cannot copy one

There is no correct pixel threshold. Anyone who gives you one is giving you a number from their product, their camera, and their exercise set. The tolerance is not a computer-vision constant at all — it is a property of your downstream decision, and it is derived like this:

1. **Name the scalar your feature thresholds.** Knee angle crossing some value. Wrist height relative to shoulder. Whatever your rep or form logic actually compares.
2. **Perturb the ground truth, not the model.** Take a labelled decision frame, displace one joint's true position by increasing amounts, and recompute the verdict each time. Find the displacement at which the verdict flips. That is the break-even for that joint, on that exercise, at that camera angle.
3. **Set the tolerance to a fraction of break-even.** The fraction is your safety margin and it is a judgement call, not a measurement. Ours is well under half, on the grounds that errors compose across joints and across frames.
4. **Express it in a scale-normalized unit.** Raw pixels are meaningless across a corpus with mixed resolutions and mixed subject distances. Apple returns recognized points in "normalized coordinates (0.0 to 1.0), with the origin at the bottom-left," which handles resolution but not how far away the person is standing. Divide by a body-derived length — torso length or person bounding-box height — so a clip shot from three metres and one shot from one metre are comparable. This is the same reasoning that produced PCKh's head-size normalization.

Run that procedure per joint and per exercise and you get a small table of tolerances that are defensible in a review, because every cell traces back to a measured verdict flip in your own product. **Any number we printed here would be wrong for you.** It has to be measured, and it has to be re-measured when the rep logic changes, which is a maintenance cost worth budgeting for honestly.

Joints your product does not read still get a tolerance, a much wider one, and they report rather than gate. They are your early warning that something changed structurally in the model even when nothing user-visible has broken yet.

## Occlusion is a label, not a miss

The most common way these suites go wrong is punishing the model for being honest. If the left ankle is behind the right calf at the bottom of a lunge, there is no visual evidence for it. A model that declines to report it is behaving correctly. A naive pixel-error metric scores that as a maximum-error failure and you will end up tuning your suite until it prefers the model that guesses.

So ground truth carries a state per joint per frame, not just a coordinate:

- **Visible** — score the position against the tolerance.
- **Occluded** — the joint is in frame but hidden. Do **not** score position. Assert instead that the model's confidence for that joint dropped.
- **Out of frame** — the joint left the image. The correct output is absence or a confidence at the floor. Apple's documented rule for its body-pose points is to "ignore any recognized points with a confidence value of 0, because they're invalid," which is the framework telling you that absence is a legitimate answer.

The asymmetry matters. A model that returns a confident, plausible-looking coordinate for a hidden joint is strictly worse than one that returns a low-confidence coordinate, and a position-only metric scores them identically whenever the guess happens to land near the truth. Scoring the *confidence* on occluded frames is what catches a version bump that made the model more assertive without making it more correct — which then breaks the confidence gate that [tuning guidance for pose accuracy](/guides/improve-pose-detection-accuracy) tells you to put in front of your angle math. Your gate threshold is only correct if the confidence signal behind it still behaves; this is the test that proves it does.

```python
# Tolerances come from tolerances.yaml, generated by the break-even procedure
# above. The file is never hand-edited: a threshold nobody measured is a
# threshold nobody can defend in review.
def score_frame(observed, truth, tol):
    for joint, label in truth.items():
        point = observed.points.get(joint)
        conf = observed.confidence.get(joint, 0.0)

        if label.state == "out_of_frame":
            # Absence, or confidence at the floor. Not a guess.
            yield Check(joint, ok=(point is None or conf <= tol.absent_confidence))

        elif label.state == "occluded":
            # Position is unscored. Assert the model admits it cannot see.
            yield Check(joint, ok=(conf <= tol.occluded_confidence))

        else:
            if point is None:
                yield Check(joint, ok=False, why="visible joint not reported")
                continue
            err = distance(point, label.point) / observed.torso_length
            yield Check(joint, ok=(err <= tol.visible[joint]), residual=err)
```

## Pin the model version so an upgrade is a visible diff

This is the whole mechanism. If the model version floats, a regression and an upgrade look identical in your git history, and the suite tells you *that* something changed without telling you *what*.

On iOS, Vision gives you an explicit handle for this: the legacy `VNDetectHumanBodyPoseRequest` documents the constant `VNDetectHumanBodyPoseRequestRevision1`, and the modern `DetectHumanBodyPoseRequest` exposes `revision`, `supportedRevisions` and a nested `Revision` type. Set the revision explicitly in production code *and* in the test, and add an assertion that the set of supported revisions is the one you expect. When an OS update adds a revision, that assertion fails, and the failure is the review prompt: someone has to look at the new revision's numbers against the corpus and decide, deliberately, to move.

If your model ships as a vendored asset file instead, hash it and assert the hash. Then record, in a committed results file, the revision or hash, the OS version, the device or runner identity, and the per-keypoint residuals. **Commit that results file.** The diff on it is the artefact a reviewer reads — a pull request that bumps an SDK and moves forty residuals is a very different review from one that moves none.

One caution: pose output is not guaranteed bit-reproducible across hardware, OS version, or compute unit. Treat the committed baseline as per-configuration. Changing the runner is a re-baseline, done knowingly, not a pass.

## Running the corpus on iOS, with the revision pinned

Feed each clip through `VNVideoProcessor`, whose `analyze(_:)` Apple documents as synchronous and whose `RequestProcessingOptions.cadence` lets you analyse only the frames your ground truth covers. [What the simulator and emulator can actually do with a camera](/test/camera-features-without-a-device) sets out the frame-source constraints and the two qualifications that come with borrowing an offline-analysis API as a harness. What belongs here is the one line the rest of this page depends on — the pinned revision:

```swift
import Vision

func poseFrames(in clip: URL, over range: CMTimeRange) throws -> [Frame] {
    var frames: [Frame] = []

    let request = VNDetectHumanBodyPoseRequest { request, _ in
        guard let obs = request.results?.first as? VNHumanBodyPoseObservation else { return }
        var points: [VNHumanBodyPoseObservation.JointName: CGPoint] = [:]
        var confidence: [VNHumanBodyPoseObservation.JointName: Float] = [:]
        for name in obs.availableJointNames {
            guard let p = try? obs.recognizedPoint(name) else { continue }
            points[name] = p.location        // normalized, origin bottom-left
            confidence[name] = p.confidence
        }
        frames.append(Frame(points: points, confidence: confidence))
    }

    // Pinned on purpose. If this constant stops being the one you support,
    // the suite should fail before the numbers do.
    request.revision = VNDetectHumanBodyPoseRequestRevision1

    let processor = VNVideoProcessor(url: clip)
    let options = VNVideoProcessor.RequestProcessingOptions()
    // Set options.cadence (FrameRateCadence / TimeIntervalCadence) so you
    // process exactly the frames your ground truth covers.
    try processor.addRequest(request, processingOptions: options)
    try processor.analyze(range)   // synchronous
    return frames
}
```

Compile that before you trust it: `frames` is a captured `var` mutated from the request's completion handler, which strict concurrency checking rejects, so collect into a class box or an actor-isolated collector instead. And do not plan your CI topology around the assumption that the body-pose request executes on the iOS Simulator at all — we found no Apple statement either way, so measure it on your own runner first.

## The pass/fail budget is a product decision

Someone has to answer "how much worse is unacceptable," and it is not the engineer who wrote the pose pipeline. The question in its real form is: is it acceptable for the counter to miss one rep in fifty for users filming from a three-quarter angle? That is a product call about a user-visible promise, and it should be made once, written down, and then encoded as the budget rather than re-litigated per pull request while a release is blocked.

The shape that has worked for us is two tiers. A **hard gate**: any per-keypoint breach on any fair-condition clip fails the build, no budget, no averaging. And a **tracked budget** on the known-hard clips: a breach rate that is allowed to be non-zero, reported on every run, and reviewed when it moves. The hard gate is binary because averaging is how a real regression gets absorbed; the budget exists because a suite that goes red for a condition you already documented as out-of-envelope trains people to ignore it.

Then the part everyone skips. **A tolerance wide enough that nothing can fail it is not coverage, it is a screenshot.** If your current model passes every clip with enormous headroom, you have not proven the model is good; you have proven the thresholds are loose. Verify the suite can fail: take a passing clip, inject a synthetic displacement at exactly the break-even you measured, and assert the suite goes red. If it does not, the budget is decorative and you should tighten it until it bites. Run that mutation check in CI alongside the real one, because it is the only thing standing between you and a green pipeline that means nothing.

## What this does not catch

Be clear with yourself about the ceiling here, because a file-driven suite is a model test, not a product test.

It does not exercise the camera pipeline at all. Thermal throttling on a long session, dropped frames when the encoder contends with inference, autoexposure hunting in a dim room, a three-year-old NPU that runs the same model at a third the frame rate — a decoded MP4 shows you none of it, and the only thing that will is real hardware. [Deciding what genuinely needs a device and what does not](/test/device-lab-and-ci) is where that line gets drawn.

It also does not tell you the rep counter is correct. Keypoint accuracy is an input to that; counting is a classification problem and wants precision and recall over a scored corpus with partial reps and tempo changes, which is [a different corpus and a different metric](/test/rep-counting).

And the labels themselves have error. Two people labelling the same ankle will not agree exactly, and a tolerance tighter than your inter-labeller spread is measuring your labellers rather than your model. Double-label a subset, measure the spread, and treat it as the floor below which no tolerance is meaningful. That measurement takes an afternoon and it is the difference between a suite people trust and a suite people mute.

Finally: this corpus is video of real people doing exercise in their homes. It is consent-bearing, retention-bearing content that will sit in your repository or object store for years. Decide who is in it and on what terms before you record it, not after.

Build the corpus small and deliberate, derive every threshold from a verdict that actually flips, score per keypoint with occlusion as a first-class label, pin the revision so an upgrade is a diff someone has to read, and make sure the thing can go red. Then the next model bump is a decision instead of a surprise.

## FAQ

### Why would a mean keypoint error improve while the rep counter gets worse?

Because the mean is averaging the joint you depend on against fifteen you never read. Your rep logic probably reads four joints. Apple's Vision body-pose observation exposes up to nineteen and other model families expose more, and most of the extra ones sit on the torso, which is the easiest region in the frame to predict. So a version that got better at shoulders and worse at ankles can move the aggregate in the flattering direction while your squat counter stops crossing its knee-angle threshold on the shallower reps. Score the per-keypoint residual instead. Rolled-up scores exist for ranking models against each other, and you are not ranking models — you are asking whether one specific joint got worse.

[Permalink](https://aifitnessapi.com/test/pose-detection-accuracy#faq-1)

### What pixel tolerance should I set for each keypoint?

One you measured, in a scale-normalized unit, and nobody can give it to you. The tolerance is not a computer-vision constant, it is a property of your own downstream decision. Derive it by naming the scalar your feature thresholds, such as a knee angle, then taking a labelled decision frame and displacing that joint's ground truth by increasing amounts until the verdict flips. That displacement is your break-even, and the tolerance is a fraction of it, with the fraction being an explicit safety margin rather than a measurement. Express the result normalized by a body-derived length such as torso length or bounding-box height, not in raw pixels, so clips at different resolutions and standing distances are comparable. That is the same reasoning behind head-size normalization in PCKh. Re-derive whenever the rep logic changes.

[Permalink](https://aifitnessapi.com/test/pose-detection-accuracy#faq-2)

### How should the suite score a keypoint that was genuinely occluded?

By scoring confidence rather than position, otherwise you will tune the suite until it prefers a model that guesses. Ground truth should carry a state per joint per frame with three values: visible, occluded, and out of frame. Visible joints get their position compared against the tolerance. Occluded joints get their position unscored and their confidence asserted to have dropped, because a hidden joint has no visual evidence and declining to report it is correct behaviour. Out-of-frame joints should be absent or at the confidence floor. Apple's documented rule for its body-pose points is to ignore recognized points with a confidence value of zero because they are invalid, which is the framework telling you absence is a legitimate answer. The asymmetry matters: a model that returns a confident plausible coordinate for a hidden joint is worse than one that returns low confidence, and a position-only metric scores them identically whenever the guess lands near the truth.

[Permalink](https://aifitnessapi.com/test/pose-detection-accuracy#faq-3)

### Should a pose accuracy regression fail the build, or just get reported?

Both, split by clip category, and the split is a product decision rather than a technical one. The real question is whether it is acceptable for the counter to miss one rep in fifty for users filming from a three-quarter angle, and that is a call about a user-visible promise. Encode it once instead of relitigating it per pull request while a release is blocked. Two tiers work: a hard gate where any per-keypoint breach on a fair-condition clip fails the build with no averaging, and a tracked budget on clips that are deliberately outside the documented operating envelope, reported every run and reviewed when it moves. The gate is binary because averaging is exactly how a real regression gets absorbed. The budget exists because a suite that goes red for a condition you already documented as hard trains everyone to ignore it.

[Permalink](https://aifitnessapi.com/test/pose-detection-accuracy#faq-4)

### How do I make a pose SDK upgrade show up as a visible diff?

Pin the model version in both production code and the test, then commit the numbers. On iOS the body-pose request carries an explicit revision, and Apple documents both the revision constant and a supported-revisions list, so you can assert that the set of revisions you expect is the set that exists. When an OS update adds one, that assertion fails and becomes the prompt for someone to look at the new revision against the corpus and move deliberately. If the model ships as a vendored asset file instead, hash it and assert the hash. Write a results file recording the revision or hash, the OS version, the runner identity and the per-keypoint residuals, and commit it, because the diff on that file is what a reviewer actually reads. One caution: output is not guaranteed bit-reproducible across hardware or OS, so a baseline is per-configuration and changing the runner is a knowing re-baseline, not a pass.

[Permalink](https://aifitnessapi.com/test/pose-detection-accuracy#faq-5)