How to Test a Rep Counting Algorithm

A user finishes a set of twelve and the counter reads eleven. There is no ambiguity here for you to hide in. This is not a VO2 max estimate the user cannot check, or a calorie number they will never verify — it is a small integer they were counting in their own head at the same time your app was counting it, and you got it wrong. They know. They will look at the number, look at the barbell, and decide the app is broken. Everything else you built is now decoration on top of a thing that cannot count to twelve.

That is the failure this page exists to catch, and it is worth being precise about why it is hard to catch before shipping. The rep counter you built is a state machine with a handful of knobs — the two hysteresis thresholds, the smoothing constant, a minimum-amplitude gate, a minimum-time gate. Every one of those knobs is global, and every complaint that arrives about them is local: "squats double-count on my phone". Someone widens the hysteresis gap, the squat ticket closes, and three weeks later a cohort of bench-press users are quietly losing a rep per set because the return threshold now sits above where their arms actually finish. Nobody re-ran push-ups. There was nothing to re-run.

So the assertion is this: a fixed corpus of labelled video, scored event-by-event against human ground truth, reported as precision and recall per exercise, and committed as a baseline that fails the build when any single clip moves. Not "I did ten push-ups in front of my laptop and it said ten."

Score at the event level, not at the end of the clip#

The first instinct is to store an expected count per clip and assert on it. Do not. A rep counter has two independent error modes that cancel: if the counter drops the fourth rep and fires a phantom on the walk back to the bar, the total is correct, the assertion passes, and you have shipped two bugs. Total-count assertions are exactly the class of test the rest of this cluster keeps warning about — an assertion that cannot fail on the thing you actually care about reads as coverage and is worse than nothing.

Concretely: each rep is an event with a timestamp; the ground truth is a list of those timestamps; the prediction is a list of the moments your counter incremented. Match them one to one within a tolerance window and you get true positives, false positives and false negatives, which is the ordinary vocabulary of a classifier. Rep counting is a classification problem — over every instant of a video, did a rep complete or not — and once you frame it that way the metrics stop being a choice.

Precision is the share of the reps you counted that really happened. Recall is the share of the reps that really happened that you counted. Report both, per exercise, per camera setup. Never pool them into one site-wide number: a pooled figure is dominated by whichever exercise you happened to film the most of, which in our experience is always squats, because squats are the easiest thing to film in an office.

The tolerance window is where this test quietly stops working#

A rep is an interval, not an instant, so both the label and the prediction need the same convention for when the rep happened. Pick the moment the counter is contractually supposed to increment — for the return-to-start state machine described in how rep counting works, that is the top of the movement — and write it into the labelling policy. If truth marks the bottom and prediction marks the top, every match either fails or succeeds by luck.

Then the window. A window that is too wide is the second way this suite becomes untestable: if the window is wider than half the gap between two adjacent labelled reps, a single prediction sits within range of both, and depending on your matching order you can score a perfect run on a counter that fired once. Our recommendation is to derive the window from the corpus rather than pick a round number, and to assert on that derivation in the harness itself:

def score(truth, predicted, window):
    """One-to-one nearest match of predicted rep events to labelled ones."""
    unmatched = sorted(predicted)
    tp, fn = [], []
    for t in sorted(truth):
        near = [p for p in unmatched if abs(p - t) <= window]
        if near:
            hit = min(near, key=lambda p: abs(p - t))
            unmatched.remove(hit)
            tp.append((t, hit))
        else:
            fn.append(t)
    return len(tp), len(unmatched), len(fn)


def assert_window_is_meaningful(truth, window):
    ordered = sorted(truth)
    gaps = [b - a for a, b in zip(ordered, ordered[1:])]
    assert all(g > 2 * window for g in gaps), (
        "matching window exceeds half the closest labelled rep pair; "
        "one prediction could satisfy two labels and the score is meaningless"
    )

The second function is not decoration. It is the guard that stops the fastest clip in your corpus — the one with the tempo change, the one you most want to test — from silently becoming unscoreable the day someone widens the window to make a flaky clip go green.

What belongs in the corpus#

Clean reps are the cheapest clips to film and the least useful. They will all pass, forever, and they will tell you nothing. The corpus earns its keep on the awkward middle ground, and the taxonomy below is the one we would build first. Every entry is a real thing people do in front of a phone.

Reps that are not reps. These are where false positives live, and they are the clips teams forget to film because nothing happens in them.

• A rep abandoned halfway. The user descends into a squat, decides it is too heavy, and stands back up. The signal traces most of a cycle. Truth: zero reps.
• Adjusting clothing mid-set — pulling a shirt down, hitching shorts, wiping a face. Arms move through large arcs at roughly rep tempo. Truth: zero reps.
• Racking, re-gripping, repositioning the feet, walking out of frame to get water and walking back.
• A stretch or a warm-up movement between sets that shares a joint with the exercise.

Reps that happened but are hard to see. These are where false negatives live.

• A paused rep. A three-second hold at the bottom of a squat is a legitimate rep and a direct attack on any minimum-time or maximum-time gate in the state machine.
• Tempo change within one set. Fatigue slows the last three reps of a set of twelve. Any gate calibrated on the first three will mis-fire on the last three, and this is the single most common realistic failure we would test for.
• Off-frame limbs. The wrists leave the top of the frame on an overhead press; the feet leave the bottom on a phone propped too high. The rep still happened.
• Occlusion by equipment — a bench, a barbell across the chest, a knee hidden behind the other knee at an oblique angle.

Scene problems. These are where the counter does something surprising rather than merely wrong.

• Two people in frame. A partner walks behind the user mid-set. Does the tracker switch skeletons? Does the count jump? This is the case where a counter fails loudly and inexplicably rather than by one, and the way you handle it is upstream of the state machine entirely.
• A mirror, which the pose stage cheerfully reads as a second person.
• A deliberately bad camera angle. Film one clip per exercise from the worst plausible position — a phone on the floor pointing up, or a forty-five-degree angle where the joint you measure projects into depth. The correct expected behaviour here is a product decision and it should be an explicit one: either the counter degrades gracefully, or it refuses and tells the user to move the camera. What it must not do is count confidently and wrongly.

And the negative clip nobody films. Camera running, person visible, moving normally, not exercising. Expected count: zero. It is the cheapest clip in the corpus and it is the one that catches the phantom-rep class of bug before a user does.

Tag every clip with which of these it exercises, because the tags are how you read a regression later. A failure across every clip tagged paused_rep is a diagnosis; a failure of "four clips" is a mystery.

Ground truth, and who is allowed to produce it#

Ground truth is what the human did. It is not what the camera could reasonably have seen, and the distinction matters more than anything else in this section. When a rep happens with the wrists out of frame, that is a rep, and it goes in the label file as a rep. The temptation to label around your implementation — to quietly mark it "not visible, excluded" — feels like fairness and is actually the act of defining away your hardest failure mode. If your recall number does not carry the cost of the off-frame rep, your recall number is measuring your own assumptions.

Our recommendation on who labels: the person who performed the set does the first pass, immediately, while they still remember. They are the only person alive who knows whether the movement at 0:34 was a rep they abandoned or a rep they finished badly, and intent is not recoverable from video. A second labeller then passes over the same clip cold, without seeing the first labels. Disagreements are never averaged — a rep is not a continuous quantity — they are escalated to the labelling policy, and if the policy does not already cover the case, the policy gets a new clause and every previously labelled clip that the clause touches is re-labelled. That re-labelling cost is the reason to write the policy before you film, not after.

The policy is a short document and it has to answer, per exercise, at least these: what range of motion constitutes a rep, what happens to an abandoned rep, what happens to a paused rep, and what happens to a rep the camera could not see. That last one is settled above. The first one is the genuinely contested one, and it is not an engineering decision. It belongs to whoever owns the product, because "was that shallow squat a rep" is a question about what your app is promising the user.

Our strong opinion on that particular question: a rep with poor depth should usually be counted and flagged, not silently dropped. Dropping it makes a form problem present to the user as a counting bug, and they will blame the count, because the count is the thing they can check. Judging depth is the job of form feedback, which has a channel for telling the user why. The counter has one integer and no way to explain itself.

Do not hand the first labelling pass to general annotation labour that has not read the policy. "Mark every frame where the bar touches the chest" is fine to outsource. "Was that a rep" is a product judgement wearing the costume of a labelling task.

The label file wants to carry enough context to reproduce the conditions, not just the timestamps:

{
  "clip": "squat_bw_sideon_paused_003.mp4",
  "exercise": "squat_bodyweight",
  "camera": "side_on, waist height, 2.5m",
  "capture": { "fps": 30, "resolution": "1920x1080", "app_build": "1.14.2" },
  "tags": ["paused_rep", "tempo_change"],
  "policy": "v3",
  "labelled_by": ["performer", "second_pass"],
  "reps": [
    { "t": 2.13, "note": "clean" },
    { "t": 5.04, "note": "3s hold at bottom" },
    { "t": 9.87, "note": "clean" },
    { "t": 13.02, "note": "shallow; counts under policy v3" }
  ],
  "non_reps": [
    { "t": 16.40, "note": "descends, aborts, stands up" }
  ]
}

Record through your own app's capture path, not through the phone's camera app. A clip shot at a different resolution, frame rate or orientation than production is scoring a pipeline you do not ship.

Precision and recall do not feel the same to a user#

They are not interchangeable and you should refuse to collapse them into a single F-score, because an F-score lets an improvement in one silently pay for a regression in the other and the two failures are experienced completely differently.

Under-counting feels like theft. The user did the work and the app took it away. The reaction is immediate and physical: they do an extra rep to get it back, which means your bug just changed their training. Then they start counting in their head as a check, and once a user is counting in their head your feature has no reason to exist. Under-counting also generates support tickets, which is the one mercy — you find out.

Over-counting flatters, and it is worse. The set ends early, the user gets a green tick they did not earn, and in the moment it feels good. They usually do not notice. What they notice is later, when a personal record in their history is one they know they never hit, and at that point the damage is not one number — it is that they cannot tell which of the last two months of sessions are real. A training log is the asset users actually care about, and over-counting corrupts it retroactively and invisibly. For a coaching product it also under-trains people, which is the opposite of the thing you sold.

So set two floors, not one target, and decide deliberately which side you lean. In our experience, free-form tracking should bias toward recall — the missed rep is the complaint that gets written down — while anything that auto-advances a set, ends a workout, or feeds a prescription should bias toward precision, because there a phantom rep makes a decision on the user's behalf. That is a judgement, and it belongs in the same policy document as the labelling rules, because the two have to agree.

The corpus is a gate, not a report#

A number on a dashboard changes nothing. Commit a baseline of per-clip results and fail the build on any clip whose true positives, false positives or false negatives moved, in either direction. Aggregate thresholds are not enough, for the same reason total counts are not enough — a change can break two clips and fix two clips and leave precision and recall untouched:

$ pytest tests/rep_corpus --baseline baselines/rep_counter.json

clip                                 tp  fp  fn   vs baseline
squat_bw_sideon_paused_003            4   0   0   unchanged
pushup_offframe_wrists_011            8   0   2   fn +2  REGRESSED
bench_partner_walks_through_004       6   1   0   fp +1  REGRESSED
curl_tempo_fatigue_027               12   0   0   fn -1  improved

aggregate precision and recall: unchanged
FAIL: 3 clips changed (2 regressed, 1 improved — update the baseline)

That last pair of lines is the whole argument. The aggregate did not move. Three clips did.

An improvement should also fail the build, and be accepted by updating the baseline in the same pull request. That makes the diff the review artifact: anyone touching the hysteresis gap has to show which clips moved and say, in words, why the trade was worth it. This is the mechanism that would have caught the bench-press regression in the opening paragraph, and it is the only reason to build any of this.

One ordering rule when the suite goes red: check the keypoint layer before you touch the state machine. The rep counter's input is pose output, so a model version bump moves the signal underneath your thresholds. If the pose accuracy corpus also regressed, tuning the counter to compensate is baking a workaround for someone else's bug into your thresholds, and it will have to be un-baked later. Version the rep baseline against the pose model version so this is visible rather than archaeological.

What you cannot automate here#

One distinction before the list, because it is easy to misread the rule above. Replaying recorded keypoint sequences captured from a real device run is legitimate and useful — that is how you regression-test the state machine cheaply, and the device lab and CI page recommends exactly that. What is circular is scoring the pose model against keypoints your own pose model produced. The test is honest when the thing under test is downstream of the recording, and dishonest when it is the recorder.

Filming it. Someone has to do the reps, in a room, in front of a camera, badly and on purpose. There is no shortcut, and the shortcut people reach for — synthesising clips or generating keypoint sequences from the same pose model you are testing — is a test that cannot fail. It scores your state machine against your own model's opinion, and the entire point of the corpus is to catch cases where that opinion is wrong.

Body and room diversity. Your corpus is filmed by your team, in your office, in your lighting, at your heights and limb proportions. That is a bias you cannot engineer away, and no amount of augmentation fixes it, because the thing that varies is anatomy and clothing, not brightness. The only real mitigation is periodic capture sessions with people from outside the team, and anything involving recorded video of real users is a consent question for your compliance owner long before it is a question for the test harness.

Deciding what counts. Already covered, and it stays a human decision permanently.

Live camera behaviour on iOS. The corpus runs through the frame-source seam, not through a camera, and on iOS it has no choice: Apple's current AVCam sample documentation states that Simulator does not have access to device cameras and is not suitable for running the app (read 2026-07-30). The file-driven substitute is VNVideoProcessor and its synchronous analyze(_:). On Android, the emulator will import a PNG or JPEG into the virtual scene (developer.android.com, read 2026-07-30), which is a still image and is consequently useless for a state machine whose entire input is a function of time. Either way, the corpus needs a seam that accepts frames from a file; testing camera features without a device covers designing that seam and the qualifications that come with borrowing an offline-analysis API as a harness, and the seam has to exist before the counter does. Retrofitting it is a rewrite.

The honest limits#

We know of no public benchmark for rep counting and no vendor tooling for testing it. Everything above is our recommendation from building this, not a standard, and the numbers your corpus produces are not comparable to anybody else's — they describe your clips. Publishing them as an accuracy claim would be measuring your own filming and calling it a fact about the world. Resist that, and resist it hardest in marketing copy.

The corpus rots in specific ways. New exercises arrive without clips. Phone defaults change frame rate and field of view. A pose model bump shifts every keypoint slightly. And the labelling policy will need a clause you did not anticipate, which means a re-labelling pass. Budget for one deliberate corpus review a year, and expect the labelling to take longer than the filming — it always does, and teams always plan the reverse.

Finally, the suite tells you nothing about the counter's behaviour on a device you did not film with, at a thermal state you did not reproduce, at a frame rate your test harness does not drop to. That is a device-lab question, not a corpus question, and the corpus should not pretend to cover it.

Build the counter first — adding rep counting walks through the state machine and where the thresholds go — then film the ugliest clips you can think of before you tune a single threshold. The tuning is only trustworthy after the scoring exists.