How to Evaluate AI Motion Tracking SDKs

Disclosure: KinesteX, this site's own product, sells in exactly this category. That conflict is why the protocol below is neutral, not despite it: the only bake-off a buyer has reason to believe is one they designed and scored themselves, so a fair protocol is the one sales pitch an honest vendor can afford. Every step runs against KinesteX on identical terms to Sency, QuickPose, Kemtai and a raw pose model you wire up yourself. Where a criterion would quietly flatter us — a shipped content library, breadth of platform coverage — it is filed under fit questions and carries no score, because a scorecard that rewards the shape of our product is a sales document wearing a lab coat. If running this tells you to buy someone else, it worked.

Why the demo does not predict your product#

Every vendor here can show you a flawless demo, and none of them are cheating. The demo is accurate about the demo.

It uses the vendor's exercises, filmed at the framing their model expects. Apple's body-pose guidance puts that boundary at a subject occupying at least a third of the image height with key body regions in view, and notes that flowing clothing reduces detection accuracy; a staged demo sits inside that envelope by construction. It runs on a recent phone, and a cold one: a pose pipeline pins camera, ISP, accelerator and display at once, and the throttle curve only appears past minute ten. And it lasts ninety seconds, one person, no mirror.

Your product is the opposite of all of that. So the protocol has exactly one governing rule: the same corpus, the same devices, the same scoring code, for every vendor, including the one that funds this page.

The two-week protocol#

Week one builds the measuring instrument; week two uses it. Integrate a vendor first and that vendor's behaviour silently becomes your definition of "working".

Week one: build the corpus#

The corpus is the deliverable; the harness is a weekend. Labelling is the expensive part, so every clip must earn its place by distinguishing a working SDK from a broken one.

Span these axes under one rule: each clip varies one thing. A clip that is dim, oblique and filmed in a baggy hoodie all at once tells you nothing when it fails.

• Your top 8 to 10 exercises, weighted by what users actually do, not by what films well.
• Camera angle — front-on, side-on, and the three-quarter angle real users default to because that is where the tripod fits.
• Lighting — daylight, a single lamp, backlit against a window.
• Body types and clothing, including subjects from outside your team. Four colleagues in a well-lit office is a corpus that passes forever while the product degrades for everyone.
• Your three worst target devices — the oldest chipset you claim to support, a mid-tier Android where the accelerator bottlenecks while the spec sheet still looks fine, and one thin-chassis phone with a thermal ceiling.

Then film the clips nobody thinks to film, where vendors diverge:

• Reps that are not reps. A squat abandoned halfway. Hitching shorts mid-set. Re-gripping, racking, walking out of frame for water. Truth: zero reps. This is where false positives live.
• Reps that are hard to see. A three-second pause at the bottom. Tempo collapse in the last three reps of twelve. Wrists leaving frame on an overhead press. A knee hidden behind the other knee.
• Scene problems. A partner walking through frame. A mirror, which the pose stage reads as a second person. The worst plausible camera position — phone on the floor, pointing up.
• The negative clip. Camera running, person visible, moving normally, not exercising. Expected count: zero. Cheapest clip in the corpus, and the one that catches phantom reps.

Label ground truth as what the human did, never as what the camera could reasonably have seen. Marking an off-frame rep "not visible, excluded" feels like fairness and is actually the act of defining away your hardest failure mode. Write the labelling policy before you film — what range of motion counts, and what happens to an abandoned rep, a paused rep, and a rep the camera missed — since a clause added later means re-labelling every clip it touches. Have the performer label first, while they still remember whether the movement at 0:34 was abandoned or merely bad, then have a second labeller pass cold, with disagreements escalating to the policy rather than being averaged.

Record two things per clip: rep events as timestamps, and per-joint state on the decision frames — visible, occluded, or out of frame. Occlusion is a label, not a miss: a model that declines to report a hidden ankle is behaving correctly, and a position-only metric punishes honesty. Our pose-estimation regression corpus page covers those states and how to derive a per-keypoint tolerance from the displacement at which your rep verdict flips. Film through a capture path matching your production resolution, frame rate and orientation, and clear footage of real people with your compliance owner before it reaches the harness.

Week two: run every SDK against the same corpus#

Build one injectable frame source and make every vendor drink from it. The constraint forcing this is documented on both platforms: Apple's AVCam documentation states the Simulator has no access to device cameras, and the Android emulator will only import a still image into its virtual scene. A file is the only frame source both can drive.

Some SDKs will not accept your frames. One that owns the capture session, or embeds a hosted camera experience, may offer no injection point at all — true of our own product's WebView architecture, and a real limitation rather than a footnote. Record the refusal as a finding and score it, then fall back to the uniform substitute: play each clip full-screen on a second display in front of the device camera. Lossy, and lossy identically for every vendor, which is all this needs.

For each SDK on each device, log the version, build, OS and date, then run three passes:

1. Cold start. Launch to first usable tracking, then the full corpus.
2. Thermal soak. One three-minute reference clip through the live pipeline, eight times back to back, from cold, screen on, no charger. Compare pass eight against pass one.
3. Offline. Airplane mode, same clip. Note the behaviour: graceful degradation, a queue, a hang, or a blank screen.

CI for an app that needs a real device explains why the soak cannot be faked in a container, and how to keep the matrix small enough to run on every build.

The metrics that decide it#

Per-clip rep precision and recall, never aggregated. Match predicted rep events one-to-one to labelled ones inside a tolerance window, and score true positives, false positives and false negatives per clip. Comparing final counts is the trap: a dropped rep and a phantom rep cancel, the total is right, and the SDK is wrong twice. Pooled precision and recall are barely better, because two clips can break in opposite directions while the aggregate sits still. Report both separately per exercise and refuse to collapse them into an F-score — under-counting feels like theft, over-counting corrupts a training log invisibly, and testing a rep counter explains why the two need separate floors.

Form-cue correctness and latency, scored apart. Did the cue fire on the deviation you staged, and stay silent on the clean rep? And how long after the movement did it arrive? A correct cue two seconds late is a cue about a rep the user already finished.

Time to first tracking. Cold start to first usable pose, on your worst device rather than your best.

Thermal soak drift. Rep count and p95 frame latency at pass eight against pass one. A pipeline that is perfect for three minutes and loses a rep per set at minute fifteen fails the only session length that matters.

Offline behaviour. Binary and observable. Note it; weight it only if your users train in basements.

The scorecard#

Download the motion SDK scorecard and add a column per vendor. The weights column is deliberately blank: whether latency outranks form-cue precision depends on whether you are building a rehab product or a home HIIT app, and any weighting we shipped would be our product's shape leaking into your decision. Set the weights as a team before you see a result — weights chosen afterwards are a justification, not a decision. Score every criterion from your own runs; a cell filled in from a vendor's deck belongs on the procurement list instead.

The procurement questions: get it in writing#

No corpus measures these, and no vendor in this category publishes them — we could not verify one from public sources, ours included. Each is a required written answer:

• Current pricing, and the unit it scales on — per active user, per session, per device, per inference-minute. The unit matters more than the number, because it decides whether your cost curve tracks revenue or usage.
• On-device and edge processing guarantees for your configuration, not the marketing sentence. Our own SDK READMEs say "optional edge processing (no cloud transmission)", and optional is the load-bearing word: the answer depends on how you are set up, so get it in writing for how you are set up. Ask every vendor the same question.
• Model update cadence, and how much notice you get. A model bump moves every keypoint slightly, which moves your rep thresholds. You need warning and the ability to pin a version.
• SLA, including the camera experience when the vendor's service degrades.
• Data retention — what is stored, where, for how long, and whether video or only derived events.
• Exit terms — what you can export, what happens to your content, and how much notice ends the contract.

Fit questions, which are not scores#

These decide the shortlist, not the ranking. They are yes-or-no matches to your requirements, and scoring them would reward whichever vendor happens to be shaped like your questionnaire.

• Shipped exercise and content library, or your own programming? The biggest differentiator in the category, and a scope question rather than a quality signal.
• White-label theming, or raw events into your own UI?
• Which platforms must be at parity? Public repos show real differences: Sency's platform table marks JavaScript as not applicable, and QuickPose's verified centre of gravity is iOS.
• Native or embedded? Sency's SMKit describes itself as a native no-UI SDK; KinesteX's SDKs are wrappers embedding a hosted camera experience via WebView or iframe. Neither is wrong; the choice decides how much of the pixel-level experience you own.
• Consumer or clinical scope? Kemtai positions toward physiotherapy, rehab and MSK, and paying for clinical scope you do not need is as much a mismatch as lacking it.

How to read a vendor accuracy claim#

Start from the fact that there is no public benchmark for this category. No leaderboard ranks camera-coaching SDKs on rep counting or form feedback, and no independent comparison of these vendors exists that we could verify. Every accuracy figure you are shown is vendor-stated.

Ours included, which is the clearest test of whether this page is honest: KinesteX's own README carries a line about models consistently exceeding 90 percent accuracy. That is a first-party marketing claim, and you should discount it exactly as hard as every other vendor's number. Kemtai states its tracking was validated against a gold-standard 3D motion lab — stronger-sounding, and still a study to ask to see.

Even a rigorous number would not transfer. As pose estimation accuracy sets out, PCK, OKS and MPJPE are measured against a specific labelled dataset under favourable conditions, and tell you nothing about your camera, your exercises or your users. The number you need does not exist until you make it — which is also the reason not to publish yours later as an accuracy claim of your own.

What this protocol will not tell you#

It will not tell you how a vendor behaves in year two, when your exercise list has doubled and their model has shipped three updates. It says nothing about support quality, and it cannot catch a failure mode you did not think to film — grow the corpus whenever production produces a bug you could not reproduce.

It also does not answer the prior question. If motion analysis is your core product and you should own the pipeline, no bake-off between vendors will surface that; build vs buy for AI motion tracking is the page for that decision, and worth settling before you spend two weeks scoring people you were never going to buy from.