How to Add Real-Time Form Feedback to a Workout App

Real-time form feedback compares the joint angles and alignments you measure from pose keypoints against target ranges you define, then surfaces one actionable cue per rep. This guide shows the geometry-only path: build the rules yourself on a free pose primitive like MediaPipe, evaluating each rep at its transition. The faster alternative is to buy a fitness coaching SDK (KinesteX, Kemtai, and similar) that ships form rules for you; this page teaches the DIY version so you understand what those SDKs are doing.

Form feedback here means one thing precisely: flag when a measured joint angle or alignment is outside a target range you define, and show a directional cue. It is geometry, not clinical assessment. The thresholds below are app-defined examples for illustration, not clinical or medical standards, and this technique does not diagnose anything or prevent injury. Pick and tune your own ranges for your exercises, camera setup, and users.

What you'll need

• A pose model producing per-frame keypoints with x, y, and visibility — MediaPipe Pose Landmarker (33 landmarks) is the reference here. See the camera pose-tracking guide for setup.
• A working rep detector that fires a callback at the bottom-to-top transition of each rep. See the rep-counting guide; this guide plugs into its onRepComplete() hook.
• JavaScript or TypeScript. The math is plain trig and ports to any language.

Throughout, keypoints use MediaPipe's landmark indices: shoulders 11 and 12, hips 23 and 24, knees 25 and 26, ankles 27 and 28. We alias them by name for readability.

type KP = { x: number; y: number; visibility: number };

// Pull named joints out of a MediaPipe landmarks array (normalized 0..1 coords).
function named(lm: KP[]) {
  return {
    leftShoulder: lm[11],  rightShoulder: lm[12],
    leftHip: lm[23],       rightHip: lm[24],
    leftKnee: lm[25],      rightKnee: lm[26],
    leftAnkle: lm[27],     rightAnkle: lm[28],
  };
}

Step 1: Capture the rep's extremes, not one frame

A single frame at the transition can miss the true bottom of the movement. Instead, accumulate the extreme values across the rep window (from the moment the rep detector enters the "down" state until it completes) and evaluate those at the end. Reset the accumulator when a new rep begins.

type RepWindow = {
  minKnee: number;        // deepest (smallest) knee angle seen this rep
  maxTorsoLean: number;   // most forward-leaning torso seen this rep
  maxValgus: number;      // worst inward knee drift seen this rep
  minVisibility: number;  // lowest confidence among driving joints this rep
};

function freshWindow(): RepWindow {
  return { minKnee: 180, maxTorsoLean: 0, maxValgus: 0, minVisibility: 1 };
}

let repWindow = freshWindow();

// Call this every frame WHILE the rep detector is in the "down" phase.
function accumulate(w: RepWindow, sample: {
  knee: number; torsoLean: number; valgus: number; visibility: number;
}) {
  w.minKnee = Math.min(w.minKnee, sample.knee);
  w.maxTorsoLean = Math.max(w.maxTorsoLean, sample.torsoLean);
  w.maxValgus = Math.max(w.maxValgus, sample.valgus);
  w.minVisibility = Math.min(w.minVisibility, sample.visibility);
}

Step 2: Compute the angles and alignments you'll grade

Every rule is built from three geometric primitives. The joint-angle helper uses the atan2 of the cross and dot products, which stays numerically stable near 0 and 180 degrees where the plain acos form can fail.

// Interior angle at vertex B for the segments B->A and B->C, in degrees (0..180).
function angle(A: KP, B: KP, C: KP): number {
  const bax = A.x - B.x, bay = A.y - B.y;   // vector B->A
  const bcx = C.x - B.x, bcy = C.y - B.y;   // vector B->C
  const dot   = bax * bcx + bay * bcy;
  const cross = bax * bcy - bay * bcx;      // z of the 2D cross product
  return Math.atan2(Math.abs(cross), dot) * 180 / Math.PI;
}

// Torso lean from vertical: 0 deg = upright, 90 deg = horizontal.
// Segment is shoulder->hip; compared to the image vertical (assumes a level camera).
function torsoFromVertical(shoulder: KP, hip: KP): number {
  const dx = hip.x - shoulder.x;
  const dy = hip.y - shoulder.y;
  return Math.atan2(Math.abs(dx), Math.abs(dy)) * 180 / Math.PI;
}

// A stable body-width reference (shoulder span in x) to normalize distances.
// Dividing by this makes distance rules invariant to how far the user stands.
function bodyWidth(k: ReturnType<typeof named>): number {
  return Math.max(Math.abs(k.leftShoulder.x - k.rightShoulder.x), 1e-6);
}

Because x and y are MediaPipe's normalized coordinates (0 to 1 of the frame), the angle is already scale-invariant. For the distance-based rule in Step 4 you still divide raw pixel-like gaps by bodyWidth so the same threshold works at any camera distance — the same idea MediaPipe's own pose classifier uses when it normalizes landmarks by torso size.

Step 3: Gate every measurement on confidence

Occluded or low-confidence landmarks jump to garbage positions and produce wild angles. Before you trust any sample, check the visibility of the joints that drive it. If it is below your gate, skip the frame and hold state rather than feeding a bad angle into the rules. This gate value is an app-defined example; tune it per camera.

const MIN_VIS = 0.6; // example gate; a common range is 0.5 to 0.7

function drivingVisibility(k: ReturnType<typeof named>): number {
  // The joints these squat rules depend on.
  return Math.min(
    k.leftHip.visibility, k.leftKnee.visibility, k.leftAnkle.visibility,
    k.leftShoulder.visibility,
  );
}

// Per-frame sampling (called while the rep is in its down phase):
function sampleFrame(lm: KP[]) {
  const k = named(lm);
  const vis = drivingVisibility(k);
  // Record confidence for the whole rep BEFORE gating, so a rep that was
  // low-confidence throughout is still detectable at the end.
  repWindow.minVisibility = Math.min(repWindow.minVisibility, vis);
  if (vis < MIN_VIS) return; // untrustworthy frame: do not measure or accumulate

  const w = bodyWidth(k);
  accumulate(repWindow, {
    knee: angle(k.leftHip, k.leftKnee, k.leftAnkle),
    torsoLean: torsoFromVertical(k.leftShoulder, k.leftHip),
    // front view, image x increases to the right: knee drifting toward body midline
    valgus: (k.leftKnee.x - k.leftAnkle.x) / w,
    visibility: vis,
  });
}

Step 4: Define target ranges and write the rules

Now express each check as "is this measurement outside the range I chose?" Here are three concrete rules for a squat. Every number is an example threshold you should tune, not a clinical value.

type Cue = { priority: number; text: string };

// App-defined example thresholds (degrees, and body-width-normalized units).
const KNEE_DEPTH_MAX = 100;  // knee angle above this = too shallow (90 ~ parallel)
const TORSO_LEAN_MAX = 45;   // torso lean above this = too far forward
const VALGUS_TOL      = 0.05; // inward knee drift (fraction of shoulder width)

function evaluate(w: RepWindow): Cue[] {
  const cues: Cue[] = [];

  // Rule A - squat depth: did they reach the bottom of the range?
  if (w.minKnee > KNEE_DEPTH_MAX) {
    cues.push({ priority: 1, text: "Go deeper — aim for thighs parallel" });
  }

  // Rule B - forward lean: torso stayed within an upright range?
  if (w.maxTorsoLean > TORSO_LEAN_MAX) {
    cues.push({ priority: 2, text: "Chest up — you're leaning too far forward" });
  }

  // Rule C - knee valgus: did the knees cave toward the midline?
  if (w.maxValgus > VALGUS_TOL) {
    cues.push({ priority: 3, text: "Push your knees out — don't let them cave in" });
  }

  return cues;
}

Each cue is directional and tied to the specific geometry that failed ("go deeper", "chest up", "push your knees out") rather than a vague "bad form". That is what makes the feedback actionable.

Step 5: Fire one cue per rep and debounce it

At the rep transition, evaluate the window, then surface the single highest-priority cue — never five corrections at once. Emit it once per rep so audio or haptics don't repeat every frame, and reset the window for the next rep. If confidence was poor across the whole rep, say so instead of grading noise.

// Wire this into your rep detector's onRepComplete() callback.
function onRepComplete() {
  if (repWindow.minVisibility < MIN_VIS) {
    showCue("Move so your full body is in frame");
  } else {
    const cues = evaluate(repWindow).sort((a, b) => a.priority - b.priority);
    showCue(cues.length ? cues[0].text : "Nice rep");
  }
  repWindow = freshWindow(); // reset for the next rep
}

let lastCue = "";
function showCue(text: string) {
  if (text === lastCue) return; // debounce repeats
  lastCue = text;
  // render the banner, speak it, or trigger a haptic — once.
}

You can also tag the rep good, partial, or bad from the same evaluation and keep a separate "good reps" counter on top of your raw rep count.

Make it reliable

• Prefer the right viewpoint. Depth and lean rules read cleanly from a side view (the sagittal plane); the knee-valgus rule needs a front view. A 2D angle only matches the true joint angle when the limb is roughly parallel to the image plane.
• Keep the camera level. The torso-lean rule compares against the image vertical, so a tilted phone biases it. For a camera-independent alternative, measure the hip–shoulder–knee angle with the same angle() helper instead.
• Normalize distance rules. Always divide pixel-like gaps by a body reference (shoulder or hip width) so a threshold tuned at one distance still holds when the user steps closer or farther.
• Smooth before you threshold. A light exponential moving average on each angle signal (alpha around 0.2 to 0.4) removes single-frame spikes that would otherwise trip a rule.
• Tune thresholds against real footage. The example numbers above will not fit every exercise, body type, or user. Treat them as starting points and adjust from recordings.

For the end user: a clear space, the full body in frame, and even front-ish lighting make the keypoints far more reliable — and reliable keypoints are the whole basis of good form feedback.