How to Track Workouts Without a Wearable (Camera-Only)

You can track a workout with nothing but a phone camera and an on-device pose model: it sees the body move, so it can measure reps, range of motion, tempo, and form directly. What it cannot measure is anything the camera never sees — heart rate, HRV, and sleep — and calorie burn from a camera is an estimate, not a measurement. This guide draws that line honestly, then walks a concrete build path: run a pose model, then turn its keypoints into reps and movement metrics.

There are two ways to build it. You can build on a free, on-device pose primitive (MediaPipe Pose Landmarker or TensorFlow MoveNet) and write the movement logic yourself, or you can buy a fitness SDK that wraps the camera, the model, and the rep/form logic behind one API. This guide takes the build-it path with MediaPipe on the web; the same ideas port to iOS, Android, and Flutter.

What a camera can and cannot measure

A camera measures geometry over time — where the joints are in each frame and how they move. Everything a camera can track is derived from that: a squat is a knee angle sweeping down and back up, a rep is one full sweep, tempo is how long the sweep took, form is whether the angles and alignments stayed in range. None of that needs a wearable.

What a camera cannot do is measure your physiology. Heart rate, heart-rate variability, sleep stages, and blood oxygen are signals the camera never sees, so they need a watch, ring, or band. Calorie burn sits in between and is the most over-claimed metric in fitness apps: a camera can estimate calories from movement, exercise type, duration, and a user's height and weight, but it is a model output, not a measurement — treat it as a rough number and label it that way in your UI. A wearable's calorie figure is also an estimate; it is just derived from heart rate instead of movement.

If your app needs the physiological column, you pair the camera with a wearable rather than replace it — see the best wearable data APIs for pulling heart rate and recovery data. For the movement column, the camera is enough on its own, and that is what the rest of this guide builds. For a deeper look at the pose layer, see camera pose tracking, explained.

What you'll need

• A device with a camera and a modern browser (or a native iOS/Android/Flutter app — the pattern is the same).
• The @mediapipe/tasks-vision npm package (MediaPipe Pose Landmarker). MoveNet via @tensorflow-models/pose-detection is an equally valid free alternative.
• No account, no cloud, no wearable. Inference runs on-device; no frames leave the phone.

Step 1: Capture frames from the camera

Get a live video stream and draw it into a video element. This is the only "hardware" you need — the camera is your sensor.

const video = document.querySelector("video");
const stream = await navigator.mediaDevices.getUserMedia({
  video: { width: 640, height: 480 },
  audio: false,
});
video.srcObject = stream;
await video.play();

Step 2: Run an on-device pose model

Load MediaPipe Pose Landmarker in VIDEO mode and run it on each frame. It returns 33 body landmarks per person, entirely on-device — this is the step that replaces the wearable's sensors with the camera.

import { PoseLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath:
      "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/latest/pose_landmarker_lite.task",
    delegate: "GPU",
  },
  runningMode: "VIDEO",
  numPoses: 1,
});

Step 3: Turn keypoints into a movement metric

Read the landmarks you care about and compute a joint angle — the raw material for range of motion and reps. Use the atan2 form of the angle at the middle joint; it stays numerically stable near 0 and 180 degrees where acos can fail. For a squat, the angle at the knee (hip, knee, ankle) tracks depth.

// MediaPipe indices: hip 23, knee 25, ankle 27 (left side).
function angle(A, B, C) {
  const bax = A.x - B.x, bay = A.y - B.y; // vector B->A
  const bcx = C.x - B.x, bcy = C.y - B.y; // vector B->C
  const dot = bax * bcx + bay * bcy;
  const cross = bax * bcy - bay * bcx;
  return (Math.atan2(Math.abs(cross), dot) * 180) / Math.PI; // 0..180
}

function renderLoop() {
  const result = poseLandmarker.detectForVideo(video, performance.now());
  const lm = result.landmarks[0];
  if (lm) {
    const kneeAngle = angle(lm[23], lm[25], lm[27]);
    // Standing knee ~170-180 deg; parallel squat ~90 deg; deep ~70 deg.
    // Range of motion (degrees of flexion from straight) = 180 - min kneeAngle.
  }
  requestAnimationFrame(renderLoop);
}
renderLoop();

The angle is scale-invariant, so it works at any camera distance as long as x and y are in the same units. Track the minimum knee angle across a rep; the flexion from straight (180 minus that minimum) is your range of motion, measured directly — no wearable involved.

Step 4: Count reps and measure tempo

A rep is not machine learning; it is a two-state machine over the angle signal, with two thresholds so noise cannot chatter across a single line (hysteresis). Count on the full down-and-up round trip, and stamp the time at each transition to get tempo.

const ENTER_DOWN = 100; // must dip below this to be "down"
const EXIT_UP = 160;    // must rise above this to be "up"
const alpha = 0.3;      // EMA smoothing on the angle

let state = "up", reps = 0, angleS = null, downAt = 0;

function onFrame(rawAngle) {
  angleS = angleS == null ? rawAngle : alpha * rawAngle + (1 - alpha) * angleS;

  if (state === "up" && angleS < ENTER_DOWN) {
    state = "down";
    downAt = performance.now();
  } else if (state === "down" && angleS > EXIT_UP) {
    state = "up";
    reps += 1;
    const tempoMs = performance.now() - downAt; // time under tension for the rep
    console.log(`rep ${reps}, tempo ${Math.round(tempoMs)} ms`);
  }
}

Smoothing (the EMA) and the gap between the two thresholds are what keep a single real rep from being counted several times. Gate on landmark visibility too — skip frames where the driving joints are unreliable rather than letting a garbage angle trip a threshold. For the full rep and form logic, see camera pose tracking, explained.

Step 5: Surface the metrics honestly

Now show what you measured — and be precise about what is measured versus estimated. Reps, range of motion, and tempo are direct readings. If you also show calories, derive them from movement, exercise type, duration, and the user's profile, and label the number as an estimate.

const session = {
  reps,                         // measured
  rangeOfMotionDeg: 180 - minKneeAngle, // measured (from Step 3)
  avgTempoMs,                   // measured
  caloriesEstimate,             // ESTIMATE — label it in the UI
};

Do not present a camera-derived calorie figure as if it were a heart-rate measurement. Users trust an app that says "estimated ~180 kcal" far more than one that quietly overclaims. If precise energy expenditure or heart rate matters for your product, add a wearable alongside the camera rather than pretending the camera can measure it.

Make it reliable

Camera tracking is only as good as the frame you feed it. A few practical rules matter more than model choice:

• Frame the whole body. Single-person models assume head-to-ankles are visible and roughly centered; cropped joints produce garbage angles.
• Use a side view for squats, curls, and lunges. A 2D joint angle is only accurate when the limb is roughly parallel to the camera; a side view keeps knee and elbow angles in-plane.
• Good, even lighting and a clean background. Backlight, silhouettes, and clutter drop landmark confidence.
• Smooth and gate. Keep the EMA alpha modest (around 0.2 to 0.4) so you kill jitter without lagging fast reps, and skip low-visibility frames instead of counting them.

For the end user: clear a space, stand so your full body is in frame, and light the room from the front. With that, a plain phone camera measures the movement side of a workout on its own — no watch, ring, or band required.