CI for an App That Needs a Real Device

A build goes green. Unit tests pass, the pose corpus matches expected keypoints, the rep counter scores perfectly on every clip in the fixture set. Then a user runs a twenty-minute session on a two-year-old mid-tier Android, the phone gets warm somewhere around minute twelve, the frame rate slides from smooth to a slideshow, and the last third of their set goes uncounted. Nothing crashed. Nothing was logged. The corpus still passes on the next build, because it runs on a Linux container in a data centre that has never been warm.

That is what a device lab is for, and "we should test on real devices" is not a test. The assertion is:

# Thermal soak, one physical phone, one clip.
run the 3-minute reference clip through the live capture pipeline
  8 times back to back, from a cold start, screen on, no charger

assert  rep_count[pass_8]      == rep_count[pass_1]
assert  p95_frame_latency[8]   <= p95_frame_latency[1] * budget
emit    per-frame latency CSV as a build artifact

Two failures, neither of which a hosted runner can produce: a count that changes because the frame interval changed, and a frame interval that changed because the silicon got hot. Everything below is about drawing the line around those two assertions as tightly as possible, because every test you push across that line costs real money and real flakiness.

Three tiers, and the line between them#

Tier 1 — hosted CI, every commit. Anything deterministic given bytes. Provider integration against recorded fixtures and injected faults; deduplication, timezone and rollup logic over deliberately ugly inputs; Health Connect's FakeHealthConnectClient for local unit tests, remembering that its aggregation is stubs you program rather than arithmetic it computes. Fault injection belongs here too: WireMock's Fault enum and Toxiproxy cover the transport nastiness between you and a provider, and the 429-storm and outage suite is where those cases live.

Note what that first assertion is and is not. It compares the same clip against itself on the same device across a thermal ramp, so it detects drift — the count changing as the silicon throttles. It is not an accuracy test, and you should not use a total-count comparison as one: testing rep counting shows why totals hide compensating errors, and accuracy belongs in that per-clip corpus, not here.

The camera-side surprise is that most of your rep-counting logic belongs in tier 1 as well. A rep state machine consumes a sequence of keypoints and timestamps; it does not care whether they came from a sensor or a file. Serialize keypoint sequences once, from a real device run of a clip you have already labelled, commit them, and the rep counter's own state-machine suite becomes pure, fast and free. The moment you can do that, the device tier only has to answer one question: do the keypoints and the timings coming off this hardware match the ones you recorded?

Tier 2 — a small real-device pool, every build or every merge. Camera capture end to end, the thermal soak above, and the accelerator. Also iOS background delivery, and here Apple is unambiguous: "Background server queries aren't supported on the Simulator. Be sure to test your background queries on a device." That sentence appears verbatim on two Apple documentation pages, on enableBackgroundDelivery(for:frequency:withCompletion:) and again on HKObserverQuery. There is no arguing with it and no workaround; the background sync test plan is built around that constraint.

Tier 3 — manual, on a calendar. Low light. A real gym with mirrors and fluorescent flicker. Real bodies in real clothing. Apple's own guidance for body-pose detection tells you exactly what tier 3 has to stage: the subject's height "should ideally be at least a third of the overall image height", "a large portion of the subject's key body regions and points should be present in the image", "a subject wearing flowing or robe-like clothing reduces the detection accuracy", and dense crowd scenes are "likely to produce inaccurate results". Every one of those is a physical staging problem — a person, a room, a phone propped against a water bottle at the wrong angle. There is no coverage number for this tier and there never will be. Put it on a schedule, write down what you looked at, and stop pretending it is automated.

Why the camera half needs silicon and the sync half mostly does not#

A sync test's inputs are bytes arriving on a socket. Its dependencies are your HTTP client, your clock and your database, all of which behave identically in a container. That is why almost the entire provider surface — token rotation races, out-of-order webhooks, duplicate day boundaries, the two-client offline conflict scenario — stays in tier 1. Android's background behaviour largely stays there too, because Google documents the constraint (in Doze, "WorkManager uses JobScheduler internally, so WorkManager tasks don't run") and documents how to force it: adb shell dumpsys deviceidle force-idle, then unforce. In our experience those commands work against an emulator as well as a handset (checked 2026-07-30; Google's Doze documentation does not say either way). iOS gets no equivalent, which is the asymmetry that decides your tier-2 budget.

A pose test's inputs are photons on a sensor. Its dependencies are a camera stack, a driver, an ISP, an accelerator, a thermal governor and a power policy, none of which exist in a container. Three specifics matter.

Thermal. A pose pipeline is the rare mobile workload that pins the camera, the ISP, the accelerator and the display simultaneously for twenty unbroken minutes. Almost everything else on a phone is bursty. Your fitness app is the sustained-load case, so it finds the throttle curve that a messaging app never touches — and the failure is not a crash. The SoC quietly drops clocks, the inter-frame interval doubles, and if your rep detection uses temporal smoothing (it does), a rep at 12 fps has a different shape than the same rep at 30 fps. The user sees a miscount at minute fifteen and blames your counter, correctly.

The accelerator. An older or mid-tier NPU is not merely slower; it is different. A delegate can fall back to CPU for a single unsupported op and take the whole pipeline with it. Quantized paths differ. The model loads, runs, and emits keypoints that are subtly not the ones your corpus expects. Sizing hardware for a shipping pose pipeline covers the production side of this; the testing matrix is the smaller, meaner version of the same question.

The camera stack. Field of view, default capture resolution, sensor orientation reporting, HDR behaviour and ultra-wide lens selection all vary by generation. The same framing code that centres a subject on one phone crops the torso on another — and a cropped torso is precisely the input Apple's guidance warns produces bad poses. This is a correctness bug in your capture layer that only a second camera generation reveals.

Running the corpus on-device changes what you can assert#

Building the labelled corpus and choosing per-keypoint tolerances is a separate job. What changes when you move the runner from a container to a phone is worth being precise about.

You gain a time axis. Per-frame latency percentiles, cold-start-to-first-pose, and the soak delta. These are the only assertions that catch throttling, and they are the reason the tier exists at all. Emit them as artifacts every run so you can diff builds rather than eyeball a pass/fail.

You gain a tolerance problem. Device B will disagree with device A on some keypoints. The tempting fix is to widen the tolerance until both pass, and if you do that twice you have built a test that cannot fail — the same defect as asserting that a revocation endpoint returned 200. Our recommendation: keep a baseline per device class, not one global tolerance, and treat "a new device class disagrees" as a review event that someone signs off on, not as a number to relax. Record the disagreement instead of dissolving it.

You have to get frames in. On iOS this is settled: the Simulator has no camera. Apple's current AVCam documentation says so directly — "Because Simulator doesn't have access to device cameras, it isn't suitable for running the app—you'll need to run it on a device." The file-driven substitute is VNVideoProcessor, whose analyze(_:) Apple documents as synchronous; the qualifications that come with it, and the frame-source seam it plugs into, are set out in full in testing camera features without a device.

On Android the emulator does have a synthetic camera. You can import a PNG or JPEG through Extended controls > Camera > Virtual scene images > Add image, and the emulator console exposes virtualscene-image {wall|table} image_path, which the adb emu shortcut makes scriptable without an interactive telnet session. Compose and test that combination yourself before committing it. For headless runners the console also has nodraw {on|off}, with two documented limitations worth knowing up front: proxy set operations fail in headless environments, and resolution adjustments are unsupported there. Be honest about what this buys you, though — it is a still image on a virtual wall, not a moving human. It proves your capture path is wired and your permission flow works. It proves nothing about pose accuracy.

All of which is the argument for a frame-source abstraction decided on day one, before there is any pose code to retrofit it into.

Picking the matrix: criteria, not model numbers#

We are not going to publish a list of phones. A model list dated mid-2026 is wrong within a year, and more to the point we have not tested your app, your model or your minimum spec — a shopping list from us would be fabricated authority. Here are the criteria we would use. Pick the smallest set of physical devices that covers them, and accept the overlaps gladly.

1. The oldest chipset your app claims to support. Whatever your minSdk or deployment target says, somebody picked that number optimistically. This device's job is to falsify the claim, and it is the one most likely to send you back to change the number.
2. One mid-tier Android with a weak accelerator. Not the cheapest handset you can find and not last year's flagship. Mid-tier is where the NPU is the bottleneck while the CPU spec sheet still looks fine, and that combination produces the "works, but slow" bug rather than the crash you would have noticed.
3. One device per camera-stack generation you support. If your install base spans two or three generations of capture pipeline, that is two or three devices, and it is the least negotiable line in the matrix.
4. One device with a known thermal ceiling. Thin chassis, small battery, no meaningful heat spreading, and ideally the model your support inbox already mentions running hot. This device owns the soak assertion.
5. One device that is genuinely popular in your install base. Weight by your own analytics, never by review-site coverage. A bug on a phone eleven percent of your users hold outranks a bug on a phone you find interesting.
6. One iPhone, because iOS is device-only for anything camera-shaped and because the Simulator cannot answer background-delivery questions either.

Re-derive the list quarterly from your own crash-free-rate and analytics. It is a distribution of your users, not a distribution of phones.

Cost and flakiness should shrink the matrix, not grow it#

Device tests are slow and expensive. Cloud device minutes are billed; self-hosted phones need somebody to re-plug them, clear storage and pin the OS version. Each additional device buys coverage once and costs build latency on every commit forever.

Device tests are also flaky in ways unit tests are not. A phone fills its storage. A USB cable degrades. The OS updates itself overnight and changes a permission dialog. A device ends up face-down on a bench, which for a camera test means every run fails on a black frame. A lane that flakes gets ignored, then muted, and a muted lane is worse than no lane at all because it still reads as coverage on the dashboard. Budget a quarantine lane and name an owner. If you cannot staff that, cut devices until you can.

Both pressures point the same way, and it leads to the position this page is really arguing: a small matrix on every build beats a large matrix run monthly. The value of a device test is dominated by time-to-detection, not by breadth. Three devices on every merge tells you which commit did it. Fifteen devices once a month tells you that something in the last four weeks did it, and now you are bisecting a release's worth of commits across physical hardware by hand. The regressions this tier exists to catch — a model version bump, an OS update that reworks the camera pipeline, a threading change that starves the inference queue — land on a schedule you do not control, so detection latency is the one variable you actually own.

What we could not verify#

Everything in this section is dated 2026-07-30 and will rot faster than the rest of the page.

• Camera and audio injection into a device cloud. Our research traced marketed camera image injection and microphone audio injection to third-party device clouds, specifically LambdaTest and TestMu AI marketing copy surfaced in search results. We did not fetch the vendors' own documentation and we have not used the feature, so treat this as a lead worth evaluating, not a verified capability. If injection works as advertised it is the closest thing to a hosted pose test that exists, which is exactly why you should verify it yourself before designing around it.
• Firebase Test Lab. firebase.google.com and cloud.google.com were both unreachable from our research environment (the proxy answered 403 to CONNECT), so we confirm nothing about Test Lab's fleet, its API-level coverage or its feature set. In particular, we do not attribute camera or audio injection to Firebase Test Lab. That attribution appeared in our own earlier scoping notes and we could not substantiate it. If you have seen the claim elsewhere, check Google's documentation directly before you build a plan on it.
• iOS Simulator camera bridges. Third-party and commercial. SimCam (software-mansion/simcam.app, as read on GitHub on 2026-07-30) is a macOS menu-bar app that feeds a Mac camera, images or videos to the Simulator with no application code changes, ships a simcamctl CLI, and has separate CI licensing; the repository was small at the time of reading. RocketSim also markets Simulator camera support, but its documentation host was unreachable, so we describe nothing about how it works. Putting a commercial vendor inside your test path is a real dependency — price it against a physical device before adopting it.
• The one first-party, scriptable camera-input path we could confirm anywhere is the Android emulator virtual scene described above.

Where this leaves you#

Push everything you can into tier 1, including the rep state machine, by making recorded keypoints a first-class fixture. Keep tier 2 to a handful of devices chosen by the criteria above and run it on every merge, with a thermal soak and a latency artifact as its reason for existing. Schedule tier 3 and write down what you saw. Then spend the argument you save on the three things linked above that actually decide whether your counter works: the labelled regression corpus and its tolerances, a frame source you can swap for a file, and an honest sizing of the hardware you are asking users to run on. The data half of your app has the same shape and a cheaper answer — a HealthKit seam narrow enough to fake puts everything except background delivery back in tier 1, which is where you want it.