Backfilling Years of Wearable Data Without Hitting Rate Limits

A user connects their tracker. Your queue picks up backfill user 4821, starts at the earliest data the provider will give you, and pages forward through 2019. Forty minutes later the worker is getting rate-limited, the user's home screen still says "no data", and when the pod restarts the job begins again at 2019. The steps they took this morning, which is the only number they actually opened the app to see, arrive last.

That is a loop. What you want is a job: ordered newest-first, chunked by time window, checkpointed after every chunk, and spending a quota you have deliberately budgeted rather than one you discover by exhausting it. Almost everything specific about health data pushes you towards that shape, and one thing about it — the Android permission wall — means the job can be correct and still stop after thirty days.

Order the work newest-first, and mean it

Recent-first is not a UX nicety bolted onto a batch job. Timescale's refresh_continuous_aggregate() ships a refresh_newest_first argument that defaults to TRUE, which is the same reasoning applied to aggregate repair: a long-running rebuild should be useful to whoever is watching while it is still running.

For a health app the argument is stronger, because of who is watching. The user granted permission ten seconds ago and is looking at an empty chart. Last week's data makes the app work. 2019 makes a chart prettier some time next Tuesday. So the ladder we would build is roughly: last 7 days, then last 30, then 90, then a year at a time going backwards, with each rung committed and rendered before the next one is claimed.

Widening the chunks as you go back is deliberate. Recent windows are small so the first useful screen lands fast; old windows are large because nobody is waiting on them and a bigger window costs fewer round trips per day of history covered. The chunk size is a knob on latency at the front and quota efficiency at the back, and those want different values.

The three shapes a backfill actually takes

Pagination, history bounds, and what a checkpoint even is differ per source. This is the table worth internalising before you write a line of worker code, because a design that assumes one row here breaks on the other two.

HealthKit is the pleasant one. Apple documents the problem — "Users may have large quantities of data saved to the HealthKit store; therefore, reading all data for a given data type might become very expensive, both in terms of memory usage and processing time" — and prescribes the fix: "Start with a nil-valued anchor, and create a one-shot query descriptor that reads a batch of data. After you process the results from one query, start a new one-shot query for the next batch. Continue reading batches until there's no new data." Apple's own sample uses limit: 100 per batch, which is an illustrative value, not a documented maximum.

The consequence is worth spelling out: on HealthKit, backfill and incremental sync are the same code path and the same cursor. You do not need date windows at all. The anchor you finish the backfill with is the anchor you start tomorrow's delta with. Nothing else in this page's design changes, but the job table gets a lot simpler. See incremental sync for what that cursor is and is not.

The catch is that Apple does not order anchor batches by date for you in any documented way, so "recent-first" and "anchor-paginated" are in tension on iOS. If the first-screen latency matters more than the elegance — and it does — run one bounded statistics query over the last week to paint the screen, then let the anchor loop grind through the full history behind it.

The Android permission wall

This is the part that is not a performance problem and cannot be engineered around. Google documents it flatly: "By default, all applications can read data from Health Connect for up to 30 days prior to when any permission was first granted."

To go further back you request PERMISSION_READ_HEALTH_DATA_HISTORY, on androidx.health.connect.client.permission.HealthPermission. Google's wording is unambiguous about the alternative: "without this permission, an attempt to read records older than 30 days results in an error." Not an empty page. An error. Your backfill worker must distinguish that error from a transient failure or it will retry the wall forever.

Three details decide your schema:

• The window is anchored to first grant, not to install or to today. So it starts moving the instant the user says yes, and the clock is per-app.
• It resets on reinstall. Google documents this with a worked example: delete the app on May 10 2023, reinstall and grant on May 15 2023, and the earliest readable date becomes April 15 2023. A user who reinstalls has silently made two months of their own history unreadable to you.
• There is a version split. On Android 14 and higher there is no historical limit on an app reading its own data, and the 30-day limit applies to other data. On Android 13 and lower the 30-day limit applies to any data.

That last point is the one that should change what you store. If your app writes to Health Connect as well as reading from it, your own writes stay readable on Android 14+ regardless of the history permission — which makes your server copy the durable one and Health Connect the transport, not the archive.

There is a second thirty in this stack, and conflating the two is a real bug. Health Connect changes tokens expire after 30 days if unused, which is the same number as the default read window but a completely different mechanism. A token that expires while the history permission was never granted leaves you able to re-read only 30 days, so the gap between the token's death and your recovery is not recoverable at all. Acquire and persist the changes token before you start the backfill, and run the delta stream in parallel with it — never behind it in the same sequential queue, where a slow multi-year backfill can outlive the token that was supposed to cover the same period.

Google also states plainly that a sync can be cut off at any moment: "your app must handle interruptions midway through a sync when reading a large amount of data from Health Connect, and continue the next time the app is opened." Read that as the contract. A completed backfill is a lucky outcome; a resumable one is the design.

The rate limit you cannot look up

Google documents that Health Connect has rate limits and does not publish the numbers. What is documented is the shape: for reads and changelogs, "a periodic limit on the number of API calls your app can make to the API" and "a daily limit on the number of API calls your app can make"; for writes, both of those plus memory limits on bulk and single insertions. Limits vary by operation type and by whether you are foregrounded, and "Background rate limiting is stricter than foreground rate limiting."

You cannot tune a fixed-rate loop against that. So do not build one. Build the loop that degrades:

// Shape only, not production code. The documented quota signal on a
// Health Connect read is IllegalStateException; back off, do not abandon.
var pageToken: String? = null
do {
    try {
        val response = client.readRecords(
            ReadRecordsRequest(
                recordType = StepsRecord::class,
                timeRangeFilter = chunk.filter,
                pageToken = pageToken,
            )
        )
        persist(response.records)          // idempotent on metadata.id
        pageToken = response.pageToken
        checkpoint(chunk, pageToken)       // committed before the next request
        backoff.reset()
    } catch (quotaError: IllegalStateException) {
        backoff.sleepAndGrow()             // resume from the same pageToken
    }
} while (pageToken != null)

Two things in there are documented rather than invented: IllegalStateException is the exception Google's own read sample catches and comments as a quota error, and ReadRecordsRequest has a default pageSize of 1000 with an explicit warning to "be careful to avoid rate-limiting concerns" when iterating pages. Everything else is our judgement about how to wrap them.

For cloud providers the numbers usually are published, and the important property is that they are typically per consented user, not per app. That cuts both ways. A ten-thousand-user backfill does not contend with itself, so you do not need a global token bucket. But a single user's multi-year backfill is genuinely slow, and no amount of infrastructure buys your way out of it — size the UX around hours, not seconds. When you do get a 429, obey the provider's own Retry-After or quota-remaining header if it sends one, and use exponential backoff with jitter when it does not. Fitbit 429 handling covers the symptom; the design point here is different.

Budget the quota, do not just consume it. Reserve a fixed share of each user's hourly allowance for the live incremental path before the backfill worker is allowed to claim anything. A backfill that spends the entire per-user budget makes today's data stale in order to make 2019 complete, which is exactly backwards from what the user can see. In our experience this single rule prevents more "the app stopped updating" tickets than any amount of retry tuning.

One provider-shape question you must answer from primary docs before designing: does the provider return backfilled data in the HTTP response, or deliver it asynchronously through the same webhook callback as live data? Community and blog sources describe Garmin as doing the latter, with a fixed multi-year cap counted from the first backfill request; we could not reach Garmin's developer portal to verify either claim, so treat it as unverified. Our answer, though, is not "go and find out before you decide" — it is that you should design as though it were true, because the two costs are wildly asymmetric. Assuming it is true buys you an ingest path that can absorb a burst of two-year-old records at any moment, which is cheap. Assuming it is false gets you an architecture that quietly depends on recency, and that is a large difference to discover after you have built the other kind. So: expect backfill and live ingest to land in the same pipe, and never write an "is this late-arriving?" heuristic whose correctness depends on the answer.

The resumable-job state

Checkpoint on a cursor, not a step counter. Store one row per unit of work and let the worker claim the highest-priority unclaimed row; a crashed worker then costs you one chunk, and "resume after a crash" and "repair after a provider retro-edit" become the same code path.

CREATE TABLE backfill_chunk (
  user_id         uuid        NOT NULL,
  provider        text        NOT NULL,
  metric          text        NOT NULL,
  window_start    date        NOT NULL,   -- civil dates, not instants
  window_end      date        NOT NULL,
  priority        int         NOT NULL,   -- lower runs first; recent windows get low numbers
  status          text        NOT NULL,   -- pending | claimed | done | blocked_by_permission
                                          --        | blocked_by_provider_cap | failed
  page_cursor     text,                   -- provider pageToken or opaque continuation
  attempts        int         NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  claimed_by      text,
  claim_expires   timestamptz,            -- so a dead worker's chunk is reclaimable
  days_covered    int         NOT NULL DEFAULT 0,
  last_error      text,
  PRIMARY KEY (user_id, provider, metric, window_start)
);

CREATE TABLE backfill_job (
  user_id             uuid NOT NULL,
  provider            text NOT NULL,
  requested_earliest  date,        -- how far back we intend to go
  authorized_earliest date,        -- how far back we are actually permitted to read
  reached_earliest    date,        -- how far back we have got
  started_at          timestamptz NOT NULL,
  completed_at        timestamptz,
  PRIMARY KEY (user_id, provider)
);

Three columns here are health-specific and are the ones a generic job table would not have.

window_start and window_end are civil dates, not instants. A backfill chunk is "the user's March", and the user's March is defined by the offset in effect at each sample's own timestamp, not by a UTC range. Chunking on instants means a traveller's chunk boundaries drift against the days you later render, which is why that column is a date and not a timestamptz.

authorized_earliest records the wall, separately from where you have reached. On iOS you get it from getEarliestAuthorizedSampleDate(for:completion:), and Apple's instruction about what it means is the whole reason the column exists: "Treat all data before that date as unknown — not an absence of data." Apple's own sample clamps query windows with max(intendedStartDate, authorizationDate). Note the asymmetry Apple documents: limited authorization is the only authorization state your app can positively identify, and full access and denied access look identical. On Android you derive the equivalent from the permission-grant date arithmetic above.

The blocked_* statuses exist because "we were not allowed to read this" and "there was nothing here" arrive as the same empty response, and a failed chunk gets retried while a blocked one must not be. Collapsing them into one status is how you end up hammering a permission wall on a backoff schedule forever, and how you end up rendering a confident zero for a month you were never permitted to see.

Every chunk write must be idempotent on the provider's own record identity, so replaying a chunk after a crash is a no-op rather than a doubling. That is not optional at any point in this design, because at-least-once is the only delivery you can build on top of a resumable job. Deduplicating health data is the mechanism.

What to show while it runs

Show the data as it lands, with a banner scoped to the range still importing. Not a global spinner, and not a percentage you invented — you frequently do not know how far back the user's history goes until you hit the wall, so a denominator is a guess.

What we would show instead:

• Progress in days covered, sourced from SUM(days_covered), and the earliest date reached so far. That is a real number derived from committed work.
• Charts labelled with the earliest date actually imported, so an empty region in March reads as "not imported yet" rather than "you did nothing in March".
• Three distinct empty states, because they are three different facts: not imported yet, not permitted to read, and genuinely no activity. Rendering the first two as a zero is a correctness bug, not a display choice.
• Streaks, personal records, "best ever" and trend badges suppressed until the range they depend on is covered. This is the one people skip and regret. A partial history produces a wrong PR, the user sees it, and they remember the number. Congratulating someone on a 5K best that their real history contradicts is worse than showing nothing for an hour.

What we would actually build

Two workers, two cursors, one job table. The incremental worker starts first and never blocks on the backfill worker; its token or anchor is acquired and persisted before the backfill claims its first chunk. The backfill worker claims chunks by priority, works recent-first with widening windows, commits a checkpoint after every chunk, backs off on the platform's documented quota signal rather than a guessed rate, and stops on a permission wall with a distinct status instead of retrying it.

Writes are idempotent on provider record identity. Daily rollups are recomputed from raw rows for the days a chunk touched, never incremented — a backfill that increments counters can never be proven correct after a partial failure, and partial failures are the normal case here.

The honest limits, and when to buy instead

This design does not make a multi-year backfill fast. It makes it interruptible, observable, and non-destructive. A single user's history still arrives at whatever rate their provider's per-user quota allows.

It also does not fix the Android permission reset. If a user reinstalls, the readable window collapses to 30 days before the new grant, and no amount of server-side engineering recovers what you never read. Your server copy is the mitigation, which is an argument for backfilling early and completely rather than lazily.

And Google's rate limits remain unpublished. Everything above is designed to degrade gracefully against a number nobody has told us, which is the right posture but not the same thing as knowing the number.

Backfill is where most teams first seriously price an aggregator, and that instinct is usually correct. The reason is not difficulty, it is variance: each provider has a different history cap, a different quota, a different pagination model, and possibly a different delivery mode. That is N one-off jobs wearing a trench coat, not one job with a config file, and the N grows every time sales asks for another integration. If you have a small team, more than two or three cloud providers, and backfill is not the thing your product is good at, buying it back is a straightforwardly good trade. Health data aggregator APIs and what an aggregator actually is are the places to start.

Two honest caveats on that recommendation. First, an aggregator solves the cloud half only. PERMISSION_READ_HEALTH_DATA_HISTORY is granted to your app on the user's phone, and HealthKit's limited authorization is granted to your app too — those walls are yours no matter who you buy. Second, you would build it yourself when you need reproducible per-source numbers that a vendor's own merge layer hides, when a single provider relationship is genuinely your product, or when your per-user unit economics do not survive per-user pricing at your scale.

Backfill is a job with a budget and a wall. Design for both, checkpoint often, and let the user use the app while it runs.