--- title: "Backfill Checkpointer: A Resumable Window Walker" canonical: "https://aifitnessapi.com/cookbook/backfill-checkpointer" cluster: "Cookbook" primary_query: "resumable backfill checkpoint 429 retry code example" last_reviewed: "2026-08-12" description: "Runnable JavaScript for a newest-first backfill: checkpoint after each window, retry the same window on 429, and record gaps instead of skipping." publisher: "AIFitnessAPI — independent, not sponsored" cite_as: "\"Backfill Checkpointer: A Resumable Window Walker\", AIFitnessAPI, https://aifitnessapi.com/cookbook/backfill-checkpointer" --- # Backfill Checkpointer: A Resumable Window Walker > A copy-and-run implementation of the resumable backfill job: newest-first civil-date windows that widen as they go back, a checkpoint written to an injectable store after each window commits rather than before, 429 and 5xx handled by retrying the same window with jittered exponential backoff and Retry-After as a floor, and an exhausted retry budget recorded as a gap carrying the window and the reason instead of silently advancing. A permission wall is a distinct terminal state that is never retried, and an unrecognised error crashes with the checkpoint intact rather than being laundered into a gap. Plain modern JavaScript, no dependencies, Node 20 and above, with a node:test suite that injects the fetcher, the sleeper, the clock and the store so the whole thing runs with no network and no waiting. - Canonical: https://aifitnessapi.com/cookbook/backfill-checkpointer - Last reviewed: 2026-08-12 - Publisher: AIFitnessAPI (https://aifitnessapi.com) — independent, not sponsored - Cite as: "Backfill Checkpointer: A Resumable Window Walker", AIFitnessAPI, https://aifitnessapi.com/cookbook/backfill-checkpointer --- A multi-year first sync is a job, not a loop: ordered newest-first, chunked by civil-date window, checkpointed after every chunk, and degrading against a quota rather than discovering one by exhausting it. The design argument is [backfilling years of wearable data](/architecture/historical-backfill), and the fault cases you have to inject to prove it works are [testing 429 and outage handling](/test/rate-limits-and-outages). This page is the walker those two describe. ## What the recipe does - **Plans windows newest-first, widening backwards.** The default ladder is the last 7 days, then the rest of the last 30, then the rest of the last 90, then a year at a time. Recent windows are small so the first useful screen lands in seconds; old windows are large because nobody is waiting on them and a bigger window costs fewer round trips per day of history covered. Bounds are civil dates, inclusive, never instants — a chunk is "the user's March", and chunking on instants makes a traveller's boundaries drift against the days you later render. - **Checkpoints after the commit, never before.** The ordering inside one window is fetch, then commit, then checkpoint. A crash anywhere in that sequence costs a re-run of that one window and nothing else, which is why the commit has to be idempotent on the provider's own record identity. - **Retries the same window on 429 and 5xx.** Exponential backoff with jitter, and Retry-After treated as a floor rather than a suggestion. RFC 6585 makes the header a MAY, so the no-header path is the normal path, not the exception. - **Records a gap when the budget runs out.** The window, its bounds, the reason, the attempt count and the last error. It never silently advances, because an advanced window is indistinguishable from a covered one and there is no later query that can tell them apart. - **Treats a permission wall as terminal, not retryable.** A 401 or 403 is recorded once and never re-asked, including on resume. Collapsing "we were not allowed to read this" into "this failed" is how you end up hammering a wall on a backoff schedule forever. - **Rethrows anything it does not recognise.** An unclassified error is a bug on your side, and laundering it into a gap makes a false claim about the provider. It crashes with the checkpoint intact instead. ## The implementation This is the canonical file. It lives in the repo at `cookbook/backfill-checkpointer.mjs` and its test file runs in CI on every commit, so what you are reading is what passes. ```js /** * backfill-checkpointer.mjs * * WHAT THIS IMPLEMENTS * A resumable historical-backfill walker. Newest-first civil-date windows * over a date range, widening as they go back; a checkpoint written to an * injectable store AFTER each window commits; 429 and 5xx handled by * retrying the SAME window with exponential backoff (honouring Retry-After) * rather than skipping it; a degrade path that records `{ window, reason }` * when the retry budget is exhausted instead of silently advancing; and a * resume that re-does at most the one window that was in flight. * * A permission wall is a distinct terminal state, not a retryable failure — * collapsing the two is how you hammer a wall on a backoff schedule forever. * Anything unrecognised is fatal and rethrown, because an unexpected bug * should crash loudly, not be laundered into a gap. * * WHICH aifitnessapi.com PAGES DOCUMENT THE PATTERN * https://aifitnessapi.com/architecture/historical-backfill * https://aifitnessapi.com/test/rate-limits-and-outages * https://aifitnessapi.com/cookbook/backfill-checkpointer * * Node 20+. Zero runtime dependencies. The clock, the sleeper, the jitter * source, the checkpoint store and the fetcher are all injected, so the whole * thing runs in microseconds with no network and no real waiting. * * MIT — from aifitnessapi.com/cookbook */ const MS_PER_DAY = 86_400_000; /** Why a window ended up as a gap. Each implies a different next action. */ export const GAP_REASON = Object.freeze({ RETRY_EXHAUSTED: "retry_exhausted", BLOCKED_BY_PERMISSION: "blocked_by_permission", BLOCKED_BY_PROVIDER_CAP: "blocked_by_provider_cap", }); // --------------------------------------------------------------------------- // Window planning — civil dates, newest first, widening backwards // --------------------------------------------------------------------------- function parseCivilDate(s) { const m = /^(\d{4})-(\d{2})-(\d{2})$/.exec(s); if (!m) throw new TypeError(`not a YYYY-MM-DD civil date: ${s}`); return Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3])); } function formatCivilDate(ms) { const d = new Date(ms); return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, "0")}-${String( d.getUTCDate(), ).padStart(2, "0")}`; } /** * Plan the windows for a backfill, newest first. * * `lookbackLadder` is a list of cumulative lookbacks in days: the default * [7, 30, 90] produces "the last 7 days", then "the 23 days before that", then * "the 60 days before that", after which windows are `thenEveryDays` long all * the way back to `from`. * * Recent windows are small so the first useful screen lands fast; old windows * are large because nobody is waiting on them and a bigger window costs fewer * round trips per day of history. Bounds are civil dates, inclusive, never * instants — a chunk is "the user's March". * * @param {{from: string, to: string, lookbackLadder?: number[], thenEveryDays?: number}} opts * @returns {Array<{id: string, start: string, end: string, days: number, priority: number}>} */ export function planWindows({ from, to, lookbackLadder = [7, 30, 90], thenEveryDays = 365 }) { const fromMs = parseCivilDate(from); const toMs = parseCivilDate(to); if (fromMs > toMs) throw new RangeError("`from` must not be after `to`"); const windows = []; let cursorEnd = toMs; // inclusive end of the next (older) window let consumed = 0; // days already planned, counting back from `to` for (const lookback of lookbackLadder) { if (cursorEnd < fromMs) break; const size = lookback - consumed; if (size <= 0) continue; const start = Math.max(fromMs, cursorEnd - (size - 1) * MS_PER_DAY); windows.push({ start, end: cursorEnd }); consumed = lookback; cursorEnd = start - MS_PER_DAY; } while (cursorEnd >= fromMs) { const start = Math.max(fromMs, cursorEnd - (thenEveryDays - 1) * MS_PER_DAY); windows.push({ start, end: cursorEnd }); cursorEnd = start - MS_PER_DAY; } return windows.map((w, i) => ({ id: `${formatCivilDate(w.start)}..${formatCivilDate(w.end)}`, start: formatCivilDate(w.start), end: formatCivilDate(w.end), days: Math.round((w.end - w.start) / MS_PER_DAY) + 1, priority: i, // lower runs first, and index 0 is the most recent window })); } // --------------------------------------------------------------------------- // Failure classification // --------------------------------------------------------------------------- const RETRYABLE_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]); const PERMISSION_STATUS = new Set([401, 403]); export function isRetryableStatus(status) { return RETRYABLE_STATUS.has(status); } /** * `retry` | `permission` | `provider_cap` | `fatal`. * * An error is retryable if it says so or carries a retryable status. It is a * wall if it says so or carries 401/403. Everything else is a bug in your own * code and is rethrown — never turned into a gap, because a gap is a claim * about the provider. */ export function classifyFailure(err) { if (err && err.permanent === true) return err.reason ?? "permission"; if (err && err.retryable === true) return "retry"; const status = err && err.status; if (isRetryableStatus(status)) return "retry"; if (PERMISSION_STATUS.has(status)) return "permission"; return "fatal"; } /** Retry-After in milliseconds, if the error carries one. RFC 6585 makes it a MAY. */ export function retryAfterMs(err) { if (!err) return null; if (Number.isFinite(err.retryAfterMs)) return err.retryAfterMs; if (Number.isFinite(err.retryAfterSeconds)) return err.retryAfterSeconds * 1000; return null; } /** * Exponential backoff with full jitter. `random` is injected so a test gets a * deterministic schedule; jitter is not optional in production, because fixed * backoff passes every single-user test and then synchronises every worker * onto the same second the moment the provider recovers. */ export function exponentialBackoff({ baseMs = 1000, factor = 2, maxMs = 60_000, jitter = 1, random = Math.random, } = {}) { return function delayFor(attempt, err) { const ceiling = Math.min(maxMs, baseMs * factor ** Math.max(0, attempt - 1)); const jittered = ceiling * (1 - jitter) + ceiling * jitter * random(); // Retry-After is a floor, not a suggestion. Never sleep less than it. return Math.max(jittered, retryAfterMs(err) ?? 0); }; } // --------------------------------------------------------------------------- // Checkpoint store // --------------------------------------------------------------------------- /** * The shape a checkpoint store must implement. Swap for a row in Postgres, * a KV entry, anything durable — the walker only calls load() and save(). */ export function createMemoryCheckpointStore(seed = {}) { const data = new Map(Object.entries(seed)); let saves = 0; return { get saveCount() { return saves; }, async load(jobId) { const v = data.get(jobId); return v ? structuredClone(v) : null; }, async save(jobId, state) { saves += 1; data.set(jobId, structuredClone(state)); }, }; } function emptyCheckpoint() { return { completed: [], gaps: [] }; } // --------------------------------------------------------------------------- // The walker // --------------------------------------------------------------------------- const realSleep = (ms) => new Promise((r) => setTimeout(r, ms)); /** * Walk the windows newest-first, committing and checkpointing as it goes. * * Ordering inside one window is load-bearing: * fetch -> commit -> checkpoint. * The checkpoint is written last, so a crash anywhere in that sequence costs * you a re-run of that one window and nothing else. `commit` must therefore be * idempotent on the provider's own record identity; at-least-once is the only * delivery a resumable job can offer. * * @param {object} opts * @param {string} opts.jobId * @param {Array} opts.windows from planWindows() * @param {Function} opts.fetchWindow async (window) => records * @param {Function} [opts.commit] async (window, records) => void; must be idempotent * @param {object} opts.store { load(jobId), save(jobId, state) } * @param {number} [opts.maxAttempts=4] attempts per window, including the first * @param {Function} [opts.backoff] (attempt, err) => ms * @param {Function} [opts.sleep] async (ms) => void * @param {Function} [opts.now] () => epoch ms * @param {Function} [opts.onEvent] (event) => void, for logging and tests */ export async function runBackfill({ jobId, windows, fetchWindow, commit = async () => {}, store, maxAttempts = 4, backoff = exponentialBackoff(), sleep = realSleep, now = Date.now, onEvent = () => {}, }) { if (!jobId) throw new TypeError("jobId is required"); if (!store || typeof store.load !== "function" || typeof store.save !== "function") { throw new TypeError("store must implement load(jobId) and save(jobId, state)"); } const checkpoint = (await store.load(jobId)) ?? emptyCheckpoint(); const completed = new Set(checkpoint.completed); const gaps = checkpoint.gaps.slice(); const gapped = new Set(gaps.map((g) => g.window)); const fetched = []; let slept = 0; let redone = 0; for (const window of windows) { if (completed.has(window.id)) { onEvent({ type: "skip", window: window.id, reason: "already_committed" }); continue; } if (gapped.has(window.id)) { onEvent({ type: "skip", window: window.id, reason: "already_gapped" }); continue; } let attempt = 0; let outcome = null; // Retry loop. Every path out of it is terminal for this window: either it // commits, or it becomes a recorded gap, or it throws. for (;;) { attempt += 1; let records; try { fetched.push(window.id); records = await fetchWindow(window, { attempt }); } catch (err) { const kind = classifyFailure(err); if (kind === "retry") { if (attempt >= maxAttempts) { // Never silently advance. Record what we could not read and why. outcome = { window: window.id, start: window.start, end: window.end, reason: GAP_REASON.RETRY_EXHAUSTED, attempts: attempt, lastError: String((err && err.message) || err), at: now(), }; onEvent({ type: "gap", ...outcome }); break; } const delay = backoff(attempt, err); slept += delay; onEvent({ type: "retry", window: window.id, attempt, delay }); await sleep(delay); continue; // the SAME window, not the next one } if (kind === "permission" || kind === "provider_cap") { // A wall, not a failure. Retrying it on a backoff schedule forever is // the bug this branch exists to prevent. outcome = { window: window.id, start: window.start, end: window.end, reason: kind === "permission" ? GAP_REASON.BLOCKED_BY_PERMISSION : GAP_REASON.BLOCKED_BY_PROVIDER_CAP, attempts: attempt, lastError: String((err && err.message) || err), at: now(), }; onEvent({ type: "gap", ...outcome }); break; } // Unclassified: a bug on our side. Crash with the checkpoint intact. onEvent({ type: "fatal", window: window.id, attempt }); throw err; } await commit(window, records); if (attempt > 1) redone += 1; completed.add(window.id); onEvent({ type: "commit", window: window.id, attempts: attempt, records: records?.length ?? 0 }); outcome = null; break; } if (outcome) { gaps.push(outcome); gapped.add(window.id); } // Checkpoint AFTER the window is terminal, never before. await store.save(jobId, { completed: [...completed], gaps }); } return { jobId, committed: [...completed], gaps, fetchedWindows: fetched, requestCount: fetched.length, retriedWithinWindow: redone, sleptMs: slept, }; } /** * Invariant 1 from /test/rate-limits-and-outages: every window in the requested * range ends in a terminal state, and none silently disappeared. */ export function coverageReport(windows, result) { const done = new Set(result.committed); const gapped = new Set(result.gaps.map((g) => g.window)); const missing = windows.filter((w) => !done.has(w.id) && !gapped.has(w.id)).map((w) => w.id); const daysCovered = windows.filter((w) => done.has(w.id)).reduce((n, w) => n + w.days, 0); const daysGapped = windows.filter((w) => gapped.has(w.id)).reduce((n, w) => n + w.days, 0); return { total: windows.length, done: windows.filter((w) => done.has(w.id)).length, gapped: windows.filter((w) => gapped.has(w.id)).length, missing, complete: missing.length === 0, daysCovered, daysGapped, }; } ``` ## Adapting it **The checkpoint store is the only thing you must replace.** It needs `load` and `save` and nothing else, so a row in Postgres, a KV entry or a document all fit. Keep the durability honest: the recipe's guarantee is that a resume re-does at most the window that was in flight, and that guarantee is exactly as strong as your store's write. **Add the provider's pagination cursor if it has one.** The walker checkpoints per window, which is the right granularity for a date-windowed REST API. Where the provider pages inside a window, carry the page token in the checkpoint alongside the window id and resume from it, so a storm mid-window costs a page rather than a chunk. On an anchor-paginated platform the anchor is itself the checkpoint and you do not need date windows at all — see [incremental sync](/architecture/incremental-sync) for what that cursor is and is not. **Do not tune the ceiling; parameterise it.** There is no published number to tune a Health Connect loop against, and cloud providers that do publish one usually count it per consented user rather than per app, so more workers buy nothing. Run your fake at a ceiling low enough that the storm case triggers every time and assert degradation behaviour rather than throughput. A job that is correct at ten requests an hour is correct at a thousand; the reverse is not true. The symptom-level version for one provider is [Fitbit 429 rate limits](/fix/fitbit-api-429-rate-limit). **Reserve quota for the live path before the backfill claims any.** The walker takes no view on this, and it should: a backfill that spends the whole per-user allowance makes today's data stale in order to complete a year nobody is looking at. Gate `fetchWindow` behind a budget that the incremental worker draws from first. **A gap is not a zero.** The gap list is the input to your empty-state logic, and "not imported yet", "not permitted to read" and "genuinely no activity" are three different facts that must render as three different things. Rendering the first two as zero is a correctness bug, not a display choice; [missing data and gaps](/architecture/missing-data-and-gaps) is the schema that keeps them apart. **Suppress derived numbers while it runs.** A partial history is a complete history as far as a personal-record calculation is concerned. Streaks, best-ever markers and trend badges stay off until the range they depend on is covered, because a wrong PR is a number the user will remember long after the import finishes. ## The test contract The suite in `cookbook/backfill-checkpointer.test.mjs` asserts: - **Windows are civil dates, newest first, contiguous and non-overlapping**, with sizes 7, 23, 60 then 365 days, and the planned days summing to the requested range exactly once. - **Failure classification separates retry, wall and bug**, and Retry-After acts as a floor: 100 ms of computed backoff becomes 5,000 ms when the provider asks for five seconds, and stays at 800 ms when our own backoff already exceeds what it asked for. - **A 429 retries the same window.** The request log reads window 0, window 0, window 0, then window 1 — never window 1 in place of window 0 — with 100 ms and 200 ms of backoff between the attempts. - **Exhausted retries record a gap and the walk continues.** The gap carries the window, its bounds, the reason and an attempt count of 3; the window is not in the committed list; the next window is still fetched; and the coverage report shows nothing missing, with the gapped days counted separately from the covered ones. - **A permission wall is asked once and never retried**, with no backoff at all, and a resumed run issues zero requests rather than re-asking. - **A crash mid-run resumes from the checkpoint.** An unclassified error rejects the run instead of becoming a silent gap; the durable checkpoint holds exactly the two committed windows; and the second run issues zero requests for either of them while every record still lands exactly once. - **A crash between commit and checkpoint re-does at most the in-flight window.** With the checkpoint write failing after the second window committed, the resumed run re-fetches exactly one window, commits it a second time, and the idempotent sink still holds one copy of every record. - **A completed run's gap list is exact** — one retry-exhausted window and one permission-walled window, in order, with no invented gaps and none swallowed. Every window in the range is terminal, the gap list survives a restart, and the restart issues no requests. ## What this does not fix It does not make a backfill fast. It makes it interruptible, observable and non-destructive; a single user's history still arrives at whatever rate their provider's per-user quota allows, and you should size the product experience around hours rather than seconds. It also cannot recover what the platform will not hand over. Where a mobile health platform resets its readable window on reinstall, the history you never read is gone, and your own server copy is the only mitigation — which is an argument for backfilling early and completely rather than lazily. And fault injection tells you nothing about how the real provider enforces its limit: whether the bucket is per user or per app, fixed or sliding, or whether repeated violations compound. None of that is in your fake because none of it is documented. What replaces it is one deliberate staging run against a real consented account, plus a permanent production signal on completion time, 429 rate and windows sitting in a gap state longer than they should. ## FAQ ### Should the checkpoint be written before or after a window commits? After, always, and the ordering is the only reason this recipe offers any guarantee at all. Fetch, then commit, then checkpoint. Written in that order, a crash anywhere in the sequence leaves at most one window to redo, and the recipe's test suite proves it by failing the checkpoint write immediately after a window has committed and asserting the resumed run re-fetches exactly one window. Written the other way round, a crash between the checkpoint and the commit loses a window permanently and nothing downstream can tell. The price of the safe ordering is that the commit has to be idempotent on the provider's own record identity, because the redo is a re-commit. [Permalink](https://aifitnessapi.com/cookbook/backfill-checkpointer#faq-1) ### What should a backfill worker do when its retry budget runs out on a window? Record the window and why it failed, then move on to the next one. Never mark it done and never let the walk stop. This implementation writes an entry carrying the window id, its civil-date bounds, a reason code, the attempt count and the last error, and it keeps that list in the checkpoint so it survives a restart. The reason the reason code matters is that each one implies a different next action: a retry-exhausted window is worth another pass later, while a window behind a permission wall is not, and collapsing them means either hammering the wall forever or abandoning recoverable history. What you must not do is advance quietly, because an advanced window is indistinguishable from a covered one afterwards. [Permalink](https://aifitnessapi.com/cookbook/backfill-checkpointer#faq-2) ### How do I stop a recorded gap from being rendered as a zero-activity day? Keep the gap list as a first-class input to the display layer rather than as a log line. The walker returns the gaps and persists them, and the coverage report tells you how many days are covered against how many are gapped, which is the number a progress banner should use. Downstream, not imported yet, not permitted to read, and genuinely no activity have to render as three different empty states; a zero in place of the first two is a correctness bug rather than a display choice. Also suppress anything derived from a range that is still incomplete. A partial history produces a confident personal record the user knows they never set, and they remember the number long after the import finishes. [Permalink](https://aifitnessapi.com/cookbook/backfill-checkpointer#faq-3) ### Is it safe to run two workers against one checkpoint store? Not as written, and that is deliberate. This walker owns its job and assumes one runner, because the guarantee it makes is about crash resumption rather than concurrency. To fan out, move claiming into the store: give each window a claimed-by and a claim expiry so a dead worker's window becomes reclaimable, and have workers claim the highest-priority unclaimed window rather than iterating a list. The other separation matters more in practice. Keep the backfill and the live incremental path in different queues with different cursors, and reserve a share of the per-user quota for the live path first, or a slow multi-year import makes today's data stale to complete a year nobody is looking at. [Permalink](https://aifitnessapi.com/cookbook/backfill-checkpointer#faq-4)