---
title: "Testing 429 Rate-Limit and Outage Handling in a Health Backfill"
canonical: "https://aifitnessapi.com/test/rate-limits-and-outages"
cluster: "Testing"
primary_query: "test api rate limit handling 429"
last_reviewed: "2026-07-27"
description: "Fault-inject 429 storms, 503s and slow responses into your provider fake, then assert a health backfill resumes from its checkpoint with no data loss."
publisher: "AIFitnessAPI — independent, not sponsored"
cite_as: "\"Testing 429 Rate-Limit and Outage Handling in a Health Backfill\", AIFitnessAPI, https://aifitnessapi.com/test/rate-limits-and-outages"
---

# Testing 429 Rate-Limit and Outage Handling in a Health Backfill

> Point the fault injection at a historical backfill rather than at your retry helper, and assert observable outcomes: no date window silently skipped, a resumed job that re-requests zero completed windows, a bounded retry budget, and recovery traffic that is spread rather than synchronised. RFC 6585 section 4 defines 429 and makes Retry-After a MAY, so a named no-header case with your own backoff is mandatory, not a nice-to-have. Provider quotas are provider-specific and frequently unpublished, so design the job to degrade instead of tuning it to a figure. Test the slow response too, because it ties up workers silently while an outage at least fails loudly.

- Canonical: https://aifitnessapi.com/test/rate-limits-and-outages
- Last reviewed: 2026-07-27
- Publisher: AIFitnessAPI (https://aifitnessapi.com) — independent, not sponsored
- Cite as: "Testing 429 Rate-Limit and Outage Handling in a Health Backfill", AIFitnessAPI, https://aifitnessapi.com/test/rate-limits-and-outages

---

Day 400 of a three-year backfill, the provider starts answering `429`. Your worker retries, the retries also 429, the pod hits its deadline and dies, and the next attempt starts at the beginning of the user's history — re-fetching the same 400 days against the same exhausted per-user quota, while the chart in the app still stops in 2023. That is the expensive failure.

The cheap one is worse. A single response comes back truncated mid-array, your JSON decoder is lenient about it, the window parses as an empty list, and your writer records a zero-step day over a Tuesday last March when the user actually walked eleven thousand. No exception, no retry, no hole for [gap handling](/architecture/missing-data-and-gaps) to notice later, because a row exists. The only thing that would have caught it is a test that fed your real client a deliberately broken body and then looked at what landed in the table.

Both are backfill failures, and that is not a coincidence. Steady-state incremental sync pulls a handful of windows per user per day and almost never touches the ceiling. A first sync pulls years for one consented user in one burst, which is the only routine moment you meet the quota head-on — and the only moment where a bad retry strategy costs a user their history rather than costing them a spinner.

## Put the faults in the fake, not in a retry unit test

The instinct is to unit-test the backoff helper: hand it a synthetic 429, assert it slept the right number of seconds. That test cannot fail. You wrote the response, you wrote the clock, and you asserted that your code computed the number your code computed. It never touches the parser, the checkpoint writer, the worker pool, or the interaction between them, which is where every real bug in this area lives.

Inject at the HTTP boundary instead — the same fake provider you already stand up for [provider test environments](/test/provider-sandboxes), with faults as a scriptable property of the fake rather than a separate test double. Named tooling, links only, because none of it is health-specific and all of it is well documented:

- [WireMock](https://github.com/wiremock/wiremock) stubs arbitrary status codes and headers, and its `Fault` enum injects `CONNECTION_RESET_BY_PEER`, `EMPTY_RESPONSE`, `MALFORMED_RESPONSE_CHUNK` and `RANDOM_DATA_THEN_CLOSE`, plus response delays (enum read from source, 2026-07-30).
- [Toxiproxy](https://github.com/Shopify/toxiproxy) degrades the transport underneath any endpoint, real or faked: latency with jitter, bandwidth, timeout, slow close, reset peer, slicer, packet loss.
- [MSW](https://github.com/mswjs/msw) for JavaScript and TypeScript; [VCR](https://github.com/vcr/vcr), [vcrpy](https://pypi.org/project/vcrpy/) and [RESPX](https://pypi.org/project/respx/) for recorded fixtures.

The split that matters: WireMock forges HTTP semantics, so it is the only way to produce "429 with a weird `Retry-After`". Toxiproxy degrades the socket, so it is the honest way to produce a slow response that exercises your real timeouts. Use both.

## The named cases

`429` is **not** in RFC 9110. RFC 9110 §10.2.3 defines `Retry-After` and names only 503 and 3xx; the string "Too Many Requests" appears nowhere in it. 429 is **RFC 6585 §4**, and there the wording is that a 429 response "MAY include a Retry-After header indicating how long to wait before making a new request." A **MAY**. A client that assumes the header is present is wrong against a compliant provider, so the no-header case is a named test, not an afterthought.

| Case | What the fake does | What you assert |
|---|---|---|
| 429 with `Retry-After` in seconds | `Retry-After: 120` | The client waits at least that long. Not less, and not a hardcoded default that happens to exceed it. |
| **429 with no `Retry-After`** | 429, no header | The client falls back to your own exponential backoff with jitter. It does not busy-loop, and it does not raise a `KeyError` on a missing header. |
| 429 with an HTTP-date | `Retry-After: Sat, 01 Jan 2028 00:00:00 GMT` | The parser handles the date form. RFC 9110 §10.2.3 permits `HTTP-date` or `delay-seconds`; a parser that only calls `int()` is non-compliant. |
| 429 storm | 429 for longer than your whole retry budget | The job parks the window in a retryable state and exits cleanly. It does not mark the window done, and it does not exhaust the pod's deadline spinning. |
| 503 mid-window | 503, sometimes with `Retry-After`, sometimes with a dropped connection | Same parking behaviour. RFC 9110 §15.6.4 notes servers MAY send `Retry-After` on a 503 and that some overloaded servers "might simply refuse the connection" instead. |
| Truncated body | `MALFORMED_RESPONSE_CHUNK` or `EMPTY_RESPONSE` | The parser raises. **Zero rows written for that window**, and the window's status is not `done`. |
| Garbage instead of JSON | `RANDOM_DATA_THEN_CLOSE` | Covers the "provider returned an HTML error page" class. Same assertion: raise, write nothing, do not advance the checkpoint. |
| Slow response | Toxiproxy latency just inside your client timeout | See below — this is the one people skip. |

One health-specific case belongs in that list and is easy to miss. On Health Connect a stalled backfill can outlive the changes token that was supposed to cover the same period, because those tokens expire when unused. Script a 429 stall long enough to cross that boundary and assert the delta stream is still alive and independently checkpointed. The design reasoning for why they must be separate queues is in [the historical backfill architecture](/architecture/historical-backfill); this page is only the assertion that the separation was actually built.

## What "handled correctly" means observably

Four invariants. All of them are assertions on state you own, not on your client's internal decisions.

**1. No data loss.** Every civil-date window in the requested range ends in a terminal state — `done`, or `parked` with a reason — and never silently disappears. Assert on the checkpoint table, not on log lines.

```sql
-- Invariant 1: no window vanished during the storm.
SELECT count(*) FROM backfill_window
WHERE user_id = :uid AND status NOT IN ('done', 'parked');
-- expected: 0
```

**2. Resumable progress.** Kill the worker mid-storm, restart it against the same checkpoint store, and assert that it re-requests nothing it already finished. This is the assertion that distinguishes a job from a loop, and it is the one that fails most often in practice.

```sql
-- Invariant 2: the resumed run did not redo finished work.
SELECT count(*) FROM fake_provider_request_log r
JOIN backfill_window w USING (user_id, window_start, window_end)
WHERE r.run_id = :second_run AND w.status = 'done'
  AND w.completed_in_run = :first_run;
-- expected: 0
```

**3. Bounded retry.** Total attempts per window and total wall clock for the job are both capped. The test is not "does it eventually succeed" — it is "does it give up in finite time and leave the range recoverable".

**4. No thundering herd on recovery.** This one only shows up with more than one user in the fixture, which is why single-user tests miss it forever. Rate-limit every user, restore 200s at a known instant, and look at the distribution of arrival times the fake recorded.

```python
def test_recovery_is_spread(fake, workers):
    fake.rate_limit_all()
    fake.restore_success_at(t0)
    counts = bucket(fake.arrival_times(after=t0), width_seconds=1)
    # A no-jitter client puts every arrival in the first bucket.
    # Choose MAX_PER_BUCKET from the quota you are willing to spend.
    assert max(counts) <= MAX_PER_BUCKET
```

Fixed backoff passes every single-user test you will ever write, then synchronises every worker you have onto the same second the moment a provider recovers.

## The slow response is the one that hurts

An outage fails loudly. The window re-queues, the worker is free in milliseconds, your error rate spikes and somebody gets paged. A response that lands just inside your client timeout does none of that: the worker sits there holding a connection, the error rate stays at zero, nothing alerts, and the only symptom is that a backfill budgeted for twenty minutes is still running tomorrow. Where a provider counts its quota per consented user rather than per app — which our own [Fitbit 429 fix page](/fix/fitbit-api-429-rate-limit) documents for Fitbit specifically — you cannot buy your way out by adding workers, so a pool full of stalled connections is a pool that is genuinely stuck. Which shape you are facing is usually undocumented, and the last section of this page is about what to do with that.

Assert a wall-clock bound on the *whole job*, not just on individual calls, and assert that a window which exceeds its share is abandoned and re-queued rather than waited on. Then check your SDK's defaults instead of assuming them. Terra's official Python SDK, for one, documents automatic retries with exponential backoff on 408, 429 and 5XX with a default of 2 attempts and a 60-second default timeout (read from the SDK README, 2026-07-30). Those are reasonable interactive defaults and questionable backfill defaults, and the only way you find that out is a latency toxic and a stopwatch.

There is a second-order trap worth one test. An SDK that auto-retries a token refresh on a 5XX, against a provider that rotates refresh tokens, can present an invalidated token and trip the provider's replay detection — which costs the user a full re-authorization. Retry policy and rotation policy have to be tested together; the token-lifecycle cases live in [testing OAuth flows](/test/oauth-flows).

## The three green results that mean nothing

Three of them show up specifically here, and each reads as coverage while proving nothing:

- **Asserting the sleep.** Covered above. Your code, your number, your clock.
- **A fake that is too generous.** If every 429 your fake emits carries a `Retry-After`, your fallback backoff is never executed and the suite is green because of the fixture's manners, not the client's correctness. That is why the no-header row in the table above is mandatory.
- **Asserting the retry happened.** Counting requests proves the client tried again. It says nothing about whether the day landed, landed once, or landed as a zero. Assert on rows, and assert on them after a resume.

## You cannot tune to a number you do not have

Do not build a test around a specific requests-per-hour figure. We could not **re-verify** any cloud fitness provider's published limit in this round (2026-07-30), because every provider documentation host was unreachable from our research environment — which is a fact about our network, not evidence that no number exists. Our own Fitbit 429 page, linked above, does carry a figure, dated and with a verify caveat attached to it, and where a provider reports your current allowance in its response headers that is the value to read at runtime. Hard-code neither. The one adjacent fact we could confirm this round is that Terra's own SDK treats 429 as retryable, which tells you 429s happen and tells you nothing about the quota.

The practical consequence is that the ceiling in your fake is a *parameter*, not a constant. Make it configurable, run the suite at a ceiling low enough that the storm case triggers every time, and assert degradation behaviour rather than throughput. A job that is correct at ten requests per hour is correct at a thousand; the reverse is not true.

## Where this stops

Fault injection cannot tell you how the real provider enforces its limit: whether the bucket is per user or per app, whether it is a fixed window or a sliding one, whether repeated violations compound into something harsher, or whether it degrades by dropping requests rather than by returning 429. None of that is in your fake, because none of it is documented.

What replaces it, in our experience, is two things. One deliberate staging run per provider against a real consented account, driven hard enough to see an actual 429 and record what the headers say — done once at integration time and again when a provider changes anything. And a production signal you keep permanently: backfill completion time per provider, 429 rate per provider, and an alert on windows sitting in `parked` for longer than they should. The fake proves your client behaves; only production tells you what it is behaving against.

For the webhook side of the same problem — duplicate and out-of-order delivery, which fails silently in exactly the same way — see [testing webhooks locally](/test/webhooks-locally).

## FAQ

### How do I test 429 handling when the provider never sends a Retry-After header?

RFC 6585 section 4 defines the 429 status code and says a response MAY include Retry-After, so a compliant provider is allowed to omit it. Add a named case where your fake returns 429 with no header at all and assert the client falls back to your own exponential backoff with jitter rather than busy-looping or raising on the missing key. Add a second case where Retry-After is an HTTP-date instead of an integer, which RFC 9110 also permits and which a parser that only calls int() will reject.

[Permalink](https://aifitnessapi.com/test/rate-limits-and-outages#faq-1)

### Should fault injection live in the retry helper's unit test or in the fake provider?

In the fake. A unit test that hands a synthetic 429 to your backoff function asserts that your code slept the number your code computed, which is a test that cannot fail. Injecting the same fault at the HTTP boundary exercises the client, the response parser, the checkpoint writer and the worker pool together, and that combination is where the real defects are.

[Permalink](https://aifitnessapi.com/test/rate-limits-and-outages#faq-2)

### How do I prove a backfill resumes from its checkpoint instead of restarting?

Have the fake log every request it serves, tagged with a run id. Run the job until the storm kills it, record which date windows committed, restart the worker against the same checkpoint store, and assert that the second run issued zero requests for windows the first run had already marked done. Then assert the final set of terminal windows covers the requested range exactly once.

[Permalink](https://aifitnessapi.com/test/rate-limits-and-outages#faq-3)

### Why is a slow provider response worse than an outage in a backfill test?

An outage fails fast, so the window re-queues and the worker is free almost immediately. A response that arrives just inside your client timeout holds that worker for its whole duration, produces no errors to alert on, and cannot be compensated for by adding workers wherever the provider counts its quota per consented user rather than per app — which shape you are facing is usually undocumented, so assume the worse one. The only observable is that the job runs far past its budget, so assert a wall-clock bound on the entire job rather than on individual calls.

[Permalink](https://aifitnessapi.com/test/rate-limits-and-outages#faq-4)

### What should a test assert about recovery after a rate-limit storm?

That the resumed traffic is spread out. Rate-limit every user in the fixture, restore success responses at a known instant, bucket the arrival times the fake recorded, and assert no bucket exceeds the share of quota you are willing to spend. Fixed backoff without jitter passes every single-user test and then synchronises every worker onto the same second the moment the provider recovers.

[Permalink](https://aifitnessapi.com/test/rate-limits-and-outages#faq-5)