Incomplete API exports are often caused by pagination assumptions rather than missing data. This guide gives you a repeatable API pagination testing workflow for cursor and offset endpoints, including request examples, boundary cases, duplicate detection, and a practical checklist for diagnosing skipped records or pages that stop too soon.
Overview
Pagination divides a large result set into smaller responses. An integration requests one page, reads the records and navigation information, then continues until the API indicates that no more data is available. Problems occur when the client guesses the wrong parameter names, treats a cursor like an offset, ignores a continuation link, or assumes that a fixed page size guarantees a complete export.
Before debugging application code, document the endpoint's actual pagination contract. Record the parameter used to request more data, the maximum permitted page size, the field that contains the next cursor or URL, and the meaning of an empty or missing continuation value. Also note whether the endpoint applies a default sort order and whether records can be added, removed, or updated while the export runs.
There are two common models:
- Offset pagination requests a page using a position and limit, such as
offset=100&limit=50orpage=3&page_size=50. - Cursor pagination requests records after a server-generated position, such as
next_cursor=eyJpZCI6.... The cursor is an opaque value and should normally be reused exactly as returned.
| Model | Typical request | Useful for | Testing concern |
|---|---|---|---|
| Offset | ?offset=100&limit=50 | Simple browsing and smaller, stable datasets | Records can shift between requests when data changes |
| Page number | ?page=3&page_size=50 | User-facing page navigation | Page numbering and maximum limits vary by endpoint |
| Cursor | ?cursor=returned-token | Large or changing datasets | Tokens may expire, be single-use, or depend on filters and sorting |
Use a JSON formatter or API debugging utility to inspect the complete response, not only the records displayed by your application. Navigation metadata may appear beside the data, inside a links object, or in response headers. For related payload issues, see the API response debugging checklist.
Checklist by scenario
When records are missing
- Capture the first request, the final request, and at least one request in the middle. Include the full query string, relevant headers, status code, and response metadata while removing secrets.
- Compare the requested limit with the number of records returned. A server may enforce a lower maximum without treating the request as an error.
- Check whether the client stops on
records.length < requestedLimit. That can be an unreliable stopping rule if the server applies filters, permissions, or a lower cap. Prefer the documented continuation signal. - Confirm that the filter, sort, tenant, and date range remain identical on every request. A missing parameter on page two can look like skipped data.
- Compare the exported identifiers with an independent count or a known test fixture. Counting only rows returned by the client will not reveal silent omissions.
When pages repeat or contain duplicates
- Log the cursor or offset used for each request and the first and last record identifier returned.
- For cursor pagination, verify that the next request uses the new cursor from the current response, not the cursor from the previous iteration.
- For offset pagination, test against a dataset that does not change during the run. If duplicates disappear in a static fixture, concurrent inserts or deletes may be shifting offsets.
- Maintain a set of stable record identifiers during the export. Flag duplicates immediately, but do not silently discard them until you understand whether the endpoint can legitimately return repeated versions.
- Check whether the API sorts by a unique, deterministic field. Sorting only by a non-unique timestamp can produce unstable page boundaries.
When pagination stops early
- Inspect the exact termination condition. Valid signals may include a missing next link, a null cursor, a boolean such as
has_more: false, or an empty result page. - Test an empty dataset and a dataset with exactly one full page, one record over a full page, and several pages.
- Check for rate-limit responses, authentication expiry, timeouts, and transient server errors between successful pages. A loop that treats any non-success response as completion hides the real failure.
- Persist the last successful continuation value so the job can resume or report its stopping point safely.
- Set a maximum page or record guard to prevent an accidental infinite loop when the API returns the same cursor repeatedly.
When the endpoint uses cursors
Treat cursors as opaque. Do not decode, increment, sort, or construct them unless the API documentation explicitly requires that behavior. A cursor is often tied to the original filter and ordering. If the response includes a complete next URL, using that URL can reduce mistakes because it preserves the server's expected parameters.
let cursor = null;
const seen = new Set();
const rows = [];
while (true) {
const url = new URL('/items', API_BASE);
url.searchParams.set('limit', '100');
if (cursor) url.searchParams.set('cursor', cursor);
const response = await fetch(url, { headers });
if (!response.ok) throw new Error(`Page failed: ${response.status}`);
const body = await response.json();
for (const item of body.items ?? []) {
if (seen.has(item.id)) console.warn('Duplicate item', item.id);
seen.add(item.id);
rows.push(item);
}
const next = body.next_cursor ?? null;
if (!next) break;
if (next === cursor) throw new Error('Cursor did not advance');
cursor = next;
}
The example is intentionally defensive: it checks HTTP status, records identifiers, stops on an explicit empty continuation value, and detects a cursor that does not advance. Adapt the field names and retry behavior to the endpoint contract.
When the endpoint uses offsets
Test both the first page and a page that begins after a known record. Confirm whether the offset counts records, pages, bytes, or another unit. If the dataset changes during extraction, prefer a server-supported snapshot, cursor, or stable time boundary. If none exists, record the limitations of the export rather than claiming that offset pagination provides a perfectly consistent view.
What to double-check
- Parameter names: Distinguish
limit,count,page_size,offset, andpage. Similar names do not guarantee equivalent behavior. - Maximum page size: The requested size may be capped, ignored, or rejected. Log the returned count and response metadata.
- Continuation location: The next value may be in JSON, a response header, or a link relation. Do not assume it is always
next_cursor. - Stable ordering: Use the documented sort field. If you control the API, prefer an ordering that includes a unique tie-breaker.
- Filter persistence: Preserve authentication context, tenant scope, search terms, date ranges, and include parameters across every request.
- Retry behavior: Retrying a failed page must not accidentally advance the cursor or append the same page twice. Make writes idempotent where possible.
- Payload shape: Confirm that an empty page is represented consistently and that a missing data property is not being mistaken for an empty successful result.
- Observability: Log request number, page size, continuation state, record count, first and last identifier, duration, and error status without logging tokens or personal data.
For integrations that feed dashboards, validate the final row count and identifier set before refreshing charts. A visually plausible chart can still be based on a partial response. The JavaScript data dashboard guide provides related guidance on data pipelines and performance.
Common mistakes
- Assuming a short page is the last page: Server-side filters, permissions, and page caps can produce short responses. Follow the endpoint's continuation rule.
- Incrementing a cursor: Cursor values are not page numbers. Reuse the returned value exactly.
- Changing query parameters between pages: Rebuilding the URL from scratch can omit a filter or sort option. Copy the documented next URL or centralize request construction.
- Using an unstable sort: Non-unique ordering makes records move across page boundaries. Add a deterministic tie-breaker when the API supports it.
- Ignoring response headers: Retry or rate-limit information may be outside the JSON body. Inspect headers when diagnosing a stalled loop.
- Deduplicating without investigation: A set can hide a pagination defect. Count and report duplicates before deciding how to handle them.
- Logging sensitive tokens: Cursors, authorization values, and URLs can contain sensitive information. Redact them in shared logs and test output.
- Testing only a large production-like dataset: Small boundary fixtures are better at exposing off-by-one errors, empty-page handling, and exact-page termination.
When pagination is part of an event-driven integration, also check whether retries or duplicate deliveries are involved. The webhook debugging guide covers related duplicate-event and retry concerns.
When to revisit
Revisit this checklist whenever you connect a new endpoint, change a filter or sort order, replace an SDK, or alter the export job's retry and storage logic. Pagination behavior is part of an API contract, so a provider's version change or a server-side configuration change can affect limits, cursor lifetime, response fields, and ordering.
It is also worth retesting before a seasonal planning cycle, a major reporting period, or any migration that depends on a complete historical dataset. Compare a controlled sample with an independent source, run the boundary fixtures, and review recent logs for repeated cursors, duplicate identifiers, short pages, and non-success responses.
Before approving an integration, use this final action list:
- Write down the endpoint's pagination contract and the expected termination signal.
- Run empty, exact-page, one-over-page, multi-page, duplicate-prone, and changing-data tests.
- Verify that every request preserves the intended filters, sort, scope, and page-size behavior.
- Track identifiers and continuation state so missing and repeated records are visible.
- Handle errors separately from successful completion, and make retries safe.
- Save a redacted request and response sample with the integration's test notes.
Keep the checklist with the integration rather than treating it as a one-time debugging document. When the endpoint, workflow, or developer tools change, rerun the same tests and compare the results with the previous baseline.