ARCNM

Building blocks

Pagination

Cursor pagination that never skips or repeats a row, plus date windows for reconciling a batch.

List endpoints page with a cursor. Ask for a page, then follow the cursor the response hands back until it stops handing one back:

Parameter Type Description
limit integer Page size. Defaults and maximums are per-endpoint — see the API reference.
cursor string Opaque position token from the previous page. Omit it for the first page.
order asc | desc Direction over the collection's ordering key. Defaults to newest-first.
created_after RFC 3339 Only rows created at or after this instant (inclusive).
created_before RFC 3339 Only rows created strictly before this instant (exclusive).
include_total boolean Also compute the total row count. Off by default — it costs an extra scan.

Endpoints that already shipped with offset still accept it, but prefer cursor: see Why not offset. Pass one or the other, never both.


Knowing when to stop

Every paginated response tells you where you are, in two places — use whichever suits your client:

In the body, on endpoints that return an envelope:

{
  "items": [ "…" ],
  "count": 50,
  "next_cursor": "eyJ2IjoxLCJrIjpbW…",
  "has_more": true,
  "total": null
}

In the headers, on every paginated endpoint including the ones that return a bare JSON array:

Link: <https://api.arcnm.io/api/v1/uploads?limit=50&cursor=eyJ2Ijox…>; rel="next"
X-Next-Cursor: eyJ2IjoxLCJrIjpbW…
X-Has-More: true
X-Page-Limit: 50

Link follows RFC 8288, so a generic HTTP client can page any of our collections without knowing the body shape. On the last page has_more is false and neither Link nor X-Next-Cursor is sent.

Stop on has_more, never on a short page. A page can come back shorter than limit mid-walk. has_more: false — equivalently, the absence of a next cursor — is the only end-of-collection signal.

count is the page, total is the collection

This trips people up, so it is worth stating plainly: on most endpoints count is the number of rows in this page. It is not a total and you cannot page against it. Ask for include_total=true and read total when you need the full figure.

A few older envelopes — parts, materials/grades, part revisions — publish count as the true total and always have; they are documented that way in the API reference and do not offer include_total. When in doubt, read the per-endpoint schema.


Iterating every page

import requests

def all_rows(url, headers, *, key="items", page=100, **filters):
    """Walk a collection to exhaustion. Filters must stay fixed."""
    cursor, out = None, []
    while True:
        params = {"limit": page, **filters}
        if cursor:
            params["cursor"] = cursor
        r = requests.get(url, headers=headers, params=params)
        r.raise_for_status()
        body = r.json()
        out.extend(body[key])
        if not body["has_more"]:
            return out
        cursor = body["next_cursor"]
async function allRows<T>(
  url: string,
  headers: Record<string, string>,
  { key = "items", page = 100, ...filters }: Record<string, unknown> = {},
): Promise<T[]> {
  const out: T[] = []
  let cursor: string | null = null
  for (;;) {
    const params = new URLSearchParams({ limit: String(page), ...filters as any })
    if (cursor) params.set("cursor", cursor)
    const r = await fetch(`${url}?${params}`, { headers })
    const body = await r.json()
    out.push(...body[key])
    if (!body.has_more) return out
    cursor = body.next_cursor
  }
}
# Follow the Link header — works on every collection, envelope or not.
next="https://api.arcnm.io/api/v1/uploads?limit=100"
while [ -n "$next" ]; do
  headers=$(mktemp)
  curl -sS -D "$headers" -H "X-API-Key: $ARCNM_API_KEY" "$next" | jq -c '.[]'
  next=$(grep -i '^link:' "$headers" | sed -n 's/.*<\(.*\)>; rel="next".*/\1/p')
done

Each request counts toward your rate limits, so very small pages cost more requests. A read is weight 1.


Reconciling a batch

The case cursors exist for: you submitted a few thousand calculations and want to check every one of them against your own records, while more are still landing.

Page ascending inside a time window:

rows = all_rows(
    "https://api.arcnm.io/api/v1/calculations",
    headers,
    page=200,
    order="asc",
    created_after="2026-07-20T09:00:00Z",
)

Ascending order puts newly created rows after your position, so the walk terminates and nothing shifts underneath it. created_after pins the window to the run you care about. Consecutive [after, before) windows tile a range without overlap, because created_after is inclusive and created_before is exclusive.

For a batch you already hold the IDs of, GET /calculations?ids=… is cheaper still: one request returns the status of up to 500 named calculations without paging at all.

For a batch submitted through POST /calculations/batch (including a multi-environment comparison grid) you don't need windows or id lists at all: GET /calculations?batch_id=… enumerates exactly that batch's rows, and GET /calculations/batches/{batch_id}/comparison is the canonical read-back — it returns the pivoted matrix plus a complete flag, so one poll tells you whether every cell has landed.


Rules for cursors

  • Opaque. A cursor encodes a position, not data. Don't parse, build, or persist one across an API version.
  • Bound to the query that made it. Keep limit free to change, but every filter and order must stay identical for the whole walk. Change one and the next request fails with 400 invalid_cursor rather than silently returning a wrong page — restart from the first page instead.
  • Not a snapshot. A walk sees rows as they are when each page is read. Deleting rows behind your position does not corrupt the walk.
  • Tenant-scoped separately. A cursor carries no tenant identity; the credential does. Replaying a cursor under another organization's key returns that organization's rows, never yours.

400 invalid_cursor

{
  "error": {
    "code": "invalid_cursor",
    "message": "cursor belongs to a different query — filters and order must stay identical for the whole walk; restart from the first page"
  },
  "request_id": "req_…"
}

Emitted for a malformed, truncated, hand-edited, or stale-version cursor, for a cursor replayed against different filters, and for passing cursor and offset together.


Why not offset

offset counts rows from the start of the result set at the moment the query runs. Two things break that:

  1. Concurrent writes. Collections are ordered newest-first, so every row created while you page shifts the window — you see some rows twice and miss others. Exactly the case reconciling a bulk run puts you in.
  2. Tied sort keys. A bulk submit writes its whole batch on one timestamp. ORDER BY created_at alone cannot break those ties, and the database is free to return them in a different order per query — so offset pages overlap even on a table nobody is writing to.

Cursors have neither problem: every collection is ordered by its sort key plus the primary key, which makes the order total, and a cursor names a row rather than a count. Deep pages also stay cheap — the database seeks straight to your position instead of counting past everything before it.

Offset remains available where it already shipped, and is fine for a paged UI over a stable collection. Don't use it to enumerate.


See also