The crawl frontier needs a state machine

Expiry does not stop the old worker

Lease expiry is permission to reclaim work, not evidence that the former worker stopped. Give each assignment a monotonically increasing generation and carry it with every result. The writer must compare that generation with the current owner in the same authoritative transaction that accepts the frontier transition. Checking ownership earlier, then writing later, leaves a race in which ownership changes between the two operations.

Burrows’s Chubby paper gives the client a sequencer describing its acquired lock and makes the recipient check validity. A once-valid lock can’t authorize a delayed request forever. I borrow that recipient-side check for the local crawler design.

An obsolete worker can append an attempt and response without changing the replacement’s current state. Named states then explain whether the URL is waiting for capacity, paused by policy, retryable, or terminal. The current-state row is a projection whose transition history should make it reconstructible.

type FrontierTransition = {
  runId: string;
  fetchKey: string;
  attemptId: string;
  from: FrontierState;
  event: string;
  to: FrontierState;
  occurredAt: string;
  policyVersion: string;
  evidenceRef?: string;
  reason: string;
};

The same address is a policy decision

Keep the observed reference, resolved request URL, and comparison key. RFC 3986 permits lowercasing scheme and host, removing dot segments, and normalizing percent-encoded unreserved characters. Deleting queries, folding path case, or equating trailing slashes needs a content-specific policy.

A documented normalizer can classify a parameter as tracking-only, content-bearing, order-sensitive or unknown. Preserve the original href so a later correction can revise the comparison key without erasing the reference the page actually emitted. The normalizer version belongs in the identity key. Fragments usually do not create another HTTP fetch, but belong on discovery edges for in-page navigation.

Admission also needs named rejection states for unsupported schemes, user information, malformed ports and out-of-scope hosts. Silent deletion makes later coverage impossible to reconcile. The run should be able to explain why an observed reference never became a job.

URL identity also determines where discovery links lead. The internal-link graph method tests crawlable edges, reachable components, and contextual relevance.

Capacity belongs to the destination origin

A global FIFO lets a deep site occupy the workers. Give each scheme-host-port origin its own eligible queue, concurrency bound and earliest dispatch time. The global scheduler chooses an origin fairly; the local scheduler chooses a URL. Transport failures, 429s and repeated 5xxs can slow one origin without freezing every other host.

A job admitted for one host may redirect to another with different scope, robots rules and available capacity. Check each hop and preserve its response. An HTTP client following the chain invisibly can bypass the scheduler the application thought it had built.

Robots retrieval needs its status, expiry, parser version and policy result. RFC 9309 defines access-failure and cache behavior; a stricter operator policy belongs alongside the protocol result as an explicit override. Backoff should retain its reason, bounded exponential delay and jitter, with applicable Retry-After honored within local safety limits.

The run also needs a finite contract before discovery starts. Calendars and facet permutations can generate more addresses than the content justifies. Bound URL count, depth and pattern cardinality, then separately bound response bytes, redirects, rendering and wall time. A suppressed pattern needs an example, reason and count so “finished” never conceals “stopped by budget.”

The executable robots.txt wildcard cases test permission for each path. Scheduling allowed fetches and retrying failures come next.

Acknowledgment follows the commit

For the local SQLite design, I prefer immutable worker result messages and one serialized writer. First persist the artifact, then use a short transaction to validate duplicate-payload equality, check current lease generation and commit the accepted attempt and projection. Acknowledgment follows that commit. The network fetch and browser work stay outside it. SQLite transactions.

A crash after commit but before acknowledgment causes redelivery. The attempt key identifies the logical submission; compare the payload hash too, and quarantine conflicting reuse of the ID. The SQL’s WHERE clause only prevents an older accepted sequence replacing a newer one. The lease-generation check must still authorize the frontier transition, under SQLite’s UPSERT semantics.

Persisting the response first can leave an unreferenced blob after a failed database commit, but committing first can leave an accepted result pointing to missing bytes, so I would accept the orphaned blob and reconcile it only after a grace period protects work still in flight. The transaction commits its immutable reference without waiting for HTTP, browser work, or storage calls.

INSERT OR REPLACE can delete and recreate a row, affecting foreign keys and erasing first observations. Named conflict targets make the allowed collision visible. A selected update guarded by attempt sequence prevents an older message from moving the current projection backward, while lease generation remains the authority check for the frontier transition.

BEGIN IMMEDIATE;

INSERT INTO fetch_attempt (attempt_id, run_id, url_id, sequence, status, artifact_id)
VALUES (:attempt_id, :run_id, :url_id, :sequence, :status, :artifact_id)
ON CONFLICT(attempt_id) DO NOTHING;

INSERT INTO url_current (run_id, url_id, sequence, status, attempt_id)
VALUES (:run_id, :url_id, :sequence, :status, :attempt_id)
ON CONFLICT(run_id, url_id) DO UPDATE SET
  sequence = excluded.sequence,
  status = excluded.status,
  attempt_id = excluded.attempt_id
WHERE excluded.sequence > url_current.sequence;

COMMIT;

For an assistant that changes saved state, tool-call timeout recovery follows a write through a lost response, its saved receipt, and a newer edit.

WAL still has one writer

WAL lets readers hold snapshots while a writer appends commits. There is still one writer at a time. I would keep transactions short and busy retries bounded, then use queue age to signal when collection outruns storage.

Long readers hold back checkpoints and can grow the WAL. Close snapshots promptly, paginate exports against a deliberate snapshot, and monitor checkpoint progress. SQLite’s shared-memory design requires participants on the same host. For independent hosts sharing a network filesystem, put write authority behind a service or use a server database.

SQLite documents that synchronous=NORMAL in WAL mode can lose recent committed transactions after power loss, while FULL syncs the WAL at commit. The artifact store needs a compatible persistence guarantee: a database reference cannot rescue an unsynced file. A process-crash test and a power-loss test exercise different promises.

SQLite’s WAL advisory describes a rare WAL-reset corruption issue fixed in SQLite 3.51.3 and later, with backports including 3.44.6 and 3.50.7. Concurrent writing and checkpointing across connections are relevant conditions. Inventory the embedded library used by the application and check the official advisory: the system sqlite3 command can belong to another runtime.

The count belongs to URL identities, not attempts

One admitted URL can have a timeout, an expired lease and a successful retry. That is one covered identity with several attempts, and the history must survive. At its cutoff, the run must reconcile terminal results with work that remains explicitly unresolved. Idle workers say nothing about whether that reconciliation is complete.

Atlas’s incremental-crawl tests cover response reuse after 304, reuse after an unchanged 200, and re-parsing when an artifact version changes. Unchanged content can need a new interpretation, so both the response and its parser or rule version belong in the record.

Interruption tests should target the actual boundaries: before artifact storage, before commit, after commit but before acknowledgment, and during export. A duplicate delivery should leave one logical result; a restore should reproduce run counts and key report queries. Integrity checks and foreign-key checks belong to the archive contract, and a live WAL backup must use a supported mechanism rather than copy only the main file.

A bounded single-worker crawl can keep pending work and its dispositions in one transactional process. It needs no distributed lease service while recovery cannot leave old and new assignments running together. Once those assignments can overlap, the generation check protects the writer from a late return. That concurrency boundary is the reason to add the protocol.

In the recovery design, the stalled worker may finish after its replacement has committed. Its response can still join the attempt history, but it cannot become the current result. Redelivery of an already accepted message should recover the same logical submission; conflicting bytes under that ID require investigation. These are the distinct outcomes that interruption tests around storage, commit and acknowledgment need to exercise.

The frontier owes an account of every admitted URL under the run’s limits. Workers can finish in any order. I would add concurrency only when the writer can preserve one current owner and a recoverable history, regardless of which late message arrives last.