Crawlee Crash Recovery: Resume Work without Duplicate Writes

The difficult crash window occurs after data commits but before a request is acknowledged. A restart may replay that request. Append-only output duplicates it; acknowledging before writing can lose it.

This tutorial combines a two-page book crawl with a persistent Crawlee queue and SQLite product identity. Its deliberate interruption switch illustrates ordering. The scope is a single-machine, single-process task, not distributed exactly-once processing.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

BOOKS_LIST_URL: the first Books to Scrape catalogue-page address.

Set these in the same Bash/WSL terminal used for subsequent commands:

export BOOKS_LIST_URL='<paste-the-actual-books-list-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install and fix task storage

Use Node.js 24 with the DatabaseSync API; consult the Node SQLite reference for the runtime's support status.

mkdir resumable-catalog
cd resumable-catalog
npm init -y
npm install crawlee@3.18.2
export CRAWLEE_STORAGE_DIR="$PWD/resume-store"
export CRAWLEE_PURGE_ON_START=0

Retain the same directory, queue name, and database for recovery. A fresh directory starts new work rather than resuming. See Crawlee configuration.

Separate queue and business identity

Request unique keys govern scheduling. UPC governs the product row. Neither replaces the other.

The example also stores source scope to reject accidental reuse under a changed entry. Only one process should own this local task directory.

Run the complete example

Create resume.mts:

import { mkdirSync } from 'node:fs';
import { join } from 'node:path';
import { DatabaseSync } from 'node:sqlite';
import { CheerioCrawler, RequestQueue, NonRetryableError } from 'crawlee';
 
const storage = process.env.CRAWLEE_STORAGE_DIR;
if (!storage) throw new Error('请固定 CRAWLEE_STORAGE_DIR');
mkdirSync(storage, { recursive: true });
const db = new DatabaseSync(join(storage, 'catalog.sqlite'));
db.exec(`
CREATE TABLE IF NOT EXISTS products(
  upc TEXT PRIMARY KEY,title TEXT NOT NULL,price_display TEXT NOT NULL,source_url TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS settings(k TEXT PRIMARY KEY,v TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS failures(
  url TEXT PRIMARY KEY,label TEXT NOT NULL,page_no INTEGER,error_type TEXT NOT NULL
);
`);
const start = process.env.BOOKS_LIST_URL;
if (!start) throw new Error('Set BOOKS_LIST_URL before running');
const scope = JSON.stringify({ start, maxListPages: 2 });
const priorScope = db.prepare("SELECT v FROM settings WHERE k='scope'").get();
if (priorScope && priorScope.v !== scope) throw new Error('任务范围已变，请使用独立任务目录');
db.prepare("INSERT OR IGNORE INTO settings VALUES ('scope',?)").run(scope);
const queue = await RequestQueue.open('catalog-resume');
await queue.addRequest({ url: start, label: 'LIST', userData: { page: 1 } });
 
const repair = process.env.REPAIR_RUN;
if (repair) {
    for (const row of db.prepare('SELECT * FROM failures').all()) {
        await queue.addRequest({
            url: String(row.url), label: String(row.label),
            userData: { page: row.page_no },
            uniqueKey: 'repair:' + repair + ':' + row.url,
        });
    }
}
const crawler = new CheerioCrawler({
    requestQueue: queue,
    maxConcurrency: 1,
    maxRequestsPerCrawl: 100,
    maxRequestRetries: 1,
    async requestHandler({ request, $, enqueueLinks }) {
        if (request.label === 'LIST') {
            if (!$('article.product_pod h3 a').length) {
                throw new NonRetryableError('列表没有商品');
            }
            await enqueueLinks({ selector: 'article.product_pod h3 a', label: 'DETAIL' });
            if (Number(request.userData.page) < 2) {
                if (!$('li.next a').length) throw new NonRetryableError('缺少下一页');
                await enqueueLinks({
                    selector: 'li.next a', label: 'LIST', userData: { page: 2 },
                });
            }
            db.prepare('DELETE FROM failures WHERE url=?').run(request.url);
            return;
        }
        if (request.label !== 'DETAIL') throw new NonRetryableError('未知请求标签');
        const fields = new Map<string, string>();
        $('table.table-striped tr').each((_, row) => {
            fields.set($(row).find('th').text().trim(), $(row).find('td').text().trim());
        });
        const upc = fields.get('UPC');
        const title = $('.product_main h1').text().trim();
        const price = fields.get('Price (incl. tax)');
        if (!upc || !title || !price) throw new NonRetryableError('商品字段缺失');
        const crash = process.argv.includes('--crash-once')
            && !db.prepare("SELECT v FROM settings WHERE k='crash_done'").get();
        db.exec('BEGIN IMMEDIATE');
        try {
            db.prepare(`INSERT INTO products VALUES (?,?,?,?)
                ON CONFLICT(upc) DO UPDATE SET title=excluded.title,
                price_display=excluded.price_display,source_url=excluded.source_url`)
                .run(upc, title, price, request.loadedUrl ?? request.url);
            db.prepare('DELETE FROM failures WHERE url=?').run(request.url);
            if (crash) db.prepare("INSERT INTO settings VALUES ('crash_done','1')").run();
            db.exec('COMMIT');
        } catch (error) {
            db.exec('ROLLBACK');
            throw error;
        }
        if (crash) {
            console.log('模拟中断：数据已提交，处理函数尚未返回');
            process.exit(99);
        }
    },
    async failedRequestHandler({ request }, error) {
        db.prepare(`INSERT INTO failures VALUES (?,?,?,?)
            ON CONFLICT(url) DO UPDATE SET label=excluded.label,
            page_no=excluded.page_no,error_type=excluded.error_type`)
            .run(request.url, request.label ?? 'LIST',
                Number(request.userData.page) || null, error.name);
    },
});
 
try {
    await crawler.run();
    const products = db.prepare('SELECT count(*) AS count FROM products').get();
    const failures = db.prepare('SELECT count(*) AS count FROM failures').get();
    const finished = await queue.isFinished();
    console.log({ products, failures, queue_finished: finished });
    if (!finished || Number(failures?.count) > 0) process.exitCode = 1;
} finally {
    db.close();
}

Normal use is node resume.mts. To inspect the discussed crash window with fresh, independent task state:

node resume.mts --crash-once
node resume.mts

The first command deliberately exits with status 99 after committing its first product. The second retains the same environment and storage. These are reader instructions, not reported execution evidence.

Commit before acknowledgement

The request handler returns only after SQLite commit. If interrupted in between, replay updates the same primary key rather than appending another product.

This is not a distributed transaction joining queue and database. It relies on allowed replay, idempotent business storage, and ordering. Other side effects need their own duplication controls.

A crashed request may remain leased or locked temporarily. Inspect pending state and wait according to storage behavior rather than creating another queue or marking work handled manually. See RequestQueue.

Check completion

With unchanged twenty-book pages and no overlap, the intended business set contains forty UPCs. Also inspect second-page samples, fields, failures, and queue completion.

The crash marker commits with the first product, so resumption does not repeat the deliberate stop. Do not delete product state while retaining a completed queue.

A subsequent run with no new work can be correct. Refreshing content needs a new batch or explicit recollection policy.

Repair terminal failures separately

After fixing the cause, submit an explicit repair batch:

REPAIR_RUN=repair-1 node resume.mts

Repair requests use a new, batch-specific scheduling key while retaining page labels and list metadata. Reusing the same repair identity does not continually enqueue another copy.

Successful processing clears the failure row; unsuccessful work remains recorded. Product upserts still prevent duplicate business rows.

Troubleshoot

Restart looks fresh: inspect working directory, storage path, queue name, and purge settings.

Queue is complete but products are empty: inspect database deletion or mismatch. Queue and output must belong to the same task state.

Products duplicate: inspect business identity and actual storage behavior, not just URL deduplication.

Failures do not rerun: explicit terminal repair differs from crash resumption.

No immediate activity after restart: inspect leases, pending requests, and storage instead of inferring completion from quiet logs.

Shared-directory workers behave unexpectedly: restore the single-process assumption before designing distributed coordination.

Reliable recovery permits repeated processing while controlling its business effect through identity, ordering, and explicit repair.