Crawlee TypeScript: Paginated Products, Detail Extraction, and JSON Export

A useful product crawler must connect three tasks: discover the intended products, extract the right fields, and explain whether anything was missed. Retrieving a page title is only the beginning.

This tutorial is for developers who know basic TypeScript and are new to Crawlee. You will collect the first two catalog pages from Books to Scrape, follow each book to its detail page, extract its UPC, full title, and tax-inclusive price, and export JSON. The example also separates discovery failures from detail extraction failures.

Books to Scrape is a practice website: its prices and ratings are demonstration data rather than real commercial information. This task uses static HTML. Login, browser interaction, and crash recovery are outside its scope.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• BOOKS_LIST_URL: the full first catalogue-page address from Books to Scrape.

Set these in the same Bash/WSL terminal used for subsequent commands:

export BOOKS_LIST_URL='<paste-the-actual-books-list-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Define the record before choosing selectors

Open the first book's detail page. Its information table distinguishes prices with and without tax. Those amounts happen to match in this example, but they represent different business concepts.

Use the following record:

The combination price_minor: 5177 and currency: "GBP" means £51.77. Shipping is not included. This example does not infer arbitrary currencies from symbols. When adapting it, establish the target site's tax, shipping, and number-format rules first.

This definition determines the implementation: read the table row named Price (incl. tax), rather than collecting every element that looks like a price.

Decide whether a browser is necessary

Use your browser's View Page Source command and search for the title or price label. If the response already contains the fields, downloading and parsing HTML is sufficient.

Crawlee's CheerioCrawler downloads HTML and supplies a parsed $ object to your handler; it does not execute the site's JavaScript. A list that appears only after an interactive action needs a browser-based approach instead. See the CheerioCrawler documentation.

For this catalog, choosing HTTP parsing avoids installing a browser and keeps the example focused on discovery and data.

Install the required environment

Use Node.js 24 and npm. Check both in your terminal:

node --version
npm --version

The first command should report v24.x.x. We use an .mts file, which Node treats as an ES module. Its type stripping supports the erasable annotations used here, without a separate compilation setup. It does not perform type checking or support every TypeScript feature. See the Node.js TypeScript documentation.

Create a fresh directory and install the library:

mkdir product-crawler
cd product-crawler
npm init -y
npm install crawlee@3.18.2

The Crawlee version is pinned to keep the installation aligned with this example. Keep package-lock.json; use npm ci when recreating that installation rather than upgrading dependencies during troubleshooting.

Create main.mts inside this directory. Append the next three TypeScript blocks in order to that same file. No separate router, helper module, or downloadable project is required.

1. Define products and coverage tracking

Start with the imports, record type, and task settings:

import {
    CheerioCrawler,
    Dataset,
    KeyValueStore,
    NonRetryableError,
} from 'crawlee';
 
interface Product {
    upc: string;
    title: string;
    price_minor: number;
    currency: 'GBP';
    source_url: string;
}
 
const START_URL = process.env.BOOKS_LIST_URL;
if (!START_URL) throw new Error('Set BOOKS_LIST_URL before running');
const MAX_LIST_PAGES = 2;
const completedLists = new Set<number>();
const discoveredDetails = new Set<string>();
const completedDetails = new Set<string>();
const products = new Map<string, Product>();
const failures = new Map<string, string>();

The collections answer different questions:

• completedLists records the list page numbers processed successfully.
• discoveredDetails records which detail URLs the lists advertised.
• completedDetails records which advertised URLs produced valid fields.
• products stores one record per UPC, even if different addresses lead to the same book.
• failures explains requests that ultimately failed.

All five collections live in the current process. They suit this small task, but do not provide restart recovery. Restarting rebuilds them, and retaining hundreds of thousands of products in memory would require a different storage strategy.

2. Connect list discovery to detail extraction

Append the crawler definition:

const crawler = new CheerioCrawler({
    maxConcurrency: 2,
    maxRequestsPerMinute: 30,
    maxRequestsPerCrawl: 100,
    maxRequestRetries: 1,
 
    async requestHandler({ request, $, enqueueLinks, log }) {
        const pageUrl = request.loadedUrl ?? request.url;
 
        if (request.label === 'LIST') {
            const listPage = Number(request.userData.listPage);
            const links = $('article.product_pod h3 a');
            const next = $('li.next a');
 
            if (links.length === 0) {
                throw new NonRetryableError(`列表没有商品链接：${pageUrl}`);
            }
            if (listPage < MAX_LIST_PAGES && next.length !== 1) {
                throw new NonRetryableError(`找不到唯一的下一页链接：${pageUrl}`);
            }
 
            links.each((_, element) => {
                const href = $(element).attr('href');
                if (href) {
                    discoveredDetails.add(new URL(href, pageUrl).href);
                }
            });
            await enqueueLinks({
                selector: 'article.product_pod h3 a',
                label: 'DETAIL',
                strategy: 'same-origin',
            });
 
            if (listPage < MAX_LIST_PAGES) {
                await enqueueLinks({
                    selector: 'li.next a',
                    label: 'LIST',
                    userData: { listPage: listPage + 1 },
                    strategy: 'same-origin',
                });
            }
            completedLists.add(listPage);
            log.info(`列表 ${listPage}：发现 ${links.length} 个商品链接`);
            return;
        }
 
        if (request.label !== 'DETAIL') {
            throw new NonRetryableError(`未知页面标签：${request.label}`);
        }
 
        const fields = new Map<string, string>();
        $('table.table-striped tr').each((_, row) => {
            fields.set(
                $(row).find('th').text().trim(),
                $(row).find('td').text().trim(),
            );
        });
        const upc = fields.get('UPC');
        const title = $('.product_main h1').text().trim();
        const priceText = fields.get('Price (incl. tax)') ?? '';
        const amount = /^£(\d+)\.(\d{2})$/.exec(priceText);
 
        if (!upc || !title || !amount) {
            throw new NonRetryableError(
                `详情字段不符合预期：${JSON.stringify({ upc, title, priceText })}`,
            );
        }
        products.set(upc, {
            upc,
            title,
            price_minor: Number(amount[1]) * 100 + Number(amount[2]),
            currency: 'GBP',
            source_url: pageUrl,
        });
        completedDetails.add(request.url);
    },
 
    async failedRequestHandler({ request, log }, error) {
        failures.set(request.url, error.message);
        log.error(`最终失败 ${request.url}：${error.message}`);
    },
});

Give each page one responsibility

The initial request has the LIST label. Product links receive DETAIL. The handler checks the label before applying page-specific rules, so a list never becomes a product record.

The selector article.product_pod h3 a identifies product title links. The selector li.next a identifies the next-page control. Calling enqueueLinks() across the entire document would also discover unrelated navigation, including category pages.

An href can be relative. Concatenating it with the site origin can produce the wrong address. The example uses new URL(href, pageUrl) to build the discovery set against the current page; enqueueLinks also handles relative links when constructing requests. See the relative-link example.

The same-origin strategy restricts link origins, but does not define business scope. Here, the two selectors provide that narrower scope. On another site, inspect whether they also match recommendations, advertisements, or search links.

Separate page coverage from the request budget

MAX_LIST_PAGES = 2 defines the task: process details on the second page, but do not enqueue page three.

Do not implement this by setting maxRequestsPerCrawl: 2. Detail pages also consume that budget. With twenty different books per list, the example processes two lists and forty details: forty-two pages.

A budget of one hundred leaves room for this small task; it does not promise one hundred products. Nor is this setting an exact count of network attempts: concurrency and retries need separate interpretation. See the crawler options.

Concurrency two and thirty request starts per minute are conservative teaching values, not general performance recommendations. One retry allows ordinary transient request failures another attempt. Field errors use NonRetryableError, since repeatedly downloading the same unexpected structure will not repair a selector.

Make parsing errors visible

The information table is read by field name, not by row number. An inserted row should not silently turn a tax amount into a product price.

The price expression accepts this site's £51.77 format. It separately converts pounds and pence to the integer 5177. Removing every non-digit character would incorrectly accept mixed text such as two different prices.

An amount such as £1,234.56, another currency, or a missing price is rejected. Adapt the parsing contract to the actual target instead of weakening it indiscriminately. Do not replace a missing price with zero: zero is a business value, while missing data means extraction did not obtain a value.

3. Run and export products plus coverage information

Append the final block:

await crawler.run([
    { url: START_URL, label: 'LIST', userData: { listPage: 1 } },
]);
 
const missingDetails = [...discoveredDetails]
    .filter((url) => !completedDetails.has(url));
const summary = {
    requested_list_pages: MAX_LIST_PAGES,
    completed_list_pages: [...completedLists].sort((a, b) => a - b),
    discovered_detail_urls: discoveredDetails.size,
    completed_detail_urls: completedDetails.size,
    unique_products: products.size,
    missing_detail_urls: missingDetails,
    failed_requests: [...failures].map(([url, message]) => ({ url, message })),
};
 
if (products.size > 0) {
    await Dataset.pushData([...products.values()]);
}
await KeyValueStore.setValue('SUMMARY', summary);
await Dataset.exportToJSON('PRODUCTS');
console.log(JSON.stringify(summary, null, 2));
 
if (
    completedLists.size !== MAX_LIST_PAGES
    || missingDetails.length > 0
    || failures.size > 0
) {
    process.exitCode = 1;
}

After crawler.run() finishes, the code writes the UPC-deduplicated records to the default Dataset. Delaying that write makes the one-product-one-record behavior clear, but also means an interrupted process loses products that have not yet been written. Long-running work needs persistent, idempotent writes.

A Dataset stores structured records. Its JSON export is saved in the Key-value store, rather than beside the individual dataset records. See result storage and the Dataset export API.

Save the file and run it from the same directory:

node main.mts

With the default local configuration, inspect:

• storage/key_value_stores/default/PRODUCTS.json: the product array.
• storage/key_value_stores/default/SUMMARY.json: list coverage, missing detail URLs, and final failures.

Individual Dataset records are under storage/datasets/default/. They represent the same products as the exported array; merging both outputs would duplicate the data.

This tutorial assumes a fresh directory, default configuration, and a new collection run. If you change storage or cleanup settings, establish which run produced the files before comparing results. Persistent request storage alone does not make these in-memory collections recoverable.

Interpret the results instead of counting successful requests

The following is an expected summary structure derived from the task, not a captured execution log. It assumes twenty books per page, no overlap between the two lists, and successful extraction of every detail:

{
  "requested_list_pages": 2,
  "completed_list_pages": [1, 2],
  "discovered_detail_urls": 40,
  "completed_detail_urls": 40,
  "unique_products": 40,
  "missing_detail_urls": [],
  "failed_requests": []
}

Read the values together. Missing list pages indicate a problem before complete detail discovery. Forty discovered URLs but thirty-nine completed URLs point to an address in the missing list. Forty completed addresses but thirty-nine UPCs could mean duplicate addresses for one product, or an incorrect UPC extraction rule.

Matching counts still do not establish correct fields. In PRODUCTS.json, find UPC a897fe39b1053632 and compare it with the source page. This is the corresponding example record; concurrent processing can change array order:

{
  "upc": "a897fe39b1053632",
  "title": "A Light in the Attic",
  "price_minor": 5177,
  "currency": "GBP",
  "source_url": "https://example.16yun.cn/catalogue/a-light-in-the-attic_1000/index.html"
}

Choose a book from the second list page, such as In Her Wake, and verify its presence too. Check the actual source links as well as the counts: forty rows could still contain the wrong forty products.

UPC identifies the entity, the source URL supports field checks, and the summary describes processing coverage. Keeping these concerns separate makes later duplicates and omissions diagnosable.

Troubleshoot at the failing stage

Module or TypeScript errors at startup

Check node --version and the main.mts extension. An older runtime, a mistakenly named JavaScript file, or CommonJS changes can prevent this example from loading.

An .mts file already selects ES modules, so adding "type": "module" is unnecessary here. After correcting the environment, rerun node main.mts. If the error instead says Crawlee cannot be resolved, check that you are running inside the directory containing the installed node_modules, rather than changing selectors.

Only the first page appears

Inspect completed_list_pages. If it contains only [1], look for the missing-next-link error and for a failed request to page two.

In page one's HTML, check that li.next a matches exactly one link whose resolved address is page two. When adapting to a site with a[rel="next"], update both the next check and the next-page enqueueLinks call. Changing only one leaves contradictory rules.

Check that the list limit remains two and the total request budget has not been reduced accidentally. Rerun: both list numbers should appear. If detail discovery remains incomplete, inspect page two's product links next.

Lists succeed but fields are empty

Open a failing detail URL and search its response source for the title and price label. Distinguish a selector mismatch from a response that is not the intended detail page.

If the fields exist, check the title selector, table selector, and exact row label. If the label changed, update the corresponding fields.get(...) key instead of collecting amounts from the entire document. Login or other unexpected responses require fixing access conditions. Data available only after JavaScript rendering requires a browser crawler and an appropriate readiness condition.

After fixing the cause, the address should move into the completed set and the missing list should shrink. Recheck the price meaning as well; absence of exceptions is insufficient.

More detail URLs complete than products export

Compare UPCs. Category routes or tracking parameters can yield multiple URLs for the same book. Request deduplication controls address scheduling; UPC deduplication controls product records. See the request storage guide.

This example retains the last completed record for a UPC. That can explain a genuine duplicate address, but not two different products accidentally assigned the same UPC. Fix the field rule in that case. For variant-rich stores, first decide whether one output row represents a product or a SKU.

The process finished but some data is missing

A drained or stopped task does not mean every detail succeeded. The failure handler records terminal failures, while the final checks preserve available products and set a nonzero exit status for incomplete work.

Use the recorded URLs to repair access or parsing, then rerun this small task. Removing the checks or increasing retries would conceal the problem rather than produce the missing record.

Adapt the task in a controlled order

Choose two pages from one category. Change the entry URL, verify the next-page path, update both list selectors, and then update the detail fields and price parser. Preserve the relationship between discovered addresses, completed addresses, and product identities.

Set the list limit to one for a single-page task. For larger runs, estimate lists plus details before changing the request budget. Do not remove all scope restrictions and assume the resulting traversal is intentional.

Dynamic rendering changes the crawler choice. Restart recovery changes state management. Larger outputs change the storage approach. The field definitions, bounded discovery, and coverage checks remain useful across those changes.