LLM Scraper: Consistent Product Data across Page Layouts

When products appear across many layouts, maintaining one CSS rule set per template can become expensive. LLM Scraper can extract a shared schema, but it still needs complete input, unambiguous field definitions, and application-side checks.

This tutorial processes three product HTML materials supplied below. It extracts a name, current price, currency, and availability, then saves accepted and failed inputs separately. Running the example calls the reader's configured model service. Credentials belong in the reader's environment, not in the article or a message to another person.

Choose this approach deliberately

For a stable site with only a few templates, deterministic selectors may be simpler. Model extraction is most useful when layouts differ, the desired fields are clearly defined, and individual outputs can be checked.

The library does not replace URL discovery, pagination, or business validation. This tutorial starts with available page material. Crawlee can discover addresses, but discovery success and extraction correctness remain separate.

Install and configure

Use Node.js 24 in a fresh directory:

mkdir product-extraction
cd product-extraction
npm init -y
npm install llm-scraper@2.0.0 ai@6 @ai-sdk/openai@3 zod@4 playwright
npx playwright install chromium

Keep the lockfile. This example uses LLM Scraper 2 and AI SDK 6's Output.object; older examples that pass a bare Zod schema are not interchangeable. See the library's official instructions.

Create a local .env:

MODEL_BASE_URL=https://api.example.16yun.cn/v1
MODEL_API_KEY=replace-with-your-key
MODEL_NAME=replace-with-an-available-model

Replace the URL, key, and model name with values from your actual service. The demonstrated address is not a working provider. The code uses an OpenAI-compatible Chat Completions provider; compatibility with that route does not automatically establish support for structured output.

Keep the file out of version control. For a different protocol, use the corresponding AI SDK provider rather than changing only the hostname.

Define current price across three layouts

Create extract.mts and start with the inputs and schema:

import { writeFile } from 'node:fs/promises';
import { chromium } from 'playwright';
import { createOpenAI } from '@ai-sdk/openai';
import { Output } from 'ai';
import { z } from 'zod';
import LLMScraper from 'llm-scraper';
 
const inputs = [
    { id: 'card', html: '<main><h1>山谷红茶 500g</h1><p>当前价 CNY 12.90</p><p>有货</p></main>' },
    { id: 'table', html: '<main><h1>山谷红茶 500g</h1><table><tr><th>原价</th><td>CNY 16.00</td></tr><tr><th>当前价</th><td>CNY 12.90</td></tr><tr><th>库存</th><td>有货</td></tr></table></main>' },
    { id: 'description', html: '<main><h1>山谷红茶 500g</h1><dl><dt>现售金额</dt><dd>12.90</dd><dt>币种</dt><dd>CNY</dd><dt>库存状态</dt><dd>有货</dd></dl></main>' },
];
const schema = z.object({
    name: z.string().min(1).describe('Complete product name; do not add a model number absent from the page'),
    price: z.string().regex(/^\d+\.\d{2}$/).nullable()
        .describe('Current publicly offered price as a decimal amount only; exclude original prices, shipping, and member-only prices; return null when missing'),
    currency: z.enum(['CNY', 'USD', 'GBP']).nullable()
        .describe('Currency explicitly provided by the page; do not infer it from language or domain'),
    availability: z.enum(['in_stock', 'out_of_stock', 'unknown'])
        .describe('Use explicit availability wording on the page; return unknown when it cannot be established'),
});

The first material uses paragraphs, the second a table, and the third a description list. The table deliberately includes an original price of 16.00. All three teaching inputs explicitly state a current price of CNY 12.90, giving you a semantic check against confusing the two.

These are embedded teaching materials, not a deployed catalog or observed model results. A nullable price expresses missing evidence. The application below rejects missing prices rather than forcing the model to invent one.

Add the complete call and cleanup path

Append this block to the same file:

const { MODEL_BASE_URL, MODEL_API_KEY, MODEL_NAME } = process.env;
if (!MODEL_BASE_URL || !MODEL_API_KEY || !MODEL_NAME) {
    throw new Error('请配置 MODEL_BASE_URL、MODEL_API_KEY、MODEL_NAME');
}
const provider = createOpenAI({
    baseURL: MODEL_BASE_URL,
    apiKey: MODEL_API_KEY,
});
const scraper = new LLMScraper(provider.chat(MODEL_NAME));
const accepted: unknown[] = [];
const failed: unknown[] = [];
const browser = await chromium.launch({ headless: true });
 
try {
    for (const input of inputs) {
        const page = await browser.newPage();
        try {
            await page.setContent(input.html);
            const { data } = await scraper.run(
                page,
                Output.object({ schema }),
                { format: 'html' },
            );
            const product = schema.parse(data);
            if (product.price === null || product.currency === null) {
                throw new Error('当前价格或币种缺失');
            }
            accepted.push({
                source_id: input.id,
                source_url: null,
                ...product,
            });
        } catch (error) {
            failed.push({
                source_id: input.id,
                error: error instanceof Error ? error.name : 'UnknownError',
                stage: 'load_or_extract_or_validate',
            });
        } finally {
            await page.close();
        }
    }
} finally {
    await browser.close();
}
await writeFile('accepted.json', JSON.stringify(accepted, null, 2));
await writeFile('failed.json', JSON.stringify(failed, null, 2));
console.log({ accepted: accepted.length, failed: failed.length });
if (failed.length) process.exitCode = 1;

Run:

node --env-file=.env extract.mts

Each input receives a separate page, which closes even on failure. The browser closes after the batch. One failed input does not prevent later inputs from being attempted.

Failure entries retain input identity and error category without dumping potentially sensitive provider request details. Inspect redacted diagnostic information in a controlled environment when distinguishing loading, provider, and validation errors.

Because these inputs use setContent, their source URLs are null. Their IDs identify the supplied materials. Do not present about:blank as a real source or ask the model to invent a product URL.

Replace embedded material with real pages

Change each input to an { id, url } object, then replace the loading statement with this local modification:

await page.goto(input.url, { waitUntil: 'domcontentloaded' });
await page.locator('main h1').waitFor();
const sourceUrl = page.url();

Also replace source_url: null in the saved record with source_url: sourceUrl. Adapt the readiness selector to the page. A title appearing before the price is not sufficient evidence that the full product material is ready.

Check redirects against your allowed source scope before sending page content to a model. Treat page text as extraction material, not as instructions to change provider configuration or perform unrelated actions.

Select the material format

The text mode targets readable content and is not universally appropriate for product pages. Inspect whether the selected projection contains the evidence before changing prompts. A more insistent instruction cannot recover material that never reached the model.

Check meaning as well as shape

An expected record has this structure; it is not captured model output:

{
  "source_id": "table",
  "source_url": null,
  "name": "山谷红茶 500g",
  "price": "12.90",
  "currency": "CNY",
  "availability": "in_stock"
}

Verify that every source ID has an outcome, then compare prices with the current-price evidence. A schema can reject incorrect types but cannot by itself establish whether a valid-looking 16.00 is the wrong price.

For real sources, maintain a small human-checked set covering original/current prices, missing prices, uncertain stock, and different currencies. Success on the three teaching materials would not establish reliability on every new template.

Troubleshoot

401 or 404 from the model route: distinguish authentication from an incorrect endpoint. Check the provider's base path and available model name. Page selectors cannot repair provider authentication.

Plain text instead of structured data: verify the model and service support the provider's structured-output mechanism. Use a matching provider or model. Extracting the first brace-delimited text from an arbitrary reply does not establish a reliable record.

Persistent null fields: check page readiness and the selected material projection. If the source genuinely lacks a public price, a rejected record is appropriate.

Original price selected: keep the conflicting-price input and strengthen the field definition. If necessary, restrict material to the product's main region before reevaluating; do not loosen numeric parsing.

Slow batches: separate browser load time from model latency. Reduce irrelevant material and inspect failure categories before increasing concurrency. The sequential example favors diagnosis over throughput.

Partial streaming fields: use them for progress display only. Persist accepted data after the completed result passes the same structural and business checks.

A shared schema can reduce template-specific maintenance, while complete input, precise field meaning, and source checks remain the basis of correctness.