Crawler Regression Tests: Detect Silent Extraction Failures

Silent failures are often worse than exceptions: a crawler finishes while collecting original rather than current prices or omitting a list item.

This tutorial builds regression checks around explicit product meaning and complete teaching HTML. It describes reader-run tests without creating or executing a separate testing project.

Install and choose the right layer

Use Python 3.12:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install pytest beautifulsoup4

Pure parsing belongs in offline HTML tests. Login, asynchronous loading, and pagination triggering need separate browser-level checks. See pytest assertions.

The expected product is p1, named red tea, with a current price of 12.90 CNY or 1290 minor units. A visible original price of 16.00 must not replace it.

Expose a wrong implementation

Create test_extraction.py:

from decimal import Decimal, InvalidOperation
from bs4 import BeautifulSoup
import pytest
 
V1 = """<article data-product-id="p1" data-currency="CNY">
<h1 data-testid="product-title">红茶</h1>
<span class="price" data-testid="current-price">12.90</span></article>"""
V2 = """<section data-product-id="p1" data-currency="CNY">
<h2 data-testid="product-title">红茶</h2>
<del class="price">16.00</del>
<strong data-testid="current-price">12.90</strong></section>"""
 
def parse_product(html):
    root = BeautifulSoup(html, "html.parser").select_one("[data-product-id]")
    if root is None:
        raise ValueError("不是商品页")
    return {
        "id": root["data-product-id"],
        "name": root.select_one('[data-testid="product-title"]').get_text(strip=True),
        "price_minor": int(Decimal(root.select_one(".price").get_text(strip=True)) * 100),
        "currency": root["data-currency"],
    }
 
def parse_catalog(html, expected_ids):
    soup = BeautifulSoup(html, "html.parser")
    ids = [node.get("data-product-id") for node in soup.select('[data-testid="product-card"]')]
    if len(ids) != len(set(ids)) or set(ids) != set(expected_ids):
        raise ValueError("目录覆盖与预期不一致")
    return ids
 
@pytest.mark.parametrize("html", [V1, V2])
def test_product_layouts_keep_business_meaning(html):
    assert parse_product(html) == {
        "id": "p1", "name": "红茶", "price_minor": 1290, "currency": "CNY"
    }
 
def test_missing_current_price_does_not_use_original():
    changed = V2.replace('data-testid="current-price"', 'data-testid="retired-price"')
    with pytest.raises(ValueError):
        parse_product(changed)
 
def test_nonnumeric_current_price_is_rejected():
    changed = V1.replace(
        '<span class="price" data-testid="current-price">12.90</span>',
        '<span class="price" data-testid="current-price">pending</span>',
    )
    with pytest.raises(ValueError):
        parse_product(changed)
 
def test_access_page_is_not_a_product():
    with pytest.raises(ValueError):
        parse_product("<h1>请完成访问验证</h1>")
 
def test_catalog_requires_both_expected_products():
    html = '<div data-testid="product-card" data-product-id="p1"></div>'
    with pytest.raises(ValueError):
        parse_catalog(html, expected_ids={"p1", "p2"})
 
def test_catalog_accepts_complete_distinct_products():
    html = """<div data-testid="product-card" data-product-id="p1"></div>
    <div data-testid="product-card" data-product-id="p2"></div>"""
    assert parse_catalog(html, {"p1", "p2"}) == ["p1", "p2"]

Run:

python -m pytest -q

The initial implementation is expected to fail the redesigned price case and the missing-current-price case: it reads the original price. The nonnumeric-price case also fails because its exception type is not yet normalized. These are code-derived expectations, not reported executions.

A returned object without an exception is insufficient. The assertions compare actual business values.

Repair meaning, not merely class names

Replace only parse_product:

def parse_product(html):
    root = BeautifulSoup(html, "html.parser").select_one("[data-product-id]")
    if root is None:
        raise ValueError("不是商品页")
    title = root.select_one('[data-testid="product-title"]')
    price = root.select_one('[data-testid="current-price"]')
    if title is None or not title.get_text(strip=True) or price is None:
        raise ValueError("缺少名称或当前价")
    try:
        amount = Decimal(price.get_text(strip=True))
    except InvalidOperation as error:
        raise ValueError("invalid_current_price_format") from error
    if not amount.is_finite() or amount < 0 or amount.as_tuple().exponent < -2:
        raise ValueError("价格不符合两位小数合同")
    if root.get("data-currency") != "CNY":
        raise ValueError("币种不符合本任务")
    return {
        "id": root["data-product-id"],
        "name": title.get_text(strip=True),
        "price_minor": int(amount * 100),
        "currency": root["data-currency"],
    }

Rerun the same command. The corrected implementation should satisfy both layouts, missing-current-price rejection, page identity, and list coverage.

Real sites may not provide these test attributes. Map the actual current-price region while retaining the semantic regression case.

Choose independent examples

Include normal material, ambiguous business structure, and explicit rejection inputs. For lists, compare expected identities rather than counts alone.

Expected values must come from reviewed source meaning, not the same parser under test. Otherwise the check reproduces its own mistake.

Preserve the structure that caused the defect when minimizing samples. Removing the original-price element would erase the regression trigger. Remove unrelated sensitive data without removing the behavior.

Add necessary browser checks

Static parsing cannot establish readiness. A browser check should open a controlled entry, perform filtering or pagination, wait for updated results, and assert business identities.

Avoid fixed counts from constantly changing public catalogs. Distinguish environment and source changes from parser regressions.

Screenshots support diagnosis but do not replace field and identity assertions.

Maintain the checks

Add the same pytest command to an existing CI pipeline for offline parsing checks. Review changes to both parser and fixtures against business intent.

Tests pass but production loses data: inspect missing discovery and readiness coverage, and add the actual failing material or flow.

Expected values are updated immediately: first establish whether business meaning changed or an extraction defect is being accepted.

Any exception satisfies a test: require an appropriate error type so unrelated import or type errors do not masquerade as business rejection.

Tests are slow: keep parser cases offline and reserve browsers for necessary interactions.

Many passing cases still provide little confidence: ask whether the targeted wrong implementation would fail. Counts alone cannot answer that.

Regression tests preserve an independent business reference against which implementation changes can be assessed.