Scrapling Adaptive Extraction: Test Completeness After a Page Redesign

Adaptive relocation can find candidate elements after a redesign. It does not establish that you recovered the entire list or selected the right business object.

This experiment uses Scrapling 0.5 and known product records. In the localhost run on October 29, 2026, the original page produced all three products. Renaming the cards recovered only p1. Adding wrappers produced three nodes whose IDs were all missing.

No LLM was used. These results describe the tested variants and version, not every website or adaptive threshold.

Establish a result independent of the selector

The target set contains three products. Prices remain strings to preserve their displayed representation.

Success requires the correct identities and fields, with no omissions or extra records. A node count alone cannot establish that.

The example repository is 16yun-cn/www-16yun-cn-v3; the runnable code is scripts/blog-ai-crawling/scrapling_adaptive.py. The neighboring serve.py implements the controlled pages.

Change the DOM in defined ways

Each variant receives a fresh baseline storage database. A previous variant's relocation result cannot update the next one's anchor.

The ordering case also renames classes. It does not isolate reordering alone. The wrapper case is likewise a combined change; retain these conditions when describing the outcome.

Obtain the exact executable example

Use Git and a GitHub CLI configured to access public repositories. The sparse checkout limits working files to the selected directories and repository root files.

gh repo clone 16yun-cn/www-16yun-cn-v3 ai-crawling-examples -- --filter=blob:none --sparse
cd ai-crawling-examples
git checkout --detach bc6828e62c417b1a3459199811fd4e2164137a5f
git sparse-checkout set scripts/blog-ai-crawling

This immutable revision contains the executed scripts and dependency locks. Run the following installation commands from its repository root; the website itself does not need to run.

Excerpts below identify their original file and line range. They omit surrounding imports, initialized variables or lifecycle code, so execute the complete checked-out script rather than concatenating fragments.

Run with pinned dependencies

The recorded runtime was Python 3.12.13 with Scrapling 0.5. Run from the repository root.

LAB_ENV="$(mktemp -d)"
uv venv --python 3.12 "$LAB_ENV/venv"
uv pip sync --python "$LAB_ENV/venv/bin/python" \
  scripts/blog-ai-crawling/requirements.lock
"$LAB_ENV/venv/bin/python" \
  scripts/blog-ai-crawling/scrapling_adaptive.py \
  --output "$LAB_ENV/results"

This path uses HTTP fetching and parsing. It does not need a browser installation or model configuration. The lock file is shared with the companion document example, while the environment remains separate from the website.

The output scrapling-report.json contains both query results, completeness flags and missing or unexpected IDs for every variant.

Save the original anchor explicitly

The baseline call enables adaptive storage, saves the selector's match and verifies the original business records.

Example source: scripts/blog-ai-crawling/scrapling_adaptive.py:25-28. The excerpt is reformatted for readability.

original = Fetcher.get(base + "/products?variant=original")
saved = Selector(
    original.html_content,
    url=base + "/products",
    adaptive=True,
    storage_args=storage,
)
baseline = records(
    saved.css(".product-card", auto_save=True, identifier="cards")
)
assert baseline == expected

The URL and identifier stay fixed for a baseline/variant pair. adaptive=True at initialization enables the storage mechanism; the query's adaptive flag requests relocation when applicable.

Without previously saved information, turning on the query flag cannot reconstruct an unspecified target.

Compare the same selector on the changed page

The changed page is fetched independently. Both queries keep the original class selector.

Example source: scripts/blog-ai-crawling/scrapling_adaptive.py:30-32. The excerpt is reformatted for readability.

changed = Selector(
    response.html_content,
    url=base + "/products",
    adaptive=True,
    storage_args=storage,
)
fixed = records(changed.css(".product-card"))
adaptive = records(
    changed.css(".product-card", adaptive=True, identifier="cards")
)

The record extractor reads data-id from the returned node itself, plus its descendant name and price. A wrong node level therefore remains visible as missing identity instead of receiving an invented replacement ID.

Inspect the business outcome

The recorded results were:

In the wrapper variant, descendant queries still found names and prices. The ID belonged to the inner article rather than the returned outer node.

That is why counting three results would have produced a false success. The output looked populated but violated the identity contract.

Connect the result to the source

At D4Vinci/Scrapling revision 0336400, scrapling/parser.py:662-680 first tries the original selector. If it matches, it returns those elements. With auto-save enabled, it saves elements[0].

Relocation is considered when the original selector has no matches and adaptive querying is enabled. A list selector's saved first element is not a complete specification for restoring every sibling record.

This also explains the distractor case. The old selector still matches ad1, so there is no relocation attempt. The library cannot infer that the matching recommendation is outside the intended product set.

Issue #259 describes a related first-item-only list recovery report on version 0.4.7. Our observation is from 0.5 and these local pages; the historical report is context, not a substitute for this run.

Validate identities and fields together

The example normalizes ordering before comparing full records.

Example source: scripts/blog-ai-crawling/scrapling_adaptive.py:34-41. This is an equivalent or shortened presentation.

canonical = lambda rows: sorted(rows, key=lambda row: row["id"] or "")
complete_and_correct = canonical(adaptive) == canonical(expected)
missing_ids = sorted(
    {row["id"] for row in expected}
    - {row["id"] for row in adaptive}
)

The fixture supplies a complete answer. On a production target, combine available evidence: required fields, unique identities, allowed values and a known total or historical range where justified.

Schema validity alone would not reject the recommendation's 999.00 price. It is a valid string attached to the wrong object.

Generating a new ID when the original is missing would hide a locator error. Preserve the failure and inspect the selected node instead.

Apply an explicit selector repair where justified

The renamed fixture retains a stable container and business attribute. A direct repair works for that variant.

Example source: scripts/blog-ai-crawling/scrapling_adaptive.py:45-47. The excerpt is reformatted for readability.

renamed = Fetcher.get(base + "/products?variant=renamed")
stable = records(renamed.css("main#products > article[data-id]"))
assert stable == expected

It returns all three expected records with correct names and prices. This is an explicit update for the known page structure, not evidence that adaptive recovery fixed the list.

The direct-child relation changes when wrappers are added. Do not advertise this repaired selector as validated across all variants.

Classify recovery failures before changing thresholds

The fixture distinguishes three questions: did a query return nodes, do those nodes represent the requested identities, and do they contain the correct fields?

These are the observed failure classes. They are not interchangeable similarity-score problems, and the experiment did not test a sweep of thresholds.

The report retains incomplete outputs rather than converting them into a passing result. In the renamed case, no unexpected ID is present, but p2 and p3 are missing; both identity coverage and field accuracy matter.

Keep anchor state separate from accepted data

Each variant saves an original baseline in a different temporary database. The queried URL and identifier stay stable within that pair, while the returned record set is evaluated separately.

That arrangement prevents a previous relocation from becoming the next variant's baseline. It also avoids mistaking a stored anchor for a complete representation of accepted business records.

In a recurring collector, decide when an anchor is allowed to change. Automatically learning from an incorrect match can make later failures harder to attribute. This example does not implement an automatic anchor-update policy.

The explicit selector repair is validated for renamed only. Extending it to a wrapper change requires a new claim and corresponding check; a prior pass does not establish that additional behavior.

Use relocation with an acceptance condition

Start with stable business attributes when the page offers them. Treat relocated elements as candidates, then verify fields and completeness before exporting them. Keep enough page evidence to diagnose a failure.

This experiment does not measure Scrapling's browser fetching, Spider pause/resume, proxy rotation, anti-bot behavior or large-scale throughput. Those capabilities need their own task-specific checks.

For pages requiring interactions, continue with the Stagehand configuration guide. It separates a working deterministic baseline from model-backed configuration that was not executed for this series.