Crawl4AI: Collect a Documentation Site with Traceable Records

A knowledge-base ingestion job needs more than Markdown. Each document needs a source, a retrieval time and a way to distinguish changed content from a failed fetch.

This example uses Crawl4AI 0.9.5 without an LLM. The localhost run on October 29, 2026 collected three valid documents. After changing the fixture, a second pass collected four: one new, one changed and two unchanged.

The example repository is 16yun-cn/www-16yun-cn-v3; the runnable implementation lives in scripts/blog-ai-crawling/. It produces JSONL for a later ingestion step; embeddings, vector storage and retrieval are outside this example.

Define the record before choosing the output format

The output contract separates provenance from extracted content.

Hashing the final representation is useful for this pipeline's change detection. It does not tell you whether the original website changed or your conversion rules changed. Keep the library and cleaning configuration fixed when interpreting a difference.

Use a fixture with deliberate failure cases

The standard-library server provides the following pages.

On revision two, the index gains /docs/new and the start document changes. The script owns the server lifecycle, so each invocation gets an isolated local instance.

Obtain the exact executable example

Use Git and a GitHub CLI configured to access public repositories. The sparse checkout limits working files to the selected directories and repository root files.

gh repo clone 16yun-cn/www-16yun-cn-v3 ai-crawling-examples -- --filter=blob:none --sparse
cd ai-crawling-examples
git checkout --detach bc6828e62c417b1a3459199811fd4e2164137a5f
git sparse-checkout set scripts/blog-ai-crawling

This immutable revision contains the executed scripts and dependency locks. Run the following installation commands from its repository root; the website itself does not need to run.

Excerpts below identify their original file and line range. They omit surrounding imports, initialized variables or lifecycle code, so execute the complete checked-out script rather than concatenating fragments.

Install into an isolated environment

The recorded environment used Python 3.12.13, Crawl4AI 0.9.5 and Playwright 1.63.0. Run the following from this repository's root.

LAB_ENV="$(mktemp -d)"
uv venv --python 3.12 "$LAB_ENV/venv"
uv pip sync --python "$LAB_ENV/venv/bin/python" \
  scripts/blog-ai-crawling/requirements.lock
"$LAB_ENV/venv/bin/python" -m playwright install chromium
CRAWL4_AI_BASE_DIRECTORY="$LAB_ENV/cache" \
  "$LAB_ENV/venv/bin/python" \
  scripts/blog-ai-crawling/crawl_docs.py \
  --output "$LAB_ENV/results"

On Linux, install the browser's required system packages if the host does not already have them. A missing browser executable is an environment failure, not evidence that the target page is incompatible.

The command writes first.jsonl, second.jsonl and crawl-report.json. It exits with an error if the business assertions fail.

Discover links within an explicit boundary

The index pass resolves relative URLs, removes fragments and restricts candidates to the same authority and /docs/ path. This excerpt is part of collect() in crawl_docs.py.

Example source: scripts/blog-ai-crawling/crawl_docs.py:20-24. The excerpt is reformatted for readability.

urls = set()
for link in index.links.get("internal", []):
    url = urldefrag(urljoin(base + "/docs/", link["href"]))[0]
    if (
        urlsplit(url).netloc == urlsplit(base).netloc
        and urlsplit(url).path.startswith("/docs/")
    ):
        urls.add(url)

The request log confirmed that /outside was not requested. Repeated links to the start document did not create duplicate output records.

This is a small site-specific rule. Query-parameter normalization, canonical URLs and destination checks after redirects require their own policy on a real site.

Wait for the data you intend to export

The dynamic fixture exposes a marker when its article is ready. The run configuration waits for that marker rather than assuming the initial HTML contains the final text.

Example source: scripts/blog-ai-crawling/crawl_docs.py:27-33. The excerpt is reformatted for readability.

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    css_selector="main",
    word_count_threshold=0,
    wait_for="css:[data-ready=yes]" if url.endswith("/dynamic") else None,
)
result = await crawler.arun(url, config=config)

The zero word threshold preserves the fixture's intentionally short documents. It is not a general production recommendation.

The assertion checks for DYNAMIC_READY in the exported Markdown. A successful HTTP response alone would not detect reading the page before its article appeared.

Choose a corresponding business condition for your target site. A test-only attribute from this fixture will not exist on arbitrary websites.

Retaining main can remove metadata sources

An early implementation read the title from result.metadata. With css_selector="main", the observed HTML contained the selected region and a wrapper, with no original head; metadata was an empty dictionary.

At unclecode/crawl4ai revision 888299f, crawl4ai/async_crawler_strategy.py:1096-1119 reads matching elements' outer HTML and builds the returned document. That explains the missing head in this configuration.

The fixture keeps the title inside main, so the implementation reads its first h1.

Example source: scripts/blog-ai-crawling/crawl_docs.py:42. This is an equivalent or shortened presentation.

title = fromstring(result.html).xpath("string((//h1)[1])").strip()

The script asserts the expected titles. If your target's only useful title is in head, preserve it through a different extraction arrangement instead of assuming metadata will remain available after narrowing the HTML.

Keep failures out of the valid record set

The local contract accepts HTTP 200 plus nonempty Markdown. It records failed responses separately from empty successful responses.

Example source: scripts/blog-ai-crawling/crawl_docs.py:35-38. The excerpt is reformatted for readability.

if not result.success or result.status_code != 200:
    failures.append({
        "url": url, "reason": "http_failure", "status": result.status_code,
    })
elif not markdown:
    failures.append({
        "url": url, "reason": "empty_body", "status": result.status_code,
    })

A real endpoint may have a different status contract. Define its redirect and error-page behavior before adapting this condition.

An empty page is not replaced with invented text. A missing document is not retried indefinitely. Keeping the address and failure reason makes a later retry decision possible.

Compare two real collection passes

The second pass re-reads the index and bypasses the content cache. It classifies records using the previous URL-to-hash map.

Additional assertions check the code text, the table's Original source text and the resolved absolute link. They establish preservation of these fixture contents, not lossless conversion of every possible document format.

This is full re-fetching followed by incremental classification. It does not implement conditional downloads, deletion synchronization or a persistent scheduler.

Distinguish updates from failed retrievals

The implementation's comparison is keyed by source URL, with Markdown hashes as values. This equivalent expansion corresponds to scripts/blog-ai-crawling/crawl_docs.py:71-75.

old = {row["url"]: row["hash"] for row in first}
changes = {"new": [], "changed": [], "unchanged": []}
for row in second:
    if row["url"] not in old:
        state = "new"
    elif row["hash"] == old[row["url"]]:
        state = "unchanged"
    else:
        state = "changed"
    changes[state].append(urlsplit(row["url"]).path)

Retrieval time is excluded from the hash. Otherwise every successful retrieval would appear to change the document, even when its content remained identical.

A missing second-pass record does not establish deletion. Discovery omissions, fetch failures and actual removal require different business handling; this example deliberately has no deletion classification.

The fixture also selects a random port for each invocation. Since Markdown contains resolved absolute links, hashes can differ across separate fixture instances. Compare the two passes within one invocation instead of hardcoding a hash across runs.

Cache hits need a freshness policy

A separate probe uses CacheMode.ENABLED, updates the server and reads again. Both Markdown outputs remain equal: the second response still contains the older cached content.

Freshness validation was not enabled in this probe. The observation should not be generalized to every cache configuration. The related cache-read path is in crawl4ai/async_webcrawler.py:265-300 at the same revision.

Choose bypassing, a documented freshness policy or server-side update signals according to your collection requirements. Only ENABLED without freshness validation and BYPASS were exercised here.

Treat the export as an ingestion boundary

scripts/blog-ai-crawling/crawl_docs.py:77-80 writes one complete JSON object per line and a separate report. A downstream reader can distinguish valid records from operational failures without interpreting text placeholders inside a document.

The script verifies a known fixture; it is not a general ingestion validator. Before adding a storage system, specify which missing fields or content losses should stop your own pipeline.

Expand after checking provenance and content

Start with a small set of known documents before expanding discovery. Retain failures, keep cleaning rules versioned and review hashes when those rules change.

A proxy changes the access path. It cannot recover content removed by a selector, fix an early read or refresh stale cached Markdown. Commercial proxy behavior, WAF handling and concurrent capacity were not tested in this example.

The next article, Scrapling adaptive extraction, applies the same business-result checks to pages whose structure changes.