Build an Offline Documentation Mirror with Working Links

A folder of page text is not necessarily an offline mirror. Images may remain online, chapter links may break, and unrelated index pages may overwrite one another.

This tutorial converts an explicit static-document list to Markdown, downloads same-origin images and common attachments, and preserves links between successfully mirrored pages. Other targets remain recorded online links rather than being falsely described as offline.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

DOC_URL_1: the Python tutorial introduction.html address.
DOC_URL_2: the controlflow.html address in the same directory.

Set these in the same Bash/WSL terminal used for subsequent commands:

export DOC_URL_1='<paste-the-actual-doc-url-1>'
export DOC_URL_2='<paste-the-actual-doc-url-2>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install and define scope

Use Python 3.12:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install httpx beautifulsoup4 markdownify

HTTPX downloads, Beautiful Soup rewrites structure, and markdownify converts content.

The example uses two Python tutorial pages. Discover and approve the intended URL range before this stage when mirroring your own documentation.

Preserve identity and anchors

A basename such as index.html is not unique. The example hashes the full defragmented URL, preserving query identity while handling fragments separately.

Markdown-generated heading anchors may differ from original HTML IDs. The example preserves explicit IDs through inline anchors, requiring a reader that supports them.

Run the complete conversion

Create mirror.py:

import hashlib
import html
import json
import os
import sys
from pathlib import Path
from urllib.parse import urldefrag, urljoin, urlsplit
import httpx
from bs4 import BeautifulSoup, NavigableString
from markdownify import markdownify
 
urls = list(dict.fromkeys(urldefrag(url)[0] for url in sys.argv[1:]))
if not urls:
    raise SystemExit("请提供明确的文档 URL")
root = Path("mirror")
root.mkdir()
(root / "pages").mkdir()
(root / "assets").mkdir()
(root / "raw").mkdir()
names = {url: Path("pages") / (hashlib.sha256(url.encode()).hexdigest() + ".md")
         for url in urls}
documents, assets, errors, external = {}, {}, [], []
 
with httpx.Client(timeout=30, follow_redirects=False) as client:
    for url in urls:
        try:
            response = client.get(url)
            response.raise_for_status()
            if "text/html" not in response.headers.get("content-type", ""):
                raise ValueError("文档响应不是 HTML")
            soup = BeautifulSoup(response.text, "html.parser")
            body = soup.select_one("main, article, div.body")
            if body is None:
                raise ValueError("没有识别到正文")
            for item in body.select("nav, script, style"):
                item.decompose()
            documents[url] = body
            (root / "raw" / (names[url].stem + ".html")).write_text(
                response.text, encoding="utf-8"
            )
        except Exception as error:
            errors.append({"url": url, "stage": "page", "error_type": type(error).__name__})
 
    for url, body in documents.items():
        for element in body.select("a[href], img[src]"):
            attribute = "src" if element.name == "img" else "href"
            absolute = urljoin(url, element[attribute])
            target, fragment = urldefrag(absolute)
            parsed = urlsplit(target)
            if parsed.scheme not in {"http", "https"}:
                continue
            local = None
            if target in documents:
                local = names[target]
            else:
                suffix = Path(parsed.path).suffix.lower()
                attachment = element.name == "img" or suffix in {".pdf", ".zip", ".txt"}
                same_origin = (parsed.scheme, parsed.netloc) == (
                    urlsplit(url).scheme, urlsplit(url).netloc
                )
                if attachment and same_origin:
                    try:
                        if target not in assets:
                            resource = client.get(target)
                            resource.raise_for_status()
                            kind = resource.headers.get("content-type", "")
                            if element.name == "img" and not kind.startswith("image/"):
                                raise ValueError("图片地址返回了非图片内容")
                            if suffix == ".pdf" and not resource.content.startswith(b"%PDF-"):
                                raise ValueError("PDF 签名不符")
                            name = hashlib.sha256(target.encode()).hexdigest() + (suffix or ".bin")
                            path = Path("assets") / name
                            (root / path).write_bytes(resource.content)
                            assets[target] = path
                        local = assets[target]
                    except Exception as error:
                        errors.append({"url": target, "stage": "asset",
                                       "error_type": type(error).__name__})
            if local is not None:
                link = os.path.relpath(root / local, (root / names[url]).parent)
                element[attribute] = link.replace(os.sep, "/") + ("#" + fragment if fragment else "")
            else:
                element[attribute] = absolute
                external.append({"page": url, "target": absolute})
 
        anchors = {}
        anchor_nodes = ([body] if body.has_attr("id") else []) + list(body.select("[id]"))
        for number, element in enumerate(anchor_nodes):
            token = f"MIRRORANCHOR{number:06d}TOKEN"
            anchors[token] = '<a id="' + html.escape(element["id"], quote=True) + '"></a>'
            if element is body:
                body.insert(0, NavigableString(token))
            else:
                element.insert_before(NavigableString(token))
        markdown = markdownify(str(body), heading_style="ATX")
        for token, anchor in anchors.items():
            markdown = markdown.replace(token, anchor)
        (root / names[url]).write_text(markdown, encoding="utf-8")
 
manifest = {
    "pages": {url: str(names[url]) for url in documents},
    "assets": {url: str(path) for url, path in assets.items()},
    "errors": errors,
    "external_links": external,
}
(root / "manifest.json").write_text(
    json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8"
)
print({"pages": len(documents), "assets": len(assets), "errors": len(errors)})
raise SystemExit(1 if errors else 0)

Run:

python mirror.py   "$DOC_URL_1" "$DOC_URL_2"

Use a new output directory per batch. Redirects are deliberately not followed: inspect a canonical destination and update the approved inputs rather than silently merging versions.

Understand the two phases

Fetch pages first, then rewrite links using the successful set. Rewriting a failed page prematurely would create a nonexistent local target.

Successfully saved assets and pages use relative paths. Out-of-scope or failed resources remain online and appear in the manifest.

The raw directory retains received HTML text, not compressed transfer bytes or complete response metadata. Archival requirements may need a separate preservation policy.

Verify offline behavior

Use the manifest to open a Markdown page in a compatible reader. Check title, image, a link to another mirrored page, and a same-page chapter link.

Repeat without network connectivity. This distinguishes local resources from a reader silently loading online content.

The image logic covers src, not every srcset, CSS background, or lazy-loading mechanism. Extend discovery deliberately when those matter.

Downloaded attachments remain files; they are not automatically incorporated as searchable Markdown text.

Update without destroying the previous mirror

Create separate batch directories, inspect new output, then switch the reading entry point. Do not delete the old mirror before a possibly incomplete download.

Compare by source identity and content, not only filesystem timestamps. Language and version belong in the chosen scope and directory arrangement.

Troubleshoot

Filename collision: use URL identity rather than titles or basenames.

Existing images do not display: check paths relative to the Markdown file and reader format support.

Chapter links fail: inspect preserved IDs, reader HTML support, and whether the target section is inside the extracted body.

A PDF contains login HTML: verify content and source access rather than trusting the suffix.

A local-looking reference remains online: inspect the successful page mapping and canonical/version differences before changing strings.

Fewer pages than inputs: inspect errors and preserve the previous batch rather than treating failures as removals.

An offline mirror is a scoped relationship between documents and resources, with explicit records of what remains outside it.