Firecrawl: Discover, Crawl, and Export a Help Center
Use Firecrawl v2 to discover links, submit a bounded crawl, retain its job ID, collect every result page, and export source-linked Markdown.
A hosted scraping API can remove browser process management from your application. It does not remove the need to retain job identity, wait for execution, retrieve all result pages, and inspect missing content.
This tutorial uses Firecrawl's v2 HTTP API directly. It discovers a documentation directory, collects up to ten pages within a specified path, and exports Markdown plus provenance. Running it requires your own account and key and may consume paid service capacity. No account allowance, price, coverage, or execution result is promised here.
Prepare the input addresses
Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.
FIRECRAWL_API_BASE: the v2 API base from your account integration settings, including its v2 path.TARGET_URL: the Python tutorial entry address with path /3/tutorial/.
Set these in the same Bash/WSL terminal used for subsequent commands:
export FIRECRAWL_API_BASE='<paste-the-actual-firecrawl-api-base>'
export TARGET_URL='<paste-the-actual-target-url>'Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.
Choose Map, Scrape, or Crawl by task
Use Scrape for one known page, Map to inspect candidate links, and Crawl for service-side discovery and collection. Map results do not contain the complete collected bodies.
The example target is TARGET_URL, restricted to that path and ten pages. This is a bounded collection task, not a complete mirror of Python documentation. See the Map and Crawl references.
Configure the task
Use Python 3.12 or later. Only the standard library is required:
export FIRECRAWL_API_KEY='replace-with-your-key'
# TARGET_URL and FIRECRAWL_API_BASE are configured above
export TARGET_PATH_PATTERN='^/3/tutorial/.*'For your own help center, change both the entry URL and path expression. The entry must match the inclusion rule too; otherwise the job may return no pages.
Keep submission separate from collection
Create firecrawl_help.py:
import argparse
import json
import os
import time
from pathlib import Path
from urllib.parse import quote, urljoin, urlsplit
from urllib.request import Request, urlopen
BASE = os.environ["FIRECRAWL_API_BASE"].rstrip("/") + "/"
API_ORIGIN = urlsplit(BASE)
if API_ORIGIN.scheme != "https" or not API_ORIGIN.netloc:
raise ValueError("FIRECRAWL_API_BASE must be an HTTPS API base")
TOKEN = os.environ["FIRECRAWL_API_KEY"]
OUT = Path("firecrawl-output")
OUT.mkdir(exist_ok=True)
def save(name, data):
(OUT / name).write_text(
json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8"
)
def api(path, payload=None):
url = urljoin(BASE, path)
if urlsplit(url).scheme != "https" or urlsplit(url).netloc != API_ORIGIN.netloc:
raise ValueError("结果分页地址不属于 Firecrawl API")
request = Request(
url,
data=None if payload is None else json.dumps(payload).encode(),
headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
method="GET" if payload is None else "POST",
)
with urlopen(request, timeout=30) as response:
result = json.load(response)
if result.get("success") is False:
raise RuntimeError("API 返回失败，请检查请求与账户状态")
return result
parser = argparse.ArgumentParser()
parser.add_argument("action", choices=["start", "collect"])
args = parser.parse_args()
if args.action == "start":
if (OUT / "job.json").exists():
raise SystemExit("已有 job.json；请先 collect 或为新任务使用新目录")
root = os.environ["TARGET_URL"]
pattern = os.environ["TARGET_PATH_PATTERN"]
save("map.json", api("map", {"url": root}))
created = api("crawl", {
"url": root, "limit": 10, "includePaths": [pattern],
"scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
})
if not created.get("id"):
raise RuntimeError("创建响应缺少任务 ID")
save("job.json", {"id": created["id"], "target": root, "path_pattern": pattern})
print("任务 ID 已保存；执行 collect 查询")
else:
job = json.loads((OUT / "job.json").read_text(encoding="utf-8"))
path = "crawl/" + quote(job["id"], safe="")
for _ in range(30):
status = api(path)
save("status.json", status)
if status["status"] in {"completed", "failed", "cancelled"}:
break
time.sleep(5)
else:
raise SystemExit("本轮等待结束；保留任务 ID，稍后再次 collect")
if status["status"] != "completed":
raise SystemExit("任务未正常完成，请检查 status.json")
documents = list(status.get("data", []))
next_url = status.get("next")
visited = set()
while next_url:
if next_url in visited:
raise RuntimeError("结果分页地址重复，停止以避免循环")
visited.add(next_url)
part = api(next_url)
documents.extend(part.get("data", []))
next_url = part.get("next")
save("documents.json", documents)
index, rejected = [], []
for number, document in enumerate(documents):
meta = document.get("metadata") or {}
url = meta.get("sourceURL") or meta.get("url")
markdown = document.get("markdown")
if not url or not isinstance(markdown, str) or not markdown.strip():
rejected.append({"index": number, "metadata": meta})
continue
filename = f"page-{number:04d}.md"
(OUT / filename).write_text(markdown, encoding="utf-8")
index.append({"source_url": url, "title": meta.get("title"), "file": filename})
save("index.json", index)
save("rejected.json", rejected)
print({"returned": len(documents), "exported": len(index), "rejected": len(rejected)})Submit once, then collect:
python firecrawl_help.py start
python firecrawl_help.py collectThe saved job file allows collection to resume without creating another potentially billable job. If the waiting loop expires, run collect again against the same ID.
The loop count and sleep interval are not an exact wall-clock deadline: each HTTP operation also takes time. The example intentionally avoids unconditional retries of creation requests, whose previous outcome may be ambiguous.
Interpret result pagination correctly
Do not append every polling response to one output: in-progress responses may contain overlapping partial results.
This example waits for a terminal state, then follows next from the completed result. That field pages the API's returned data, not the target website. Large results may require it even after execution ends. See the status endpoint.
Because result requests carry a token, the helper checks their API origin before attaching credentials. Extend error handling according to the service contract rather than forwarding authorization to arbitrary returned addresses.
Inspect the export
The output directory contains task identity, raw status, returned documents, individual Markdown files, and an index mapping files to sources. This is an illustrative index entry, not captured API output:
[
{
"source_url": "https://example.16yun.cn/3/tutorial/index.html",
"title": "页面实际标题",
"file": "page-0000.md"
}
]Check source paths, corresponding body content, and whether the ten-page budget truncated the intended collection. Reaching the budget does not establish that only ten pages exist.
Compare path-matching Map candidates with the collected sources as a diagnostic aid. Map and Crawl may discover different sets, so Map count is not automatically an expected total.
Likewise, a completed status or matching completed/total counters does not by itself establish that every discovered page succeeded. Inspect Crawl Errors and account for attempted sources that were not returned. The script exports data; it does not certify a complete knowledge base.
Troubleshoot by stage
401 or 402: inspect credentials or account usage conditions respectively. Neither is a page-selector problem.
A job exists but no bodies export: confirm the saved ID, terminal state, path inclusion, status metadata, and rejected entries. A created file alone proves little.
The limit was reached: decide whether the intended scope needs expansion, then submit a new task with a deliberate budget. Editing the old job file cannot change work already submitted.
The job remains scraping: preserve its ID and continue status inspection later. Repeated submission creates separate jobs rather than extending the previous wait.
Many duplicate pages: inspect language, version, and query-string variants. Deduplicate sources carefully; similar text does not automatically mean duplicate document identity.
Empty Markdown despite an apparently successful request: inspect metadata, page identity, access conditions, and format selection. Login pages, challenges, unreachable resources, and conversion failures need different fixes.
Hosted collection changes who manages browsers. Job identity, coverage, and interpretation of the returned data still belong to the application.
Need an enterprise proxy plan?
We can tailor architecture to your target domains, concurrency, and reliability goals.