webclaw: Deploy an Internal Web Crawling Service

When several internal tools need page material, a shared extraction service can avoid duplicating browser and parser setup in every caller. webclaw provides both CLI and MCP access.

This tutorial extracts book titles and prices locally, then exposes the capability through an authenticated, loopback HTTP MCP server. That address carries MCP; it is not an arbitrary REST endpoint accepting any JSON URL request. The entry points follow the official webclaw documentation.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• WEBCLAW_REPO_URL: the clone address of the official 0xMassi/webclaw repository.
• BOOKS_LIST_URL: the Books to Scrape product-list address.

Set these in the same Bash/WSL terminal used for subsequent commands:

export WEBCLAW_REPO_URL='<paste-the-actual-webclaw-repo-url>'
export BOOKS_LIST_URL='<paste-the-actual-books-list-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install the CLI and server separately

The Linux path requires stable Rust/Cargo and native build prerequisites. For Debian/Ubuntu:

sudo apt install -y pkg-config libssl-dev cmake clang git build-essential
cargo install --git "$WEBCLAW_REPO_URL" --locked webclaw-cli
cargo install --git "$WEBCLAW_REPO_URL" --locked webclaw-mcp
webclaw --version
webclaw-mcp --help

See the Rust installation instructions if needed. These Cargo commands build the repository's default branch. Preserve the actual version and build origin for a deployment rather than combining old release instructions with newer features.

The CLI package installs the executable webclaw; the MCP package installs a separate executable. A successful CLI command does not establish that the MCP server was installed or started.

Establish a deterministic extraction first

Each practice-site card is article.product_pod. Its title attribute preserves the complete name, while .price_color contains the displayed amount:

webclaw "$BOOKS_LIST_URL"   --select 'article.product_pod'   --field 'title=h3 a::attr(title)'   --field 'price=.price_color' > products.json

The outer selector defines repeated objects. Field selectors run inside each object. Selecting the whole body instead would change the record scope.

Expect an array shaped like this illustrative record:

[
  {"title": "Complete product title", "price": "£51.77"}
]

The price remains source text. It is not automatically a normalized numeric amount with a tax and currency contract.

Create check_products.py:

import json
 
with open("products.json", encoding="utf-8") as stream:
    rows = json.load(stream)
if not isinstance(rows, list) or not rows:
    raise ValueError("提取结果不是非空数组")
for index, row in enumerate(rows):
    if not isinstance(row, dict):
        raise ValueError(f"第 {index} 条不是对象")
    if any(not isinstance(row.get(key), str) or not row[key].strip()
           for key in ("title", "price")):
        raise ValueError(f"第 {index} 条缺少标题或价格")
print({"records": len(rows), "fields": ["title", "price"]})

Run python check_products.py, then compare representative records with the page. A structural pass does not prove semantic correctness or second-page coverage.

Start HTTP MCP with authentication

Generate a local token and retain it in your own secret configuration:

export WEBCLAW_MCP_TOKEN="$(openssl rand -hex 16)"
webclaw-mcp --http 127.0.0.1:8000

The endpoint is http://localhost:8000/mcp. Do not disable authentication merely to fix a client configuration error.

An HTTP-capable MCP client may use this configuration shape:

{
  "mcpServers": {
    "webclaw": {
      "type": "http",
      "url": "http://localhost:8000/mcp",
      "headers": {
        "Authorization": "Bearer replace-with-the-server-token"
      }
    }
  }
}

Replace the placeholder with the server's token. Exact transport configuration varies by client; map the endpoint and Authorization header into its supported settings.

Complete one bounded client task

Inspect the discovered tool list first, then request a single-page task:

Read the page at the configured BOOKS_LIST_URL.
Process only product cards on the current list and extract titles and displayed prices.
Return the source URL and a records array. Do not follow category links or expand to the whole site.

This is a natural-language task description, not a fixed API request or guarantee of tool selection. For deterministic use, inspect the actual tool schema and supply the selectors already checked through the CLI.

Basic fetching and selector extraction do not require a model. Semantic extraction and summarization have separate configuration requirements. Avoid adding those dependencies for a two-field deterministic task.

A shared process and token do not establish tenant isolation. Cross-machine access requires a deliberate network, TLS, and authorization arrangement, rather than merely changing the listener to a public interface.

Manage scope and process lifetime

Before moving to crawl, define path, depth, and page budgets against the installed version's supported options. Start with a small directory. The presence of a crawl tool is not evidence of complete coverage.

Keep protocol output separate from diagnostics. In stdio mode, stdout carries MCP traffic; arbitrary log output can break it. For this HTTP example, stop the server through Ctrl+C or its actual process manager.

Diagnostic records should retain target, tool, redacted parameters, and outcome without storing authorization headers. Page text is external material, not permission to change server configuration or broaden the task.

Troubleshoot

CLI works but the client fails: verify the server executable, port, /mcp path, and HTTP MCP transport. An ordinary HTTP GET is not an MCP handshake.

Authentication failure: compare the startup environment token with the client's Bearer value. Update the client when a restart changes the token.

Missing tools: reconnect and inspect client logs and binary versions. Do not invoke capabilities copied from documentation for a different version.

Empty extraction: check the same selector through the CLI, then inspect page identity and dynamic rendering requirements.

Null fields: verify selector scope and whether a value comes from an attribute or text.

Parameter type errors: use the numeric, boolean, and string types declared by the actual schema rather than quoting every value.

A checked CLI extraction gives you an understandable baseline before MCP adds protocol and authentication concerns.