Spider for Rust: Build a Site and Resource Inventory

A site inventory needs more than addresses: response status, saved material, and the resources each page declares all help later migration and diagnosis.

This tutorial subscribes to Spider's page stream, saves up to twenty bounded pages, and lists image, script, and link resources found in their HTML. It does not download those resources or certify a complete site mirror.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

BOOKS_LIST_URL: the Books to Scrape homepage address.

Set these in the same Bash/WSL terminal used for subsequent commands:

export BOOKS_LIST_URL='<paste-the-actual-books-list-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Create the Rust project

Install stable Rust/Cargo through the official instructions, then:

cargo new site-inventory
cd site-inventory
cargo add spider@=2.52.4 --features sync
cargo add serde_json scraper url

Retain Cargo.lock. Spider traverses pages, scraper parses the saved HTML, and url resolves relative references. See the pinned Rust API documentation.

Define inventory scope

The practice-site entry has depth one and a twenty-page budget. These are not category or product counts: links of different purposes can be discovered.

The inventory describes materials obtained under these settings. A real report should retain entry, scope, and budget rather than implying site-wide completeness.

Replace src/main.rs:

use std::{collections::BTreeSet, error::Error, fs, io::Write};
use scraper::{Html, Selector};
use serde_json::json;
use spider::{tokio, traits::PageData, website::Website};
use url::Url;
 
type AnyError = Box<dyn Error + Send + Sync>;
 
#[tokio::main]
async fn main() -> Result<(), AnyError> {
    fs::create_dir("output")?;
    fs::create_dir("output/pages")?;
    let entry = std::env::var("BOOKS_LIST_URL")?;
    let mut site = Website::new(&entry);
    site.with_limit(20).with_depth(1).with_delay(500);
    site.with_respect_robots_txt(true);
    let mut pages = site.subscribe(64);
 
    let consumer = tokio::spawn(async move {
        let selector = Selector::parse("img[src], script[src], link[href]")
            .map_err(|_| "invalid resource selector")?;
        let mut index = fs::File::create("output/inventory.jsonl.part")?;
        let mut count = 0usize;
        loop {
            let page = match pages.recv().await {
                Ok(page) => page,
                Err(tokio::sync::broadcast::error::RecvError::Closed) => break,
                Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => {
                    return Err::<usize, AnyError>(
                        format!("page consumer missed {n} messages").into()
                    );
                }
            };
            let source = page.get_url().to_string();
            let html = page.get_html();
            let path = format!("output/pages/{count:04}.html");
            fs::write(&path, &html)?;
            let base = Url::parse(&source)?;
            let document = Html::parse_document(&html);
            let mut resources = BTreeSet::new();
            for element in document.select(&selector) {
                let value = element.value().attr("src")
                    .or_else(|| element.value().attr("href"));
                if let Some(value) = value {
                    if let Ok(resolved) = base.join(value) {
                        if matches!(resolved.scheme(), "http" | "https") {
                            resources.insert(resolved.to_string());
                        }
                    }
                }
            }
            let record = json!({
                "url": source,
                "status": page.status_code().to_string(),
                "html_file": path,
                "html_bytes": html.len(),
                "declared_resource_urls": resources,
            });
            writeln!(index, "{record}")?;
            count += 1;
        }
        index.flush()?;
        Ok::<usize, AnyError>(count)
    });
 
    site.crawl().await;
    site.unsubscribe();
    let count = consumer.await??;
    fs::rename("output/inventory.jsonl.part", "output/inventory.jsonl")?;
    println!("saved_pages={count}");
    Ok(())
}

Run:

cargo run --release

The output directory must be new. Move an earlier batch or choose a separate location rather than allowing a failed run to overwrite old evidence.

Understand completion and lag

Subscribe before starting traversal; a subscription created afterward should not be expected to replay historical results.

Traversal and consumption run concurrently. The broadcast buffer is finite. Lagged messages explicitly fail this example and leave a partial index instead of silently ending the consumer.

After crawling returns, unsubscribe, await the consumer, and rename the completed index. Traversal completion, message consumption, and durable file writing are distinct stages.

A URL-append queue is a different channel from page subscriptions. Dropping one sender is not a universal traversal termination signal.

Interpret the records

An illustrative JSONL record is:

{"url":"https://example.16yun.cn/docs/","status":"200 OK","html_file":"output/pages/0000.html","html_bytes":1234,"declared_resource_urls":["https://example.16yun.cn/style.css"]}

The status text and numbers are examples; exact representation comes from the library. HTML bytes are not readable-word count or compressed transfer size.

The resource selector covers explicit image, script, and link elements. It does not capture every CSS reference or runtime network request, and a declared URL is not evidence of successful download.

Resources deduplicate within each page while retaining cross-page relationships. A complete browser network inventory or resource download check needs a separate, bounded collection stage.

Diagnose scope and rendering

Inspect unsuccessful statuses and saved bodies, including HTTP 200 responses that contain unexpected access pages.

The default HTTP path does not execute every dynamic behavior. A browser-rendered configuration changes dependencies, readiness, resources, and budget; enabling a feature alone does not certify SPA completeness.

Troubleshoot

Status method unavailable: import PageData and use the pinned API's status_code rather than inventing get_status_code.

Only a partial index appears: inspect lag, parsing, and disk errors. Do not rename incomplete data as complete.

Fewer than twenty pages: a cap is not a minimum. Inspect scope, discoverability, robots, and responses.

Resources are absent: determine whether they were dynamically generated or referenced outside the HTML elements inspected here.

Consumers lag: reduce production speed or redesign persistence with explicit memory and throughput constraints; increasing a buffer indefinitely only postpones the problem.

Output directory exists: preserve batch separation and compare sources under recorded settings.

Streaming connects traversal and storage, while explicit scope and failure evidence make the inventory interpretable.