Browser4: Collect Dynamic Pages and Recover Failed Work

Dynamic extraction needs to distinguish the loaded page, rendered data, and any stored snapshot. Otherwise a browser can display new content while a query keeps returning an older copy.

This tutorial reads two public dynamic quote pages with deterministic CLI queries, preserves their results, and explains adaptation to detail pages and batch recovery. The main path requires no model.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• QUOTES_LIST_URL: the Quotes to Scrape JavaScript list entry with path /js/.
• QUOTES_PAGE_2_URL: the second page on that practice site with path /js/page/2/.

Set these in the same Bash/WSL terminal used for subsequent commands:

export QUOTES_LIST_URL='<paste-the-actual-quotes-list-url>'
export QUOTES_PAGE_2_URL='<paste-the-actual-quotes-page-2-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install both CLI and runtime

Prepare Node.js and a supported Chrome/Chromium environment:

npm install -g browser4-cli
browser4-cli install
browser4-cli --help

Package installation and runtime preparation are separate. Retain the installed versions and upgrade them together where required. See the official Browser4 instructions.

The tutorial uses CLI operations rather than inventing a universal REST endpoint. AI extraction and agent commands introduce separate model configuration.

Inspect the rendered page

browser4-cli open --headed "$QUOTES_LIST_URL"
browser4-cli snapshot
browser4-cli get text title

Check title, source, and actual quote content. If loading is incomplete, inspect again when the relevant content appears; an early zero count is not an empty-site conclusion.

Attribute reads require their name:

browser4-cli get attr 'li.next a' href

Resolve relative values against the actual page rather than treating them as complete addresses.

Read correlated fields

Create read_page.js:

(() => {
  const rows = [...document.querySelectorAll('.quote')].map(card => ({
    text: card.querySelector('.text')?.textContent?.trim() ?? null,
    author: card.querySelector('.author')?.textContent?.trim() ?? null,
    tags: [...card.querySelectorAll('.tags .tag')].map(tag => tag.textContent.trim())
  }));
  if (!rows.length) throw new Error('目标列表尚未出现或选择器失效');
  if (rows.some(row => !row.text || !row.author)) {
    throw new Error('名言文本或作者缺失');
  }
  return { source_url: location.href, title: document.title, rows };
})()

Save the command output:

browser4-cli eval --file read_page.js > page-1.txt

The expression returns a value. Logging inside the page is a separate diagnostic channel.

The saved text is CLI output, not an assumed SDK envelope. For machine integration, inspect the installed version's typed JSON output and extract its actual business value. Do not recover arbitrary brace-delimited fragments from mixed logs.

Read and verify a second page

browser4-cli goto "$QUOTES_PAGE_2_URL"
browser4-cli snapshot
browser4-cli eval --file read_page.js > page-2.txt

Compare source addresses and representative quote content. Two evaluations or two filenames do not establish two distinct page results.

For product details, replace the selectors and field contract with product-specific identity, price, and currency rules. Preserve relationships within each object rather than zipping unrelated page-wide arrays by position.

Choose live or stored material deliberately

Save HTML when useful:

browser4-cli htmlsnapshot export --file page-2.html

Eval reads the live DOM. Snapshot reads with an expiry window can intentionally use stored content. Investigate that distinction when the browser and extraction disagree.

Interaction snapshots provide current references; HTML snapshots support content reading and material retention. Old references are not permanent selectors.

Batch operations and targeted repair

Try an explicit fail-fast sequence:

browser4-cli batch --bail   "goto $QUOTES_LIST_URL"   'snapshot'   'get text title'

For JSON input mode, save this array as commands.json:

[
  ["goto", "<QUOTES_LIST_URL>"],
  ["snapshot"],
  ["get", "text", "title"]
]

Then run browser4-cli batch --json < commands.json. The option selects stdin-driven command input, not simply output formatting for arbitrary positional commands.

Inspect individual step outcomes. Retain the failing source and stage, then rerun the unresolved work after repair. This is application-side continuation, not a claim that CLI batches provide a durable business queue.

Check and close

An extracted row should preserve the text/author relationship:

{"text":"页面实际名言","author":"对应作者","tags":["页面标签"]}

This is illustrative data. Close the session with browser4-cli close; that does not mean every stored profile or runtime file was deleted.

Troubleshoot

Open fails: inspect runtime, browser, and startup before selectors.

Eval returns null: verify an actual return value and explicit missing-data handling rather than console-only code.

Old values persist: compare live page, source identity, snapshot, and expiry policy.

Batch JSON is rejected: check the nonempty array, command tokens, and absence of conflicting positional inputs.

A click target expires: obtain a fresh snapshot.

Both page files match: inspect navigation, source URL, and readiness before accepting coverage.

Separating opening, observation, querying, and saving makes dynamic-page failures local and understandable.