Pydoll: Search Dynamic Pages and Collect Paginated Results

A click, an idle network, and an updated result list are different states. Fixed sleeps can waste time on fast pages and still read stale content on slow ones.

This tutorial opens the dynamic Quotes to Scrape list, selects the life tag, and collects at most two result pages. It preserves text, author, and provenance, saving a diagnostic screenshot if the browser remains available after a failure.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• QUOTES_LIST_URL: the Quotes to Scrape JavaScript list address with path /js/.

Set these in the same Bash/WSL terminal used for subsequent commands:

export QUOTES_LIST_URL='<paste-the-actual-quotes-list-url>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install the Python package and browser prerequisite

Use Python 3.12 with an installed Chrome or Edge:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install pydoll-python==3.0.0

Pydoll does not download the browser and does not need WebDriver. This tutorial uses v3 imports and element APIs rather than mixing earlier examples or another library's methods. See the official project.

Tie actions to their expected state

Each quote container holds its text and author. After clicking the tag, the example waits for the intended URL and then for the list.

This site navigates. An application that updates data without changing its URL needs a content or readiness marker instead.

Run the complete task

Create collect_quotes.py:

import asyncio
import os
import csv
from pathlib import Path
from urllib.parse import urljoin
from pydoll import Chrome
 
async def main():
    out = Path("pydoll-output")
    out.mkdir(exist_ok=True)
    rows, seen = [], set()
    async with Chrome() as browser:
        tab = await browser.start()
        try:
            await tab.go_to(os.environ["QUOTES_LIST_URL"])
            link = await tab.query('a.tag[href="/tag/life/page/1/"]', timeout=15)
            target = urljoin(await tab.current_url(), link.get_attribute("href"))
            await link.click()
            await tab.wait_for_url(target, timeout=20)
 
            for page_number in range(1, 3):
                await tab.query(".quote", timeout=15)
                cards = await tab.query(".quote", find_all=True)
                source_url = await tab.current_url()
                for card in cards:
                    text = (await (await card.query(".text")).text()).strip()
                    author = (await (await card.query(".author")).text()).strip()
                    if not text or not author:
                        raise ValueError("名言或作者缺失")
                    key = (text, author)
                    if key not in seen:
                        seen.add(key)
                        rows.append({
                            "text": text, "author": author,
                            "source_url": source_url, "page_number": page_number,
                        })
                if page_number == 2:
                    break
                next_link = await tab.query("li.next a", timeout=2, raise_exc=False)
                if next_link is None:
                    break
                next_url = urljoin(source_url, next_link.get_attribute("href"))
                if next_url == source_url:
                    raise ValueError("下一页仍指向当前页")
                await next_link.click()
                await tab.wait_for_url(next_url, timeout=20)
            await tab.take_screenshot(str(out / "last-page.png"), beyond_viewport=True)
        except Exception:
            try:
                await tab.take_screenshot(str(out / "failure.png"), beyond_viewport=True)
            except Exception as shot_error:
                print("截图未保存：", type(shot_error).__name__)
            raise
 
    if not rows:
        raise ValueError("没有取得任何目标记录")
    with (out / "quotes.csv").open("w", encoding="utf-8-sig", newline="") as stream:
        writer = csv.DictWriter(stream, fieldnames=["text", "author", "source_url", "page_number"])
        writer.writeheader()
        writer.writerows(rows)
    print({"unique_quotes": len(rows)})
 
asyncio.run(main())

Run:

python collect_quotes.py

The browser context manager owns cleanup. Screenshot errors do not replace the original collection exception.

In this version, get_attribute is synchronous while text and query are asynchronous. Similar-looking methods do not imply identical invocation.

Inspect the export

Verify life-tag source paths and compare text and author inside the same repeated object. Independent page-wide arrays can become misaligned.

Actual row count depends on source content. A two-page cap does not promise that two pages exist; verify a missing next link really describes the end state.

The text/author key is suitable for this practice task, not a universal product identity. Product collection needs a business identifier such as SKU.

The final CSV is written only after successful completion. An old file in the same directory must not be treated as the current failed run's result. Use distinct batch output locations in a recurring integration.

Adapt the workflow

Change entry and filtering action first, then repeated-object selectors and fields, then termination. Search-input tasks need the installed version's input API plus evidence that results updated.

Add proxy settings only for an actual network requirement. Establish a direct extraction baseline before combining access, identity, and network changes.

Virtualized scrolling requires a different traversal strategy: current DOM count may represent only the visible window.

Troubleshoot

Browser unavailable: install or configure the actual executable; reinstalling the Python package cannot supply it.

Tag missing: inspect URL, rendering, and the current link. A new filter requires matching selectors and expectations.

Click does not update content: determine whether the URL changes and use the relevant update signal.

Page two duplicates page one: compare stored source, next href, and text. Deduplication alone can conceal the missing page.

Attribute calls fail: distinguish synchronous attributes from asynchronous text and queries, and check the pinned version.

Failure screenshot is unavailable: retain the original error. A disconnected browser cannot guarantee an artifact.

CSV looks corrupted: compare in-memory text and a read-back before changing extraction rules.

Useful dynamic automation connects each action to evidence that the corresponding new data is ready.