Playwright Dynamic Lists: Responses, Scrolling, and Cursors

An infinite list may retain only visible nodes. Reading the DOM once at the end can miss earlier data, while an unchanged count may indicate failure rather than completion.

This tutorial supplies a complete browser page and three response batches. It collects responses before triggering scroll, follows next cursors, and merges product IDs. One repeated item demonstrates why deduplication alone cannot prove progress.

Install

Use Python 3.12:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install playwright
python -m playwright install chromium

The example routes a teaching domain internally rather than accessing a live service. It collects list records, not detail pages.

Define explicit response material

Create cursor_list.py:

import asyncio
import json
from pathlib import Path
from urllib.parse import parse_qs, urlsplit
from playwright.async_api import async_playwright
 
PAGES = {
    "": {"items": [{"id": "p1", "name": "商品一"}, {"id": "p2", "name": "商品二"}], "next_cursor": "second"},
    "second": {"items": [{"id": "p2", "name": "商品二"}, {"id": "p3", "name": "商品三"}], "next_cursor": "last"},
    "last": {"items": [{"id": "p4", "name": "商品四"}], "next_cursor": None},
}
HTML = """
<style>.card {height:600px;border:1px solid #ddd}</style>
<h1>教学列表</h1><div id="list"></div><p id="state"></p>
<script>
let cursor = '', loading = false, ended = false;
async function load() {
  if (loading || ended) return;
  loading = true;
  try {
    const response = await fetch('/api/items?cursor=' + encodeURIComponent(cursor));
    if (!response.ok) throw new Error('API failed');
    const data = await response.json();
    const list = document.querySelector('#list');
    for (const item of data.items) {
      const card = document.createElement('div');
      card.className = 'card'; card.textContent = item.name;
      list.appendChild(card);
    }
    cursor = data.next_cursor; ended = cursor === null;
    list.dataset.next = ended ? 'END' : cursor;
  } catch (error) {
    document.querySelector('#state').textContent = '加载失败';
  } finally { loading = false; }
}
window.addEventListener('scroll', () => {
  if (innerHeight + scrollY >= document.body.scrollHeight - 10) load();
});
load();
</script>
"""

Cursors are opaque service values. Real tokens may be signed or otherwise nonnumeric; preserve them instead of deriving page numbers.

Observe and scroll

Append:

async def main():
    records, cursors = {}, set()
    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        try:
            context = await browser.new_context(viewport={"width": 1000, "height": 400})
            page = await context.new_page()
 
            async def respond(route):
                parsed = urlsplit(route.request.url)
                if parsed.path == "/api/items":
                    cursor = parse_qs(parsed.query, keep_blank_values=True).get("cursor", [""])[0]
                    data = PAGES.get(cursor)
                    await route.fulfill(
                        status=200 if data is not None else 400,
                        content_type="application/json", body=json.dumps(data or {"error": "cursor"}),
                    )
                else:
                    await route.fulfill(content_type="text/html; charset=utf-8", body=HTML)
 
            await page.route("https://example.16yun.cn/**", respond)
            def is_items(response):
                return urlsplit(response.url).path == "/api/items" and response.request.method == "GET"
 
            async with page.expect_response(is_items) as first:
                await page.goto("https://example.16yun.cn/")
            response = await first.value
            for _ in range(10):
                if response.status != 200:
                    raise ValueError("分页请求失败")
                data = await response.json()
                if not isinstance(data.get("items"), list):
                    raise ValueError("分页响应不是预期结构")
                for item in data["items"]:
                    if not isinstance(item.get("id"), str) or not item.get("name"):
                        raise ValueError("商品 ID 或名称缺失")
                    if item["id"] in records and records[item["id"]] != item:
                        raise ValueError("同一 ID 跨页数据冲突")
                    records[item["id"]] = item
                next_cursor = data.get("next_cursor")
                marker = "END" if next_cursor is None else next_cursor
                await page.wait_for_function(
                    "marker => document.querySelector('#list').dataset.next === marker",
                    arg=marker,
                )
                if next_cursor is None:
                    break
                if not isinstance(next_cursor, str) or next_cursor in cursors:
                    raise ValueError("分页游标无效或循环")
                cursors.add(next_cursor)
                async with page.expect_response(is_items, timeout=15000) as pending:
                    await page.mouse.wheel(0, 5000)
                response = await pending.value
            else:
                raise ValueError("超出本轮分页预算")
        finally:
            await browser.close()
    Path("items.json").write_text(
        json.dumps({"source_kind": "teaching_material", "items": list(records.values())},
                   ensure_ascii=False, indent=2), encoding="utf-8"
    )
    print({"unique_items": len(records)})
 
asyncio.run(main())

Run:

python cursor_list.py

The material implies three batches and four unique products. The repeated p2 does not add identity but does not stop cursor processing.

Register observation before acting

Starting response observation after scrolling can miss fast requests. The waiting context establishes observation first. See the Playwright network guide.

A completed response also does not guarantee rendering is finished. The teaching marker confirms the UI appended the batch before another scroll. Real sources require their own equivalent signal.

Path matching is sufficient only for this controlled material. Production observation may need method, API identity, filters, and expected cursor to exclude prefetched or unrelated responses.

Use contractual termination

Null cursor is the demonstrated end condition. Duplicate items, empty batches, failed requests, or unchanged visible DOM do not independently mean the end.

An API may allow empty batches with another cursor; follow its actual contract with finite budgets and loop detection.

Response collection can avoid a virtualized DOM's window limit, but inspect whether the response really belongs to the intended filter and accessible data scope.

Troubleshoot

No response after scrolling: inspect trigger position, loading, path matching, and actual termination before treating a timeout as completion.

Repeated cursor: retain request and response evidence, then inspect filters and cursor handling.

Only duplicates accumulate: inspect cursor progress and identity. Deduplication does not establish new-page discovery.

Fields exist only in the response: determine whether they are relevant data or unrelated prefetch material.

Pagination uses POST: include body semantics in observation and request handling, not just the URL.

An old export remains after failure: this example writes only on completion and does not provide durable recovery. Track batch state separately.

Dependable dynamic collection connects response identity, cursor progress, and business records instead of counting scrolls.