Monitor Web Changes without Treating Failures as Deletions
Store normalized snapshots in SQLite, identify additions and modifications, preserve failed details, and confirm absence through explicit complete observations.
Missing from today's output does not necessarily mean deleted. Incomplete discovery, detail timeouts, and broken selectors can all reduce a batch.
This tutorial compares collector-produced inventory and detail outcomes, preserving known products while distinguishing additions, modifications, failures, and tentative removal.
Separate discovery from detail retrieval
Expected IDs describe inventory discovery. Results describe detail processing. Only a verified complete traversal can establish the complete flag.
An item still listed but failing to refresh keeps its previous content. An item absent from a complete inventory gains an absence count. This example marks removal after two complete missing observations while retaining historical payload.
Prepare explicit inputs
Use Python 3.12 and its standard library. Save baseline.json:
{
"discovery_complete": true,
"expected_ids": ["p1", "p2"],
"results": [
{"id":"p1","status":"ok","name":"红茶","price_minor":100,"currency":"CNY"},
{"id":"p2","status":"ok","name":"绿茶","price_minor":200,"currency":"CNY"}
]
}Save update.json:
{
"discovery_complete": true,
"expected_ids": ["p1", "p2", "p3"],
"results": [
{"id":"p1","status":"ok","name":"红茶","price_minor":110,"currency":"CNY"},
{"id":"p2","status":"failed"},
{"id":"p3","status":"ok","name":"乌龙茶","price_minor":300,"currency":"CNY"}
]
}These are teaching snapshots. p2 remains listed, so its failed detail must not become a deletion.
Compare and persist atomically
Create monitor.py:
import argparse
import hashlib
import json
import sqlite3
from pathlib import Path
parser = argparse.ArgumentParser()
parser.add_argument("snapshot")
parser.add_argument("--run-id", required=True)
args = parser.parse_args()
raw = Path(args.snapshot).read_text(encoding="utf-8")
batch = json.loads(raw)
input_hash = hashlib.sha256(raw.encode()).hexdigest()
expected = batch["expected_ids"]
if len(set(expected)) != len(expected) or any(not isinstance(x, str) or not x for x in expected):
raise ValueError("目录 ID 缺失或重复")
if not expected and not batch.get("empty_confirmed"):
raise ValueError("空目录需要明确确认，不能自动覆盖")
results = {}
for item in batch["results"]:
key = item["id"]
if key not in expected or key in results:
raise ValueError("详情 ID 不属于目录或重复")
if item["status"] not in {"ok", "failed"}:
raise ValueError("未知详情状态")
if item["status"] == "ok":
if not item.get("name") or type(item.get("price_minor")) is not int:
raise ValueError("详情结构不符合约定")
if item["price_minor"] < 0 or item.get("currency") != "CNY":
raise ValueError("金额或币种不符合约定")
results[key] = item
db = sqlite3.connect("monitor.sqlite")
db.executescript("""
CREATE TABLE IF NOT EXISTS items(
id TEXT PRIMARY KEY,payload TEXT NOT NULL,misses INTEGER NOT NULL,state TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS runs(
id TEXT PRIMARY KEY,input_hash TEXT NOT NULL,report TEXT NOT NULL
);
""")
try:
prior = db.execute("SELECT input_hash,report FROM runs WHERE id=?", (args.run_id,)).fetchone()
if prior:
if prior[0] != input_hash:
raise ValueError("同一 run-id 不能对应不同输入")
print(prior[1])
else:
old = {row[0]: row for row in db.execute("SELECT id,payload,misses,state FROM items")}
events = []
with db:
for key in expected:
item = results.get(key)
previous = old.get(key)
if item is None or item["status"] == "failed":
events.append({"id": key, "change": "fetch_failed"})
if previous:
db.execute("""UPDATE items SET misses=0,state=CASE
WHEN state IN ('deleted','restore_pending') THEN 'restore_pending'
ELSE 'active' END WHERE id=?""", (key,))
continue
payload = json.dumps({field: item[field] for field in
("name", "price_minor", "currency")}, ensure_ascii=False, sort_keys=True)
change = "added" if previous is None else (
"restored" if previous[3] in {"deleted", "restore_pending"} else
"modified" if previous[1] != payload else "unchanged"
)
events.append({"id": key, "change": change,
"before": json.loads(previous[1]) if previous else None,
"after": json.loads(payload)})
db.execute("""INSERT INTO items VALUES (?,?,0,'active')
ON CONFLICT(id) DO UPDATE SET payload=excluded.payload,misses=0,state='active'""",
(key, payload))
if batch["discovery_complete"] is True:
for key, previous in old.items():
if key not in expected:
misses = previous[2] + 1
state = "deleted" if misses >= 2 else "pending_delete"
db.execute("UPDATE items SET misses=?,state=? WHERE id=?", (misses, state, key))
events.append({"id": key, "change": (
state if previous[3] != state else "still_absent"
), "state": state})
report = json.dumps(events, ensure_ascii=False, indent=2)
db.execute("INSERT INTO runs VALUES (?,?,?)", (args.run_id, input_hash, report))
print(report)
finally:
db.close()Run:
python monitor.py baseline.json --run-id baseline
python monitor.py update.json --run-id update-1The baseline adds p1 and p2. The update changes p1, adds p3, and records p2's failure without discarding its old value.
Reusing a run ID with identical input returns its saved report rather than counting another observation. Reusing it for different input is rejected.
Demonstrate removal states
Save missing.json:
{
"discovery_complete": true,
"expected_ids": ["p1", "p3"],
"results": [
{"id":"p1","status":"ok","name":"红茶","price_minor":110,"currency":"CNY"},
{"id":"p3","status":"ok","name":"乌龙茶","price_minor":300,"currency":"CNY"}
]
}Process two teaching observations:
python monitor.py missing.json --run-id missing-demo-1
python monitor.py missing.json --run-id missing-demo-2The logic moves p2 through pending deletion to deleted. In a real system, relabeling one cached response with two IDs does not establish two independent observations. Choose observation timing and evidence according to source behavior and deletion risk.
Deleted is a retained database state, not physical destruction. A successfully refreshed reappearance becomes restored.
A deleted item rediscovered with a failed detail becomes restore_pending, retaining its old payload until a successful refresh can report restored. Continued absence in the same deletion state reports still_absent instead of repeatedly announcing a new deletion.
Compare business fields rather than arbitrary HTML
The example compares normalized name, price, and currency, excluding volatile navigation or collection timestamps. That avoids declaring every advertising change a product update.
It also means inventory or variant changes are outside the current contract. Add relevant fields deliberately, with explicit missing and zero semantics.
The transaction stores state changes together with their report. See the SQLite transaction documentation.
Troubleshoot
Failures cause mass deletion: repair upstream completeness classification before changing thresholds.
Listed failures become removals: retain discovered IDs separately from successfully parsed details.
Reprocessing increases absence counts: keep one stable observation ID.
Every item changes daily: remove volatile or inconsistently normalized values from the comparison payload.
Old content survives a failed refresh: that is deliberate preservation, not evidence of fresh data. Track freshness separately.
A genuinely empty inventory: require explicit upstream empty confirmation and preserve its evidence.
Change monitoring becomes useful when business change is distinguishable from collection uncertainty.
Need an enterprise proxy plan?
We can tailor architecture to your target domains, concurrency, and reliability goals.