Reconcile Product Data across Websites

Two websites selling “classic black tea” may offer different products or packages. Even 500 g and two 250 g boxes are not necessarily interchangeable offers.

This tutorial starts with results from two existing collectors. You will map their different fields, normalize units, and group offers only where the inputs support a shared product identity. The outputs separate confirmed groups, records requiring review, and invalid records. This is entity reconciliation rather than URL deduplication.

Keep product identity separate from an offer

A product can have multiple prices across websites, markets, and observation times. Preserve those offers rather than choosing one value and overwriting the rest.

The example requires a shared GTIN plus consistent brand and package weight for automatic grouping. Missing identifiers go to review. The supplied identifiers are teaching inputs; the example does not verify check digits or prove that a source's identifier is genuine.

A price comparison also needs compatible currencies and package meanings. Unknown shipping or tax cannot be converted into a claim about the cheapest total purchase.

Prepare and run the complete example

Use Python 3.12 or later; only the standard library is needed. Create merge_products.py with the following code.

Source A uses name/grams/price; source B uses label/weight/amount. For the identified tea product, 0.5 kg and 500 g should normalize to the same package weight. Two other tea gift boxes lack identifiers and must remain unconfirmed.

import json
import unicodedata
from collections import defaultdict
from decimal import Decimal, InvalidOperation
 
feeds = {
    "A": [
        {"id": "a1", "name": "山谷红茶 500g", "brand": "Valley",
         "gtin": "0123456789012", "grams": "500", "price": "12.90",
         "currency": "CNY", "url": "https://example.16yun.cn/a/a1"},
        {"id": "a2", "name": "茶叶礼盒", "brand": "Valley",
         "gtin": None, "grams": "500", "price": "18.00",
         "currency": "CNY", "url": "https://example.16yun.cn/a/a2"},
    ],
    "B": [
        {"id": "b1", "label": "Valley black tea", "brand": "Valley",
         "gtin": "0123456789012", "weight": "0.5", "unit": "kg",
         "amount": "11.90", "currency": "CNY",
         "url": "https://example.16yun.cn/b/b1"},
        {"id": "b2", "label": "茶叶礼盒", "brand": "Valley",
         "gtin": None, "weight": "1", "unit": "kg",
         "amount": "25.00", "currency": "CNY",
         "url": "https://example.16yun.cn/b/b2"},
    ],
}
observed_at = "2026-10-28T09:00:00Z"
 
def normalized_key(value):
    return " ".join(unicodedata.normalize("NFKC", value).casefold().split())
 
def normalize(source, raw):
    if source == "A":
        title, grams, price = raw["name"], raw["grams"], raw["price"]
        grams = Decimal(grams)
    else:
        title, price = raw["label"], raw["amount"]
        multiplier = {"g": Decimal(1), "kg": Decimal(1000)}[raw["unit"]]
        grams = Decimal(raw["weight"]) * multiplier
    amount = Decimal(price)
    if not grams.is_finite() or grams <= 0:
        raise ValueError("包装重量必须是正的有限值")
    if not amount.is_finite() or amount < 0:
        raise ValueError("价格必须是非负有限值")
    brand = raw["brand"].strip()
    currency = raw["currency"]
    if not title.strip() or not brand or currency not in {"CNY", "USD", "GBP"}:
        raise ValueError("名称、品牌或币种不符合约定")
    return {
        "source": source, "source_id": raw["id"],
        "title_raw": title, "brand_key": normalized_key(brand),
        "gtin": raw.get("gtin"), "grams": str(grams.normalize()),
        "price": str(amount), "currency": currency,
        "source_url": raw["url"], "observed_at": observed_at,
    }
 
groups = defaultdict(list)
review = []
rejected = []
for source, items in feeds.items():
    for raw in items:
        try:
            item = normalize(source, raw)
        except (KeyError, ValueError, InvalidOperation) as error:
            rejected.append({"source": source, "raw": raw, "reason": str(error)})
            continue
        if not item["gtin"]:
            review.append({"reason": "missing_identifier", "offer": item})
            continue
        groups[item["gtin"]].append(item)
 
matched = []
for gtin, offers in groups.items():
    signatures = {(row["brand_key"], row["grams"]) for row in offers}
    if len(signatures) != 1:
        review.append({"reason": "identifier_conflict", "gtin": gtin, "offers": offers})
    else:
        matched.append({"gtin": gtin, "offers": offers})
 
result = {"matched": matched, "review": review, "rejected": rejected}
with open("merged.json", "w", encoding="utf-8") as stream:
    json.dump(result, stream, ensure_ascii=False, indent=2)
print({"matched_groups": len(matched), "review_items": len(review),
       "rejected_items": len(rejected)})

Run it with:

python merge_products.py

From the supplied inputs, expect one confirmed group, two review entries, and no rejected records. These are derived example results, not an observed matching accuracy measurement.

Understand field mapping and normalization

The source adapter translates field names into an internal representation. It retains title_raw, because a cleaned title cannot explain later whether important package information was removed.

Weights and amounts are parsed from strings using Decimal, avoiding a binary floating-point detour. This makes the 0.5 kg to 500 g conversion explicit. See the Decimal documentation.

The brand matching key applies NFKC normalization, case folding, and whitespace consolidation. That key is separate from display names and original identifiers. Compatibility normalization can change presentation, so do not apply it blindly to legal names or model identifiers. See the Unicode normalization documentation.

Total weight is insufficient for multipacks. Add package count, individual-unit weight, and package type before handling two 250 g boxes. A business decision that two packages are substitutes is a separate analytical relationship, not proof of identical identity.

Check an identifier before accepting its group

A source might collect the wrong code or reuse a product-level identifier across variants. The code first creates candidate groups by GTIN, then compares brand and weight.

A conflicting group goes to review with every offer preserved. The implementation does not replace a minority value with the majority value. Inspect the original pages to determine whether the identifier describes a product, a variant, or something unrelated.

Likewise, the two unnumbered gift boxes cannot be confirmed from their shared title, particularly when their weights differ. Additional model numbers, manufacturer fields, and package evidence can support candidate generation, but a similarity score should not silently become a confirmed match.

Compare offers without losing their meaning

Within the confirmed example group, both prices use CNY and the same package weight. The displayed amounts 12.90 and 11.90 can be compared. Nothing in the input establishes delivered cost, because shipping information is absent.

The example uses a fixed observation timestamp to illustrate the field. A real collector should supply its actual observation time. Do not substitute the merge job's start time for the time the page was collected.

For ongoing data, identify an offer by source, source record ID, and any relevant market or variant dimension. Preserve observations over time. GTIN links sources; it should not erase the history of each source's offer.

Troubleshoot the reconciliation

Expected matches remain separate: distinguish a missing identifier from a conflicting one. Collect the manufacturer's identifier from the detail source where available. Do not immediately weaken the rule to title equality. After correction, both original offers should appear in one confirmed group.

Weights differ by a factor of a thousand: verify the source unit and fix the adapter. Reprocess the original values. Do not clamp outliers to an average, and do not default unknown units to grams.

One identifier has different brands: inspect which page element supplied it. It might be a platform category code rather than GTIN. Correct the identifier type or extraction; a brand-alias dictionary cannot resolve an unrelated code.

Amount parsing fails: the source may contain currency symbols, grouping separators, or a “from” price. This adapter expects a previously separated decimal amount. Resolve that format and meaning in the source adapter rather than removing every non-digit character.

A source disappears from the export: inspect review and rejected entries as well as confirmed groups. Reconciliation does not require every input to receive an automatic identity.

The useful output is a traceable relationship between products and offers. Retaining uncertainty makes later correction possible without reconstructing lost sources.