Multilingual Scraping: Decode Pages and Export Correct CSV

When Chinese text becomes unreadable, locate the damaged stage first: response decoding, a later conversion, or the application opening the exported file. Saving already corrupted text as UTF-8 preserves the corruption.

This tutorial is for Python developers who can download pages and need to handle sources with different encodings. You will process equivalent UTF-8 and GB18030 materials, keep their original bytes, export CSV, and read it back. A download function shows where actual website responses enter the same workflow.

Separate bytes, text, and files

Gzip is transport compression, not a character encoding. The online helper requests an uncompressed response and rejects a compressed one. Decompress according to Content-Encoding before applying this workflow if your source still compresses responses.

Successful decoding does not uniquely identify an encoding: the same bytes may be valid in multiple encodings. Use a source-specific agreement, response declaration, or page declaration. Without evidence, retain the material instead of guessing silently.

Prepare the environment and complete example

Use Python 3.12 or later. No third-party packages are required. Create encoding_flow.py in a new directory and add the following code. Its default path builds two explicitly encoded teaching inputs, rather than requiring a demonstration server.

import csv
import hashlib
import json
import re
from pathlib import Path
from urllib.request import Request, urlopen
 
OUT = Path("encoding-output")
OUT.mkdir(exist_ok=True)
ALLOWED = {"utf-8", "utf-8-sig", "gb18030", "gbk"}
 
def fetch(url):
    request = Request(url, headers={"Accept-Encoding": "identity"})
    with urlopen(request, timeout=20) as response:
        if response.headers.get("Content-Encoding", "identity") != "identity":
            raise ValueError("响应仍被压缩，请先按 Content-Encoding 解压")
        return (
            response.read(),
            response.headers.get_content_charset(),
            response.geturl(),
        )
 
def decode_html(raw, declared=None, override=None):
    if override:
        selected, reason = override.lower(), "site_override"
    elif raw.startswith(b"\xef\xbb\xbf"):
        selected, reason = "utf-8-sig", "bom"
    elif declared:
        selected, reason = declared.lower(), "http_header"
    else:
        head = raw[:4096].decode("ascii", errors="ignore")
        match = re.search(r'charset\s*=\s*["\x27]?([\w-]+)', head, re.I)
        if not match:
            raise ValueError("缺少编码依据，请提供站点编码配置")
        selected, reason = match[1].lower(), "html_meta"
    if selected not in ALLOWED:
        raise ValueError(f"尚未支持的编码：{selected}")
    return raw.decode(selected, errors="strict"), selected, reason
 
def extract_title(text):
    match = re.search(r"<h1>(.*?)</h1>", text, re.S | re.I)
    if not match:
        raise ValueError("演示页面缺少 h1")
    return re.sub(r"\s+", " ", match[1]).strip()
 
materials = [
    ("utf8", '<meta charset="utf-8"><h1>中文采集指南</h1>'.encode("utf-8")),
    ("gb", '<meta charset="gb18030"><h1>中文采集指南</h1>'.encode("gb18030")),
]
rows = []
for name, raw in materials:
    (OUT / f"{name}.bin").write_bytes(raw)
    text, encoding, reason = decode_html(raw)
    rows.append({
        "source": name,
        "title": extract_title(text),
        "encoding": encoding,
        "encoding_reason": reason,
        "raw_sha256": hashlib.sha256(raw).hexdigest(),
    })
 
with (OUT / "records.json").open("w", encoding="utf-8") as stream:
    json.dump(rows, stream, ensure_ascii=False, indent=2)
with (OUT / "titles.csv").open("w", encoding="utf-8-sig", newline="") as stream:
    writer = csv.DictWriter(stream, fieldnames=list(rows[0]))
    writer.writeheader()
    writer.writerows(rows)
 
with (OUT / "titles.csv").open(encoding="utf-8-sig", newline="") as stream:
    restored = list(csv.DictReader(stream))
if [row["title"] for row in restored] != ["中文采集指南"] * 2:
    raise ValueError("CSV 回读内容与提取结果不一致")
print(json.dumps(rows, ensure_ascii=False, indent=2))

Run it with:

python encoding_flow.py

The small title extractor applies only to the simplified HTML defined here. For a real page, decode first and use your normal DOM parser afterward. This regular expression is not intended to parse nested markup or arbitrary HTML entities.

Understand strict decoding and raw material retention

Ignoring decoding errors deletes bytes; replacing them can insert replacement characters. Those policies may be suitable for display, but they can hide damage in collected records.

The example ignores non-ASCII bytes only while locating the ASCII encoding declaration. Actual text decoding is strict. A failure remains visible and can be investigated against the saved material. See the Python codec rules.

The two .bin files retain the inputs for later correction. Their hashes identify the exact material, not whether its contents are correct.

The priority implemented here is a deliberate policy, not a guarantee that declarations are truthful. A known source with a misleading header may need override="gb18030". Keep that override attached to the source rather than cycling through many encodings until something returns a string.

Connect an actual response

Replace the demonstration loop when introducing a real input. This snippet connects downloading and decoding; replace the placeholder address with an actual page:

raw, declared, final_url = fetch("https://example.16yun.cn/article")
text, encoding, reason = decode_html(raw, declared=declared)
print(final_url, encoding, reason)
print(extract_title(text))

The placeholder does not provide an article endpoint. For a source known to use GB18030 despite an incorrect UTF-8 response header, pass override="gb18030" and compare the title and a body passage with the visible source.

Do not apply Latin-1 re-encoding tricks to already correct Unicode text. Such repairs require evidence of the exact earlier conversion; otherwise they can create a new error.

Interpret the files

Both titles should be identical, while the recorded input encodings differ. This is the expected field structure, not a captured run; the complete records also contain the selection reason and byte hash:

[
  {"source": "utf8", "title": "中文采集指南", "encoding": "utf-8"},
  {"source": "gb", "title": "中文采集指南", "encoding": "gb18030"}
]

For JSON, ensure_ascii=False controls character escaping. The file's encoding comes from its open call. A legal Unicode escape in JSON is not automatically mojibake.

The CSV uses utf-8-sig to provide a BOM for readers that benefit from one. Machine-to-machine exchange may instead agree on plain UTF-8. Either choice requires a matching reader.

The newline="" argument lets the CSV module handle line endings. It also avoids the temptation to construct rows by joining strings with commas, which would fail for quoted fields. See the CSV documentation.

The read-back check establishes that this writer and reader preserve the expected titles. It cannot establish that every spreadsheet application will select the correct import settings. If Python reads the file correctly but a spreadsheet does not, import it explicitly as UTF-8 before changing the crawler.

Troubleshoot by stage

UnicodeDecodeError: determine whether it came from decoding downloaded bytes or opening an output file. For response decoding, inspect declarations and preserved bytes. For file reading, verify the file's actual encoding. Do not hide either error with an ignore policy.

Readable but incorrect characters: a wrong encoding can still produce valid text. Compare a known title with the source and investigate conflicting declarations. A returned string alone is insufficient evidence.

Replacement characters: check for an earlier replacement policy or an upstream service that already damaged the text. Reprocess the original bytes. CSV settings cannot recover information already discarded before collection.

Nonbreaking spaces or full-width characters: distinguish formatting from encoding. Normalize a search key only when appropriate; retain original names and identifiers where their exact form matters.

Blank lines or split titles in CSV: use the CSV writer with explicit newline handling. Check row counts and restored fields, then verify the importing application's delimiter settings.

Keeping original bytes, recording why an encoding was selected, and checking the export gives each failure a specific place to investigate.