Python Selenium Chrome 隧道代理：浏览器自动化集成爬虫代理

Selenium + 爬虫代理的特殊性

Selenium Chrome 配置代理认证的方式与其他工具有所不同：Chrome 不支持通过命令行参数直接传入代理用户名密码，需要通过 Chrome 扩展 注入代理认证信息。

环境准备

export PROXY_HOST=t.16yun.cn
export PROXY_PORT=31111
export PROXY_USERNAME=your-username
export PROXY_PASSWORD=your-password
export TARGET_URL=http://httpbin.org/ip

Chrome 扩展注入代理认证

通过动态生成 Chrome 扩展来配置代理和认证信息：

import os
import string
import tempfile
import zipfile
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def create_proxy_extension(host, port, username, password):
    """生成包含代理配置的 Chrome 扩展 ZIP"""
    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Chrome Proxy",
        "permissions": ["proxy", "tabs", "unlimitedStorage",
                        "storage", "<all_urls>", "webRequest",
                        "webRequestBlocking"],
        "background": {"scripts": ["background.js"]}
    }
    """
    background_js = string.Template("""
        var config = {
            mode: "fixed_servers",
            rules: {
                singleProxy: {
                    scheme: "http",
                    host: "${host}",
                    port: parseInt(${port})
                },
                bypassList: ["<local>"]
            }
        };
        chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
        function callbackFn(details) {
            return {
                authCredentials: {
                    username: "${username}",
                    password: "${password}"
                }
            };
        }
        chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ["blocking"]
        );
    """).substitute(host=host, port=port, username=username, password=password)

    tmpdir = tempfile.mkdtemp(prefix="chrome-proxy-")
    plugin_path = os.path.join(tmpdir, "proxyauth_plugin.zip")
    with zipfile.ZipFile(plugin_path, "w") as zp:
        zp.writestr("manifest.json", manifest_json)
        zp.writestr("background.js", background_js)
    return plugin_path

启动 Chrome 并加载扩展

def main():
    host = os.getenv("PROXY_HOST", "t.16yun.cn")
    port = os.getenv("PROXY_PORT", "31111")
    user = os.getenv("PROXY_USERNAME", "user")
    pwd = os.getenv("PROXY_PASSWORD", "password")

    options = Options()
    options.add_argument(f"--proxy-server=http://{host}:{port}")
    options.add_argument("--headless=new")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")

    # 加载代理认证扩展
    extension_path = create_proxy_extension(host, port, user, pwd)
    options.add_extension(extension_path)

    driver = webdriver.Chrome(options=options)

    try:
        driver.get(os.getenv("TARGET_URL", "http://httpbin.org/ip"))
        print(driver.page_source[:500])
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

使用 HTTP 目标（http://httpbin.org/ip）避免 HTTPS 证书问题。如需 HTTPS，可添加 --ignore-certificate-errors 参数。

页面操作与数据提取

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://example.com")

# 等待元素加载
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "content")))

# 提取文本
text = driver.find_element(By.TAG_NAME, "body").text

# 截图
driver.save_screenshot("page.png")

# 执行 JavaScript
ip = driver.execute_script("return document.body.innerText")

四种场景对照

Selenium 是浏览器级工具，强在页面交互和截图。HTTP 级别的 IP 控制（Proxy-Tunnel）需要通过其他工具（requests/httpx）配合实现。

错误排查

curl -x http://$PROXY_USERNAME:$PROXY_PASSWORD@$PROXY_HOST:$PROXY_PORT http://httpbin.org/ip