Python pyppeteer & Selenium Firefox Tunnel Proxy: Browser Automation

pyppeteer and Selenium Firefox integrating 16Yun Crawler Proxy.

16Yun Engineering TeamMay 9, 20261 min read

pyppeteer

pyppeteer is the Python port of Puppeteer, handling proxies via the --proxy-server parameter and page.authenticate.

import asyncio, os
from pyppeteer import launch
 
host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")
 
async def main():
    browser = await launch({
        "headless": True,
        "args": [
            f"--proxy-server=http://{host}:{port}",
            "--no-sandbox",
            "--ignore-certificate-errors",
        ],
        "ignoreHTTPSErrors": True,
    })
    page = await browser.newPage()
    if user and pwd:
        await page.authenticate({"username": user, "password": pwd})
 
    await page.goto("https://httpbin.org/ip", {"waitUntil": "domcontentloaded", "timeout": 30000})
    html = await page.content()
    print(html[:500])
    await browser.close()
 
asyncio.run(main())

Selenium Firefox

Selenium Firefox configures proxies via set_preference:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager
 
host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")
 
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", host)
profile.set_preference("network.proxy.http_port", int(port))
profile.set_preference("network.proxy.ssl", host)
profile.set_preference("network.proxy.ssl_port", int(port))
profile.set_preference("network.proxy.socks", host)
profile.set_preference("network.proxy.socks_port", int(port))
 
options = Options()
options.profile = profile
options.add_argument("--headless")
 
driver = webdriver.Firefox(
    service=Service(executable_path=GeckoDriverManager().install()),
    options=options,
)
 
driver.get("https://httpbin.org/ip")
print(driver.page_source[:500])
driver.quit()

Browser Tool Proxy Comparison

FeaturepyppeteerSelenium FirefoxSelenium Chrome (previous)
Proxy Config--proxy-server parameternetwork.proxy.* preferencesChrome extension injection
Auth Methodpage.authenticate()Firefox does not support HTTP proxy authChrome extension
HTTPS Tunnel SupportNo (browser limitation)NoNo
Use CaseChromium automationFirefox Gecko engine testingChrome automation

Need an enterprise proxy plan?

We can tailor architecture to your target domains, concurrency, and reliability goals.