Python pyppeteer & Selenium Firefox Tunnel Proxy: Browser Automation
pyppeteer and Selenium Firefox integrating 16Yun Crawler Proxy.
16Yun Engineering TeamMay 9, 20261 min read
pyppeteer
pyppeteer is the Python port of Puppeteer, handling proxies via the --proxy-server parameter and page.authenticate.
import asyncio, os
from pyppeteer import launch
host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")
async def main():
browser = await launch({
"headless": True,
"args": [
f"--proxy-server=http://{host}:{port}",
"--no-sandbox",
"--ignore-certificate-errors",
],
"ignoreHTTPSErrors": True,
})
page = await browser.newPage()
if user and pwd:
await page.authenticate({"username": user, "password": pwd})
await page.goto("https://httpbin.org/ip", {"waitUntil": "domcontentloaded", "timeout": 30000})
html = await page.content()
print(html[:500])
await browser.close()
asyncio.run(main())Selenium Firefox
Selenium Firefox configures proxies via set_preference:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager
host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", host)
profile.set_preference("network.proxy.http_port", int(port))
profile.set_preference("network.proxy.ssl", host)
profile.set_preference("network.proxy.ssl_port", int(port))
profile.set_preference("network.proxy.socks", host)
profile.set_preference("network.proxy.socks_port", int(port))
options = Options()
options.profile = profile
options.add_argument("--headless")
driver = webdriver.Firefox(
service=Service(executable_path=GeckoDriverManager().install()),
options=options,
)
driver.get("https://httpbin.org/ip")
print(driver.page_source[:500])
driver.quit()Browser Tool Proxy Comparison
| Feature | pyppeteer | Selenium Firefox | Selenium Chrome (previous) |
|---|---|---|---|
| Proxy Config | --proxy-server parameter | network.proxy.* preferences | Chrome extension injection |
| Auth Method | page.authenticate() | Firefox does not support HTTP proxy auth | Chrome extension |
| HTTPS Tunnel Support | No (browser limitation) | No | No |
| Use Case | Chromium automation | Firefox Gecko engine testing | Chrome automation |
Need an enterprise proxy plan?
We can tailor architecture to your target domains, concurrency, and reliability goals.