Python urllib Tunnel Proxy: Standard Library Four Scenarios
Python standard library urllib implements four tunnel proxy scenarios with custom HTTPSConnection for HTTPS Proxy-Tunnel.
16Yun Engineering TeamMay 13, 20261 min read
urllib: No Third-Party Dependencies
urllib is a Python standard library module requiring no third-party packages for proxy support. By overriding the HTTPSConnection.set_tunnel method, you can inject Proxy-Tunnel headers during CONNECT to implement HTTPS Proxy-Tunnel.
Scenario A: Force IP Switch
import urllib.request, os, ssl, time
host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")
proxy_url = f"http://{user}:{pwd}@{host}:{port}"
ctx = ssl.create_default_context()
for i in range(1, 4):
handler = urllib.request.ProxyHandler({"http": proxy_url, "https": proxy_url})
https_handler = urllib.request.HTTPSHandler(context=ctx)
opener = urllib.request.build_opener(handler, https_handler)
req = urllib.request.Request("https://httpbin.org/ip",
headers={"Connection": "close", "Proxy-Connection": "close"})
with opener.open(req, timeout=15) as resp:
print(f"Request {i}: {resp.read().decode()}")
time.sleep(0.5)Scenario B: Keep-Alive
handler = urllib.request.ProxyHandler({"http": proxy_url, "https": proxy_url})
opener = urllib.request.build_opener(handler, urllib.request.HTTPSHandler(context=ctx))
for i in range(1, 4):
req = urllib.request.Request("https://httpbin.org/ip",
headers={"Connection": "keep-alive", "Proxy-Connection": "keep-alive"})
with opener.open(req, timeout=15) as resp:
print(f"Request {i}: {resp.read().decode()}")
time.sleep(0.2)Scenario C-HTTPS: Proxy-Tunnel (Custom HTTPSConnection)
urllib requires overriding the set_tunnel method to inject CONNECT headers for HTTPS targets:
import http.client, base64, json
class TunnelHTTPSConnection(http.client.HTTPSConnection):
def set_tunnel(self, host, port=None, headers=None):
if headers is None:
headers = {}
headers["Proxy-Tunnel"] = os.getenv("PROXY_TUNNEL", "urllib-demo")
super().set_tunnel(host, port, headers)
proxy_url = f"http://{user}:{pwd}@{host}:{port}"
conn = TunnelHTTPSConnection(host, int(port), context=ctx)
auth = base64.b64encode(f"{user}:{pwd}".encode()).decode()
conn.set_tunnel("httpbin.org", 443, headers={"Proxy-Authorization": f"Basic {auth}"})
conn.request("GET", "/ip")
resp = conn.getresponse()
print(resp.read().decode())Four Scenarios Comparison
| Scenario | urllib Implementation | Key API |
|---|---|---|
| A | Create new opener each time | ProxyHandler + Connection: close |
| B | Reuse opener | Same build_opener instance |
| C-HTTP | Request header Proxy-Tunnel | Request headers |
| C-HTTPS | Custom set_tunnel | HTTPSConnection.set_tunnel |
For a simpler HTTPS Proxy-Tunnel approach, use
httpx.Proxy(headers=...)or a customrequests.HTTPAdapter.proxy_headers().
Need an enterprise proxy plan?
We can tailor architecture to your target domains, concurrency, and reliability goals.