Python requests Tunnel Proxy: Four IP Control Scenarios for Crawler Proxy

Four IP Control Scenarios

16Yun's Crawler Proxy (tunnel mode) supports four IP usage patterns:

Setup

export PROXY_HOST=t.16yun.cn
export PROXY_PORT=31111
export PROXY_USERNAME=your-username
export PROXY_PASSWORD=your-password

Verify Proxy

curl -x http://$PROXY_USERNAME:$PROXY_PASSWORD@$PROXY_HOST:$PROXY_PORT https://httpbin.org/ip

Scenario A: Force New IP

Each request creates a new connection, forcing a different exit IP.

import os, requests

host = os.getenv("PROXY_HOST", "t.16yun.cn")
port = os.getenv("PROXY_PORT", "31111")
user = os.getenv("PROXY_USERNAME", "user")
pwd = os.getenv("PROXY_PASSWORD", "password")

proxies = {"http": f"http://{user}:{pwd}@{host}:{port}",
           "https": f"http://{user}:{pwd}@{host}:{port}"}

for i in range(1, 4):
    resp = requests.get("https://httpbin.org/ip", proxies=proxies,
        headers={"Connection":"close","Proxy-Connection":"close"}, timeout=15)
    print(f"Request {i}: {resp.json()['origin']}")

Scenario B: Keep-Alive Same IP

Reuse the same requests.Session — requests stay on the same exit IP within the proxy session.

session = requests.Session()
headers = {"Connection":"keep-alive","Proxy-Connection":"keep-alive"}

for i in range(1, 4):
    resp = session.get("https://httpbin.org/ip", proxies=proxies,
        headers=headers, timeout=15)
    print(f"Request {i}: {resp.json()['origin']}")
session.close()

Scenario C-HTTP: Proxy-Tunnel for HTTP

Add a Proxy-Tunnel header. Same value = same exit IP, different value = different IP.

import random
tunnel = os.getenv("PROXY_TUNNEL") or str(random.randint(1, 10000))

session = requests.Session()
for i in range(1, 4):
    resp = session.get("http://httpbin.org/ip", proxies=proxies,
        headers={"Proxy-Tunnel":tunnel,"Connection":"keep-alive"}, timeout=15)
    print(f"Request {i}: {resp.json()['origin']}")

Scenario C-HTTPS: Proxy-Tunnel for HTTPS

requests does NOT send custom headers during HTTPS CONNECT by default. Override HTTPAdapter.proxy_headers():

from requests.adapters import HTTPAdapter

class TunnelAdapter(HTTPAdapter):
    def __init__(self, tunnel_value=None, *args, **kwargs):
        self.tunnel_value = tunnel_value
        super().__init__(*args, **kwargs)

    def proxy_headers(self, proxy):
        headers = super().proxy_headers(proxy)
        if self.tunnel_value:
            headers["Proxy-Tunnel"] = self.tunnel_value
        return headers

tunnel = os.getenv("PROXY_TUNNEL") or str(random.randint(1, 10000))

session = requests.Session()
adapter = TunnelAdapter(tunnel_value=tunnel)
session.mount("https://", adapter)

for i in range(1, 4):
    resp = session.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
    print(f"Request {i}: {resp.json()['origin']} (HTTPS Proxy-Tunnel)")

Scenario Comparison

Troubleshooting

curl -x http://$PROXY_USERNAME:$PROXY_PASSWORD@$PROXY_HOST:$PROXY_PORT https://httpbin.org/ip