proxy.py: Build a Custom Forwarding Gateway for Crawlers

A forwarding gateway can centralize a request rule that several collectors would otherwise implement independently. proxy.py provides Python plugins for this purpose.

This tutorial starts a local gateway, adds a task header only for an explicit test host, and checks what the destination receives. It modifies ordinary HTTP requests and does not enable TLS interception.

Install and establish basic forwarding

Use Python 3.12 and Bash:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install proxy.py
python -m pip show proxy.py
proxy --hostname 127.0.0.1 --port 8899

Record the installed version. Plugin interfaces can change, so an upgrade requires checking the methods used by your plugin. See the official project.

Keep it running and use another terminal:

curl --max-time 20 --noproxy ''   --proxy http://localhost:8899 http://httpbin.org/headers

Verify a header-echo JSON response. If the network blocks ordinary HTTP, use an HTTP echo endpoint you control. Do not silently switch to HTTPS and assume the later header rule remains equivalent.

Stop this basic instance before starting the plugin instance on the same port.

Implement a scoped rule

Create gateway.py:

import logging
from typing import Optional
from proxy.http.parser import HttpParser
from proxy.http.proxy import HttpProxyBasePlugin
 
logger = logging.getLogger(__name__)
 
class JobHeaderPlugin(HttpProxyBasePlugin):
    def before_upstream_connection(self, request: HttpParser) -> Optional[HttpParser]:
        return request
 
    def handle_client_request(self, request: HttpParser) -> Optional[HttpParser]:
        if request.host and request.host.lower() == b"httpbin.org":
            request.add_header(b"X-Collection-Job", b"docs-demo")
            logger.info("applied job header to host=%r", request.host)
        return request

Launch it with the current directory available for module imports:

PYTHONPATH="$PWD" proxy --hostname 127.0.0.1 --port 8899   --plugins gateway.JobHeaderPlugin

The plugin argument is a module and class name, not a file path. The handler returns the request to continue forwarding. Header arguments are bytes, matching the parser API.

The rule targets only httpbin.org. Sending internal task markers to every destination is unnecessary. Real applications should define allowed hosts and task identifiers deliberately.

Verify the destination's observation

Repeat the curl request and find X-Collection-Job with value docs-demo in the returned headers. Header-name casing may be normalized by the echo service.

This checks the client route, plugin loading, modification, and destination observation together. A gateway log alone does not establish that the destination received the header.

When replacing the echo host, update the plugin's host condition too. Forwarding can work even when the rule does not match.

Understand the TLS boundary

An ordinary HTTP proxy can inspect a plaintext request. HTTPS commonly uses CONNECT to establish a tunnel; the destination request headers remain inside TLS.

Changing the test URL to HTTPS therefore does not establish that this plugin can add a header to the encrypted destination request. For business headers, client-side configuration is often simpler.

TLS interception has separate certificate, trust, and maintenance requirements. Supporting an HTTPS tunnel is different from modifying HTTPS content.

Connect collectors deliberately

Configure each supporting client with the gateway address. Environment-variable support varies across libraries; verify the route rather than assuming HTTP_PROXY is always honored.

Across containers, localhost identifies the current container, not the gateway's container. Define the service address and permitted listener scope before sharing it across machines.

The example logs only the matching host, not paths, authorization, or bodies. More detailed diagnostics may expose credentials or query data and need appropriate handling.

Troubleshoot

Module not found: confirm the file, PYTHONPATH, working directory, and environment used to launch the installed package.

Plugin loads but no header appears: check the host condition, plaintext HTTP, and whether the client bypassed the proxy. Establish the provided curl baseline before changing the collector.

Connection refused: inspect the listener, port ownership, and network namespace before debugging plugin logic.

HTTP 200 contains an unexpected page: inspect the response type and echo fields rather than trusting status alone.

Custom plugin flags are ignored: inspect the installed version's parameter registration and help. This minimal example avoids custom flags to keep the main path clear.

The port remains occupied: stop the actual gateway instance through its terminal or process manager. Individual collectors should not arbitrarily shut down a shared gateway.

A forwarding layer is justified when centralized rules provide value. Host scope, TLS limits, and destination-side checks keep that value understandable.