GOST v3: Share a Proxy Gateway across Crawlers

Maintaining the same upstream settings across Python, Node, and browser clients makes switching and diagnosis harder. GOST can expose shared client-facing listeners and forward their traffic to a chosen upstream.

This tutorial builds HTTP and SOCKS5 listeners over one HTTP upstream, checks both paths, and saves the resulting configuration. It covers GOST v3 forward proxying, not transparent routing.

Prepare the image and upstream

Use Docker and an upstream you are authorized to use:

docker pull gogost/gost
docker run --rm gogost/gost -V

The default image tag changes over time. Verify that the reported release is the intended v3 series and retain its image identity for deployment. Native binaries are also available from official releases.

Replace this format example with working upstream credentials:

export UPSTREAM_URL='http://user:pass@proxy.16yun.cn:8888'

Encode special credential characters as URL components. The local listener option and forwarding-node option serve different roles. Multiple forwarding nodes can describe sequential hops, not automatically a random pool. See the v3 CLI reference.

Start both listeners

docker run --rm --name crawler-gateway   -p 127.0.0.1:18080:8080   -p 127.0.0.1:11080:1080   gogost/gost   -L 'http://:8080'   -L 'socks5://:1080'   -F "$UPSTREAM_URL"

Container ports 8080 and 1080 map to host ports 18080 and 11080. The host bindings are loopback-only.

From another terminal:

curl --max-time 20 --noproxy ''   --proxy http://localhost:18080 https://httpbin.org/ip
curl --max-time 20 --noproxy ''   --proxy socks5h://localhost:11080 https://httpbin.org/ip

The SOCKS hostname variant asks the proxy side to resolve the destination. An HTTP proxy can also tunnel an HTTPS destination using CONNECT; the listener protocol and destination scheme are distinct.

Export and reuse a configuration

Stop the initial instance, then export its configuration:

docker run --rm gogost/gost   -L 'http://:8080' -L 'socks5://:1080'   -F "$UPSTREAM_URL" -O yaml > gost.yaml
chmod 600 gost.yaml

Inspect services, chains, hops, and upstream authentication. The exported file can contain credentials and should remain a private local configuration.

Start from that file:

docker run --rm --name crawler-gateway   -p 127.0.0.1:18080:8080   -p 127.0.0.1:11080:1080   -v "$PWD/gost.yaml:/etc/gost.yaml:ro"   gogost/gost -C /etc/gost.yaml

Repeat both checks. Exporting avoids guessing YAML structure, but still requires checking the intended behavior. See the file reference.

Do not retain conflicting command-line forwarding options while expecting file edits to control everything. Inspect startup arguments and effective behavior after changes.

Connect host and container consumers

Host clients use the published host addresses. A different container's localhost does not reach the gateway.

For container-to-container access, create a dedicated network:

docker network create crawler-net

Add the gateway's startup option --network crawler-net, and attach its consumers to that network. Configure the HTTP proxy host as crawler-gateway and port as 8080, using the service name and internal port rather than the host's 18080 mapping.

Limit network membership and configure listener authentication according to the sharing arrangement. Upstream authentication does not automatically authorize clients at the local listener.

Interpret checks and switch upstreams

The echo response should contain an origin. Two listeners need not produce the same IP when the upstream itself rotates. The gateway does not automatically create sticky egress.

Before switching, preserve the old configuration. Restart with the new upstream, repeat connectivity checks, then verify the actual collection page. A running container proves process liveness, not correct target data.

If a task binds login state to its exit environment, do not change its upstream midway without an explicit session transition.

Troubleshoot

Host connection fails: check process startup, port publishing, and conflicting instances before investigating credentials.

Only one protocol works: verify client protocol support, listener port, and intended DNS behavior.

Upstream authentication fails: inspect the forwarding-node credentials, not an unrelated client-listener setting.

Configuration edits have no effect: check the mounted path, restart, and command-line overrides.

The proxy works but data is wrong: inspect page identity, session, market, and extraction. Routing cannot repair business fields.

Stop or roll back: use the owning terminal or docker stop crawler-gateway. Restore the saved configuration and repeat connectivity plus business checks without leaving competing listeners.

A shared gateway reduces repeated client configuration while keeping routing, sessions, and data correctness explicit.