Issue Playbook: Support for socks5 proxy in Session Consistency and

Context and Problem Definition

This article targets one concrete production failure: Session inconsistency causing short-cycle blocking. It is not broad guidance; it is an executable Scrapy fix path with clear acceptance and rollback boundaries.

Typical symptom: After login, captcha spikes within 10-30 minutes because the same account drifts on IP/UA/Cookie. The root cause is usually policy-level inconsistency, not just proxy quality.

Issue signals used for this exact problem:

scrapy/scrapy#747: Support for socks5 proxy (comments: 54)
scrapy/scrapy#7060: Fix flaky test_download_with_proxy_https_timeout() (comments: 25)
scrapy/scrapy#4821: DOWNLOADER_CLIENT_TLS_METHOD only supports TLS 1.2 and lower (comments: 21)

External evidence supplements (only when local evidence has gaps):

No external evidence supplement (no gap)

Insight Framework

This is an identity-state-machine problem, not a single-parameter problem.
Most blocks come from identity drift, not just burst concurrency.
Session policy must be centralized in middleware, not fragmented in spiders.

Method Path

Define a hard binding key for account, UA family, and cookie jar.
Apply domain-tier sticky TTL instead of per-request rotation.
Split 403 and 429 policy: rotate identity for 403, backoff for 429.
Validate with session lifetime, captcha ratio, and block ratio.

Architecture and Data Flow

Scheduler -> SessionKeyBuilder -> StickySessionPool
         -> Downloader Middleware -> Target
                   |                    |
                   v                    v
             Fingerprint Store     Risk Classifier
                   |                    |
                   +-----> Rotate Controller

Operational constraints:

Do not switch UA family inside the same session_key.
Cookie jars move only with session lifecycle, never cross-job reuse.
Every rotation must emit audit logs with old/new IP and trigger reason.

Configuration Matrix

Config	Recommended Value	Why	Bad Pattern
`SESSION_STICKY_SECONDS`	180	keep login identity stable	rotate proxy every request
`SESSION_MAX_ERRORS`	3	force rotate after consecutive failures	infinite retries on same identity
`COOKIE_JAR_PARTITION`	account+domain	prevent cross-account contamination	single global cookie jar
`UA_FAMILY_LOCK`	true	limit fingerprint drift	random UA on each request
`ROTATE_ON_403`	true	403 means identity is recognized	keep retrying same identity
`ROTATE_ON_429`	false	backoff first for 429	immediate rotate for every 429

Key Code Snippets

# middleware/session_key.py
from hashlib import sha1

def build_session_key(account_id: str, domain: str, ua_family: str) -> str:
    raw = f"{account_id}|{domain}|{ua_family}"
    return sha1(raw.encode("utf-8")).hexdigest()[:16]

# middleware/session_consistency.py
class SessionConsistencyMiddleware:
    def process_request(self, request, spider):
        account_id = request.meta["account_id"]
        domain = request.url.split("/")[2]
        ua_family = spider.ua_pool.family_for(account_id)

        session_key = build_session_key(account_id, domain, ua_family)
        session = spider.session_pool.pick(session_key, sticky_seconds=180)

        request.meta["session_key"] = session_key
        request.meta["proxy"] = session.proxy
        request.headers["User-Agent"] = session.user_agent
        request.cookies.update(session.cookies)

# middleware/risk_policy.py
def handle_response(response, request, spider):
    status = response.status
    if status == 403:
        spider.session_pool.force_rotate(request.meta["session_key"], reason="403")
    elif status == 429:
        spider.backoff.schedule(request, reason="429")
    return response

Failure Cases and Troubleshooting

Failure scenario: Login and detail flows shared one cookie jar but used different UA pools, causing frequent session signature changes.

Troubleshooting sequence:

Sample 200 requests and verify proxy/UA stability per session_key.
Check whether UA family changed within 20 seconds before 403.
Verify forced rotation triggers on 403 only, not on every 429.
Compare captcha_ratio and session_lifetime_p95 before/after fix.

Performance Metrics and Load Testing

Load tests should cover baseline, peak, and anti-bot escalation profiles.

Acceptance thresholds:

success_rate >= 92%
captcha_ratio <= 3%
session_lifetime_p95 >= 12 minutes
403_ratio <= 4%

Vendor Comparison and 16Yun Positioning

Only issue-relevant capabilities are kept here:

Crawler Tunnel Proxy: 跨IDC架构, 毫秒级检测, IP自动切换
Dynamic Residential Proxy: 住宅IP轮换, 全球覆盖, 城市级定位
Static Residential Proxy: 长期固定IP, 城市级定位, 住宅网络可信度

For this issue, controllable stickiness and rotation matter more than raw pool size; 16Yun tunnel plus dynamic residential combo is better aligned.

Rollout Checklist

single policy entry implemented in middleware
key configs split by priority and environment
regression load test passed all thresholds
rollback can be executed within 10 minutes
alerts set for 429/403/latency thresholds
change audit log recorded for this rollout