2020 — Single Points of Failure Are the Biggest Enemy of Stability

What Happened That Year

In 2020, proxy demand grew rapidly. User expectations shifted from "do not break often" to "never break." But many providers still ran single-server or small-cluster setups -- when one machine went down, an entire batch of users lost connection.

That year, we focused on one thing: eliminating single points of failure.

Why SPOFs Became Especially Important in 2020

2020 had a unique context -- the pandemic drove a surge in remote work and online business. Users relied more heavily on proxy services and had less tolerance for interruptions. An unstable proxy service could directly impact a company's remote work efficiency and data collection continuity.

At the same time, the proxy industry itself was changing rapidly. User numbers grew, traffic grew, and stability requirements became more demanding. In the early stages with fewer users, occasional faults could be mitigated through manual handling and apologies. But once the user base reached a certain scale, every fault affected a large number of users, and manual handling was no longer fast enough.

This is precisely why we invested heavily in high-availability improvements in 2020. Not because we suddenly had a new insight about stability, but because our user scale had reached a critical threshold -- without high-availability architecture, we could no longer maintain service quality.

Fundamentals of High-Availability Architecture

Before diving deeper, it is worth introducing the core concept of high-availability architecture: System availability = MTBF / (MTBF + MTTR), where MTBF is mean time between failures and MTTR is mean time to recovery.

This formula teaches us two things:

First, to improve availability, you either extend failure-free operation time (MTBF) or shorten recovery time (MTTR).

Second, for proxy services, MTTR has a greater impact on user perception than MTBF. A service that occasionally fails but recovers within seconds may be completely invisible to users. But a service that rarely fails but takes hours to recover each time -- users will notice every single time.

Our 2020 strategy was: without significantly reducing MTBF, reduce MTTR from minutes to seconds. This is why we invested heavily in automatic failover -- because manual handling measures time in minutes, while automatic failover measures time in seconds.

What Is a Single Point of Failure

A single point of failure (SPOF) is a component in the system whose failure causes the entire system to become unavailable. In proxy services, SPOFs take many forms:

• A proxy server fails due to hardware issues -- if you only have that one server, all users are affected
• An upstream resource pool becomes unresponsive due to API rate limiting -- if you only use one upstream, all dependent users go down
• A network egress goes down due to carrier issues -- if you only have that one egress, the entire service is interrupted
• DNS resolution fails due to cache pollution -- if your DNS resolver fails, users cannot resolve the proxy address

Every type of SPOF means service interruption under a non-redundant architecture. The solution is simple in principle: do not have any "single" dependency.

Our High-Availability Architecture Design

In 2020, we carried out a comprehensive high-availability overhaul of the proxy architecture. The core principle: redundancy at every layer.

Access layer. We added a load balancer between users and proxy servers. User requests first reach the load balancer, which distributes them to different proxy servers based on backend node health and load. If a proxy server goes down, the load balancer automatically routes traffic to other healthy nodes.

Proxy layer. Proxy servers themselves were deployed redundantly. Each region has multiple proxy server nodes that back each other up. When a node fails, other nodes immediately take over its traffic.

Resource layer. Upstream resource sources were expanded from single-source to multi-source. Each source has independent resource pools and access links. When one source encounters problems, the system automatically switches to other sources.

Monitoring layer. The health status of all nodes is continuously monitored. Monitoring data drives load balancing and automatic failover decisions.

Load Balancing Strategy

The load balancing strategy we adopted in 2020 evolved through several stages:

Stage one: Round-robin. The simplest strategy -- requests are distributed to backend nodes in sequence. Simple to implement, but does not consider actual node load or health. The result was that some nodes were already overloaded while requests kept being sent to them.

Stage two: Least connections. Requests are sent to the node with the fewest active connections. More intelligent than round-robin, but assumes all nodes have the same processing capacity. In reality, different nodes may have different performance and configurations.

Stage three: Weighted least connections. Distribution is weighted based on both node processing capacity and current load. Higher-performance nodes receive more requests, high-load nodes receive fewer. This was our final choice and the most effective.

Automatic Failover Mechanism

Automatic failover is the core capability of a high-availability architecture. The speed of failover directly determines how long users are affected.

We designed a three-tier failover mechanism:

Tier one: Single node failover. When a proxy node fails consecutive health checks, the system automatically marks it as offline and stops distributing new requests to it. Existing connections are either completed normally or migrated to other nodes. Failover time is controlled within 10 seconds.

Tier two: Resource pool failover. When an upstream source's availability drops below a threshold, the system automatically switches traffic to backup resource pools. This failover has a broader impact since it involves switching a large batch of nodes simultaneously. Failover time is controlled within 30 seconds.

Tier three: Regional failover. When an entire region experiences network problems (e.g., a data center power outage in a specific city), the system automatically switches traffic to nodes in other regions. This is the most extreme case, but having a plan is essential.

Challenges of High-Availability Architecture

Although the concept of high-availability architecture sounds simple -- "add redundancy, automate failover" -- implementation presented several challenges:

State synchronization. When multiple proxy nodes run simultaneously, they need to share state information. If state is not synchronized, the same IP could be assigned to different users, or the load balancer could make decisions based on outdated data.

Smooth traffic migration. When switching from a failed node to a healthy one, overly aggressive switching can overwhelm the healthy node with a sudden influx of traffic. We adopted a "slow start" strategy -- new nodes initially accept only a small number of requests after switching, then gradually ramp up.

Data consistency. Core data maintains strong consistency, while operational data allows eventual consistency.

Health Check Mechanism

Health checks are the foundation of failover. Without accurate health checks, automatic failover has no basis for decisions.

Our health check system consists of several components:

Active probing. A probe request is sent to each node every 5 seconds, measuring response time and success rate. If 3 consecutive probes fail, the node is marked as "suspected abnormal." If 6 consecutive probes fail, the node is marked as "failed."

Passive detection. Beyond active probing, the system also analyzes user request failure rates. If a node's user request failure rate suddenly rises -- even if active probing shows normal results -- the system initiates investigation. Some faults only affect real user traffic without impacting probe requests.

Correlation analysis. When multiple nodes show anomalies simultaneously, the system performs correlation analysis to determine whether these are individual node issues or a global problem (such as resource pool or network link failure).

How to Judge Proxy Stability (Part 4): Node Redundancy

In 2020, we advised users to evaluate a provider's node redundancy strategy:

• Is the proxy service single-node or multi-node?
• How are failed nodes detected?
• After a failure occurs, how is traffic switched to healthy nodes?
• How long does failover take? Seconds, minutes, or hours?

A simple test method: shut down one proxy node and see if your business still runs normally.

Results of High-Availability Improvements

After completing the 2020 high-availability overhaul, service stability improved significantly. Key metric changes:

• Average fault recovery time dropped from approximately 3-5 minutes in 2019 to approximately 30 seconds
• Service interruptions caused by single points of failure dropped from multiple occurrences in 2019 to nearly zero
• User-perceptible "outages" essentially disappeared -- even when backend nodes were switching, the frontend experience was nearly seamless

What We Built That Year

Technology upgrades completed in 2020:

• Introduced intelligent load balancing that dynamically distributes traffic based on node health and load
• Implemented a three-tier automatic failover mechanism
• Upgraded the entire architecture to a fully redundant design with no single points of failure on any critical path
• Shortened health check intervals to the second level
• Established correlation analysis capability to distinguish between local and global faults

Real-World Fault Cases

During the 2020 high-availability overhaul, we experienced several typical faults that were important for understanding single points of failure:

Case one: Cloud provider regional failure. A cloud provider's availability zone went down due to a power fault, affecting nodes deployed in that region. Because we had cross-region redundant deployment, the fault was automatically switched to other regions. Users barely noticed -- only the monitoring system recorded the switch. With single-region deployment, this fault would have caused widespread service interruption.

Case two: Upstream API rate limiting. An upstream resource source triggered API rate limiting due to traffic growth, causing a large batch of proxy nodes from that source to become unavailable. The automatic failover mechanism detected the availability drop and switched traffic to other sources. If we had relied on only this one source, the rate limiting would have directly caused service interruption.

Case three: Load balancer failure itself. If the load balancer itself is a single point, it becomes a new SPOF. We made the load balancer layer redundant too -- two load balancers in active-standby mode, with the standby automatically taking over if the primary fails. Load balancer redundancy is often overlooked but critically important.

These cases illustrate an important principle: In system design, when you eliminate one single point of failure, watch out for new ones emerging. For example, you eliminate server-level SPOF but introduce a load balancer. If there is only one load balancer, it becomes the new single point. Every layer needs redundancy. Every layer needs to answer the question: "What happens if this component fails?"

Cost Analysis of Single Points of Failure

When undertaking high-availability improvements, we needed to balance investment and return. Not every SPOF needs to be eliminated -- some have a very low probability of failure and limited impact, making large resource investments uneconomical.

But the cost calculation for SPOFs in proxy services is unique:

• Direct costs: User churn and refunds caused by service interruption
• Indirect costs: Decreased user trust, damaged brand reputation
• Opportunity costs: Users switching to other providers after a single outage

In our 2020 competitive analysis, we found that users ranked provider selection factors as: stability > price > features > service. Stability came first. This reinforced our determination to invest in high availability.

Failover Timeline

In 2020, we optimized the complete failover timeline, ensuring clear expected timing for each stage:

The foundation of this timeline is that every stage is automated. Any stage requiring manual intervention would significantly extend recovery time.

Year-End Reflection

In 2020, we accomplished a lot technically, but the biggest takeaway was not technical. It was a consensus the team formed during the year: Stability is designed, not operated.

This means: you cannot expect the operations team to compensate for architectural deficiencies by handling faults quickly after they occur. Truly effective stability assurance starts at the design stage -- planning not just for normal operation, but also for fault tolerance and recovery under abnormal conditions.

The 2020 high-availability overhaul was the first step in putting this principle into practice. Every subsequent year would continue to build on this foundation.

One Piece of Advice

When evaluating whether a proxy service is stable, start with this question: How many nodes do you have, and what happens when one fails? A provider that can clearly answer this question truly cares about stability. If a provider says "we have redundancy" but cannot specifically describe the redundancy layers and failover timing, that redundancy likely exists only on the marketing page and has never been proven in production.