Introduction
Recently, both Alipay and Ctrip experienced issues preventing users from accessing their services normally. Although the causes of these incidents differed, they have sounded the alarm on operational security for internet companies. Consequently, disaster recovery (DR) capabilities, which were previously overlooked, have returned to the spotlight.
On the afternoon of May 27, Alipay suffered a large-scale outage, leaving many domestic users unable to transfer funds or make payments via both PC and mobile platforms. This incident lasted approximately two hours. Alipay stated that the cause was municipal construction work that severed an optical cable in a specific area of Xiaoshan District, Hangzhou, thereby affecting one of its primary data centers.
The following day, Ctrip faced a similar crisis. Starting from noon, Ctrip’s official website and app became inaccessible. More seriously, it took 12 hours to resolve the issue, sparking widespread speculation about Ctrip’s reliability. Ultimately, Ctrip attributed the incident to an employee error where execution code on the production server was accidentally deleted.
Such events are not new. In July 2013, a severed optical cable in Shanghai caused WeChat to malfunction in many regions for seven hours. Similarly, in October 2014, a failure in a Shanghai data center led to a nationwide WeChat outage lasting two hours.
Although disasters are low-probability events, their extensive impact and long duration can lead to significant user churn, severely degrade user experience, and cause substantial business losses.
Overview of Disaster Recovery
A disaster recovery solution involves establishing two or more systems with identical functionalities in the same city or different locations. These systems synchronize data in real-time and feature health monitoring and automatic failover capabilities. If one system fails due to an accident, the entire application can switch to another site to continue providing services.
Implementation Methods
Common implementation methods include “Active-Active Across Two Sites” or “Three Centers in Two Locations.” By leveraging various products from Alibaba Cloud or AWS, a primary site is established in one location with the main system deployed, while a cloud computing node in another location serves as the disaster recovery site. The same application system is deployed on cloud servers at both sites. Data synchronization is achieved through underlying data replication. The backup system is capable of taking over business operations at any time and can also support read-only workloads during normal operations. The specific failover process is tailored to the characteristics of your own business.
Conclusion
The Alipay incident was largely due to accidental external factors, making prevention difficult, whereas the Ctrip incident was entirely an internal issue. However, both incidents highlight deficiencies and loopholes in the disaster recovery practices of these two enterprises. Some businesses mistakenly believe that disaster recovery increases costs; in reality, if disaster recovery is not properly implemented, the losses incurred from an incident far outweigh the investment in DR measures.
It is crucial to prepare disaster recovery plans before systems go live, addressing risks proactively to keep them within the smallest possible scope.