Alibaba Cloud was among the companies impacted by a fire that broke out at one of Digital Realty’s three data centres in Singapore on the morning of Tuesday, September 10.
According to a notice on the Alibaba Cloud website, the company detected network access anomalies in Availability Zone C of the Singapore region by 10:20 AM Beijing time, causing interruptions to cloud services.
In a statement, Digital Realty said it “became aware of a fire alarm being triggered at [its] SIN11 data centre at 7:45 AM on 10th September 2024.” By 8:15 AM, all onsite personnel at its 3 Loyang Way facility had been safely evacuated, with no casualties reported, the company noted.
“The Singapore Civil Defence Force (SCDF) responded to the incident and is working to secure the premises and stabilise the situation,” the statement further read.
Digital Realty also stated that it notified impacted customers to activate their business continuity plans.
In a follow-up report, Alibaba Cloud said an explosion of lithium batteries in the Singapore data centre caused the fire. “Most of the cloud network products and cloud security products completed the proactive switch by 10:55 AM. Other cloud product services are still being processed. Please migrate your production workloads as soon as possible,” Alibaba Cloud told customers.
By midday, the high availability versions of cloud products such as Cloud Database Redis/MongoDB/RDS MySQL, OSS, OTS, and MaxCompute, had successfully completed disaster recovery, Alibaba Cloud said.
Around 2:20 PM, the temperature in the data centre remained high despite the fire situation being brought under control. By 8:04 PM, the fire alarm had not been fully cleared.
“Due to fire safety control, operation and maintenance engineers are unable to enter the building in which the data centre is located, and the risk of continuously rising temperature in the data centre cannot be eliminated for the time being. Currently, some network equipment in the data centre has experienced abnormalities in high-temperature environments, affecting the network connectivity of some cloud products. If the subsequent temperature rise is not effectively controlled, the possibility of a complete network outage in Availability Zone C of Singapore will increase. If your business is deployed in Availability Zone C of Singapore, we will assist you in migrating your business as soon as possible,” Alibaba Cloud said.
At around 1:45 AM of September 11, Digital Realty shut down some electrical systems, “following advice from Digital Realty’s Licensed Electrical Worker (LEW) and the SCDF.” The company added that it had advised all impacted customers to “continue to maintain their business continuity plans in effect.”
After a preliminary evaluation of the building, the Building and Construction Authority (BCA) has identified minor structural damage, and confirmed that its overall structural integrity is not affected, Digital Realty announced in a September 12 follow-up statement.
“As a precautionary measure, the BCA has issued a Dangerous Building Order (DBO) and a Closure Order (CO) for specific areas of the facility. Digital Realty is complying fully with these orders and has commissioned a Professional Engineer to conduct a full assessment of the building’s structure,” the company said.
* Article updated September 13, 2024, 10:33 AM to reflect Digital Realty’s latest statement.
