Amazon Cloud Outage Exposes Corporate Risk of Vendor Dependency

Amazon Web Services’ outage last week served as a costly reminder there’s often little companies can do to mitigate risk when a major provider they rely on shuts down.

At least hundreds of companies, including Robinhood Markets Inc., Salesforce Inc., and Epic Games Inc., suffered disruptions and lost business when the world’s largest cloud provider suffered a 15-hour outage on Oct. 20.

The crash is the latest example of the fragility of digital supply chains, coming only a year after a faulty update in CrowdStrike Holdings Inc.’s software caused global computer outages that halted air travel and paralyzed banks and businesses across the world.

Amazon’s IT freeze also shows inevitable risks created as companies turn to a long list of third-party vendors to support their operations. The interconnectedness of all those vendors means even contractual agreements, insurance policies, and technical remedies offer limited relief from internet-wide incidents.

“Our dependencies have been moving that way,” said Bob Maley, chief security officer at Black Kite, a cyber risk management provider. “We’re only as good as one tiny little failure somewhere in that complex world of technology to prevent you from getting into your bank, to prevent you from playing your game or getting on your communications device,” he added.

AWS in a statement defended its track record. “Customers choose AWS because we provide the most innovative, secure, and scalable services,” a spokesperson said.

No Way to Vet

Few security professionals were surprised by the scope of the outage. The widespread disruptions were due, in part, to having only a few cloud providers currently underpinning the internet.

AWS alone supports about a third of the cloud market.

“It’s definitely a wake-up call for folks who are assuming that very large infrastructure providers are immune,” said Yogesh Badwe, chief security officer at Druva, a security solutions provider.

As businesses rely on more and more vendors to offer services or support their payroll, HR, or security functions, there isn’t an easy way for them to truly assess risks across the supply chain.

“It’s not just about third parties. There is a cascading dependency of vendors on each other,” said Badwe, who previously oversaw security at Okta Inc. and Salesforce.

Contractual agreements with vendors can help companies obtain some guarantees from the providers they work with—whether it’s assurances they follow strict security standards or liability protections in case something goes wrong.

“I use the contractual abilities that we have to their fullest extent, because ultimately, the contract is a way for us to get reparations after an incident,” said Aimee Cardwell, chief information security officer in residence at Transcend, which offers data governance solutions. “But I also don’t rely on anything that a vendor said as the truth.”

Contractual terms with giants such as Amazon may offer limited, if any, relief to smaller companies.

“When you’re looking at AWS, it’s the gorilla in the room,” said Melissa Ventrone, leader of Clark Hill’s cyber, data protection, and privacy practice. “So your ability to negotiate a higher threshold is probably pretty limited, unless you’re also a giant company.”

Resiliency Options

Companies may have to find resiliency in other ways, security professionals said.

Businesses can rely on two vendors providing the same service to have a back-up ready in case of an incident. They can invest in prevention, detection, and response services. They should test out incident response plans and audit their vendors when possible.

In some cases, the best thing to do during a vendor outage may be to wait it out.

“It’s a risk perspective, right? If the outage is more than 12 hours, then we would need to fail over” to a back-up system, said Cardwell, former CISO at UnitedHealth Group Inc. “But if the outage is four hours, is it worth the money? I don’t know. Your company needs to make that risk calculation.”

Companies may instead choose to invest more heavily in the resilience of applications critical to their operations. A health-care entity, for example, may prioritize ensuring patients can access health services over keeping their billing function online.

“There are a whole bunch of companies who had that sort of application resilience, and they weren’t screaming,” Cardwell said.

The ‘Near-Miss’

Insurance, the traditional way companies can offset the cost of unforeseen incidents, serves a purpose, but it’s limited.

“We realize that there’s going to be technology failures, and there’s going to be widespread ramifications,” said Matthew McCabe, the US and Canada cyber coverage leader at insurance broker Marsh. “What cyber insurance in its most conventional aspect does is pick up the risk transfer for the financial impact of that.”

Large-scale incidents such as last week’s outage mean that supply chain risk is “more of a conversation now than ever” for companies buying cyber insurance, said Alex Clark, cyber solutions practice leader at Hylant, another broker.

A soft cyber insurance market has resulted in policy language that largely favors policyholders, industry professionals said. Despite widespread concern about the potential impact of a systemic incident, generally cyber policies contain few restrictions on coverage for such risk because insurers are hesitant to narrow policy terms in a competitive marketplace.

However, policies typically impose a “waiting period” before coverage kicks in for business interruption losses resulting from a third-party incident, which may limit coverage. Whether losses from the AWS outage will be picked up by cyber insurers will vary across policies.

Cyber risk analytics provider CyberCube in a preliminary estimate said losses from the outage could range from $38 million to $581 million. It said the incident would likely only have a moderate impact on the insurance industry due to a quick recovery and potential reimbursements by Amazon for the losses.

“I don’t think it’s going to be the white whale event we’ve been expecting for the last couple years,” said Jason Curreri, general counsel and head of product at managing general agent Elpha Secure. “I think it’s another near miss.”