How Certificate Expiration Led to Some of the Largest IT System Failures of the Last Two Years

By Tim Callan, Senior Fellow, Sectigo

Our modern IT landscape depends fundamentally on digital certificates. Certificates are nearly ubiquitous in contemporary computing systems and permeate every aspect of our digital lives. They are essential to the secure functioning of our business processes, communication, retail purchasing, utilities, transportation systems, personal electronics, and so much more. Virtually no digital processor device would securely operate without the use of certificates.

Each certificate authenticates the identity of a machine, device, or software operation to ensure that only the intended connections are occurring, and most systems won’t enable encryption unless certificates are available. This latter fact is because encryption on its own does not constitute protection if the encrypted information might wind up in the hands of the wrong party.

Certificates must be issued by a Certificate Authority(or CA), which is the trusted authority for identity on that particular network. For internal uses like IoT networks or enterprise device certificates, the company that owns the devices can be the Certificate Authority. But for the public internet (including use for web sites, server-to-server connections, or email) certificates need to come from a public CA that has roots universally trusted by the systems on the internet.

With so much depending on certificates, it may not be surprising that an unexpected expiration can cause an application to stop working or security to lapse. in fact, it was revealed in December that the expiration of two certificates disrupted the lives of hundreds of millions of people. Early in the month, mobile service outage for tens of millions of customers using O2, Softbank, and other services ultimately owed itself to the expiration of a certificate that was part of the backend data service Ericsson provided to mobile service providers around the world. And then the following week, the House Oversight Committee released its report on 2017’s Equifax data breach.

December’s mobile outage affected carriers in eleven countries for as long as a day. The consequences of the carriers were huge. O2 gave all affected customers a credit worth two days of their data plans. Softbank experienced this outage a day before its IPO– a tremendous black eye exactly when the technology giant was looking for investor confidence. And it is reported that O2 could penalize Ericsson up to $100 million for failure to meet its SLA.

The Equifax breach, among the largest thefts of PII in history, involved the loss of 148  million people’s data; nearly 45% of the US population. The thieves sat inside Equifax’s infrastructure and harvested data on 265 occasions during the course of roughly 70 days. This extended attack was possible due to a lapse in service from a data exfiltration monitoring tool set up to guard against this kind of theft, and that tool’s failure to operate owed itself to an expired certificate. It turns out this service was offline for an astounding 19 months and that the certificate in question was one of at least 324 expired certificates operating in the Equifax infrastructure.

So how did these errors happen? Certificate management is a tricky business. All certificates have expiration dates to ensure they are current, and these expirations are indispensable to the security certificates provide. That means administrators must track and renew certificates to prevent expirations form creating the kind of errors described above.

This administration has always been a headache; one that only gets worse as the scope and complexity of the enterprise digital applications and computing ecosystems continue to increase. Virtualization, containerization, public and private cloud, and “software-defined everything” just adds to the complexity of what must be managed. Furthermore, as centralized IT breaks down into embedded functions within lines of business using DevOps methodologies, it becomes harder and harder to even know the full set of certificates running in the enterprise’s systems. Even the most diligent network administrator can be caught unaware of previously unknown certificates.

Automated monitoring and replacement of certificates are essential for protecting against unexpected expirations. Such a system can make administrators aware of upcoming expirations and can take hassle and error out of lifecycle management and renewal for all certificates in an environment. These benefits only apply to certificates that are under management, of course, so a certificate discovery system is a key component of the successful enterprise certificate management strategy.

A certificate discovery system crawls the organization’s network and catalogs all certificates it finds. Discovered certificates are now available for monitoring and automated replacement just as another certificate. Some IT organizations build certificate automation functionality for themselves, and some employ third-party certificate management platforms to handle these needs for them.

By using software to automate the discovery, lifecycle management, and renewal of certificates, IT departments can vastly mitigate the risk of expiration-based outages, potentially saving their companies embarrassment, customer service problems, lost revenue, data loss, and even millions of dollars in financial penalties. All enterprises should be evaluating how to put automated certificate management.

About the Author

As Senior Fellow, Tim Callancontributesto Sectigo’sstandards and practices effort, industry relations, product roadmap, and go-to-market strategy. Tim has two decades of experience in the SSL and PKI sectors and has served extensively as a strategic marketing and product leader for successful B2B software and SaaS companies. A security blogger since 2006, he is a frequently published author of technology articles and has spoken at many conferences including the RSA Security Expo, ClickZ, Search Engine Strategies, Shop. org, and the Internet Retailer Conference and Expo