Developers Hold the New Crown Jewels. Are They Properly Protected?

By Aaron Bray, CEO. Phylum

Cybersecurity has changed dramatically in the last four years. During the pandemic, organizations around the globe found themselves faced with accelerating digital transformation initiatives, remote workforces, and a whole host of other concerns that have dramatically changed the attack surface they must defend. However, most organizations have a security posture that hasn’t yet metamorphosed in response to the changes their operations, processes, and systems have undergone.

One of the biggest impacts of these changes has been a dramatic shift in the tradeoff concerning where the proverbial security “crown jewels” of the organization lie, and the cost an attacker faces in trying to gain access to them. In the past, organizations consisted largely of traditional, on-premise workforces who primarily used workstations while at work, which connected to domain controllers that lived on-premise, and were tended by system administrators. This effectively centralized power in the hands of the system administrators, which made them the primary target of any potential attacker.

Compromising an account with domain administrator privileges would enable access to virtually anything connected to the network. Additionally, consider that the security posture of this type of organization, from basic hygiene concerns like patch maintenance and breach detection to higher order concerns such as managing security policy and direction, all reside entirely with security staff, executives, and system administrators retained by the organization. The ability to prevent attackers from successfully breaching systems, moving laterally, and ultimately gaining access to these “God” accounts is entirely a function of how much the organization prioritizes security, and in some cases, how much legacy baggage to which they are subjected.

As Attack Surface Expands, Targets Shift

Contrast this with modernized organizations: What does the attack surface look like when many business-critical assets have now been shifted to the cloud, organizations are increasingly leveraging third parties to handle what were previously core business IT functions (i.e., the adoption of products like Office365 and G Suite), many employees are working remotely, and more development processes have migrated from slow, gated releases to being effectively continuous? Fundamentally, it means that the traditional system administrator accounts likely don’t have credentials centralized in quite the same way they were before, and traditional access vectors outside of more unsophisticated approaches like phishing, such as exploitation, are more expensive than they were previously. It becomes much more challenging, for example, to find an unpatched mail server or domain controller to compromise when that entire function has now been effectively outsourced to Microsoft.

Additionally, engineering teams now have far more access than they enjoyed previously, with far fewer security controls. Many organizations have no endpoint protection on developer systems or have policy exceptions for build and test directories. Cloud infrastructure is often maintained “as code.” Many modern, business-critical assets are now described by a set of scripts that are read, modified, tested, and deployed through Continuous Integration / Continuous Deployment (CI/CD) solutions.

The same is generally true of most modern development processes. Large enterprises now field hundreds or thousands of builds every day. From a security perspective, software developers and CI/CD systems now have access to a tremendous amount of business-critical functionality. In order to operate, developers often have administrative access to cloud infrastructure, credentials for production data, and intellectual property, not to mention systems featuring little-to-no security tooling, almost universal local administrative privileges, and unfettered access to download things from the open internet.

What this means is the balance has shifted. In the past, system administrators were the quick, easy targets attackers used to gain broad access and the ability to operate with impunity throughout an organization. Those targets have become harder and much more expensive for attackers to reach, with now substantially-reduced rewards.

On the other hand, modern organizations face an absolutely staggering amount of risk in the activities performed by their development workforces and processes, including CI/CD infrastructures, and shockingly few tools exist to help mitigate or even give insight into the challenges faced.

On top of this, attackers have taken note – leveraging these open channels to steal credentials, production data, intellectual property, and more. Perhaps the most frightening aspect of most of these incidents is the fact that many organizations lack even basic visibility into these activities, or try to rely on more traditional controls such as standard endpoint products, which don’t at all align with the threat model these types of attacks represent. This means that many more such breaches are likely already underway, albeit undetected.

So how do attackers use these channels to gain initial access, and how does this really change the threat model? One of the most active vectors that adversaries are currently leveraging to gain a foothold in organizations is the Open Source Software (OSS) ecosystem. While this may seem a bit unintuitive at first blush, let’s consider briefly how OSS development and consumption has evolved, and what that statement really entails.

Open Source Software: Rewards and Risks

OSS has risen to prominence in the last two decades, despite having a relatively rich history well before. These days, nearly every organization in existence, from governments to regulated industries to advertising agencies, rely on it for business-critical functions. It has led to many great things, including massive cost reductions for development and faster time to field. However, it does come with sharp edges.

While many of the more traditional issues, like problematic licenses and unpatched vulnerabilities, are relatively well understood, there is also an entire strata of new issues that have risen to the forefront in the last few years as well – driven by a migration of processes from static to continuous, backed by systems that were fundamentally designed without security in mind.

What does the open-source ecosystem as an access vector really mean, and how does this relate to developers? Consider that most modern software projects of any sophistication contain thousands of third-party, open-source software packages. These packages are published, managed, and maintained by tens of thousands of volunteer software developers from all over the globe. While some may have corporate backing from a large enterprise, there is fundamentally no sort of traditional supplier relationship between those who create the OSS, and those who consume it. Everything is effectively supplied as-is.

These packages are pulled down and incorporated into business-critical systems in a massive, messy web. For example, a software developer may install a popular package to solve a common business problem. That package may depend on two or three other packages, which will also silently get pulled down and installed. Each of those two or three packages, in turn, will likely depend on more packages, and so on. From a security practitioner’s perspective, that single software developer installing that one, innocuous, popular package has now effectively integrated a supply chain of software spanning thousands of possible individual software packages, maintained by tens of thousands of strangers that have no relationship or vetting process of any sort with your organization, into a business-critical piece of software. Worse yet, each one of those thousands of software packages gets the ability to execute code each and every time the software developer or CI/CD runner hits “install” or “update.”

An analysis of the largest, most active software package managers developers pull packages from found that in Q2, an average of about 28,000 packages were published every day – a staggering amount when considering that most of these package managers have a veritable skeleton crew on staff of usually one to three individuals responsible for trying to manage and triage all of the software being published. While these ecosystems are the primary locations developers pull packages from in order to build software, the contents are largely unvetted. In fact, bursts of bad activity have actually triggered some of these ecosystems to shut down portions of their functionality altogether.

Phylum’s research reports for Q1 and Q2 Security researchers have noted an increase in incidents in which bad actors have pumped out hundreds of thousands of packages that are either spam or actively malicious incidents in which bad actors have pumped out hundreds of thousands of packages that are either spam or actively malicious; the vast majority of which targeted software developers and CI/CD infrastructure. While some of these attacks take a spray-and-pray approach, many are much more targeted. Attacks like dependency confusion enable malicious actors to surgically target organizations through their software supply chains, and a massive rise in typosquatting attacks, which may come in a surprising variety of flavors, tend to target popular open-source packages in use by organizations of interest. Additionally, threat actors have also employed attacks against the package maintainers themselves in order to gain a foothold.

Software Developers: The New Keepers of the Crown Jewels

Software developers have become the new high-value targets, owning much more privilege, with much less security and oversight than in the past. Attackers are capitalizing on this, as evidenced by the dramatic increase in software supply chain-borne attacks and compromises in recent years, and targeted attacks on software developers in many recent breaches.

Organizations have their work cut out for them in adjusting security posture to match organizational changes driven by digital transformation. A rapid focus on closing these security gaps is now critically important, as it is now more a matter of when, rather than if, a breach as a result of a malicious package installed from the open-source ecosystem occurs without intervention.

About the Author

My name is Aaron Bray, CEO and Co-Founder, Phylum. Aaron has 14 years of experience working in software engineering and information security. He spent 11 years working within the U.S. Intelligence Community before joining Sony to lead development for the Global Threat Emulation cell. Aaron’s past research has focused on program synthesis, malware diversity, software anomaly detection, and the application of natural language processing techniques to binary analysis. Aaron can be reached online at [email protected] and https://www.linkedin.com/in/aaron-bray-422ba06a/ and at our company website http://www.phylum.io