How DHS fell silent when a hack threatened the U.S. power grid
A month after hackers blacked out power in western Ukraine, a team of U.S. security experts touched down in Kiev to piece together the extraordinary assault.
Interviews, cellphone video evidence and a crash course in Soviet-era grid equipment helped the dozen or so Americans untangle the Dec. 23, 2015, cyberattack on three utilities. The investigators traveled thousands of miles with one big question in mind: Could the methods used to hack the Ukrainian power distributors, or the hidden code behind the strike, pose a threat to the U.S. electric grid?
But two days into the five-day mission, analysts working in an opaque intelligence aggregator at the U.S. Department of Homeland Security reached their own conclusion. The Ukraine case did not pose any particular risk for U.S. systems, according to a Jan. 27 DHS memo marked “For Official Use Only.”
Weeks later, a separate branch of DHS flipped that conclusion on its head, delivering the first in a series of stark warnings to electric utilities and other operators of U.S. critical infrastructure.
The conflicting and drawn-out response to the hack has triggered pointed criticism about DHS’s ability to deliver cyberthreat intelligence outside the walls of government. The agency is supposed to spread the word about fast-moving online threats to the networks that underlie everything from the bulk power grid to car factories. But in the case of the Ukraine hack, the first of its kind, it took two months for DHS to disclose lessons from the incident and three more months to provide additional guidance accounting for the attackers’ techniques.
“There was a credible threat to the U.S. grid, with realistic mitigations that could have been applied, and instead [DHS] decided to sit on the information,” said Robert M. Lee, founder of Dragos Security LLC and a co-author of an influential SANS Institute analysis of the Ukraine case.
“In the midst of the first attack on a power grid that was public, there was no public word from the government,” he said.
The war that had been raging in Ukraine for two years was a major source of frustration for U.S.-Russia relations. The agency was struggling to field requests from the utility industry and private analysts to share what the U.S. government considered sensitive information.
Some industry officials had an inside track on earlier attack details, including executives with security clearances and members of the CEO-level Electricity Subsector Coordinating Council, the industry’s principal liaison with the U.S. government on security issues. But the broader power sector would have to wait.
The hackers in Eastern Europe had preyed upon equipment and technological vulnerabilities also present in North America’s energy infrastructure, even repurposing a malware strain that was unearthed in U.S. systems in 2014.
As DHS officials kept largely quiet, utilities relied on private cybersecurity firms and media reports to fill in the blanks about the methods hackers used. Experts say the early lack of widely shared, actionable data could have left some companies exposed. And that has put DHS at the center of concerns about the effectiveness of cyberthreat-sharing from the U.S. government to the private sector, which controls the vast majority of the nation’s critical infrastructure.
“If the U.S. government is seeking to achieve a real partnership with the private sector, what is their value-added proposition?” said Susan Hennessey, a fellow in national security law at the Brookings Institution and managing editor of Lawfare.
Unimportant or ‘imperative’?
DHS was still trying to pin down details of the Ukraine attack a month after it happened.
During the on-the-ground investigation in western Ukraine from Jan. 25 to 29, DHS’s Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) joined representatives from the Department of Energy, the FBI and the North American Electric Reliability Corp. (NERC), which develops and enforces cybersecurity rules for the high-voltage bulk power transmission grid.
The itinerary was secret. But the fact-finding mission came as no surprise, given that the hourslong grid takedown in Ukraine was without precedent in the brief history of cyber conflict. Private security firms had already concluded in early January that evidence pointed to computer hackers, not some other form of sabotage or human error.
Still, officials at DHS headquarters saw no reason to wait for investigators to return from Ukraine before issuing a threat assessment. On Jan. 27, with the investigators on the ground, DHS’s Office of Intelligence and Analysis (I&A) published an analysis titled, in bold letters, “Damaging Cyber Attacks Possible but Not Likely Against the U.S. Energy Sector.”
The report from I&A, which reports directly to DHS Secretary Jeh Johnson, said it “is unable to confirm the event was triggered by cyber means,” citing “limited authoritative reporting.”
I&A is tasked with analyzing top-secret intelligence, and it’s charged with being a DHS conduit to state and local authorities. Its direct access to DHS’s chain of command also puts it at the center of gravity as the agency considers rising threats. But the office has faced sharp criticism from Congress about its effectiveness, and it has fought turf battles with the FBI over who is tasked with distributing information about domestic threats.
The I&A report, which was later leaked and published by the Public Intelligence accountability and transparency research project, concluded that “this incident does not represent an increase in the threat of a disruptive or destructive cyberattack on U.S. energy infrastructure, which I&A assesses is low.”
In explaining the reassuring finding in a footnote, I&A said it was based on the earliest views of the attack expressed at a Jan. 4 meeting that included DHS and industry officials.
But the I&A outlook crumbled fast. DHS’s view switched 180 degrees two weeks after the U.S. team returned home. In a February alert pushed out to electricity providers, DHS officials warned of a potential threat against utilities. The seriousness of DHS alerts to industry only escalated from there.
On March 7, the department released a detailed breakdown and alert about the attack and cited an “urgent need” for grid operators and other critical infrastructure owners to take “enhanced cyber measures” to protect themselves.
On the same day, Andy Ozment, DHS assistant secretary for cybersecurity and communications, and Greg Touhill, the deputy assistant secretary in the same office, stated that while there was no evidence of a Ukraine-level attack underway in the United States, it was “imperative” to raise defenses against what happened there.
The DHS alert put the risk in stark terms.
“It is the assessment of ICS-CERT that critical infrastructure [industrial control system] networks, across multiple sectors, are vulnerable to similar attacks,” the alert said.
DHS officials rejected repeated requests from EnergyWire for interviews and information about the department’s response to the Ukraine attack and any lessons the agency learned.
By spring, senior DHS officials had switched gears from silence about the threat to elevating Ukraine to a top priority.
“It is incredibly important,” said Suzanne Spaulding, DHS undersecretary for the National Protection and Programs Directorate (NPPD), in an April 12 podcast interview with a Washington law firm. “We are beginning a multi-city campaign across the country to make sure we get the word to critical infrastructure owners and operators about what happened there.”
Spaulding said the “good news” is that the U.S. government knows how to protect against and mitigate a Ukraine-style attack on critical control systems. “But folks have to take steps. They have to take action. They have to understand this is not just something that has the potential to affect the electric grid,” but something that could affect any Internet-connected critical infrastructure organization, she said.
NERC, the U.S. grid overseer, has maintained that the impact to the U.S. bulk electric power system would be blunted by best practices and binding federal critical infrastructure protection standards, the latest version of which took effect this month. But the standards rarely trickle down to small electric utilities.
“The grid in North America is larger and more diverse in the design and configuration of its equipment, including industrial control systems,” NERC spokesman Martin Coyne said in response to EnergyWire’s emailed questions. “As part of the industry’s best practices, these systems run on licensed software and are routinely screened for potential threats including malware, which is not the case in Ukraine.”
A BlackEnergy link
But there is at least one known and ominous similarity between the Ukraine systems and U.S. electric utilities — the presence of BlackEnergy, a powerful, elusive intrusion malware that can give attackers a hidden opening to victims’ systems. DHS has issued a series of warnings that BlackEnergy 2 has broken into the U.S. grid.
The similarities between the U.S. and Ukraine strains were so striking that DHS reposted the technical indicators in its original 2014 alert on BlackEnergy 2 to help companies root out its newer cousin, BlackEnergy 3, which was spotted on the Ukraine system.
NERC said BlackEnergy 3 has not made its way across the Atlantic.
