Inside the diabolical Ukrainian hack that put the U.S. grid on high alert
Eastern Europe was blanketed in a heat wave last summer. In Kiev, Ukraine, a state of desperate resignation had set in as fighting intensified between pro-Russia rebels and Ukrainian forces to the east. Separatists closed highways and attacked ports. Meanwhile, a silent incursion had started to worm its way into the email accounts of employees at media outlets, national railroads and power distributors in the western half of the country.
The digital-era Trojan horse looked like a call to arms from the nation’s embattled capital. The subject line read simply, “Mobilization.”
As Ukraine’s civil war raged, a few mouse clicks at three local power companies set in motion the covert intrusion. It was the first successful attempt at planting a bug, then disabling an electric grid serving hundreds of thousands of people.
At 3:30 p.m. on Dec. 23, 2015, lights winked out in parts of the Ivano-Frankivsk regional capital. A minute later, another part of the grid went down. Soon, a third utility — and almost one-quarter of a million households and businesses had lost electricity.
Workers at the Prykarpattyaoblenergo, Kyivoblenergo and Chernivtsioblenergo utilities watched helplessly as cursors moved across their workstation screens at the intruders’ commands, shutting down substations. Other hidden commands destroyed vital equipment. The attackers were invisible and precise, and they showed the world how fragile critical infrastructure is when hacking is used as a weapon of war.
Ukraine’s battle to wrest control from the hackers elevated the story of frequent blackouts in a poor country to the latest in a series of cyberattacks with implications for the United States. Months in the making, it represented an escalation in attack methods that frightened U.S. authorities and executives. The hack methodically corrupted standard programing and subverted controls. It laid bare the work of persistent planners.
Seven months after the Ukraine attack, U.S. security officials are still trying to understand whether the much larger, and more sophisticated, North American power grid is equally as vulnerable to a determined, insidious assault. A more ominous warning has been sounded to utilities and federal agencies: Step up preparations to recover from a cyberattack that may one day break through.
Hackers didn’t simply crack a code and pull the off-switches at local substations — they rendered some crucial station devices inoperable. Then, they corrupted software and servers designed to turn the power back on.
The unparalleled grid strike in Eastern Europe has led to stronger, more frustrated complaints by industry and security experts about the performance of the U.S. Department of Homeland Security as a source of rapid, actionable cyberthreat intelligence for the electricity sector. It also has raised concerns that federal guidelines applicable to the high-voltage interstate grid don’t guarantee the security of local utilities that distribute power to millions of homes and businesses.
A four-part investigation by EnergyWire found that relationships between DHS and outside experts with deep knowledge about grid security became badly frayed in the weeks after the December hack. For several months, DHS put out conflicting internal and public messages about the dangers posed by the Ukraine hack, compounded by tug of wars around the use of closely held information inside the diffuse intelligence community.
What resulted was a slow and halting response by the U.S. government in the aftermath of the Ukraine takedown.
In an age that pits state-sponsored hackers against private companies like power utilities, critics of congressional inaction and government secrecy are starting to hammer at what they view as glaring failures around threat-sharing.
Ukraine is one of a cluster of cyberattacks in the past two years that grabbed headlines. The November 2014 attack on Sony Pictures Entertainment mushroomed into a national security and free-speech entanglement with North Korea. The U.S. Office of Personnel Management (OPM) disclosed last summer that computer breaches included the theft of Social Security numbers of 21 million Americans. Hackers also stole fingerprints of government workers and compromised security clearances.
The Obama administration in September 2015 publicly acknowledged suspicions that China was the source of the OPM breach.
In the Ukraine case, top administration officials have kept quiet, refusing to give credence to experts’ widely held view that Russian hackers likely planned and executed the first-known takedown of a power grid.
Reconnaissance
The email messages snuck through the Ukrainian utility servers in droves last summer, asking employees there to “enable content” to read the attached document. When they clicked, the file unloaded the BlackEnergy attack software on their office computer systems, burrowing deep into the information technology side of the business.
Once out of sight, BlackEnergy established hidden lines of communication the hackers could use to extract information and download malware, all under the noses of the grid operators. There was plenty of time to inventory systems and search for passwords and pathways that would take the intruders from business-side computers into the protected heart of the utilities’ operations, including the grid operators’ control room.
Over many months and perhaps after additional runs of “spear phishing” emails, the hackers were able to find utility officials with credentialed access to the operating systems and steal their passwords, too. With login info, the hackers could escalate from espionage to attack. All they needed was a trigger.
Incitement
On the eve of the assault, utility employees in Ukraine, Russia and a few former Soviet states were slapping each other on the back for a job well done.
It was Energy Day, a holiday to recognize and celebrate electricity workers in parts of Eastern Europe. In a Dec. 22, 2015, speech, Russian President Vladimir Putin congratulated his country’s power industry professionals. He singled out their “great, strenuous and highly demanding” work keeping the lights on in Crimea.
It was not an offhand comment. Putin’s forces annexed the predominantly Russian-speaking peninsula in 2014, setting off a violent and ongoing territorial conflict that leaked into eastern Ukraine. Crimea, despite its de facto status as a Russian republic, remains connected to Ukraine’s electric transmission network. Power outages in the contested territory have been attributed to Ukrainian saboteurs.
Observers have suggested those blackouts set the stage for Russian vengeance, a theory that would explain the timing and rationale for the Dec. 23 cyberattacks, and why the biggest impacts were confined to pro-Western portions of the country.
The message in that scenario: You hit our grid in Crimea, we hit yours.
Attack
The takedown itself was quick and clinical, backed as it was by months of planning.
Western Ukrainian grid operators could only watch as hackers booted them from workstations, dragging cursors around control system screens to achieve their own harmful ends. The attackers changed passwords so the Ukrainians couldn’t log back in to grab the reins.
Utility industry workers were sidelined during those first frantic minutes. In hacker parlance, they had been owned.
No one online could be trusted. The computers, however, were inherently trusting. They dutifully carried out the hackers’ commands to open high-voltage circuit breakers at dozens of substations across western Ukraine, knocking out power. The machines had never been programmed to question why so many users would simultaneously log in from unusual internet protocol addresses. The virtual network tied to the operational workstations asked for a username and password — nothing more — to grant unfettered access.
With credentials in hand, the attackers still had to understand Ukraine’s Soviet-era electricity infrastructure to do any real damage. After all, they were dealing with three different power distribution management systems at three different companies.
