Oldsmar, Florida, experienced one of the biggest fears in cybersecurity Friday — hackers looking to poison its water supply.
It’s the kind of breach that has been warned about for years, but rarely seen. And while the hack was quickly addressed, it offers what experts say is a prime example of why the cybersecurity of the U.S. water supply remains one of the greatest risks to the nation’s infrastructure.
And like the U.S. election system, it tends to be a sprawling and varied challenge.
“Water facilities are particularly problematic,” said Suzanne Spaulding, the former chief cybersecurity official at the Department of Homeland Security under former President Barack Obama. “When I first came into DHS and started getting the sector-specific briefings, my team said, ‘here’s what you’ve got to know about water facilities: when you’ve seen one water facility, you’ve seen one water facility.’”
There’s approximately 54,000 drinking systems in the U.S., which are run independently, either by local governments or small corporations. And that means thousands of different security setups, often run by generalists who are responsible for the technology of their particular water system.
“I’ve been to numerous water treatment facilities where there is one IT person or two IT people,” said Lesley Carhart, a principal threat analyst at the cybersecurity company Dragos. “And they have to handle everything from provisioning computers and devices that keep the infrastructure running to trying to do security.”
“Most are very conscious of it, but they’re just drowning,” she said. “They don’t know how to accomplish all the things they’re required to do to both keep things running from an IT perspective and also fill compliance checkboxes.”
All of the city’s cybersecurity services, including that of the water treatment plant, are managed by one man, City Manager Al Braithwaite, Assistant City Manager Felicia Donnelly said in an email.
In the case of the Oldsmar attack, all the hackers needed to gain access was to log in to a TeamViewer account, which lets remote users take full control of a computer, which was associated with the plant. That let them open and toy with a computer with a program that sets the chemical content for the underground water reservoir that provides the drinking water for nearly 15,000 people. While the facility does have backup alarms in place to measure unsafe chemical levels, the hacker was at least briefly able to order the plant to poison the water.
With a few clicks, they told it to raise the levels of lye in the water from 100 to 11,100 parts per million. Anything more than 10,000 can lead to “difficulty swallowing, nausea/vomiting, abdominal pain, and potentially even damage to the gastrointestinal tract,” Dr. Kelly Johnson-Arbor, a medical toxicology physician at the National Capital Poison Center, said in an email.
Bryson Bort, a cybersecurity consultant who helped start ICS Village, a nonprofit that raises awareness of cybersecurity for industrial systems, said that such a practice — setting up a computer program to allow users to take control of those sensitive industrial systems — is extremely common for industrial systems that don’t have the means to employ a staff of experts to be on call at all hours.
“If you think about it, you have a challenge both technically and resourcewise with being able to manage things,” he said in a phone interview. “So the ability to get an alert light at 3 a.m. and get that one expert has value. People are always mystified that this is the way it is, but this is the way it is. It’s the convenience of these resource constraints. You don’t have a choice.”
Foreign government-sponsored hackers regularly target U.S. industrial systems, which often are labyrinthine enough that a simple intrusion usually doesn’t give them the ability to shut down infrastructure. It is unclear who or what was behind the Oldsmar hack.
Federal officials have long fretted over a potential “cyber Pearl Harbor” incident, in which hackers could wreak physical damage to American infrastructure. While that hasn’t happened, the U.S. is keen to push back in the instances it finds an adversary nation getting too close.
In 2013, a hacker broke into computers that controlled the Bowman Dam in Rye, New York, and potentially could have got access to its controls if it wasn’t offline for maintenance. Three years later, the Justice Department charged an Iranian national for the hack, saying he worked for a company tied to the Iranian Revolutionary Guard Corps.
And in 2020, the U.S. Treasury Department sanctioned a Russian government institution for its suspected creation of a powerful, destructive program, called Triton, that targets industrial systems. But there’s no public evidence of an American company having been seriously harmed through Triton.
That doesn’t mean that those countries’ hackers don’t try to exploit the open holes in American infrastructure, Carhart said. It means they know better than to cause cavalier damage.
“The foreign state hackers are there. They are in the water utilities, I promise you. But they know better than to poke buttons today,” she said.
“They’re going to wait until they’ve got a really good reason to poke buttons. They’re there. We find them all the time.”