This is the fourth blog in our series focused on real-world problems in Operational Technology (OT)/ Industrial Control Systems (ICS) cybersecurity learned from our experiences with clients across the globe. The intent of this series is to cast light on common problems found across client environments - which pose both security and operational concerns.
In this blog, we focus on the security implications of the practice of using online edits for minor adjustments to running batches.
Changes to process vary in scale from full batch changes (Project Downloads) that alter multiple parameters and logic within a single or multiple controllers, to minor adjustments executed through Online Editing.
From our observations across client environments, Project Downloads occur with far lower degree of frequency than Online Edits and as such as more controlled and less succeptible to hijacking by a malicious actor. Somewhat alarmingly, the use of Online Edits for fine tuning appears to be a frequent and somewhat widespread practice.
Early into our engagement with a global Chemical company, we encountered the use of Online Edits exceeding more than 100 edits per day. This is a large amount of traffic that opens up the real and dangerous potential for malicious activity to be hidden in seemingly benign communications.
figure 1: screenshot of site INSCT_7 capturing multiple Online Edits
Project Downloads correspond to production batches. As such, their occurrence and timing can easily be monitored and forced to comply with strict change governance policies based on the approximate duration of each batch. The more strictly the policy is followed, the harder it is for an attacker to leverage this tool to perform process changes. If you know and control exactly when these Project Downloads are happening, any malicious Project Download would stand out against the legitimate ones.
However, control is much harder to apply to Online Edits used for real-time adjustments while the batch is running. The adjustments are not pre-defined (as they are in the case of Project Downloads) - and in many cases, they are left solely to the discretion of the engineering teams. And we've found a fairly widely used practice related to Online Edits which will likely send some cybersecurity minds reeling. It appears that a common operational shortcut is to leave a permanent connection running - enabling engineers to apply their own, real-time judgement on how and when Online Edits occur. This is a huge governance oversight and a potential "Mack truck" sized opening for an adversary to exploit.
In the best case scenario, the use of any Online Edits would follow a tightly controlled change-request policy which would control who can log in to a controller, for what purpose, at what time and with what duration. Without this, it is nearly impossible for operational and security teams to ensure that Online Edits aren't hijacked to send malicious commands.
The cybersecurity challenges introduced by the "anything goes" style use of Online Edits are clear. The lack of a definite blueprint defining ‘good’ Online Edits poses significant difficulties in discerning legitimate edits from malicious ones, enabling attackers to mask their editing activities during the allegedly benign approved connection timeframe. Obviously, the risk intensifies in networks where engineers leave a permanent connection which creates a practically permanently exposed attack surface.
OT networks that rely on Online Edits for process adjustments are more exposed than ones which perform these adjustments with Project Downloads. Of course, it’s not as if one can just replace one practice with the other. Reliance on Online Edits characterizes processes that require mass real-time adjustments as a matter of course. Operators and security teams of such networks should acknowledge this security gap and allocate resources to secure it by setting up well defined policies on when and from which machine an Online Edit can take place. Eliminate the practice of leaving a permanent connection in place and for any Online Edits, as proactively have security and engineering teams examine them to detect any possible malicious interference.