If you ask any cybersecurity professional about their most trusted information source for best practices and practical advice, most will say it’s their peers…other cybersecurity professionals. Hey, we get it; Claroty was created by OT cybersecurity insiders.
So, with that in mind, we will periodically feature real-world insight and commentary from people in the trenches who are solving real OT cybersecurity problems for their companies.
Our first installment features Greg, an Industrial Automation & Control System Superintendent with a large oil & gas drilling operator (In keeping with cybersecurity best practices, we’ve omitted certain details regarding Greg and his employer to maintain their anonymity)Claroty: Greg, you’ve been in industrial electrical operations and field applications for 30 years, the last 14 of those within the offshore oil and gas exploration industry. What led to your responsibility for industrial control system cybersecurity there?
Greg: One of our directors approached me with several documents which contained new customer requirements and asked me to review and assess the security requirements within a couple of days. His transition guidance was to have me answer the question “we do all this, right?” Once I completed the digest, it was clear we needed to make some improvements. So, I was chartered with an 18-month project to get our operations in alignment with requirements. It was pretty demanding, and given the nature of our operations, we had a lot of infrastructure changes to make all while keeping production operational 24x7x365. Suffice it to say there were lots of moving parts.
Claroty: Your organization ultimately decided to create a dedicated security operations center (SOC) within your Operations Technology (OT) organization, rather than not combine it with the IT department. How did this come about?
Greg: Ours is a multi-million dollar, non-stop exploration business with multiple ICS vendors in our mix such as Rockwell, ABB, GE, Kongsberg and others. We also had 5 mission-critical applications running between rigs and ships in the fleet. This was sort of the easy part. The hard part is that I needed to know the status of the assets at all times. I needed to know if any configuration changes had occurred and any security issues that may arise without altering the known-working environment. And, I needed to be notified immediately of suspicious or errant behaviors that I could respond to quickly and efficiently. Plus, we had a small team of two (including me) to run and respond to all this. One of the few ways to meet our customer’s requirements and bring it all together for the needed visibility and security was to create and manage our own security operations center.
Claroty: That sounds incredibly challenging.
Greg: It was – it still IS. We built the SOC itself within 6 months of establishing the security project, but obviously it has to evolve with the assets, applications and risks.
Claroty: Many industrial organizations don’t actually have their own budget for ICS security or even a program to begin with – does your budget come from IT – is that your reporting structure – an OT guy reporting up through IT?
Greg: This all is funded and reports up to the Operations business unit through the maintenance group. Our Director of Maintenance reports to the Vice President of Maintenance, HSE, & Quality. The initial goal was to gain real-time visibility into the systems in one central location. This led to a room full of monitors, but we’ve grown far beyond that. With Claroty’s technology we’ve been able to identify our assets, their configurations and vulnerabilities as well as our communications and protocols - including visibility down to each PLC slot and I/O. Additional tools have also been important to this effort.
Claroty: Prioritization is a challenge for most operations organizations. You, as the OT specialist with hands-on experience could make priority calls around what risks there are and what might happen. However, usually that can’t be a one-person decision. How did you marry the business priorities to the potential risk outcomes?
Greg: In our case we led the effort out of OT because we had to move quickly, and you’re right – we probably knew best. However, we had to create and get agreement on policies, draw in our chief security officer, add 4 folks from the IT team and create a sort of virtual office for collaboration and upward visibility to our activity. This is still ongoing today but that was our starting point.
Here’s a rough summary of our process:
- We know the business value of each of our rigs – that’s important for prioritization, uptime, and efficiency. We chose to stagger our prioritization overall to begin with the most automated and complex environments with the highest number of assets and systems. Everyone tells you to start small, but we didn’t have that luxury. We had to dive in so to speak.
- Next, we realized we were highly silo’d – every rig was not accessible at all times, and to achieve the required visibility over our security we had to install a fiber optic backbone. This could give us a real-time view to each system and platform.
- We started with passive monitoring of our most mission-critical assets and automated processes. We also had to work closely with our vendors for additional equipment and applications so we could have a non-production test environment adjunct to the SOC. We’re talking drilling and engine control systems and creating a new ability for assuring valid “proof-of-concept” evaluations, a testbed for patches and functional verification, and internal training for OT personnel with hands-on capability, etc. It was monumental and has borne out to be one of the best things we ever did.
- Our other priorities then began to fall in line around securing and controlling remote access to our environment including all our third-party vendors and refining our monitoring to give immediate real-time notifications to specific things we uniquely needed. And of course, it’s always evolving.
Claroty: Now that you’ve been on this journey for the last four years, what insights you can share?
Greg: Well, I’ve caught vendors who were logging in and making configuration changes in the equipment without us previously knowing about it. I think that’s probably par for the course. We also found assets and communications going on that we didn’t have visibility into before – much more than we thought. There are other incidents I can’t share, but one of the best things about your solution is the deep multi-vendor visibility we have gained because usually each vendor only can see or monitor their own assets and proprietary protocols. This has been a huge benefit for our field applications.
One of the most interesting industry-specific use cases enabled out of our SOC is that on drilling rigs and ships, sometimes things fall overboard. Obviously, recovery efforts have to ensue and it’s time-consuming - a big deal. Due to our technology upgrades and maintenance, we’ve been able to use our SOC as a command-center so our executives could observe the retrievals through live video streams when they occur.