In the same diligent way Dale Peterson held control system vendors to account for security design flaws a “few” years back, he has been trying to help asset owners sort out ICS threat detection technologies and vendor claims. It took Dale, ICSSecure and aeSolutions teams months of hard work to prepare the contest and we, and the rest of the industry, really appreciate their efforts! When you are willing to provide “non billable” hours to make it happen, you are in it for the mission! Thank you, Dale Peterson, Eric Byres, John Cusimano, Ron Brash and everyone else that chipped in - you are helping raise the bar!
We applaud all the competitors who decided to enter the fray. Deciding to jump into a first-ever competition of this sort was a hard decision for all of us and we challenge the other vendors join in next!
We’ve put together a highlight reel below which covers how the competitors were judged as well as the final results.
The challenge was split into two parts–Identification and Detection.
On Tuesday, contestants consumed PCAPs taken from fifteen different sites at a West Coast oil and gas company. The contestants were asked to use their products to identify assets on the network and provide a view of how the assets were communicating. This was a blind challenge–none of the vendors knew anything about the network other than the fact that it was from an O/G pipeline company. Contestants were rewarded for discovering assets, identifying specific details about them and were given bonus points for unique insights. Scoring was designed to test the products themselves, rather than the teams, with more points awarded for quick answers. Claroty won the day.
It was a difficult challenge since the PCAPs were taken from fifteen segments and only included one hour of network traffic. Claroty Continuous Threat Detection performed very well. But we also made a few tactical errors that held our day 1 score down. For example, we didn’t submit some of the assets we discovered for scoring–we initially classified them as “ghost assets” and we err on the side of accuracy.
On Thursday, for part two, contestants received a live stream of network data. Ron Brash noted that it “was it was a ruthless and fast paced attack; all within 52 minutes.” The data included various known malware including known ICS threats (e.g., Havex and Stuxnet) port scans (fast and slow), policy violations (e.g., plaintext passwords), logic changes and firmware, etc. This is where the rubber meets the road. The ability to detect attacks on ICS networks that can impact industrial processes and pre-cursor activity, early in the “cyber kill chain” that can signpost an impending attack. Again, our Continuous Threat Detection product performed outstandingly and Claroty won the day.
During the award ceremony judges Eric Byres, John Cusimano, Ron Brash discussed the challenge with the Claroty Research team (Amir “Jumbo” Preminger, Tal Keren and Nadav Erez). The judges noted just how difficult it was for the contestants–with Eric Byres remarking that the contest was “much harder than the real world because of the limited time sample, lack of context and the use of only one sensor”. In a nod to Claroty and the overall threat detection category, Byres also noted that he was “pleasantly surprised just how well all the products performed”. We were not surprised–visibility and threat detection for ICS is definitely ready for prime time–and we look forward to competing next year!