Jon Holzbauer on IT/OT's Divergent Approaches to Asset Security

Jon Holzbauer, OT Systems Manager at Silgan Containers, joins the Nexus Podcast to discuss where IT security teams and OT operations run into challenges in protecting these diverse complicated environments in manufacturing. A clash of approaches may lead to rash decisions around cybersecurity that could disrupt key processes or impact safety and reliability. 

This interview was pulled from Episode 3 of Nexus Digest, a monthly recap of content published on Nexus. 

Subscribe and listen to the Nexus Podcast here. 

Transcript

SPEAKER_00 0:12

All right, welcome to episode three of Nexus Digest, where we recap the month's content published on our thought leadership website called, of course, Nexus. I'm Mike Momoso, the editor of the website, and joining me today is one of our contributors, John Holzbauer. John is the OT systems manager at Stilgen Containers, and he has a lot of experience in the convergence of IT and OT, especially within manufacturing enterprises. He's also written a couple of great articles for Nexus. And today we're going to talk about his latest contribution, which focuses on the IT OT cybersecurity skills gap and what's happening there. So let's bring in John. How are you doing? Good to see you, man. Thank you.

SPEAKER_01 0:53

Hey, good to see you again.

SPEAKER_00 0:55

So maybe it's a good place to start by telling the viewers just a little bit about your day-to-day role and maybe some of the experience you have operating in these converged environments.

SPEAKER_01 1:05

Yeah. So we uh as a manufacturing um business, we have 121 global sites. We have, like many other companies, realized that visibility of our OT environment is critical to our uh protection and security. So we've we've started to roll out uh clarity as our primary IDS tool. Um, and my responsibility there is to work with a partner to make sure that those implementations occur on time and on and under budget. And uh and then also take the data that comes from those activities and communicate that back to the business so we can understand the what the environments look like, uh types of devices we're seeing, and the types of risk that we're exposed to. And then in combination with the sites that we've implemented, we're trying to understand how we can get out in front of the risk for sites that maybe don't have clarity yet with some proactive uh approaches. But it um it's been it's been a challenge to uh to try to get through each of these sites because they're so unique. Every single one of our manufacturing sites, the OT environment is not standardized, uh, even within a given business unit. Um, so yeah, it's been it's been good, but uh we have a long way to go.

SPEAKER_00 2:22

So all right, so let's jump into your article, which I thought did a really great job of establishing obviously that there's a skills gap when it comes to cybersecurity, but on both the IT and OT side, IT lacks that that process knowledge that they would need to kind of secure industrial assets. OT lacks overall formal cybersecurity training for for the most part. So from your experience, maybe explain what's happening with this dynamic and of, of course, some of the challenges that you see arising from from both sides of this equation.

SPEAKER_01 2:52

Yeah, so on the on the IT side, I think while there might be a better baseline understanding of cybersecurity, there tends to be a gap when you get to the plant floor level of how the types of devices that live in a plant floor environment can interact with each other and the types of tools that can communicate with them. Uh, and a simple example of this would be um an active scan across a network that includes things like PLCs, could could brick machines, shut down production. Um, that's a major risk, right? And somebody who's only been in an IT security environment might not be exposed to that type of issue. Um, and on the flip side, when you have OT staff who maybe come from an automation background or a maintenance background and are more familiar with the internal workings of a machine network, they may not have the historical knowledge of how the IT network needs to be constructed to be safe. Um, whether it's making sure that things are not communicating out to the outside world through third-party vendors, or maybe it's the way that two devices on uh a shared network have the same IP address. For example, we see that a lot. You know, public IP gets um consistently put into equipment where it makes the manufacturing process faster to put all my PLCs, they are going to be 192, 168.1.1. Well, then I put those two on the same network, and if they see each other, I now have IP redundancy. So those types of things basically it's just not having a foot in both worlds, makes it difficult to even anticipate what you might run into. And that's I think coming to the forefront now when we look at trying to put cybersecurity into an environment that wasn't built on it.

SPEAKER_00 4:43

Right. I mean, there's a lot of structure too when it comes to security within IT, even if it's just you know regular patching cycles, regular procurement cycles, and that just doesn't exist in OT, right?

SPEAKER_01 4:55

It definitely not. And and some of the stuff too, we when production is the biggest production uptime is the the most important factor here in a lot of the decision making. We might make uh short-sighted decisions from a security perspective, like running a legacy device, uh, because I don't have a way to quickly engineer a solution to get it back up to a modern option. Um, those types of decisions obviously have cybersecurity impacts, but even worse, the support needed for those devices eventually gets worse and worse. Um, we've heard stories either at our facilities or at other companies where people are now turning to eBay or even buying from competitors to try to get motors, drives, uh, you know, devices that are specific to their equipment because they've put off maintenance for so long.

SPEAKER_00 5:47

Right. And what are some of the consequences that you potentially see or try to avoid that, you know, maybe can be directly linked from this kind of lack of mutual understanding of of each other's worlds?

SPEAKER_01 6:02

Yeah. So first of all, there's a there's a vulnerability and and risk to having these legacy systems, whether it's the operating systems of shop floor PCs that might be outside of the IT infrastructure. Um, we've seen things like Windows CE still popping up at times, definitely Windows XP, Windows 7, and Windows 10 has recently been uh moved out of the main support structure. Um, but you know, additionally, when we do have an issue, the it's a lot longer time periods that we're talking about to fix these things because we've waited so long and the engineering required to deal with a legacy solution is is worse. Um, but uh the the other thing, and manufacturing is maybe behind in some of the compared to some of the other markets, but the regulatory requirements coming down from you know the SEC and other groups that say, hey, these are the things you need to do, or cybersecurity insurance. You know, you have to have these parts of your network covered. Um, so yeah, it's kind of coming at us from all angles, but without a plan, um it's it's gonna be difficult to see us get past sort of status quo.

SPEAKER_00 7:12

Are you hearing or having, you know, starting more conversations about the risks around internet-facing assets, for example, and ensuring that they're, if they're online, that they're online securely, you know, either behind a VPN or with without default credentials, et cetera. How big of a problem is this phenomenon of connecting assets to the internet?

SPEAKER_01 7:37

Yeah, and I think it pops up in interesting ways. Um, a lot of the devices that have been uh highlighted for us are things that aren't even directly related to the operations. We're finding things like automated thermostats, um, connected devices like security cameras and vending machines, right, popping into parts of an OT network, but they have a direct line back to maybe that that vending machine provider that puts snacks in the in the break room. Um so I think it's just getting a broader picture in mind of what does the OT network really include? And all the devices, just like we have at home, many people don't realize that even when they buy uh an oven or or a refrigerator, some of those devices are connectable, right? You you could actually have internet connection going to appliances now. So um it's really understanding what the implications are of having that capability. Generally, we're being sold as manufacturing plant floor employees that that connectability means support. It means uh the ability to reach out to a third party if we need maintenance. Um, and while that might be true, we also could be giving something up to get that.

SPEAKER_00 8:54

So, how has how much has your job changed or the internal conversations have changed in the last few years because of digital transformation, because of this growing uh connectivity of assets to the internet?

SPEAKER_01 9:09

I mean, personally, my when I was hired to to come in, um, I was doing business intelligence work and making sure that we had our ERP and MES systems and CRMs communicating so we could build dashboards. And now a large majority of my time is spent looking at OT cybersecurity, and that really speaks to how quickly this has moved in the general sphere of cybersecurity, right? We we have more devices than ever and more coming in that are connectable. Um, the desire to quickly use third parties for remote support has gone up and up and up, and with that, the landscape of the complexity of the tool sets. So there's dozens of different ways that people can connect to these machines now. Um, so trying to just put some some boundaries around any of that has been challenging. And then you you amplify that by a number of sites, right? So even if we were just doing this at one site, it'd be challenging, but we have 121 sites. So uh it's hard to keep up.

SPEAKER_00 10:12

It's a lot of complexity. Um, so just as a last question as we wrap up, like any pieces of advice in terms of how do we fix this? Where where should companies start? What should they think about? What should first steps be?

SPEAKER_01 10:27

Yeah, I think the the first part is is understanding and and realizing that there is a unique new skill need. So there's a hybrid role or a set of roles that are in between IT and OT. Um, it's a it's a unique combination of being able to speak the shop floor language and understand how systems communicate with each other in a shop floor environment, as well as having some basic understanding of the IT network structure and cybersecurity. Um, those roles, uh, other companies are doing things like upskilling, finding people who have three quarters of that skill set or half of that skill set and training on the other half. Um, I I personally believe that within a few years here, we're gonna start to see colleges form majors related to these things because that's typically the way that the pendulum swings. Um, but it's really about putting people in positions where their dedicated role is OT-specific cybersecurity. Instead of, I think what companies have historically tried to do is either take the IT people in cybersecurity and try to ask them to stretch into covering this, um, but not fully commit to it. Or we're asking plant floor individuals that maybe have an automation background to try to hold themselves accountable. And there's a bit of a conflict of interest there, right? Um, so I think first of all, it's it's acknowledging that there's a difference and then putting some resources behind it. And it's a tough sell uh because it's a it's it's not really a direct ROI, right? It's more of a cost avoidance. Um, but in order to do it well, it's definitely the first and major step that needs to be taken.

SPEAKER_00 12:12

Yeah. And you wrote about some of those conflicts of interest, those competing incentives, and I believe it was your first article for Nexus, right?

SPEAKER_01 12:20

Yeah. And it again, this is why this is not an easy conversation. I think a lot of businesses are struggling. There's a lot of competing priorities right now for spend. Um, companies are are forced to have to deal with things like AI initiatives, um, you know, other insurance type things to protect themselves. Um, and when this topic comes up, it's one of those those difficult ones that doesn't I can't I can't calculate you uh uh a cost savings ROI, right? It's a cost avoidance. Uh it's effectively like buying more insurance. But um, the companies that have had issues where they've been breached in their OT environments, I mean, there's a lot of information out there to show the impacts of that can be great and difficult to unwind. So I think it's just a balancing act where the executive boards need to figure out what risk they are willing to live with and get educated on the the gaps that they have in their organization and how to fill them most appropriately.

SPEAKER_00 13:17

Right. All right, John, that's a good place to leave it, I think. Thank you so much. Appreciate your time and uh and the good work for next week.

SPEAKER_01 13:25

Thank you very much.