Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh.
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.)
While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged.
If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.
A microcontroller based solution is rarely a good replacement for a PLC. You can find integrators willing to come out and troubleshoot your device in the field a lot more easily than embedded progammers and vendor support is usually excellent. When your plant is down, that sort of thing matters. PLCs handle harsh environments for decades, whereas with a microcontroller all the thermal issues and power conditioning are all on you. When something fails, it's usually just a matter of pulling a spare module off the shelf and plugging it in.
If that's an old GE 90-30 (a common FANUC system), it's time to replace it. You're currently in that period when the CPU has reached end of life but the modules are still supported and the vendor has an easy upgrade path. Emerson owns the brand now, and can sell you a new unit that'll run your old program with minimal changes. You can call them and get a list of integrators in your area. It'll be pricey, but you won't need to do it again for another quarter century.
This system barely deserves a PLC. The modules are only a couple of extremely basic analog inputs and outputs, plus a wildly outdated and unsupported HMI. And the entire plant could possibly be replaced with something more modern for less money than upgrading the PLC. (Or I could get something cheap from AutomationDirect - I have no idea how good their PLCs are.)
"but you won't need to do it again for another quarter century."
Hmm, so how do they handle the new cyber threats, since most of these are connected? It seems upgrades would need to be performed to address new threats. And just because something is not internet connected doesn't make it immune (eg stuxnet).
They shouldn't be connected is the point of the article. All PLC and DCS manufacturers are constantly working on threat mitigation and pushing out updates to address vulnerabilities. It's on the Process Controls team to decide what is an actual threat that requires patching.
The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such?
That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way.
If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment.
And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
> You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such?
This guy is very happy when someone literally does that: https://www.youtube.com/@Cursed_Controls (great channel btw), because usually he has to deal with worse.
> And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
You'd think Joe from shipping wouldn't be given a key to the cabinet with the PLC.
Real locks in such systems are rare, sometimes even disallowed. Usually, a lock with a real key is only used to temporarily prevent switching on a machine during maintenance or such.
> The PLC is usually in a control cabinet that can be opened by a control cabinet key:
As far as I can tell, that thing's $50. Why would Joe from shipping have one?
> Usually, a lock with a real key is only used to temporarily prevent switching on a machine during maintenance or such.
Most locks aren't really meant to resist and adversary. They're meant to communicate who is allowed and who is not, and back that up by making violations a little non-trivial.
That's the first search engine hit I've pasted. The cheap ones that work just as well are single-digit prices. And usually there is one in every toolbox on the shop floor.
Even more so, if it’s just there you’re making it easy for the customer to find someone bidding less to execute new projects. Guess who they’re calling if they don’t have the plc project (or don’t understand they need it). Job security.
Plus, yea, someone could really make a mess even if they have the best intention. A guy from my company almost uploaded the wrong program to a large bioreactor. At best it would have made the bioreactor just unusable until corrected. At worst it could have broken the equipment or caused a safety issue. This guy had been around the block a few times too. Luckily someone noticed before it was too late.
I don't get into PLCs, per se, but I've done some stuff with other building-scale gear.
Like: We took care of the centralized controls for the inmate portion of a jail that was built in the 1980s. We rejiggered the controls for that whole jail back around 2010 or so with new kit and had been taking care of it since then.
All the wiring for all of the individual doors and intercoms was documented in one old 3-ring binder. (There were probably 3 copies originally -- for the architect, the maintenance room, and the equipment room, but just 1 remained and it had hand-written notes.)
One day, around 2020-ish, they called us because the book was missing. I didn't have it (I never once left the equipment room with it), and the co-workers who had been there insisted that they didn't have it either.
As time passed, we'd get occasional phone calls of "Hey, we really could use the book for our building back if you guys ever find it." They weren't angry calls, but sheesh: That book had real value and getting stuff done without it was difficult. So every time when they'd call again, I'd ask around our shop -- again -- if it had turned up anywhere.
And then, finally: One of our guys cleaned out his truck and found it. He must have taken the book out to look at and have a smoke about some problem or other, and then it just stayed in his truck. For years.
This shit happens. People are imperfect and sometimes we screw stuff up.
> If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment.
Is that really an issue? Do PLC vendors have concerns about other vendors turning up and yanking USB drives out of critical infrastructure with reckless abandon? Do the facilities themselves see no issue with potential vendors yanking operational utilities out of their systems while they are running?
It's just a thumb drive, on a lanyard, hanging in a cabinet. It's not like it has guards posted. :)
Maybe Joe (from shipping) took it back to his desk to have a squiz at it and try to passively learn something new during some otherwise-downtime and never returned it -- years ago. That cabinet is in his area, so he has the key.
Maybe Tim (the plant electrician) borrowed it to get something else done, and it unintentionally disappeared somewhere along the way. Tim has all of the keys for all of the things.
Maybe Emily (the ubergeek who prides herself on being able to code her way out of a mess on any system) had it in her office before she got hit by a car and lost the ability to communicate with polysyllabic words. She borrowed Joe's cabinet key after he mentioned an issue.
Either way, it was just a thumb drive that had been inside of the cabinet, and now it is gone.
Things like thumb drives physically connected to cabinets can just disappear in your environment with no followup whatsoever? And that's regular, expected, and desired?
Hospital IT got a call asking where the little binder with the DVDs for the CT scanner were from someone in Radiology 'cause the Siemens tech showed up.
No, we didn't have them, and they told us they last saw them in the back of a cabinet in their own work era.
Turns out those were the unit-specific reload DVDs for half the reconstruction computers and some serial-specific license stuff.
If it's a custom industrial installation then the source code is part of the documentation just like the electrical schematics, specifically so that any trained electrician can maintain it. On the other hand Joe from shipping wouldn't even know what a PLC is and if you let untrained staff maintain electrical systems I think you have some more serious problems.
> On the other hand Joe from shipping wouldn't even know what a PLC is
Joe from shipping is undefined. It is possible that Joe knows more about PLCs than you think you know about shipping, and that he has a god complex worse than that of a grizzled Haskell programmer. People all over have jobs that don't align with their interests, education, and prior work experiences. That's pretty normal; the bills still need paid and people find work where they can.
> if you let untrained staff maintain electrical systems I think you have some more serious problems.
Or maybe Joe from shipping is a departmental director who has this role because he is a strong leader who takes the initiative to embrace and overcome new challenges. We don't know.
> The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.
Relatedly, I made an embedded product. I built a live CD (yes this was a while ago) for my employer with the entire development environment. As long as you can find an x86 machine with CD-ROM you could build, compile, test, and program using the exact environment I did twenty years ago.
Maybe today I'd do micropython on ESP32. Download text file from device, edit, upload back on.
There's at least one customer out there who has an RPi with NixOS on it from me. Probably still works to this day. It was the best use of NixOS I think I've ever found.
If it's a custom Yocto thing or it involved buildroot at any state of provisioning it gets the moniker "quasi-embedded" to me. Doubly so if it never actually connects to the internet, unlike these water controllers, apparently.
I've switched up to NUC-ish hardware which WAS cheap until recently. All the components just a bit higher quality than the lowest-bidder you get on the Pi. Run nixos on that and you've got a really solid platform.
An entire class of controller with really shit security now (as in scanning a network with PLC controllers has a non-trivial chance of rebooting or freezing any number of devices on the network) trained to just plug USB sticks in?
I understand the idea, and its laudable. But the entire model of PLC security needs root and branches overhaul.
Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough.
>>> The pay is also so much worse in my experience.
The result is that anybody who can program well enough to develop software is doing so. Those with any programming skills who remain behind are smart-but-undisciplined programmers, people who have a reason to be in a particular geographic area, too old to think about changing careers, or just plain nuts.
This is what "lack of investment in manufacturing, research, and infrastructure" looks like. Sow what you reap.
Disclosure: I might be one of those people. I came out of grad school in physics research, and programmed an entire plant. Thankfully, that was before it was economical to put every controller on the Internet.
To me that speaks to a fundamentally incorrect economic structure. I bet PLC programmers could command much higher salaries - at much higher levels of risk - if they held more liability for things going wrong. That, in turn, would spur a lot of economic development around unfucking the field.
On the other hand, markets are generally smarter than I am. Much like how US citizens living in the EU are basically barred from investing in index funds due to two fundamentally incompatible regulatory regimes, there are almost certainly factors here I'm not seeing and wouldn't see without a couple weeks of research and talking to people who have tried and failed as this.
Deming wrote, "quality control is a management problem." When you see a plant where workers are cutting corners on quality, they're probably under pressure to do so.
Factories face liability already, either due to the cost of "line down" or getting sued. They could pass this liability on to their employees, who would probably cover it through insurance, like doctors.
Not only is the pay worse (like literally half of what you make doing webdev for a corp), but they mandate you be on site at a factory working first shift like you're on the production floor or something, clocking in and out when you're a salary worker. It's an awful culture.
Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
>
I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID
<
The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc.
So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?
I've seen similar things in this decade. Lots of stuff still running where the last change in design was 20+ years ago, the current team knows how to keep the lights on and replace parts, and calls a vendor if something goes end of life and asks for a replacement part, then configures it the same way that the older one was.
It's quite reliable, and does not require paying for a team of people who wants to try new things. Once this happens for a long while there's no one in the building left who's going to advocate for change, since the only thing all that fancy new technology brings is the next 5 years of nobody getting a bonus because operational metrics weren't being met anymore.
You have to try really hard to convince these people that the thing that they have seen on a daily basis to work just fine for the past twenty years is no longer good enough, because all their evidence says that it is. And you should also listen to them much more than people who just want to use the latest thing typically do, because they do have the decades of experience seeing how things actually fail.
Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure.
I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold.
The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.
A lot of PLC controllers can certainly be unforgiving. I mentioned overflowing a 16 bit float point because that's exactly what happened at a waste water treatment plant I was called out to and the resulting regulatory fiasco that ensued probably cost the company tens if not hundreds of thousands of dollars.
I'd love to see more formal methods come to the field. Part of me says I'll return one day, maybe if AI kicks me out of my software field, but I'd really want to come back at a position I could healthily influence towards safety and correctness.
The industry/consulting term is "digital twin". Nothing off the shelf, since every plant is different. Even for a single process it gets complicated really fast. You probably want to track the environment itself too like temperature, humidity, leaks, etc. All the machinery and if they don't have necessary sensors, gotta retrofit them. Oh and human workers too. Stuff like "John Doe pressed a manual override button at 9:25 AM" or "it takes 12 minutes for a forklift operator to move a pallet from zone A to C".
I was involved in a project that made such a twin for a warehouse. It was simpler in theory because there was no manufacturing process. Just moving objects from one place to another. It was still expensive and took a long time. We tracked traffic heatmap but also collected data on and modelled luminosity for safety. On top of that there was a standard warehouse space optimization.
You're going to run into similar problems as with testing and documenting code. Eventually something will change - a process, a machine, a sensor. You have to stay on top of things and update your model too. Otherwise they will drift apart and not be as useful.
This is starting to be done for very high value plants (think chip fabs, large datacenters, that kind of thing. When the infrastructure is a $10b expense it's worth it).
Small plants in the middle of nowhere, especially municipal ones? You're lucky if it was designed this century and not 50-100 years ago. Everyone's just trying to keep old stuff running at absolute minimum cost.
It can become really complex. As an example, try to predict resonant frequencies on an entire plant full of rotating equipment. It can be done, but it often requires very expensive equipment and skilled technicians capable of using it to profile a plant. And that's just one potential problem you can run into.
We're often talking about custom plants here, or plants made in very low volume. Making a simulator for that can be prohibitively expensive, since you can't amortize the cost. For particularly important/expensive plants it still sometimes gets done. Though note the simulator might not be able to capture all the nuances of the real world deployment, so it might not be sufficient.
Meanwhile having a separate copy of the plant is also not often viable, once the plants get to any kind of size.
edit: After having 'someone' fact check and remind me: there are quite a number of simulation tools available that can sometimes partially do the work.
The trouble is the simulation tools are either proprietary and thus have only a small mind share among software developers, or academic, and either way, PLC programmers have not gone near them yet.
But a decent LLM should be able to translate the description of a plant to a TLA+ model (or Alloy, or something similar.)
A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
Yes, the fortune 50 company I spoke of was running some of the most advanced data centers in the world. There was literally a team of world class IT specialists walking the same corridors as the industrial automation team but they might as well have spoken different languages.
I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure.
I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
integration tests (which can be trickier when it requires a hardware test bed.)
I use this as a fizzbuzz-type test when I'm interviewing at hardware companies: do they have development hardware in a rack with programmable power supplies and mini-PCs (or similar)? It's a low, low bar for testing, and rules surprisingly many companies.
They'll often just have The Guy running manual tests instead.
Wow. I’d kill for even a guy running my hardware tests. Most of my jobs barely seem to leave time to write the code, before it needs to be scraped off my monitor and poured into the customer’s systems. The ones like you describe with actual hardware labs are a dream when contrasted to those nightmares.
Does management sell unrealistic deadlines to the vendors that buy your solutions? Are you bound by impossible support contracts? Did the firm just let go of a lot of people and expect everyone to finish work quickly? Curious how the economic/management pressures here work as I abandoned hardware very early in my career (from an internship) for similar reasons.
Sounds like we have a similar development path. People fail to understand how complex some of these systems can be, especially since you have to have the project to understand what registers/variables are what. Some when Joe the contractor does a job and walks away, you better make sure they give you the project and you store it in a nice safe place.
A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel.
This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).
I enjoy watching Cursed Controls on YouTube. For him, the gold standard is a flash drive with the program on it physically hanging out in the control cabinet.
Yep, everything is lacking. No version control, ladder diagrams instead of written code, goto statements as primary control flow in code. A $500 PLC with less processing power than a 1985 086 and 1 whole MB of memory. Upgradable with a $200 license to 1.5MB, only programmable with a 20GB monstrosity of a piece of software in which download means copying a program.e to the PLC, and upload means downloading a backup to your laptop. That world needs some disruption really really bad
I was on the cybersecurity side of that, "advising" huge, multinational companies that hold the world together.
Oh boy.
I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here".
I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway.
I often wondered over the years why we have so little significant incidents in the utilities sector.
> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Same with SCADA: just as bad as what you describe.
We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.
Storytime: Once upon a time, I was a young grasshopper at a small consulting firm at which every employee was issued a Windows machine, our source control system was "\Customers" on C: drive, and our deployment strategy was "putty + copy-paste + hot-reload".
One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry.
I gave boss^2 a heads-up that his hard drive might be failing, and that he might want to run a S.M.A.R.T. check on the poor thing. I also asked where the backup hard drives were, because I was brand-new and assumed that I just hadn't been issued one yet.
I checked the supply closet, found no hard drives, popped over to the office admin person, and recommended a deal I'd seen on some WD black drives before I realized that the place had gone quiet.
Everyone looked at me like I was from Mars.
Exactly 7 days later, boss^2's hard drive failed. We lost a week of work and had to zero out a 5 days x 3 employees worth of billable hours. We also ended up delivering late.
The client was pissed.
Based on the nasty looks that I got afterwards, it appeared that the standard assumption was that I had tampered with the drive to prove a point. (Um, nope).
I have since learned to ask prospective employers about their backup strategy.
The worst part -as I mentioned in another comment few days earlier- is most of the network is ancient, dialup, poor architecture designed by people who barely touched a computer, very old OSes, one utility had windows 3.1 a couple years ago! For them as long as it’s working, don’t touch it, because a downtime is far more costly and it will bring many eyes and attention to them than just sit and hope nothing will happen. Physical security is a major gap too, I have been to many locations where the field server panels are just behind the door, the RTUs are just closed with panel keys, even the whole facility are using some old HID cards that you can unlock in few seconds, this is even in R&D that beyond automation but also in robotic ones.
I agree. The problem is this: most companies using PLCs work in isolated networks with internet disconnected, based on confidentiality. There are many difficulties even with using git, because the structure forces you to implement features in a way that makes it hard to separate them. Branching, in particular, doesn't work well with PLC ladder logic—PLC uses proprietary binary formats that make diff tracking impossible, so you're forced into a unified development approach
> This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.
There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.
If that means you need to at least be physically present then I'd say that's a lot of protection already. Means someone in a foreign country can't simply get lucky fuzzing.
Alternatively, small drones exist, and so do low power devices you could slingshot/shoot into a secure area and set to auto wipe after they've done their job. Maybe could even build it out of biodegradable material so it'll clean up in short time.
Why bomb something, which carries its own risk profile and success criteria, when you can make some piece of equipment malfunction just enough to wear out in a year instead of 20 years?
Modern warfare is sneaky. What's the worst you can do without starting an actual war? Plus what's the point of retaliating if the actual combatant was just "an insane person from another country". IE - There's no way to prove he's working for the CCP or IRGC, for example, maybe he's just a crazy person.
hell, you could even fedex them your own remote hands a la LTE modem tied to an rpi.. you could do what you need to by the time someone gets around to opening the box
Looking at whats happening in europe, thats not a barrier. all it takes is a few hundred euros, some time on a dipshit forum and you have a willing gopher who'll go to a given location and run an unspecified payload.
You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
Okay but then is the RF connection really your biggest concern?
I'm kind of worried (probably unnecessarily) that posting ideas would get me on some list, but it seems like there would be many simpler terrorism opportunities once you have physical access.
WWII provides some interesting data points. On the one hand, yes, as Germany occupied numerous neighbours people - even sometimes in the face of group punishment, kept sabotaging the German military and its logistics. In some cases there are literally enemy agents, the Special Operations Executive† but often they're local partisans either working with the SOE or on their own.
But on the other hand, even though the Americans suspected that people who merely looked Japanese might be traitors, AFAIK there aren't any clear examples where the people who were sent to camps actually were enemy spies who'd have sabotaged America given the chance. And in Britain the counter-intelligence operation was so successful that when captured German spymasters revealed their list of agents in Britain, every name was already either working for Twenty Department (20 = XX = Double Cross, we can't resist a pun) or in prison for espionage or dead.
† notably in WWII if as a woman you say you want to be on a ship of the line, or crew front line aircraft and attack the Nazis you will be told women can't serve front line roles and at most you'll be doing delivery runs in relative safety. But if you know the right people to become a spy they will cheerfully send you behind enemy lines even though if caught you will almost certainly be horribly tortured and then probably killed and the government which sent you won't even acknowledge you existed for years.
So, maybe the risk from people who are already where your infrastructure is is much lower that you'd think if you haven't invaded them and occupied their land. On the other hand remote adversaries are definitely always a risk.
Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?
All the siblings might talk a lot about "but ruggedized!", "but realtime!", "but supported!".
But in many cases, it is just a case of vendor lock-in. Ruggedized is just a casing and decoupled I/O-ports, you can get that for your Raspberry Pi. Realtime is a concern, but modern PLCs aren't really realtime anymore. Those are usually some big multicore non-realtime ARM, plus some small realtime-capable I/O-processors that do the really time-critical stuff. Or even just a CPLD to implement some trivial ladder logic cycles. Support matters, but if your PLC vendor goes bankrupt you've drained a few millions. Might be better to get something cheap where you can just stock up a few hundred replacement parts for cheaper.
That's why actually, a lot of smaller PLCs are Raspberry Pi CMs now. The vendor won't tell you, but if you open them up, you'll see.
And the big customers also wised up to the PLC scam and are now moving towards putting their control tasks on standard x86 hardware in the shop floor basement plus realtime ethernet with cheap I/O extenders on the shop floor. Which is why the big PLC vendors now try to sell "virtual PLC" to those customers, so they get at least a little money instead of none ;) https://www.siemens.com/en-us/products/simatic/s7-1500v/
The term "PLC" is a lot more specific than "computer with some GPIO ports." No one that works with PLCs would consider a Raspberry Pi to be one.
These things are walled gardens. You never see the operating system. You can only change their behavior using the vendor's software. They generally run a single program (that you write using the vendor's software) on a fixed scan cycle. They read the inputs, run your program, write the outputs, then repeat.
While you're giving up the nearly infinite possibilities that an SBC gives you, the benefits more than make up for the lack of flexibility. They run (and have parts and support available) for decades. Modules are easy to diagnose and replace. An electrician who isn't a programmer can follow ladder logic and troubleshoot problems. Integrators can quickly come up to speed and understand your code.
There's a reason companies will pay tens or even hundreds of thousands of dollars for these things.
The point I was making is that PLCs have a software stack. You have to know the software and programming languages defined in IEC 61131-3. For all intents and purposes they are just very repeatable compute in an industrial setting, but if you don’t know ladder logic or how to interface with them you might be SoL. Most don’t have any sort of web interface and if they do it operates outside the controllable logic. Same goes for shells. If you want to program it you typically have to use the vendor software.
PLCs are ruggedised and built to survive for years in the kinds of hostile environment that are frequently found in industrial settings.
And PLC vendors typically provide guaranteed support periods that are much longer (like a decade or more) than those commonly provided by conventional computer OEMs.
> If a Raspberry Pi had industrial environment ratings and was certified hard real time, then yes it could be considered a PLC.
Ostensibly yes, but so far I haven't seen anyone really use a PLC in a way that requires hard real time (so far). The cycle on eg a siemens S7-1200 is anyway much too slow for anything really exciting, and a Pi might very well be more reliable in actual practice, were it not for the very unfortunate tendency to eat SD cards. :-P
(And revolution pi actually ships a hardened Pi for industrial use. So that's one way to go about it. I'm not a big fan of that brand, but it's a data-point. Meanwhile in personal experience some regular pi's left in industrial cabinets for one-off emergency monitoring purposes have managed to stay annoyingly alive over time.)
Not really, sure you can use a rpi to control some hw but they are not the same, a PLC usually run rtos, is deterministic (you can predict timing) while rpi relies on linux OS and its scheduler, the PLC also uses ladder language or function block compared to rpi high level language, and obviously PLC industrial grade I/O both analog and digital that also deal with voltage noise that usually happens by field sensors, and environmentally rugged and rated to run non stop compared to rpi.
Now, if you really want to use rpi as a plc, you need something like openplc or codesys as a runtime, add some HATs for I/O, and use protocols like modbus. It will be a software plc but you are missing the hardware certification and other features. Rpi is good as edge computing rather than plc, like processing vision or data logging, it’s why in drones you need the autopilot AND rpi or companion computer, each does certain functions.
I used to think codesys on raspberry pi was pretty much the pessimal application of an ARM processor (and nothing has changed my mind so far). Here you have a chip that is famous for staying stone-cold and doing interrupts in microseconds , and instead you run it hot on a 50Hz (default) PLC loop instead.
Inside this box you have access to the “production” network, if you will. Unfortunately, most SCADA systems implicitly trust their RTUs/PLCs, so this has always been a weak point for the system. Hopefully the situation has improved.
The reality is that critical infrastructure is rarely tested against genuine hostility, except in times of war. There is “cyber” activity going on all the time, but attacks that require physical proximity will probably only happen when things have escalated to hardware. Hopefully the NSA’s of the world run “pen testing” for these companies from time to time.
You can also find all kinds of interesting water infrastructure, often with no electronics, all around town. It rarely has any sort of security beyond a padlock.
With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
> How is it NSA/DHS negligence? Neither is responsible for securing the infrastructure of state and local governments nor private companies.
It is not a duty nor their responsibility. It is however bad for nation overall if companies don't dedicate some resources to security, so a sane administration will do something to advocate for it.
DHS has the Cybersecurity & Infrastructure Security Agency. Their job is to help organizations in the US with protecting public services and physical infrastructure. They are partially responsible for the outcomes.
Unlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.
NSA's job is SIGINT, so it's at least partly their responsibility when the threat is foreign.
For example, the NSA should be successfully infiltrating every major foreign hacking team in the world and monitoring and/or disrupting their activities.
> the NSA should be successfully infiltrating every major foreign hacking team in the world and monitoring and/or disrupting their activities.
I'm sure this exists, but I'd expect CIA to be more involved in any sort of covert operations across nation state lines, especially if undercover humans are involved.
He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.
Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
People start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
The physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
Its not like we lack the labor to do it. Plus how many years or decades of wages for someone to stand around would be required to exceed the cost of infastructure going down even just a few days?
No such thing. There are levels of security, where higher levels mean more cost. And even with the most secure facility, there are no guarantees. And certainly we don't have the funding to provide that level of security for every single facility.
A well designed and deployed VPN link from the satellite location back to HQ (to machines isolated for just this use) would be orders of magnitude safer then what they seem to have been doing, and a reasonable compromise between security and utility.
I mean first you need to figure out the cost of a large municipality getting stuxnetted, and then you need to figure out where to hide when your constituents find out you made that kind of call.
> Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea.
This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem.
No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand.
When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
You can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice.
If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
Right, enclose it in meters thick reinforced concrete walls and let no one near it.
While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access.
Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for.
Do something in between, if you even just apply normal network security you'll be ahead of the pack.
Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem.
Rather than "on/off" I think we need to distinguish between at least four things:
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
The big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.
Things that don’t have a compelling need to be connected to the internet should never be connected to the internet. When there is a need, connectivity should be restricted to the bare minimum required for the task.
A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though.
Obviously we need open source designs.
Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)
If you want to do that sort of thing in a cheap way, then you can.
I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it.
But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork.
On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!
Data diode hardware isn’t terribly expensive, but the software runs into the hundreds of thousands precisely because these are niche products.
To DIY it you need three NICs (or an optical isolator, which is honestly more work than it’s worth) because the transmitting NIC needs to see a link up before it transmits anything. That link is most easily provided by the second port on the transmitting side.
Developing software to reliably move data across the link, and ensure that it’s properly formatted etc will still be your problem.
Well yes this is true but air gapping isn't perfect security, it's important to understand that too.
And true air gapping isn't possible because it'll need to be monitored somewhere central so there'll have to be some vpn or mpls whatever. Meaning it can be hacked.
Having it exposed to the public internet is not good practice but should be far from the only layer in its security.
And really, when a PLC is found unfirewalled on the public internet, you can bet that's far from the only security screwup in that infrastructure. If they won't even handle the low hanging fruit.
Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
If we can beat a security-state airgap by sprinkling USB drives in the parking lot, I think the Iranians can beat an "internal" network by getting someone to click on an email attachment or visit the wrong website on their municipal issued Windows machine.
It's very common for the proprietary software for interfacing with ancient, expensive machines to break after OS upgrades, so they're probably unpatched... you might not even need to burn a 0-day.
Which would then tunnel through the VPN and still hit the PLC. Right. "But we need it on our windows computer" is not an entirely unreasonable ask. But security is only partially served.
We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.
No infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.
what does belong on the internet in a post-mythos world?
one argument: only services which need to be available to unauthenticated endpoints should be default reachable.
all other services should be default unreachable (no data plane until authorized ...then use internet and other networks to establish the connections).
yes, that is not always easy. it is much more possible than it used to be.
and arguably we now need to commit to the tradeoffs of default unreachable services.
I sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.
The security vulnerabilities the NSA would use to spy on a system, or even just use as a jumping point to another system, can also be used by bad actors for other purposes, such as screwing with people's water.
Another vulnerability is natural gas pipeline compressor stations are all internet-connected, often via wireless because they are in remote locations. Simple DOS attack against one will force it to shutdown flow, causing things like power stations to drop offline.
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged.
If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.
If that's an old GE 90-30 (a common FANUC system), it's time to replace it. You're currently in that period when the CPU has reached end of life but the modules are still supported and the vendor has an easy upgrade path. Emerson owns the brand now, and can sell you a new unit that'll run your old program with minimal changes. You can call them and get a list of integrators in your area. It'll be pricey, but you won't need to do it again for another quarter century.
Hmm, so how do they handle the new cyber threats, since most of these are connected? It seems upgrades would need to be performed to address new threats. And just because something is not internet connected doesn't make it immune (eg stuxnet).
That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way.
If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment.
And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
This guy is very happy when someone literally does that: https://www.youtube.com/@Cursed_Controls (great channel btw), because usually he has to deal with worse.
> And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
You'd think Joe from shipping wouldn't be given a key to the cabinet with the PLC.
https://www.knipex.de/schaltschrankschluessel?v=2921
Real locks in such systems are rare, sometimes even disallowed. Usually, a lock with a real key is only used to temporarily prevent switching on a machine during maintenance or such.
As far as I can tell, that thing's $50. Why would Joe from shipping have one?
> Usually, a lock with a real key is only used to temporarily prevent switching on a machine during maintenance or such.
Most locks aren't really meant to resist and adversary. They're meant to communicate who is allowed and who is not, and back that up by making violations a little non-trivial.
Edit: if you need proof, 2 for 6.50€, shipping included: https://www.ebay.de/itm/277671545941
Plus, yea, someone could really make a mess even if they have the best intention. A guy from my company almost uploaded the wrong program to a large bioreactor. At best it would have made the bioreactor just unusable until corrected. At worst it could have broken the equipment or caused a safety issue. This guy had been around the block a few times too. Luckily someone noticed before it was too late.
Like: We took care of the centralized controls for the inmate portion of a jail that was built in the 1980s. We rejiggered the controls for that whole jail back around 2010 or so with new kit and had been taking care of it since then.
All the wiring for all of the individual doors and intercoms was documented in one old 3-ring binder. (There were probably 3 copies originally -- for the architect, the maintenance room, and the equipment room, but just 1 remained and it had hand-written notes.)
One day, around 2020-ish, they called us because the book was missing. I didn't have it (I never once left the equipment room with it), and the co-workers who had been there insisted that they didn't have it either.
As time passed, we'd get occasional phone calls of "Hey, we really could use the book for our building back if you guys ever find it." They weren't angry calls, but sheesh: That book had real value and getting stuff done without it was difficult. So every time when they'd call again, I'd ask around our shop -- again -- if it had turned up anywhere.
And then, finally: One of our guys cleaned out his truck and found it. He must have taken the book out to look at and have a smoke about some problem or other, and then it just stayed in his truck. For years.
This shit happens. People are imperfect and sometimes we screw stuff up.
Is that really an issue? Do PLC vendors have concerns about other vendors turning up and yanking USB drives out of critical infrastructure with reckless abandon? Do the facilities themselves see no issue with potential vendors yanking operational utilities out of their systems while they are running?
It's just a thumb drive, on a lanyard, hanging in a cabinet. It's not like it has guards posted. :)
Maybe Joe (from shipping) took it back to his desk to have a squiz at it and try to passively learn something new during some otherwise-downtime and never returned it -- years ago. That cabinet is in his area, so he has the key.
Maybe Tim (the plant electrician) borrowed it to get something else done, and it unintentionally disappeared somewhere along the way. Tim has all of the keys for all of the things.
Maybe Emily (the ubergeek who prides herself on being able to code her way out of a mess on any system) had it in her office before she got hit by a car and lost the ability to communicate with polysyllabic words. She borrowed Joe's cabinet key after he mentioned an issue.
Either way, it was just a thumb drive that had been inside of the cabinet, and now it is gone.
Yes. I follow up. No one ever knows.
> And that's regular,
Yes
> expected
Yes
> and desired?
No.
And the thumb drive is still gone, isn't it?
---
No, it's not regular. It's not desired.
Does shit never go wrong in your world? If not, then: Perhaps you should start expecting it to. :)
No, we didn't have them, and they told us they last saw them in the back of a cabinet in their own work era.
Turns out those were the unit-specific reload DVDs for half the reconstruction computers and some serial-specific license stuff.
Whoopsie...
Joe from shipping is undefined. It is possible that Joe knows more about PLCs than you think you know about shipping, and that he has a god complex worse than that of a grizzled Haskell programmer. People all over have jobs that don't align with their interests, education, and prior work experiences. That's pretty normal; the bills still need paid and people find work where they can.
> if you let untrained staff maintain electrical systems I think you have some more serious problems.
Or maybe Joe from shipping is a departmental director who has this role because he is a strong leader who takes the initiative to embrace and overcome new challenges. We don't know.
This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.
Maybe today I'd do micropython on ESP32. Download text file from device, edit, upload back on.
An entire class of controller with really shit security now (as in scanning a network with PLC controllers has a non-trivial chance of rebooting or freezing any number of devices on the network) trained to just plug USB sticks in?
I understand the idea, and its laudable. But the entire model of PLC security needs root and branches overhaul.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough.
The pay is also so much worse in my experience.
The result is that anybody who can program well enough to develop software is doing so. Those with any programming skills who remain behind are smart-but-undisciplined programmers, people who have a reason to be in a particular geographic area, too old to think about changing careers, or just plain nuts.
This is what "lack of investment in manufacturing, research, and infrastructure" looks like. Sow what you reap.
Disclosure: I might be one of those people. I came out of grad school in physics research, and programmed an entire plant. Thankfully, that was before it was economical to put every controller on the Internet.
On the other hand, markets are generally smarter than I am. Much like how US citizens living in the EU are basically barred from investing in index funds due to two fundamentally incompatible regulatory regimes, there are almost certainly factors here I'm not seeing and wouldn't see without a couple weeks of research and talking to people who have tried and failed as this.
A guy can dream, though.
Factories face liability already, either due to the cost of "line down" or getting sued. They could pass this liability on to their employees, who would probably cover it through insurance, like doctors.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
Would it be feasible for people to flush their toilets with bottled water for a week?
> I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID <
The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc.
So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?
It's quite reliable, and does not require paying for a team of people who wants to try new things. Once this happens for a long while there's no one in the building left who's going to advocate for change, since the only thing all that fancy new technology brings is the next 5 years of nobody getting a bonus because operational metrics weren't being met anymore.
You have to try really hard to convince these people that the thing that they have seen on a daily basis to work just fine for the past twenty years is no longer good enough, because all their evidence says that it is. And you should also listen to them much more than people who just want to use the latest thing typically do, because they do have the decades of experience seeing how things actually fail.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.
I'd love to see more formal methods come to the field. Part of me says I'll return one day, maybe if AI kicks me out of my software field, but I'd really want to come back at a position I could healthily influence towards safety and correctness.
I was involved in a project that made such a twin for a warehouse. It was simpler in theory because there was no manufacturing process. Just moving objects from one place to another. It was still expensive and took a long time. We tracked traffic heatmap but also collected data on and modelled luminosity for safety. On top of that there was a standard warehouse space optimization.
You're going to run into similar problems as with testing and documenting code. Eventually something will change - a process, a machine, a sensor. You have to stay on top of things and update your model too. Otherwise they will drift apart and not be as useful.
Small plants in the middle of nowhere, especially municipal ones? You're lucky if it was designed this century and not 50-100 years ago. Everyone's just trying to keep old stuff running at absolute minimum cost.
Meanwhile having a separate copy of the plant is also not often viable, once the plants get to any kind of size.
edit: After having 'someone' fact check and remind me: there are quite a number of simulation tools available that can sometimes partially do the work.
But a decent LLM should be able to translate the description of a plant to a TLA+ model (or Alloy, or something similar.)
I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure.
I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
They'll often just have The Guy running manual tests instead.
A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel.
This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).
Oh boy.
I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here".
I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway.
I often wondered over the years why we have so little significant incidents in the utilities sector.
Same with SCADA: just as bad as what you describe.
Never will they ever get my money again for anything. To wit, I miss the days when APC was a brand that I could trust.
One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry.
I gave boss^2 a heads-up that his hard drive might be failing, and that he might want to run a S.M.A.R.T. check on the poor thing. I also asked where the backup hard drives were, because I was brand-new and assumed that I just hadn't been issued one yet.
I checked the supply closet, found no hard drives, popped over to the office admin person, and recommended a deal I'd seen on some WD black drives before I realized that the place had gone quiet.
Everyone looked at me like I was from Mars.
Exactly 7 days later, boss^2's hard drive failed. We lost a week of work and had to zero out a 5 days x 3 employees worth of billable hours. We also ended up delivering late.
The client was pissed.
Based on the nasty looks that I got afterwards, it appeared that the standard assumption was that I had tampered with the drive to prove a point. (Um, nope).
I have since learned to ask prospective employers about their backup strategy.
Incompetence will cost them their job.
If I had fuck you money I'd just call them incompetent fucks and laugh at them.
As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.
And nuclear isn't exactly a great option.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
I'm kind of worried (probably unnecessarily) that posting ideas would get me on some list, but it seems like there would be many simpler terrorism opportunities once you have physical access.
But on the other hand, even though the Americans suspected that people who merely looked Japanese might be traitors, AFAIK there aren't any clear examples where the people who were sent to camps actually were enemy spies who'd have sabotaged America given the chance. And in Britain the counter-intelligence operation was so successful that when captured German spymasters revealed their list of agents in Britain, every name was already either working for Twenty Department (20 = XX = Double Cross, we can't resist a pun) or in prison for espionage or dead.
† notably in WWII if as a woman you say you want to be on a ship of the line, or crew front line aircraft and attack the Nazis you will be told women can't serve front line roles and at most you'll be doing delivery runs in relative safety. But if you know the right people to become a spy they will cheerfully send you behind enemy lines even though if caught you will almost certainly be horribly tortured and then probably killed and the government which sent you won't even acknowledge you existed for years.
So, maybe the risk from people who are already where your infrastructure is is much lower that you'd think if you haven't invaded them and occupied their land. On the other hand remote adversaries are definitely always a risk.
All the siblings might talk a lot about "but ruggedized!", "but realtime!", "but supported!".
But in many cases, it is just a case of vendor lock-in. Ruggedized is just a casing and decoupled I/O-ports, you can get that for your Raspberry Pi. Realtime is a concern, but modern PLCs aren't really realtime anymore. Those are usually some big multicore non-realtime ARM, plus some small realtime-capable I/O-processors that do the really time-critical stuff. Or even just a CPLD to implement some trivial ladder logic cycles. Support matters, but if your PLC vendor goes bankrupt you've drained a few millions. Might be better to get something cheap where you can just stock up a few hundred replacement parts for cheaper.
That's why actually, a lot of smaller PLCs are Raspberry Pi CMs now. The vendor won't tell you, but if you open them up, you'll see.
And the big customers also wised up to the PLC scam and are now moving towards putting their control tasks on standard x86 hardware in the shop floor basement plus realtime ethernet with cheap I/O extenders on the shop floor. Which is why the big PLC vendors now try to sell "virtual PLC" to those customers, so they get at least a little money instead of none ;) https://www.siemens.com/en-us/products/simatic/s7-1500v/
These things are walled gardens. You never see the operating system. You can only change their behavior using the vendor's software. They generally run a single program (that you write using the vendor's software) on a fixed scan cycle. They read the inputs, run your program, write the outputs, then repeat.
While you're giving up the nearly infinite possibilities that an SBC gives you, the benefits more than make up for the lack of flexibility. They run (and have parts and support available) for decades. Modules are easy to diagnose and replace. An electrician who isn't a programmer can follow ladder logic and troubleshoot problems. Integrators can quickly come up to speed and understand your code.
There's a reason companies will pay tens or even hundreds of thousands of dollars for these things.
PLCs are ruggedised and built to survive for years in the kinds of hostile environment that are frequently found in industrial settings.
And PLC vendors typically provide guaranteed support periods that are much longer (like a decade or more) than those commonly provided by conventional computer OEMs.
All the ones I've encountered in the wild ran VxWorks
Ostensibly yes, but so far I haven't seen anyone really use a PLC in a way that requires hard real time (so far). The cycle on eg a siemens S7-1200 is anyway much too slow for anything really exciting, and a Pi might very well be more reliable in actual practice, were it not for the very unfortunate tendency to eat SD cards. :-P
(And revolution pi actually ships a hardened Pi for industrial use. So that's one way to go about it. I'm not a big fan of that brand, but it's a data-point. Meanwhile in personal experience some regular pi's left in industrial cabinets for one-off emergency monitoring purposes have managed to stay annoyingly alive over time.)
Honestly, I think it's an advantage at this point. Way too much diversity to easily attack remotely.
Now, if you really want to use rpi as a plc, you need something like openplc or codesys as a runtime, add some HATs for I/O, and use protocols like modbus. It will be a software plc but you are missing the hardware certification and other features. Rpi is good as edge computing rather than plc, like processing vision or data logging, it’s why in drones you need the autopilot AND rpi or companion computer, each does certain functions.
Na, bro! Today we call this "public private partnership" :-D
In water/wastewater much of infrastructure is physically remote and physical security is the typical engineering trade offs. https://validmfg.com/product/lift-station/
Inside this box you have access to the “production” network, if you will. Unfortunately, most SCADA systems implicitly trust their RTUs/PLCs, so this has always been a weak point for the system. Hopefully the situation has improved.
The reality is that critical infrastructure is rarely tested against genuine hostility, except in times of war. There is “cyber” activity going on all the time, but attacks that require physical proximity will probably only happen when things have escalated to hardware. Hopefully the NSA’s of the world run “pen testing” for these companies from time to time.
The largest threat isn't bombs falling on our heads, it's incompetent fools leaving the door open to their enemies.
These aren't mistakes that can be excused. Failing in one's duty to steward important infrastructure must mean immediate replacement of leadership.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
They should provide guidance, but I’m not sure we really want the NSA inside of networks more than they already are.
If voters and CEOs don’t want to spend the money required to secure their infrastructure, that’s on them.
It is not a duty nor their responsibility. It is however bad for nation overall if companies don't dedicate some resources to security, so a sane administration will do something to advocate for it.
https://www.cisa.gov/
https://www.cbsnews.com/news/cybersecurity-agencys-top-recru...
By mandating backdoors.
“Have no access to” I have doubts
While that’s not the job of the NSA, they do produce a lot of good cybersecurity guides.
For example, the NSA should be successfully infiltrating every major foreign hacking team in the world and monitoring and/or disrupting their activities.
I'm sure this exists, but I'd expect CIA to be more involved in any sort of covert operations across nation state lines, especially if undercover humans are involved.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
Folly to think otherwise.
You can keep saying it’s more secure than currently, and that’s good and all. It still doesn’t matter against the adversaries being mentioned.
Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them?
We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem.
No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand.
When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access.
Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for.
Do something in between, if you even just apply normal network security you'll be ahead of the pack.
Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem.
[0] https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR...
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
Obviously we need open source designs.
Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)
I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it.
But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork.
On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!
To DIY it you need three NICs (or an optical isolator, which is honestly more work than it’s worth) because the transmitting NIC needs to see a link up before it transmits anything. That link is most easily provided by the second port on the transmitting side.
Developing software to reliably move data across the link, and ensure that it’s properly formatted etc will still be your problem.
And true air gapping isn't possible because it'll need to be monitored somewhere central so there'll have to be some vpn or mpls whatever. Meaning it can be hacked.
Having it exposed to the public internet is not good practice but should be far from the only layer in its security.
And really, when a PLC is found unfirewalled on the public internet, you can bet that's far from the only security screwup in that infrastructure. If they won't even handle the low hanging fruit.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
It's very common for the proprietary software for interfacing with ancient, expensive machines to break after OS upgrades, so they're probably unpatched... you might not even need to burn a 0-day.
one argument: only services which need to be available to unauthenticated endpoints should be default reachable.
all other services should be default unreachable (no data plane until authorized ...then use internet and other networks to establish the connections).
yes, that is not always easy. it is much more possible than it used to be.
and arguably we now need to commit to the tradeoffs of default unreachable services.
>nsa: what no, stop that
sounds like cope for a bunch of felons that management, its director and congress can't get a handle on.