Interviewee: Justin Shneck, Co-Author of NervesProject Software Engineering Fellow at Very
Security is a big subject, but that doesn’t mean you can’t learn a lot in a short amount of time. We know people are busy, but for those that are still hungry for better ways to approach securing their organizations or clients, we’ve created Small But Mighty.
Twice each month, hosts from Attila Security hold a 15-minute coffee chat with innovators and thought leaders in the cybersecurity space to provide big insights in a bite-sized format.
Watch the video conversation here, or check out the summary or full transcript below.
Learn more about the topics discussed in this video:
- Hardware vs. Software for IoT Security
- What Engineering Leaders Need to Know About IoT Security
- Manual vs. OTA Firmware Updates for IoT
This is just the beginning! If you are interested in learning more about Justin's work or over the air firmware updates, join Justin's session at our IoT Integrator's Summit in July!
In this Episode
Meet Justin Schneck
Justin, coauthor of an IoT framework called Nerves and a software engineering fellow at Very, joins Joe Gatto and Stacy Willis to discuss the current environment of security as it relates to over-the-air firmware updates for IoT devices.
Motorcycles to coffee & IoT in-between
The discussion with Justin kicks off with a confession that his introduction to IoT security came from his personal interest in leveraging his mobile development experience to build a framework that would allow him to remotely start his motorcycle from his phone. This was clearly before the days of commercials featuring consumers starting their Buicks from their airplane seats!
Challenges of IoT Device Security
“Security is something that is always on our mind when it comes to how we protect ourselves….unfortunately, it becomes a little bit [vaguer]....a little less personal feeling when it comes to the things that we don't see.”
Unlike personal devices that are in-use daily, an increasing number of devices are not being used or managed by a human - sometimes for years. It’s that rise of remotely-managed devices that presents unique IoT security challenges.
First, there isn’t a human-user to select the “update software” prompt, potentially leaving devices not designed to accept updates without interference from a user vulnerable. Leveraging cybersecurity best practices during the design and development of these devices can mitigate some of this.
While this sounds like a fairly straightforward challenge, the truth is, the bar of cybersecurity continues to be raised when it comes to creating self-sustaining technologies. The ever-evolving demand from consumer and business use of IoT requires developers to continue to push their understanding and solutions.
But, if the solution for the first challenge of user-less devices is to create devices that are self-sufficient and regulating, this gives birth to a secondary, and arguably more difficult problem.
The question must be asked, “How do you protect secrets on a device that doesn’t have a user to log in and type a password?”
The common answer is by using a crypto authentication processor.
The use of a crypto authentication processor
A crypto authentication processor is a chip found on the board and when a device is manufactured, you’ll have it create a key inside of itself and then share the part of that key that's public with users that need access.
These are known as public-private keys.
The device provides public keys verifying the authenticity of the device and information it’s sharing, but it never reveals what’s actually stored inside.
These processors are popular and are often required in modern-day systems. For example, any devices that seek to be included in Apples’ HomeKit must have a crypto authentication processor.
This is a best practice as we continue to evolve how we protect communications with IoT devices that communicate automatically with the outside world.
Using a crypto authentication processor is also interesting from a security perspective, because it’s leveraging hardware to create a root of trust (RoT), essentially protecting in two directions.
Not only does it stop access into the IoT device that's out in the field -- it’s also the central location collecting data from that device. This makes it critical that it can be 100% assured that that data is coming from the real device and not from someone in the middle trying to feed fake information.
What does this mean for firmware updates?
Firmware updates are secured in this environment because the updating server has the ability to verify that the device it's expecting is actually the device it is connecting to.
Leveraging these keys, you can even use part of that key to be able to sign the updates that go through, making it so you can encrypt the updates themselves to provide extra levels of security.
This allows you to make sure that, in those cases, the software that goes across the wire can't be inspected because it's protected by the SSL certificates that you're also handing back and forth, ultimately creating a system where your communication with the outside world is protected.
This also means the contents of the disc that you have locally can be protected, with this encryption using a similar key signing mechanism as well.
All of these are steps that are taken to obfuscate things and make it much more difficult for a malicious actor to make you out to be a target. It's very important to be able to at least take these steps to increase your IoT device security.
How can we ensure updates are being applied reliably?
Everything is digitally signed, so nearly everything can be verified at the highest level. But more importantly, when applying updates, it's important that we treat these systems differently than we’re used to.
When you update your computer, you usually download the files and apply them on top of the current operating system, and then reboot and things will magically work. Sometimes, if you have a bad day, they won't, and you’ll just recover from a backup. That's fine because you are human and you were there to fix that problem.
But, if you deploy an update to a device that exists in a shed in the middle of nowhere, there’s the potential for weather-related interference that prevents an update or an update wasn't necessarily tested for that specific model of the device and it was accidentally deployed to the field.
Justin shared that in those situations, you need to be able to have some serious contingency plans in place.
“With Nerves, we institute what's known as a blue-green deployment mechanism onboard. This comes, this term comes from the web world. And really it's just AB partitions.
And, so we have like a slots system where you have an A slot and a B slot for firmware that goes in from where we string the entire thing as an entire package, the entire operating system, and application all at once into the slot, that's not currently booted.
So, if you're booted on A, we'll stream it into B and then what happens, we say, okay, now we're done streaming into the B slot. Switch reboot into B, and, if you fail to boot into B, you have a last known good working firmware to fall back to so you can try again or you know, not become the dreaded brick.”
What can happen without a system in place?
Other than the obvious security nightmare of NOT having a crypto authentication processor in place, allowing for unfiltered access to remotely-managed devices and their data, the biggest problem is that a lot of these IoT devices need to have remote data links to function as intended.
The volume of data that is required to maintain security updates can be massive. So, how do you access massive amounts of data?
The easiest thing to reach for is, "Oh, I'll just put a cellular modem on it and call it a day."
But, connecting to a cellular network as the permanent solution can be expensive. The monetary investment that would be needed to update multiple devices via a cellular network isn’t a feasible option for an organization running hundreds or even thousands of devices, or even five devices for a consumer’s home. At $15 per device, per update, it isn’t a sustainable model.
Justin says this is why Nerves exists.
“For one, with Nerves hub, our firmware device management and firmware update network, we are applying some additional features to it to help alleviate this problem, at least one, on expensive connections.
For one, we're working or we're allowing you to then resume download of firmware. So if you request a firmware update from a server and the server starts to download updates, and you're on a cellular link, and let's say the tower kicks you off for a brief moment and you come back online, that you don't have to stop and then start over. You can resume with minimum bit loss so that those expensive bits are not lost. They're saved on the device.
And then a next step to take is to do something a little bit more extreme, which is to be able to support.
Because we implement from the start this AB partition strategy, know that on the other partition when we stream in, that we can balance the two by writing the patch and then having the patch pull the data that it needs from the other side to populate itself to fill in the gaps.
And once that happens, the original files that have then been reconstituted on the other end are digitally signed. We can check all their signatures and make sure that they're appropriately rehydrated in a way. And so that can save a considerable amount of data.”
IoT presents exciting new features and functions but also brings with it serious cybersecurity challenges that MUST be addressed.
When it comes to over-the-air firmware updates, Justin, a coauthor of an IoT framework called Nerves and a software engineering fellow at Very, suggests the usage of a Crypto Authentication Processor.
This allows the device to self-authenticate with the updating server without revealing any internal data.
Justin also shares how Nerves solves the data-expense problem by making the data transfer as efficient as possible.
Stacy Willis(00:00): Welcome to the inaugural episode of Small But Mighty. Security might be a big subject, but that doesn't mean people can't learn a lot about security in a short amount of time. We know people are busy, but they're still hungry for ways to approach securing their organizations. And so twice each month we're going to be hosting coffee chats with innovators and thought leaders in the cybersecurity space to provide big insights in a bite-sized format.
Today we are here with Justin Schneck, a software engineer at Very.
Hey, Justin, tell us a little bit about yourself and what you've been working on lately.
Justin Schneck (00:34): Hi. Yeah, I've been working with IoT devices for quite some time now. I sort of become obsessed with them a long time ago. Early on I was a mobile developer, but then I decided that I wanted to be able to connect that together with some things that I love to do, like motorcycling. I had this idea to build a system that I could remotely start my motorcycle from my phone and found that there was a lot of complexity in connecting all those pieces together and making sure that you can communicate effectively with cloud devices and mobile phones.
So I've spent the last eight years or so building and coauthoring a framework for building IoT devices called Nerves. And today we have plenty of companies that are launching products on our platform and deploying them out into the field with nerve sidebar.
Stacy (01:25): Are any of them remote start motorcycles?
Justin (01:28): I actually you know, things amazingly come full circle. When you write a system that helps you build the thing that you dreamed to make a reality, you get so invested in building the thing that helps you build that product that you kind of put that to the side just long enough that you find somebody out there that goes, "Hey, look what I'm doing." And you're like, ‘that's so cool. I wanted to do that’. So yes, there is somebody who's working on that. There is a Japanese colleague that we have that's working on that
Stacy (01:58): Well, then that dream will shortly become a reality.
Justin (02:02): Yes. Yeah, and it's so great because I could facilitate that coming to existence.
Stacy (02:08): Well, that's interesting and I'm definitely going to come back to our remote start motorcycle at the end of our conversation. But today I know we wanted to talk a little bit about how we can secure over the air for more updates for IoT devices. So to start with, can you tell us a little bit about why this particular topic is so important and who might need to be worried about it?
Justin (02:31): Sure. Security is something that is always on our mind when it comes to how we protect ourselves. And unfortunately it becomes a little bit more vague or I should say a little less personal feeling when it comes to the things that we don't see.
Like the digital space in our lives that connects our devices to the cloud and to services and providers and, and, and we need it there more than anything because of the exposure that these devices have.
And you know, when we talk about security we should clarify that. There are two aspects of this too that we can, that we can talk about the physical security of a device like that, things that somebody has physical access to it or digital security of the device, and that that would be like communicating with servers and external systems.
We're not going to get into physical security and all that. I mean, if you can have physical access to a device that's, that's complex protecting that, but digitally, yeah, that, that space it's really important because there's overtime software, it's malleable and it's running out there in the wild.
Every day you'll notice there are updates on for things like your phone or your laptop or computers like those that are highly connected and available. That's a security fix that was identified and patched. And it's important that those fixes get applied to these devices so that we can all move forward and stay protected.
But it's a whole other problem when it comes to how to maintain those in the wild for these boxes that might be installed. You know, in a shed, in the middle of a field that runs a power set up or maybe helps operate some farm equipment for sensors or, or readings and things like that.
Stacy (04:17): And I would imagine most of the devices are unmanned, right? So you don't have the person that's saying, remind me to put this update on later. You have to do it automatically.
Justin (04:24): Yeah, exactly. And, and you know, that's a really good point actually, because when it comes to the securing of these devices, yeah. Then, the human, when an update comes into your computer, you, you authorize that, that action. Right? But these, these IoT things, they're like, they're, there are no humans that operate them. They're supposed to operate on their own. And so how do you protect secrets on these, these things, these tiny little computers. And so there's, there's best practices ways to be able to go about this that unfortunately have a higher, a higher bar to entry.
It's just more difficult to be able to do this kind of programming. And so oftentimes you'll find that like young developers or people who aren't really versed in the security vulnerabilities that can be, that can be created or opened.
If you were not to embrace best practices they can naively put these into play and make it easy for somebody to be able to go in and, and get into devices. The most common one is ‘shared credentials’, right?
So we, we always hear about situations where you buy a home router for your home and you go and install it and for a long time there they would come with, "Oh yeah, you logged into the home, username admin, the password is ‘password,’” and no one would change it, right?
So then everybody knows how to be able to get into everybody else's router if they have access to some sort of part of the network. Right. And then, and then the interesting part about that is California passed a law around 2018 or so that bands from being able to be sold those devices from being able to be sold. So they changed it.
But there's, there's another step now. There's another law that came about that protects against keys. They were, they would go, okay, I won't put us, I won't put a known password on it, but I'll put this crypto cryptography key just on the device and it will be the same one on every device. Well then there were, that was, that was just a workaround of the same problem. Right. so anyways,
Stacy (06:29): You know, one, you know them all.
Justin (06:30): Exactly. You want to minimize the blast radius if something were to happen, if somebody were to compromise one of these devices, cause really security is not a binary thing. It's, it's like how secure you are versus, you know, in the, in the case of IoT device security. So all of this, and to say like the crux of this problem.
How do you protect secrets on a device that has no human to be able to like internally, like login, like type a password in like and, and so that answer commonly is found by using what's known as a crypto authentication processor. So it's this chip they put on the board and that chip when, when you manufacture the device, you say here "create a key inside of yourself, create some secret, and then give me part of it that's public."
That's not the private one. It's, it's known as public-private keys, right? So give me the public ones so that I can verify that anything you tell me is accurate, but I'll never see the secret that exists inside of you. And, and so these, these processors are popular and often the requirements in modern-day systems like anytime you have an Apple device that participates in-home kit for example, that does interactive home mechanism requires to have one of these crypto authentication processors.
And so that's, that's like the best practice way that we can at least get started in protecting the communication that devices have with the outside world.
Stacy (08:00): That's interesting. And if you actually think about it, it's security. Using hardware to create a root of trust essentially is security in two directions, right? Not only am I stopping you from getting onto the IoT device that's out in the world out in the field, but also I as the central location collecting data from that device can be 100% assured that that data is coming from the real device and not from someone in the middle trying to feed me fake information.
This is really interesting. It came up in another interview we did with an IOT specialist over at HP. So we talked to them about something similar using hardware to create that, that root of trust in IoT.
Justin (08:37): Yeah. that's referred to commonly as a client-side SSL in public key infrastructure. And it's very, it's just like an additional step that happens on what we're used to today when visiting websites that are like HTTPS, that server-side SSL lets you trust that you're going to the site that they claim to be.
And then yes, the second step is then that the server would ask you for your certificate the client and validate that you are who you say that you are. Right. That would mean that the client just holds a certificate as well. And those are the certificates that get buried inside of that crypto authentication processor.
Stacy (09:16): Very interesting. And so then how does this apply to firmware updates over time? Like how do you typically see this now? Now used in the use case that we're talking about.
Justin (09:25): So firmware updates are secured in that case by means of allowing the server, the firmware update server to verify that the person who's requesting the firmware can validate that they're who they say they are. That's the device that they expect it to be.
And then when you apply those, you can even use part of that key to be able to sign the updates that go through so you can even encrypt the updates themselves and go extra levels of security to be able to make sure that in those cases the software that goes across the wire can't be inspected because it's protected by the SSL certificates that you're also handing back and forth. And so your communication with the outside world is protected.
And then also the contents of the disc that you have locally can be protected, with this encryption using a similar key signing mechanism as well. So that way physical access as a brief touchpoint can have a little can rest a little bit more assured.
Stacy (10:34): Yeah, you don't need a lock and key as much. Great.
Justin (10:39): All of these are just steps to just obfuscate things and make it much more difficult for somebody to make you out to be a target. It's very important to be able to at least take these steps because you know, you're not going to be easy prey at that point then.
Stacy (10:54): Right. And so how then can we be ensured that updates are being applied reliably, right? So if there's not a human interacting with the device to make sure that the update happened and then use it, what, you know, what mechanisms do you usually see to put in place to make sure that those items are actually being applied and like patches or anything else that's coming across over the air is working correctly?
Justin (11:19): Well, everything is digitally signed, so we can check some, all of it at the top layer. But more importantly, when we apply updates, it's important that we have that we treat these systems a little bit, with a little bit more of a special case. Right.
And when you update your computer usually you'll download the files and then you'll just apply them on top of itself and then you'll reboot and things will magically work. And sometimes if you have a bad day, they won't, you'll just recover from a backup. But that's fine because you are human and you were there to fix that problem.
But you know, if you, if you deploy an update to a device that as we've talked about would exist in the shed in the middle of nowhere, there’s weather system and that device has a bad experience applying that update or that update wasn't necessarily tested for that specific model of the device and it was accidentally deployed to the field.
Justin (12:08): Then in those situations, you need to be able to have like some serious contingency plans in place. With Nerves, we institute what's known as a blue-green deployment mechanism onboard. This comes, this term comes from the web world. And really it's just AB partitions.
And so we have like a slots system where you have an A slot and a B slot for firmware that goes in when. We stream from where we string the entire thing as an entire package, the entire operating system, and application all at once into the slot, that's not the that's not currently booted.
So if you're booted on A, we'll stream it into B and then what happens is we, we say, okay, now we're done streaming into the B slot switch reboot into B, and if, if you fail to boot into B, you have a last known good working firmware to fail back to so you can try again or you know, not become the dreaded brick.
Stacy (13:05): Right, right. Which is still just a brick.
Joe (13:11): Justin, I was going to ask that. Is there some sort of retransmission capability built in there? So if the, if it, if you're sending the update, you sense of failure and it's not happening. This all automatic, this AB swapping sort of thing.
Justin (13:28): Yeah. In our systems we have them designed for a lot of the industrial platforms to be able to do some more of an automatic fallback approach. We do support some DIY maker hardware like Raspberry Pi systems that currently in those cases don't out of the box, but we're working to add that support as well.
Now our OS will run on most anything that'll support embedded Linux. And so it's not just like the Raspberry Pis and things of that nature or what we offer. Currently as far as base, like devices we support. But building that future in is, is not terribly difficult to do. And we're starting to work documenting how to do that on additional custom systems. Gotcha.
Stacy (14:18): And so the alternative to what we're talking about today, so what, what might be happening if they don't have a system in place? Like what you're talking about is people might just be leaving devices without updates in the field, right? Is that pretty much what's happening? If they don't have something like this setup.
Justin (14:35): Yeah. and there's a lot of reasons that just happens because the biggest, the biggest pro problem is that lots of these IoT devices need to have remote data links. And the easiest thing to reach for is, "Oh, I'll just put a cellular modem on it and call it a day." But connecting to a cellular network can be expensive.
And it's funny that I say that, but as a consumer, I go like, “Oh, like $50 a month? I can get like unlimited data on whatever, whatever mobile network in the United States and I can stream whatever I need to all month long and never worry about my data usage.” And then we have global SIM network providers. Like, there's a handful of them out there, but, the rates are on average costing, let's say 20 megabytes, which is not a lot of data.
Justin (15:33): You know, it's a size of like maybe a handful of pictures. Streaming that to a device costs in 2015, $20 or so and on, on a lot of these popular networks. If you were to just sign up and be a nobody, you know, and not sending like you just signed up for service.
And so if I were to update an entire system, like a big OS system you think of like ubuntu or desktop systems, windows, Mac OS, these updates are like hundreds of megabytes. Now, you know, when it comes to our, our software, we try to promote really lean ones. So we're trying to be able to say like hello ass and everything, like 20, 30 megabytes. But even still, if you, if you push 30 megabytes, that's being like $15. Like that's a lot of money to put out an update.
So there's no, the businesses are not being incentivized to keep these devices up to date. Either big telecom was not ready to bring on all of these devices at once or they're just completely corrupt and they're not providing the right business case to be able to keep consumers devices secure. I mean, people are more incentivized to stream movies like Sharkando on Netflix for hours at a time for $50 a month in their data plan to stream gigabytes of data than it is to be able to keep some of our most important devices that are out in the field up to date and secure. Right.
Joe (16:58): It sounds like, yeah. The business models are not quite suited for devices especially, you know, you talk about the expense, but then with, the volume of data, but then there's the breadth of all the sensors, right? That was just one sensor and you'd multiply it times hundreds and then it gets really expensive.
Justin (17:18): I could feel for them, if you're a telecom company and you're onboarding like thousands of new SIM numbers and devices, these towers reach a certain capacity. Just like, we see Wi-Fi equipment in our homes start to slow down. The more devices show up.
Or you know, if you've ever been in a public place where there just is lots of phones all around you your service ends up suffering just because of all of the noise in the area. That's the same situation now. So is the answer to, you know, set up your own private LTE network or, or something like
Stacy (17:50): That. I mean, I imagine you can only do that if you're of a certain size and have the, you know, the wherewithal to be able to execute something like that. But is that the only answer that people have or
Justin (18:00): There's a couple of ways to approach it. For one with Nerves hub, our, our firmware device management, and firmware update network for Nerves we are applying some additional features to it to help alleviate this problem, at least on, on expensive connections. For one, we're working or we're allowing you to then resume download of firmware. So if you request a firmware update from a server and your server and the server starts to download updates and you're on a cellular link and let's say the tower kicks you off for a brief moment and you come back online that you don't have to stop and then start over, you can resume with minimum bit loss so that those expensive bits are not lost, they're saved on the device. And then a next step to take is to do something a little bit more extreme, which is to be able to support.
Justin (18:52): We're adding support for a binary Delta firmware update. So patch updates. And the way that this works is the server itself will know what version of firmware is on your device currently and will know what version of firmware we're trying to send to your device and on the server what we'll do is we'll capture the differences in the in all of the bits between the two, the systems and the most efficient way possible to create just the content that changed as a binary representation. And send only that down the wire to the other end.
Because we implement from the start this AB partition strategy, we, we know that on the other partition when we stream in and you try to, and once you stream into there that, that we can balance the two by, by writing the patch and then having the patch pull the data that it needs from the other side to populate itself to fill in the gaps.
And then, and then once that happens, because the original, the original files that are, that have then been reconstituted on the other end are digitally signed, we can check all their signatures and make sure that they're appropriately rehydrated in away. And so that can save a considerable amount of data.
Stacy (20:14): Right. So it's essentially about finding a way to make the process as efficient as possible is the best solution here.
Justin (20:20): Yeah. LTE might not be the answer for you. You might not need that long-range solution. And there are other options available. Like LoRaWAN with long-range radio or Wi-Fi, mesh is even becoming popular as well if you need to spend long distances.
Joe (20:38): Yeah, I even heard that satellite providers are now very big into IoT using their bandwidth for that given their et cetera.
Justin (20:49): Yeah, we'll all hopefully be able to play with Starlink soon. Right.
Stacy (20:54): We can dream well that is all the time that we have for today. Been really great grabbing you for this 15-minute block and really pulling a ton of information out of your brain in such a short amount of time. But I do hear that you are quite the coffee connoisseur, so no coffee chat would be complete without discussing some important opinions about coffee. Do you have a recommendation as we for all of our watchers that that may be interested in a good coffee?
Justin (21:24): Sure. Well, today I'm drinking Abracadabra Coffee Co., cold brew from Vermont. It's a nice roaster that I found in my area, well close by. They've been really nice in shipping out during this time. And but when it comes to brewing coffee I was at the end of last year I ended up spending some time in Japan and I had the pleasure of being introduced to what I believe is the most scientifically accurate or I should say consistent [method for brewing coffee].
It can produce the most consistent cup of coffee, the siphon brewing method and that anyways, to break it down immersion styles brewing techniques like using a siphon or an arrow press are far superior to drip-style coffee techniques like pour-over or drip systems as far as minimizing the number of variables that go into coffee producing a cup of coffee. The siphon controls most of them and will, it will give you a very consistent cup. Now you can still do it wrong. So you can, yeah, but, but at least you'll have a consistently bad cup of coffee though.
Stacy (22:44): Well, so I know what my Amazon shopping list is going to include next time. I've been on the quarantine, everything from Amazon, so they probably are wondering what I'm buying next. But now I know. Awesome. Well, thank you so much for your time, Justin. We appreciate it and we'll talk soon.
Justin (23:02): Thank you.