Hello Congress! So happy to be here today.
I'm Marius, also known as NSR, as day job an assistant professor
over at the University of Birmingham, but all of this, what
we are talking about today, was done as a side project.
My research mostly focuses on a lot of security of embedded systems, a little bit
of binary exploitation, microarchitectural exploits and all the fun low-level stuff.
Cool. Yeah, my name is Thomas Roth aka Stacksmashing.
I'm a founder at hextree.io together with LiveOverflow.
And yeah, I'm a security researcher for all kinds of fun stuff.
And yeah, I try to run Doom on as many devices as possible. Cool.
Yeah. So let's have a look at what do we have brought with us today?
What will this presentation be about?
We first introduce a little bit the RP2350, then move on talking
about the related hacking challenge, and then introduce all the five
different attacks which were found, and really zoom in on two of these.
Spoiler, these two attacks were found by us.
And to end the talk, we will later on talk a little bit about the lessons learned,
and really what we thought comes out positive out of these hacking challenges.
So, let's get started.
stacksmashing: Yeah.
Let's talk about the RP2350. So, if you've followed me in the past, it's kind of
a meme that I try to hack everything with the cheapest microcontroller possible.
And so, I did a lot of stuff with the RP2040, the first chip of Raspberry Pi.
And so, one morning, I woke up, and I had an email
saying, "Hey Thomas, can you do some hacking for us?
We are building a secure RP2040"
Fun fact, I actually almost missed that email because I read it in bed
and I thought it was spam because I get a lot of very similar spam.
Luckily, I scrolled to the bottom and saw CTO,
Raspberry Pi, and I said, oh, I should reply with yes.
And so, yeah, the RP2350 is a Cortex-M33 microcontroller.
It has two cores running at 150 MHz.
It's an ARM chip, but also it's a RISC-V chip because
apparently they had some transistors left over and decided
to also, just for the hell of it, put a small RISC-V core in.
But that's kind of the boring part, at least to me as a security
researcher, because it also has a lot of security features.
It has TrustZone-M, it has glitch detectors, a redundancy coprocessor, secure boot, and OTP.
And, you know, TrustZone-M is very familiar to me – back in Leipzig,
I talked about this a couple of years ago.
So, very interesting microcontroller, and even better,
it's just $1. And so, it's a very, very interesting chip at
that price point, especially the active glitch detectors.
Because in the past years, if you've seen attacks on devices,
almost all devices nowadays, you know, disable JTAG, you
can't read out the flash externally, and so on and so forth.
And one technique that is very commonly used now is called fault injection,
more precisely voltage fault injection, or glitching, as we call it.
Now, the idea of voltage fault injection is that we take the power supply
to the chip, and then we drop it for a couple of nanoseconds at just the
right point in time, and trying to hit a single instruction with this.
For this, we need a trigger event, so something to base our
pulse off of, a very precisely calculated delay, and then a
pulse, you know, where we drop the voltage, the supply voltage.
Funnily enough, we used to do this a lot with, like, FPGAs and more expensive
equipment, but the predecessor of the RP2350, the RP2040, is actually nowadays a
very common fault injection tool, because it happens to have very precise I/O timing.
In fact, it's so common for fault injection, that if
you take a look at, for example, the Nintendo Switch mod
chips, they actually glitch the main SOC using the RP2040.
And so, yeah, it kind of went from being this lab attack that is only available
to, let's say, experts, to something that you can kind of buy off the shelf.
nsr: Now, coming back to the RP2350, or in fact, also the RP2040
already, there's something amazing about these chips.
They're not only cheap and useful for fault injection,
also Raspberry is quite transparent about them.
They release a lot of information.
For instance, about the RP2350, they have this data sheet published.
If you look all the way up there, very small, it's written 1,357 pages.
That's the right amount of length of a data sheet for a chip.
It has most of the information you want just in there.
But even more, also the boot ROM, so the part of code which is on the read-only
memory of the chip, is not only documented, the source is openly released.
And not only the source, also the binary artifact.
So, no need to dump the boot ROM.
We can look at the source code and the binary code alike, see if they match.
And this is really a nice, nice entry point for security research.
We will get into this later.
Then, additionally, going even more transparent, Tom
quickly mentioned that it's an ARM and a RISC-V chip.
And the RISC-V design running on the RP2350 is likewise open source.
So, you can go to GitHub and look everything up about the Hazard3 core.
Now, with this transparency, we could dive really deep
into the countermeasures which are deployed on the chip.
There were five of them, if I recall correctly.
So, let's go through them one by one.
One of the core security features the RP2350 had is the one-time programmable memory.
So, this is an anti-fuse memory which is implemented
to be written to only once and really only once.
So, it holds all the different configurations of the chip, the important bits
which are needed both for hardware bring up and also the boot ROM later on.
So, for instance, it enables the different cores, right,
whether we want to run from the ARM core or the RISC-V core.
It selects which of them are active or selectable early on during boot.
It also stores information whether debug access is enabled or disabled.
So, you can write to the one-time programmable
memory, set a flag and say debug should be disabled.
Then, it also configures the glitch detectors in different types of strengths.
We will get to that.
And it also has, most importantly, the flag whether secure boot is enabled or disabled.
Once secure boot is enabled, it can never be disabled.
And the idea of secure boot for the RP2350 is that only signed firmware can be running.
So, the hashes of the cryptographic keys used are
also stored on the one-time programmable memory.
And to make sure that this is really a secure boot chain, this OTP also supports locking.
So, besides just writing it once, you can also write configurations
to lock down the different pages so that they cannot be written again
or even that they cannot be read out again if you're, for instance,
storing some secrets there and are worried about them being extracted.
The second important security feature present on the RP2350
is the redundancy coprocessor, or RCP in short, and it
brings a lot of different additional security features.
It's just a small coprocessor running aside and has multiple features.
For instance, hardware enforced stack-canaries.
So, no need to program in the secure boot chain or
in the boot loader any type of stack canary check.
This can be done directly by coprocessor accesses.
It also allows to add randomized delays, mainly, again, as
a countermeasure to fault injection or glitching attacks.
Because if you have random delays, it gets harder for an attacker
to really synchronize and to know where to inject the glitch later.
We will look into that later into the talk.
Then it also has something which is called sequence count checking.
So, different calls to the coprocessor can make sure whether the
code is running in the programmer intended sequence of operation.
Idea here is if you were to mount somehow, if you can bypass the
hardware for stack canaries and can mount code reuse attack, you would
end up probably having issues with the sequence count checking because
the code does not really execute anymore in the intended sequence.
It also allows in-hardware integer validation.
So, it can check if two registers have the same values
or if they have a certain, well, true or false values.
More simple, they are more complex than just zero or one.
Again, as countermeasure against fault injection, because glitching
allows us to very easily switch values to all zeros or all ones.
And then probably the most important feature
this coprocessor has is the panic functionality.
As soon as something goes wrong, we panic, execution stalls, nothing has happened.
The chip needs to reboot and start from scratch.
stacksmashing: And these are all custom implementations.
Like the RCP is designed by Raspberry Pi.
It's not an IP that they bought in, but it's designed by them.
They also have – which is something that is very unique in the
microcontroller world, at least at like the regular, let's say
commercial off-the-shelf microcontrollers – glitch detectors.
And the idea is that if somebody were to mount a fault attack
on this chip, it could be detected and it would reset the chip.
Now, the way the glitch detector works is actually fully
described on this level in the data sheet, which is really nice.
And it's essentially 2D flip-flops that are running off the system clock.
And now if we look at the gated output, we can see
that it feeds back into the input of the flip-flop.
However, the second flip-flop has a slight delay.
And this delay is configurable and essentially sets the sensitivity of the glitch detector.
Now, the idea is that the output of these two flip-flops should always be the same.
Now, because the timing of when they switch is slightly different, the idea is
that if there was a fault attack, only one of them would potentially get corrupted.
And so, we could detect that, hey, somebody glitched.
Let's better panic the chip and reset.
And in the end, we end up with a nice classical Swiss cheese model. Right?
We have the glitch detectors.
We have the RCP with randomized delays, stack canaries, and so on.
And we have a hardened boot ROM that actively tries to have code
that is, you know, hardened against fault injection attacks.
Now, where do you best launch a secure chip?
Any ideas? Yes?
Audience: Under your own physical control?
Yes, that's a good idea. But Raspberry had another idea.
stacksmashing: They decided to use the biggest – but only second
greatest – hacking conference in the world, DEF CON.
So, the batch in that year was based on the RP2350. And so, they immediately
gave it in the hands of, like, 15 ,000 hackers, which is very cool.
And we actually brainstormed with Raspberry Pi.
So, they hired us to do a lot of the fault injection testing and so on.
And they asked us, how can we get hackers interested?
And so, we teamed up to launch the RP2350 hacking
challenge, essentially a bug bounty on this chip.
And so, the idea is very simple.
If you manage to hack the chip, you win $10,000. And kind of the problem is, if
you have a nice batch, nobody wants to, you know, destroy it by soldering on it.
And so, we asked them, hey, would you let us produce a board that
has, like, everything important to attack the chip already exposed?
Like, you know, to glitch the core voltage of the chip, the main voltage rails, and so on.
And they said, yeah, let's produce, you know, a
couple of hundred and give them away to researchers.
And then we had a nice booth at DEF CON, and this guy walked by, and I was like, hey.
nsr: Yeah, that's how I got nerd-sniped.
From my perspective, I was enjoying DEF CON, was looking
forward to have summer and autumn with not so much work.
And then this guy comes and tells me, look at this cool chip. You want to hack it?
I said, no, I don't want to hack it.
But he pushed on, come on, just have a little look.
So, I started the little look, downloaded the very same data sheet, and
was immediately nerd-sniped, and didn't get sleep for a way too long time.
So, besides the data sheet, I also looked into how the challenge was set up.
So, what they cooked up. And the main idea of the challenge is the following.
There is a secret, which is hidden in the one-time programmable memory, the OTP.
And secure boot is enabled, and the secret is marked as non-readable.
Only if you have code execution on the chip in the privileged
domain, you are able to read out the secret from the OTP.
So, the task is simple.
Use any means we can come up with to read out the secret from the OTP.
So, this explicitly puts fault injection and
similar attacks into scope, which is really cool.
But looking into it from another way, from the challenge perspective,
it looks something like this, like really broken down to the basics.
Read out 16 bytes from the one-time programmable memory, win $10,000.
And even better, like even accounting for inflation
and so on, that's a full whopping $625 per byte!
So, yeah, that was the main motivation.
And then multiple months of hacking went on and continued.
As Tom said, it was released at DEF CON in August.
And in January, there was a blog post on the Raspberry Pi website, which would
say, "Security through transparency: RP 2350 Hacking Challenge results are in".
On this blog post, Raspberry went through all the five
different found attacks and the four challenge winners.
And we are here to now talk a little bit more in detail about them.
So, there were five different attacks found.
The first one was actually presented already last year here at CCC.
It was attacking the one-time programmable power on state machine.
The second one was about abusing boot vectors.
The third one would do some really crazy and nice laser fault injection setup.
Then the fourth would abuse a double glitch to
read out the OTP memory in the bootloader code.
And the last of the five attacks was disregarding all of
this fault injection stuff anyhow, and just used FIB, focused
ion beam to directly extract the secret from the antifuse.
So, think about it as a very, very fancy microscope.
We will now zoom in into the two attacks we found, which
is forcing a vector boot and the OTP memory double glitch.
So, let's get started and talk about the boot process
to understand how this first attack really works.
It's easy, right? Like just, nothing to see here.
Now, the amazing part of this is the upper part of
this figure is actually just as this in the data sheet.
So, this can be read from the data sheet and describes the power on state
machine on hardware level, like how the system is brought up on the hardware.
We only added two additions here, which is the lens and the flash symbol
indicating the target of our attacks and what we glitch in some points.
The lower half of this figure of the boot process is the software part of the boot process.
So, the bootloader code stored in ROM is executing all those different stages
early on, before any of the user-defined or secure boot, secure code is running.
So, it would generally start going through all of the hardware bring
-up, then going into the software, starting the boot ROM, and then
very early on in the boot ROM, it will check the power manager vector.
Now, I read this in the data sheet, saw it in
the code, already at DEF CON, and I was puzzled.
I thought, hey, this power manager, it's a kind of special register
and memory we can write to, and then it would boot from an user-defined
program counter and stack pointer, so from user-defined entry point.
And I thought, this cannot be secure.
That doesn't sound right if all the secure boot validation is
only happening later on, because if we were able to write to this
peripheral, we could just carry out this vector boot and start
executing code where we want, bypassing all of the secure boot chain.
So, I was very excited about this, and at DEF CON, it was not only Hextree
with the booth, but also Raspberry itself was there, and I went with the
data sheet waving in my hand and said like, "Look, that doesn't look right.
Isn't there something wrong?"And they told me, "No, no, it's all fine.
We talked about this. Don't worry."And I was like, okay. Hmph.
Still, I couldn't let loose and continued looking through the different boot chains.
So, we see then, in the yellow part, in the lower part of the figure, how
the different types of boot, other boot types are tried to carry it out.
So, the RP2350 first tries to boot from RAM, then from the one-time programmable memory.
You could also store your full firmware in OTP if it's very small.
And only afterwards it tries to boot from flash.
And in each of these three scenarios, if secure boot
is enabled, the signature of the firmware is verified.
So, it's made sure that, indeed, whatever firmware is
to be loaded was signed by the according developers.
If none of this worked out, or if the BOOTSEL button on the chip
was pressed, or the line was high, it would go into the bootloader.
So, there it first prepares for the boot selection.
And then, based on which boot mode is selected, would
either go into a USB boot loader or a UART bootloader.
And this is then where even more of this boot ROM code is
accessible, because all of this lower part is still implemented
in boot ROM, including the full-fledged USB bootloader.
So, yeah.
Let's have a look at the USB bootloader a little bit more, what happens there.
So, first of all, we remember, this is a Cortex M33 chip with
TrustZone, so it has the secure and the non-secure execution state.
And the USB boot loader, which, by definition, is a lot of attack
surface, right, because it is a full-blown USB stack which needs
to be implemented there, it runs in the non-secure execution state.
So, already good, because this means if we had any bug in
the USB boot loader itself or could launch a code reuse
attack, we are still in the non-secure execution state.
We could not read out the secret yet.
Interestingly, some of the functions in boot ROM are used as APIs,
which a firmware developer can also call from normal software.
There are different APIs like a flash operation, for instance,
where we could read or erase some flash, so if we want to re-flash.
This could be done, of course, from the bootloader, but also from firmware itself.
Similar, there could be OTP memory accesses, so writing
to or reading from the OTP, all of this as a nice API.
Reboot, so the chip could be rebooted using different boot modes, for
instance, boot from RAM or boot from OTP, and just a generic secure call.
Now, the interesting part about these different API functions is that they can
be called both from the secure world and the non-secure world, and based on
which context they are called from, they have slightly different functionality,
which is quite interesting and something we looked into a little bit more.
So, yeah, firmware running in the secure world can also access this, and the last
part of the USB bootloader is that it implements all the interaction with picotool.
So, if you ever work with the Raspberry Pi Pico, the main microchip
using the RP2040 and RP2350 for the Pico 2, there is a special tool,
a command line utility called pico tool, which allows you to interact
with the chip early on, or more specifically, with the USB bootloader.
So, it has custom commands for flashing the firmware, speaks
a custom binary protocol which goes on over the USB stick.
It can do different things, and one of them is reset.
Looking a little bit more at reset, again, here the data
sheet is fantastic because it documents all of this.
It turns out that the RP2350 implements multiple reboot types.
So, rebooting normally, rebooting from RAM, rebooting from flash, all the normal ones.
But there is also one very interesting boot type noted, which is reboot type PC_SP.
So, it says that reboot to a specific program counter and stack pointer, and then,
more importantly, note that this is not allowed in the ARM non-secure variant.
So, this is a reboot API call, which we cannot trigger via the USB boot
loader, because we run in the non -secure world, but the code lives there.
If we were someway able to get to the secure world and call reboot with this special
boot type, we could completely tell where to start execution from on a reboot.
And it turns out that under the hood, all of this is implemented
using the special boot vectors we saw earlier in the boot chain.
So, let's look a little bit more into the command boot flow during the reset command.
So, it starts in the bootloader.
In the non-secure world, we receive the reboot command.
The bootloader gets the reboot command, and then
it jumps into the secure world reboot entry.
So, there is a transition from the non-secure world
to the secure world through secure call gateways.
Then, the first part, really the first thing which happens
in the secure world is that the arguments are sanitized.
This includes looking for the reboot type.
So, if for whatever reasons we are coming from the non-secure
world and asked to reboot with the reboot type program
counter stack pointer, this would fail at this point in time.
The bootloader would catch this and say, hey, that looks wrong.
Even more amazing, all of this is fault injection hardened.
So, the check is split up over multiple instructions with the idea that if
you could glitch a single point here, well, then there are still enough other
instructions which would make this argument visible and have the chip failing.
Then, once the sanitization happens, it jumps to the shared reset code.
So, this is the boot ROM API reset.
This code dispatches all types of reboot types, including the disallowed one.
However, it once again verifies that the boot registers are matching.
As an additional countermeasure, the boot type is stored
in two different registers of the ARM architecture.
And here, even after we already sanitized the argument, the chip
will check once more that these both registers are matching,
that everyone agrees that the boot type is the exact right one.
And then, after this is the case, it executes the requested boot type.
So, this was a logical overview. Now, let's have a look at the code.
If we ask to reboot via USB, we assume the boot type cannot be PC_SP.
If we come from the USB boot loader, there's too much
stuff going on, we cannot have this boot type being set.
However, if we start looking at the assembly, something interesting comes up.
Now, this is the entrypoint of the reboot type of the boot API.
In the upper part, it does the register verification and so on.
And then, very interesting here, it does a comparison of register 5 with 0x0d.
0xd is the internal value for indicating the reboot type PC_SP.
And if r5 is not 0xd, it will jump to all the other reboot types.
Otherwise, if it's 0xd, it will do the reboot type PC_SP.
So, looking in pseudocode, it looks something like this.
If boot type is not equal to program counter stack pointer (PC_SP),
do the other reboot types and otherwise do reboot type PC_SP, right?
There's a fall-through with this if condition.
And this, now for fault injection, becomes very interesting
because we could try glitching this particular instruction.
If we glitch that instruction, we would fall through to the
do_reboot_type_pc_sp(), even though we could not specify it.
We as the USB bootloader end up in the very valuable
branch for us as an attacker because suddenly we can boot
to a program counter and stack pointer under our control.
The other part we could glitch here is also do with the other reboot types, right?
If we never jump really to this part where we do the other reboot
types, so if we glitch the jump instruction, we would also fall through.
So, even better, we don't only need to hit one instruction,
but one out of two here in this code would be sufficient.
So, with this in mind, let's try glitching.
And now the first question you have may be, but what about the glitch detectors, right?
Like we discussed, there were those glitch detectors
with a different delay cycle trying to understand whether
something funky is happening in the power domain of the chip.
And this is where I started first profiling of the chip in a very
controlled environment where I had full control over the chip.
I tried to understand how effective those glitch detectors actually are.
So, we see here some of the RP2350 playground boards which
were given to me, together with a Faultier board allowing
the RP2040 to carry out the voltage fault injection.
So, the experimental profiling looked something like this.
It's not entirely well visible on this figure, but still, the experimental
profiling showed that the glitch detector runs pretty well if the chip is
clocked at around 150MHz, which is the recommended operating speed of the chip.
Now, interestingly, the USB boot loader runs from a different clock,
I think from around a 48MHz clock to be better synchronized to USB
to allow the communication to be more efficient, more flawless.
And now, if we think about it, right, we measure whether
a glitch comes between two clock cycles more or less.
Now, if the clock gets slower, suddenly the window of measurements get wider, and
we could try to inject a fault in between, undetected by the glitch detectors.
And this was not only the idea, we were able to do this in practice.
So, this figure shows the glitch detector running at a lower, or,
the full chip running at the same clock speed as in the bootloader.
And we can see in a lot of cases, the red dots here, the glitches were not successful.
But then, at these two points, actually, we could see successful glitches.
And with a little bit more profiling, we could bring the success rate even higher.
In general, for fault injection, I think it's a really good tip to do some
figures like this, plot your results to understand where you have vulnerable
points and where glitches are working, even in the presence of glitch detectors.
Now, with this, we could come up with the attack strategy.
The attack strategy looked something like this.
We load the unsigned firmware in RAM.
This unsigned firmware will just read out the secret
from the OTP to allow us to win the challenge.
Then, we send the USB reset command to the attack code.
So, the reset command is just a normal reset command
without the program code stack pointer (PC_SP) flag.
However, in the parameters of the boot command, because you can
pass on parameters, we already put the right program counter
and the stack pointer, which will point to the firmware in RAM.
Then, we glitch the critical instruction, and if everything works well, we
would end up in the unsigned RAM firmware, which we injected after the reboot.
stacksmashing: So, five minutes tops, I guess.
Yeah, exactly. Fault injection, very fast.
At this point, I was sure, okay, it's possible, one more day, and we have it.
And then, time passed, and it was not working.
My setup got gradually more complex.
Here, I tried to break out the USB connection to have a custom
-built trigger to see when to inject, and.. it was not working.
And I was like, okay, at this point, because the glitch was not working
on the boot ROM, I thought, maybe let's try to verify feasibility.
So, I attached a debugger, and in GDB, in our control chip, I had it running
up to the point of the critical instruction, skipped it manually with the
attack code in place, and just see if I would get into the attack code.
And, indeed, that worked.
So, I was more positive again, and I thought, okay,
now it's only five minutes until we have the attack.
Uhm... not really. The setup got more and more complex.
And, unfortunately, still no cake.
We now have additional components in here, for instance, a PhyWhisperer to
trigger better on the USB, a logic analyzer to just see the USB communication.
And, yeah, I don't know. My expectation looked really something like this, right?
Like, I build the setup, I execute the glitch, I run the attack code, I win the challenge.
So, there was a mind model, but as we see, reality looks quite different, right?
Like, we kind of got stuck here. We didn't get the glitch working.
Really early on, I didn't even see a single basic glitch.
And I was calling Tom and saying, like, hey, I think your tooling is broken.
Like, the attack board just doesn't work, it doesn't receive
any glitches, even without glitch detector and so on.
He asked me, okay, can you send me a picture? What are you trying to do?
I sent him a picture, and the response was, " use different jumper wires, you idiot".
It wasn't "you idiot", but no, an important bit here is that wires do matter
in the scenarios, and cheap jumper wires can cause glitches not to happen.
So, a lesson learned in, I guess, two hours of banging my head against the chip.
Then, at some point, I wanted to debug the USB
traffic, and I was just not seeing what's going on.
I was like, okay, I really know there's USB traffic going over the line, right?
I can see something is happening, but I cannot decode.
Like, I should see the reset command at least happening.
And I was trying and trying, and in the end, a colleague just looked
over my shoulder, asked what's going on, and I told him, and he
said, can you open the sample rate settings of your logic analyzer?
And in this moment, when I opened it, I thought, oh no.
So, my logic analyzer was not sampling on the highest rate by default, so
I under-sampled, which, of course, prevented me from seeing a good signal.
So, yeah, another two hours, well spent, I guess, or two weeks, rather.
Then, yeah, another part of this fun journey is the front fell off.
So, you know how these airplanes, they are not
built to last these days, fronts are falling off.
Turns out, also, PCBs and microchips.
So, fitting to the motto of this year's CCC, I
needed to POWER CYCLE the chip very often, right?
Because I flashed new code, I tried to understand what was going on.
So, it was indeed hitting the reset button so often that it just fell off of the PCB.
Audience: (Laughter)
stacksmashing: It's an intentional security feature.
Yes. And then, I'm there with, okay, only one board, right?
Because I only took one with me for the challenge.
I was like, hmm, what am I doing?
The only sane solution, I take a jumper wire and
manually emulate the button by hitting the jumper wire.
And more and more and more.
So, the lesson here really is that these things can be
tedious and everyone can make mistakes at various levels.
And it's fun!We can always learn from them. ...
Audience: (Laughter)
nsr: It doesn't get simpler.
And I was, I think at this point, close to losing my sanity, and still no visible glitch.
Like, I know the attack should work and it would work in theory, but it just didn't work.
And Tom told me, like, seeing how my sanity degraded over time, like, hey, maybe
just send what you have now to Raspberry and let them know that you have something.
And I thought, good idea.
The challenge description said, like, when you think you
found something, we will send you a board with a secret on it.
So, I thought, okay, maybe I can at least get the special board.
So, I wrote everything up nicely, supported by some clean pictures of my setup.
And sent it to Raspberry. And Raspberry replied, yep, that will do it.
That's fine. We accept this as a valid entry. I was like, okay.
So, still no final glitch, unfortunately.
But Raspberry did agree there was this overlook in the way the boot ROM was compiled down.
And this fall-through branch is indeed exploitable.
stacksmashing: And we kept your sanity.
nsr: Yeah, with this, I could finally relax a bit and say, okay,
at least someone else also thinks that this is a valid attack.
stacksmashing: Cool.
Audience: (Applause)
Yeah. What's better than one glitch that doesn't work?
Two glitches that don't work. So, to give you some context.
During the boot of the chip, essentially, the boot ROM has to have access to OTP.
It can't be fully locked down.
So, that means that it's readable during early boot to the boot ROM.
And then when we enter the later boot stages, we lock down the OTP, essentially,
to make it either, you know, read only or not readable at all anymore.
And so, if the chip is secured, you will not be
able to read the OTP, at least not the pages.
And via picotool, aka the user, we can only read the non-locked OTP values.
Now, Raspberry Pi is nice enough to actually document the
entire lock progression on how that works in the data sheet
and tells us exactly which bits need to be where to lock those.
And we can see that there's a lock progression.
We have 0, which is read-write, 1 , which is read-only, and we
can't go back to read-write, and then 2, which is inaccessible.
And there are three different places where this is stored.
There's the secure lock state, which essentially tells
the chip which OTP can be accessed from the secure world.
There's the non-secure lock state. Same thing, but for non-secure world.
And then there's the PicoBoot lock state that
defines what we can read via the USB bootloader.
And there's some nice code in the, again, in the boot ROM code.
And we can see that this is implemented in this function
here called s_otp_advance_( bootloader)_to_(secure)_value.
And if we look at the disassembly of that function, we
can first off see that there are two weird coprocessor
interactions at the beginning and at the end of the function.
These are the RCP calls that essentially check the stack canary.
And they also introduce a random delay between 0 and 127 cycles.
And so that's very important, because if you want to glitch
in here, we actually have to deal with random delays, which is
not so much fun, because we want to hit a single instruction.
If there's a single offset, then we are already past it in time.
But what's really interesting are these two instructions down here,
which essentially do nothing if you look at it on the first look,
because it's just a logical shift left and then a logical shift right.
And essentially, this takes the lock value, so the bits that set how we lock
down the OTP, and it shifts it first to the left, and then again to the right.
And then the idea is essentially that if any bits in the upper
part of this uint32 were set, they would be cleared to zero.
Now, interestingly, if we were to, you know, do the first shift,
and so just shift it to the left, and then we glitch and skip the
second shift, then we change the value that is written back into
those secure bits, and essentially we would keep the OTP unlocked.
And interestingly, even if you have these upper bits set, the hardware will accept it.
And so that's very interesting, because it essentially means that this shifting thing
kind of does nothing and wasn't necessary, but it gives us a great glitch target.
Now, unfortunately, as you hopefully realize by now, Raspberry Pi really
doesn't like us glitching the chip, and so they call this function twice.
Unfortunate. And so, easy fix, we just glitch it twice, right?(Laughter)
Because one is not enough, because there's even a coprocessor comparison at
the end that checks whether both functions return the same value and so on.
So, they really don't like us glitching this, unfortunately.
So, yeah, our attack strategy is we set up the boot type to boot
select mode, we bypass the glitch detector, then we glitch the critical
instruction twice, and then we read out the secret via picotool.
So, to bring up our cheese model again, we double the cheese on the glitch detectors,
and then we double the cheese on the RCP, because we have to bypass the delays
twice, and the glitch detector twice, and then we still have to deal with the
hardened boot ROM, so we can't just randomly corrupt it, we need to really skip.
Now, the problem is, you saw earlier, that, like, with voltage
fault injection, we need a lot of tries to get a successful glitch.
So, I'm not a very lucky person, so waiting for that long didn't seem feasible.
So, I decided to use a different fault injection attack, which is called EMFI.
And so, this is the actual setup mounted on my CNC.
And the idea behind EMFI is really cool.
We essentially have a high voltage pulse generator, in this case a ChipSHOUTER,
and then a small coil in front of it, and this will create a short electromagnetic
field that will collapse into the chip, and we can use that for glitching.
And so, the idea is very simple.
We point that magic device to the chip, we zap it, and the chip is hacked. Simple.
And even better, we can actually localize where
we glitch by positioning the coil very precisely.
And that's why, what you see on this on the setup here, I can move it around.
And so, the nice thing is, the glitch detectors
are at specific physical locations in the chip.
And so, with EMFI, we can just drive past the glitch detector and glitch
over there, essentially, which might not be picked up by the glitch detector.
And so, I started doing this, and I noticed, so
this is like an XY top-down view of the chip.
I noticed that I can only glitch the bottom-left corner of the chip.
I asked Raspberry Pi, I was like, hey guys, what's there?
It turns out the internal voltage regulator is there, and so
I was glitching the voltage regulator, which means I was doing
voltage fault injection using electromagnetic fault injection,
Audience: (Laughter)
stacksmashing: which is not great.
And then, so what's interesting is my coil was polarized, I think, clockwise.
And then I tried a counterclockwise coil, which changes
the polarization of the field, and I got perfect results.
And this is with the glitch detector enabled.
The white spot means I get 100% bypasses on the glitch detector, which is really cool.
Now, the next problem is, we need a trigger for our glitch
to work, but this is in the middle of the boot ROM after
like 100 different RCP delays, and so on and so forth.
And so, I started measuring the power of the chip, and I noticed these dips
in power, and so I figured, hey, what's gonna consume very little power?
Turns out that if you do a delay and let the chip do nothing...
Audience: (Laughter followed by Applause)
stacksmashing: So, the RCP delays were my trigger, and gave me a perfect beginning of the function.
Now, my trigger setup was very janky, and so I
could only have a single trigger, unfortunately.
And so, what about the random delays?
Well, there are just two delays, and so, you know, just let it run.
Because in the end, it's a statistical distribution.
You just try the middle value, and now we wait.
And so, yeah, after doing this for a long time, we eventually got a success, and we
can see that we read out some nice c0ffee from the OTP, and yeah, that will do it.
Cool.
Audience: (Applause)
stacksmashing: Now, what's interesting is how did we actually find this?
And so, we used something that essentially does fault injection emulation
and fuzzing, so we characterize the chip, we see what effects can we
cause, and then we let it run, and we essentially, you know, defined
the target of, hey, is the OTP readable at the end of the boot process?
And then we emulate this, let it run, and so on and so forth.
And interestingly, because of a bug, I was permanently switching this single instruction.
And so, essentially, I accidentally found the double glitch because of a bug in my software.
And yeah, that was that. And we had the RP2350 glitched. Yeah.
nsr: Yeah.
Audience: (Applause)
nsr: So, very quickly, before coming to the end of this talk,
let's have a small look at the three other attacks.
The first one was attacking the OTP power-on state machine by Aeden
Cullen, and it was actually quite an instructive and interesting attack.
It's based on the fact that the OTP has a separate power
rail, so it's not connected to the same power rail.
And if we were to successfully power fault this power rail just for the OTP during
hardware bring up, the last read out data would be returned again for future reads.
And this, by accident, aligns so well in this attack that when done during
early boot, when the hardware reads out how everything is configured, when it
reads out the OTP, it will just bring out the RISC-V core with debug enabled.
So, that's a very nice and easy way to get through it.
And if you're interested in this, really, really check out the 38C3 presentation on this.
It's a fantastic presentation. Please go watch it.
Yeah. And the next attack, which is insanely cool, is by Kévin Courdesses.
And it's seriously the best blog post you will probably ever read.
Because he will say, well, how do I attack this chip?
And he doesn't work in security, right?
He just decided to 3D print a laser fault injection setup.
And, like, his blog post is full of animations, 3D
models, and you can see, download every part and so on.
It looks like this.
And then the problem is that you can't get the laser through the chip.
So, what do you do?
You take a Dremel, and you just Dremel off the back of the chip by hand,
remove the metal, and then you solder that chip onto a special carrier PCB.
And the nice thing is that the back side of the chip is transparent to infrared.
And so, you can just, you know, actually do a nice chip scan.
And then you do.. the best way to hack chips is to shoot lasers at it.
And so, he just blasted lasers at it and figured out that if
you, you know, blast the laser in this rectangle, he even had
this nice GIF, you actually can bypass the security of the chip.
So, yeah, seriously, go out, read that blog post. It's amazing.
nsr: Yeah.
And the third of the other attacks was by Andrew Zonenberg and his team at
IOActive, where they just went along and said, no, no fault injection whatsoever.
Let's use microscopy, passage voltage contrast
in combination with FIBs, so focused ion beams.
And this would allow, if particles are aimed at the right area of the chip,
you can get reflections back and read out, of course, with a lot of tricks
and magic around it, what's actually stored in the antifuse memory itself.
So, as we will see here in the image, there is a little bit of annotation.
There are, like, visibly different patterns which you could use to
understand which bits are set and which are non-set, which is, of
course, another valid way to read out what's stored in the chip, right?
Just literally read it out.
It's very easy. You spend $50,000 of microscope time to win $10,000.
nsr: But seriously, it's also a very interesting attack.
There is a dedicated paper on this, both an academic and a white paper.
Again, recommendation here, go read it if this interests you.
stacksmashing: And then, what's really cool is that all of this, except, you
know, the crazy part with the FIB, got fixed in new chip revisions.
And so, it was really nice because the results were in in January,
and then in August, they already had, like, new silicon and
open errata and just, like, super transparent about all of this.
And, yeah, lots of lessons learned for me who does this professionally is,
like, include every reboot path and also the external IP in your fault models.
Hardware mitigations will fail, right?
It's just if you can try often enough, you might
find a bypass, even in a nice layered model.
Complicated boot paths are difficult to deal with.
And laser fault injection is much cheaper than you might think.
And, yeah, chip bounties work because people came up with crazy ideas.
And so, it was really nice to see.
Yeah. So, in other words, really, security through transparency is awesome.
We think this was a really great effort by Raspberry Pi, securing the
chip even more because we could find a safe way to experiment and find
vulnerabilities and report it to them in a very nicely controlled environment.
So, this is amazing, and we hope that more chip
vendors follow similar approaches in the future.
And..
Audience: (Applause)
..there's more. One second. There's still a little bit more.
Yeah. So, what's really cool is that there's a second hacking challenge.
It's essentially there's a new AES implementation that
has, like, this side channel, hopefully side channel safe.
And, yeah, if you manage to bypass that, you can, again, win a chip bounty.
And I just got confirmation that it will be extended.
So, the site still says end of December, but we'll extend it a bit.
And so, yeah, we would love to see your attacks.
And if you think you want to try it, hit me up later, maybe not immediately
after the talk, but later when I have my backpack, because I'm still
giving out these boards that have, like, it's a new revision that actually
exports everything you need to reconstruct all the attacks already.
So, the OTP power rail is separate and so on.
And we are happy to give you one if you're interested in trying this yourself.
Also, check out the paper that we all wrote together. And thank you.
nsr: Yep.