ilja: All right, so our talk is called Escaping Attainment, a Security Analysis of Freebies EJLs.
Before we dive into that, we'll introduce ourselves. Hi, my name is Ilya.
I work at a company called IOActive. We do computer security stuff.
The CCC conference is my absolute favorite conference in the world.
I've been going to every single one of them since 18C3.
I think this is 21 or 22 for me.
I've spoken here a couple times before on fuzzing and code review and
some Linux stuff and PST stuff and Windows stuff and a few other things.
And this is sort of my next one.
Michael: Hi, I'm Michael.
I'm an independent programmer and system administrator for 20 years.
And my first conference was in 2004. And the second one is this one, 21 years later.
And they were actually both with Ilya. So yeah, thank you.
Computers are my hobby and my work as well. It's intermingled.
I'm not too tied up on programming languages or operating systems.
I just like flipping bits, you know. Sweet. Okay.
ilja: So yeah, who's the autos for this talk?
Essentially, if you're a security geek and you like security stuff and code reviews
or you like operating system stuff or you like defensive security or you like
offensive security, there's going to be bits and pieces in here that you like.
But even if you're not a security geek, if you're
a BSD guy, you're going to like some of it.
You may not like other bits of it, but there'll be stuff in here you like.
If you're into virtualization and containerization, there's stuff in here as well.
If you're a general OS nerd, there's things in here you like as well.
So I think bits and pieces of this will speak to a broad audience.
Yeah, what's this about?
I mean, in a nutshell, essentially, it's sort of a technical deep dive
into the security boundaries of FreeBSD Jails, as the title has said.
And we focus on, okay, can we escape this stuff? What's the attack surface?
What does a jailbreak look like?
And then near the end, we have some observations,
conclusions, and call to action in terms of future hardening.
Our agenda is actually really simple. I'll be doing some intro stuff.
And then we'll go over some issues that we discovered.
And then sort of halfway through, we will look into some of these issues we
discovered and we'll show you an exploit and sort of how we created that.
And then the last section of our presentation is essentially observations,
conclusions, call to action, things where we think stuff can be improved in the future.
Yeah, before I want to continue, I want to thank the FreeBSD security team.
They have been so incredibly kind and good to us.
The interaction with FreeBSD has been amazing.
I'll talk about some that process a little bit later.
But the interaction and sort of the almost real
-time communication has been really, really amazing.
And I'm sure I've left people off, but these are people
that I've gotten responses from and emails from and so on.
They've all been absolutely wonderful. They've been fantastic to work with.
They absolutely understand what we're doing. They were pretty supportive.
They have been fixing bugs bugs and so on. So this whole process has been amazing.
And so, yeah, we wouldn't have been able to do any of this without them, really.
Which gets me to sort of a second slide.
So this is sort of the defenders and these are sort of the attackers, the security hackers.
This is, of course, we, certainly for the exploit stuff,
we, you know, relied upon those that came before us.
Some of these, some of this work goes back to 25 years ago.
Some of it's more recent.
Some of it is like PS4, PS5 hacking and then a whole bunch of stuff in the middle.
And no doubt I've left out a bunch of people.
If you have done interesting PSD or FreeBS security
work and your name is not in here, that's my fault.
I should have, I should have added your name to it.
And then, yeah, we decided to add ChatGPT in there.
It has been useful in a few cases when it wasn't hallucinating stuff.
Yeah, so why did we do this? Right?
And for me, this has been a quest that has been going on for
about 25 years. 25 years ago, I was a teenager and I was on an IRC
channel with a friend of mine who was a big fan of FreeBSD and he
loved FreeBSD4. And one of the things he really liked was jails.
And from that moment until this, I've always
been like, how hard would it be to break a jail?
And so finally, earlier this year, I found some
time and I started digging into FreeBSD jails.
And I come more from sort of a code background.
And then, so Michael and I have known each other for, I don't know, late
90s, 20 something years, 25. And we had kind of lost touch for a while.
And then earlier this year, we had reconnected.
And Michael was telling me that he was doing FreeBSD,
DevOps work and admin work and security work at an ISP.
And this was just as I was looking at some of the code.
And I was like, oh, wouldn't it be great if we could collaborate on this?
I could bring some of my code knowledge and then Michael could
put it all together and make it look beautiful and get all these
scripts working and it'll help all these exploits working and so on.
And yeah, a few days later, we started collaborating.
And we'll be presenting basically what we've been doing in the last three, four months.
So, okay, let's start off with one big assumption, right?
We, our assumption is we assume a jail that is compromised, right?
So you have a jail that, I don't know, runs some
service, whatever, someone rooted your jail, right?
How secure are you, right?
Can you escape a root jail and get access to the host, right?
And what are the risks associated with that, right?
That is what we're examining in this presentation.
A little bit of history around jails.
As I said, this was introduced in FreeBSD4, early 2000s.
The essential idea was, is we want to run normal Unix things on a
single machine, but your root sucks and we want something better.
And Poole Henningkamp was like, let me go build jails. And then he did.
And certainly for the time, they were kind of ahead of their time.
These days, everybody has some kind of thing like
jails, but 25 years ago, that certainly wasn't the case.
The other thing with jails, of course, is that, you know, V1 is never perfect
and it has many issues, but it's certainly matured and hardened over time
and they fixed a whole bunch of things and they added new features and so on.
Early jails' networking was very limited. That was fixed.
Conflicts are now dynamic and flexible. Didn't always used to be the case.
We have much better tools to manage jails.
So there's been sort of an evolution over time.
And certainly from a usability perspective and sort of things you
can do with jails, things have improved compared to 25 years ago.
But it started all in FreeBSD4. At a high level, how do jails work, right?
So essentially what you do is you create a jail configuration
and then at a syscall level, you call jail_set() and there's
layers in the middle that convert one into the other.
But essentially in your jail.conf, you can define your host name
and your root file system and the mounts and rlimits and exposed
devices and host name and all those kind of things, right?
That's how you set it up.
And then when you create a jail and you sort of, you can attach stuff to
it and, you know, that creates internally like this prison structure and
that says, even though in kernel this process is root, it's still contained
to this prison and so it's not allowed to do anything that root can do.
It enforces the chroot, it enforces the file system access,
it enforces mount points and does a number of other things.
One interesting thing kind of on a side note is that jail_attach()
is something, well, that's obviously a useful and needed syscall.
It does require some care.
So generally, if you look at it from an abstract point of view, you can see
a little bit as execve, where if you have an SUID, like set user ID binary
and you run a program, the resources you have get passed on to your child.
That is mostly true for jails. There are exceptions, there is some scrutiny.
But for example, if I have a file descriptor to a device or to
a file, and I do jail_attach(), that file descriptor goes along.
And let's say you have a config and you say, you know, your config says this
jail does not have access to this device, but you happen to pass that file
descriptor along, all of a sudden your jail has access to that device, right?
So there's a little bit of care there if you're doing jail_attach() stuff.
Generally, if you're doing just this admin stuff,
you're golden because the tools do that for you.
But if you end up building your own tools, you got to be a little bit careful here.
Right, so the crux of jails is essentially syscall mediation, right?
Once the jail is created, you have all of these system calls.
And for each syscall, if it needs to do a privileged operation,
the kernel basically looks at it and says, okay, are you,
yes, you're jail, yes, you're root, but you're in a jail.
Are you in that jail allowed to do this particular operation, right?
The main point of that is a function called prison_priv_check(),
which gets called by priv_check_cred(), which is sort of
the main privilege checking API in the FreeBSD kernel.
And this is basically where jail policy restricts privileges and enforces
that root is not, rooting your jail is not rooting the host, right?
This policy is enforced jails.
It can only ever use jail, or it tends to basically look at
privileges and the privileges you are potentially allowed
to have are for subsystems and drivers that are jail aware.
Generally, if it's not jail aware, you're not going to get that privilege.
Although on occasion, there can be mismatches, but in general, that's how it works.
What's also really nice is that function has this very, very large
switch statement for all these privileges that are potentially allowed
for a jail, and that was our starting point to audit this stuff.
It's like, okay, these are all the things that jail is allowed to do.
Let's go play with them.
On top of this, so those privilege checks are potential, right?
That is, you are potentially allowed to do that,
but you still are bound by a jail.config, right?
And if a jail config says, so for example, one of the
privileges you are allowed to have is kmem read, right?
But that is conceptually true.
But in order to actually do kmem read, the config still has
to say, I'm exposing kmem read to you as the jail, which most
of them don't, thank God, because that's a terrible idea.
But technically, kmem read is jail aware.
Anyway, these are a list of privileges, and it's not even complete, but that's 120-ish.
It's lots and lots and lots of privileges that are potentially allowed by jails.
Not surprisingly, a lot of it centers around network stuff.
All three firewalls are jail aware.
If the interface is jail aware, NFS is jail aware, IP stack is jail aware.
The Wi-Fi stack is jail aware.
Credential stuff, process stuff, scheduling, and so on.
As I said, kmem read is jail aware, which there are
sort of reasons for that, but it's just a bad idea.
Linux subsystem is jail aware, which I think is
problematic too, but I can see there being a need for that.
Yeah, so basically, my office is essentially looking at these
privileges, and then I sort of cherry-picked the ones that look
most interesting to me, and then started playing around with them.
I did not complete everything.
Like, I did not do a full audit of the Linux subsystem, because that thing is ginormous.
Most of the work was centered around networking, because that seemed very interesting.
What I did mostly is I looked at handlers that said,
okay, only root is supposed to do these things, right?
And the idea there is, if you are on the host and
non-root, you can't touch these things, right?
And my thinking was, that code was probably written 20, 30 years ago.
Someone wrote it going, well, it's root only, who really cares?
Do we really need security? It's all just fine.
And so, the other thing is, older code tends to have more trivial security bugs,
because there are things we know today, when you write code, especially C code,
that we didn't quite understand as well in the, let's say, mid or late 90s.
Yeah, so, issues. We found 40-ish issues.
Not all of them are fixed yet, although quite a few of them have been fixed, which is great.
We're not going to talk about the ones that haven't been fixed
yet, but I will talk about the ones that have been fixed.
As I mentioned, most of the stuff we looked at has been in IPsec, CARP,
Wi-Fi, interface, IOC rules, NFS, and then the firewalls, basically.
And the firewalls, in particular, is where there's very interesting stuff happening.
As I said earlier, I did mention the FreeBSD security team.
And I reported issues with them before, and usually the way this works,
and not just FreeBSD, but any security team, is, you know, you have an
email address, and you just give them your list, and at some point they
mail you back and say, we fixed them, we're good, here's the advisories.
And this time, Michael and I, basically, we were using
GitHub to track our work, because we were collabing.
And so we put all of our bugs in there, but also our POCs, and our
notes, and our documentation, and our VMs, and all of these things.
And what we did is, when we emailed the FreeBSD folks,
we said, hey, by the way, we have this GitHub thing.
Do you want us to add you to it?
And they said yes, and out of that followed sort of them
commenting on our stuff, and us looking at their stuff,
and them pointing us to fixes as they're being written.
And so you got the sort of real-time collaboration going with their security team.
It was fantastic. I loved it.
I don't think I've ever worked, when reporting external security
bugs, I don't think I've ever had a process that worked this well.
So I'm incredibly, incredibly happy that the FreeBSD folks were willing to sort
of work with us and kind of comment on some of the GitHub stuff we were doing.
Yeah, so sampling of issues.
I'm going to go over this really fast, because the
really interesting stuff is what Michael's doing.
So, yeah, I'll just go over them really fast. Ah, perfect. Awesome.
So, yeah, one of the first issues we found was in the IPFW, which is one of the firewalls.
It's an IOCTL where you add a rule, does a malloc,
then copies in stuff from user-land into it.
The copying, the error is captured, but it is never used.
So the error is discarded, so the copying can fail.
The malloc wasn't memory initialized, so now you're operating under-sized kernel memory.
That's obviously bad.
You can have side channels, and you may leak under-sized kernel data. Not great.
Similar to that code, the IPFW firewall, the format
for the rules changed between FreeBSD 7 and 8.
And so for 8, there was a function added that says,
hey, take these old 7 rules and convert them to 8 rules.
And this conversion basically has, you look at that for loop, and
there's embedded length fields, and there's a missing bound check for
that f_len, and when that B copy happens, memory corruption could occur.
And so the source from your rule 7 to your dest in rule 8, the
dest in rule 8 can be smaller, and you can have memory corruption.
Obviously that's bad.
This bug is fixed by simply the FreeBSD folks basically going, we don't need this anymore.
This is really, really old. FreeBSD 7, no one's using that.
And so they just cut the whole thing, and that fixed the bug, which is great.
Yeah, this was not a bug we found. This was in CARP.
This was actually really cute.
So this is basically, it's ioctl, and you can do this S-I-O-C-G-V-H.
And this ioctl basically says, hey, and give me, for every CARP
interface you have, give me information on this CARP interface.
And they have the structure CARP R, which is REC CARP record, and
you copy it in there, and then it gives you back to userland.
Problem is, CARP record could only hold one record.
And yet the feature allows for giving a count for number of records.
And so my first thought is like, what's going on?
They're kind of like, how does this work? Do you corrupt memory?
And then the actual work is done by carp_ioctl_get().
And carp_ioctl_get() is, I guess, it still tries to do multiple
records, but somewhere along the line someone was like, we can't
copy more than one record, because that would corrupt memory.
So what they do is they enumerate all the records, and then they keep
copying it into the very first entry, so you lose all the others.
But then they still give you an updated count.
So when it's time to copy out your CARPR, you copy out of bound.
And that may not sound very exciting, but we'll
show you in a minute why this is really cool.
This particular primitive allows you to incredibly
reliably leak the stack cookie, which is a key piece of
information we're going to need for some of our exploitation.
In particular, this bug, which is also in the IP filter firewall,
which is one of the more older ones, goes back to the mid-90s.
It has a device where if you write to it, basically you write to the
structure, and the structure has a linked field, and they use that
linked field, and they copy into this data structure on the stack.
That's 2,048 bytes, no bound check whatsoever, and it has this beautiful comment
on top that says, this must be sufficiently large to store any possible data type.
It's like, hmm, great.
There's, I'm getting a bit short on time, so I'm going to, there's
other samples here, but I'm just going to run over these really quickly.
So unbound malloc causes a panic.
Info disclosure, info disclosure, integer underflow.
Out of bound read, out of bound read, information disclosure.
Out of bound read and out of bound write.
This is in the fix up, which is a pretty cute bug. Same out of bound write.
More memory. Oh, this is a remote memory disclosure over NFS. Yeah.
And this is, the spec says, when you do a write over NFS, the
server responds and says, hey, you actually wrote this many bytes.
And the spec says the number you give it has to be an unsigned int.
And the function they use to extract it is an
unsigned int, but they store it in a signed int.
And then they check, oh, if it's bigger than the buffer, then they bail out.
But if it's smaller, they just basically subtract it.
And so what happens is, you say, you wrote minus five bytes.
And so it goes, we know the buffer, minus five.
And then it writes before the buffer over the network.
And then you can, you can, eventually you'll
crash because you got up, you'll go out of bound.
But you can do byte, byte, byte, byte, byte, and
just keep saying minus one, minus two, minus three.
And so you can really leak as much data as you want until it crashes.
This one was really cute.
When I found out this bug, and I'm looking for it, and
it turns out, so we found this in September this year.
Turns out the OpenBSD folks ran into this bug 2019, fixed
it, called it a reliability fix, not a security fix.
They said it can crash.
I don't think they realized that this can also info leak for you.
Because, obviously, if your negative value is too large, hidden on that page, you crash.
But if it's not large enough, you end up sending
uninitialized memory from the kernel over the network.
So, yeah, OpenBSD fixed it. No one knew. It's NFS, so it's shared code.
It's in Solaris. It's in illumos. It was in OpenBSD. It was in FreeBSD.
It was in NetBSD.
When we told the FreeBSD folks, they were like, yeah, yeah, let
us go talk to NetBSD folks and we'll make sure they know too.
And so that patch at the bottom is not the FreeBSD patch.
That is the OpenBSD patch from six years ago. Or six and a half. Okay. So, yeah.
This is only sampling. There are many more bugs that haven't been fixed yet.
A lot of that is the juicy stuff, the memory corruption and so on.
While we're looking at this code, we found some
bugs in other places that aren't jail related.
So, we found some elf bugs, some exec bugs, some netmap bugs.
And they're interesting, but I'm not going to talk
about them because it's not really jail related.
Sweet. Let's go. The juicy bits are coming up.
Yes. All right. Let me see. I'm not a professional in this thing.
I learned a lot from Ilya and I thank him a lot for allowing
me to go on this journey with him in the last couple of months.
I have been developing and doing system administration for 20 years.
So, I tried to use some of that experience in getting proof of concept exploits, basically.
That's what we want to get to next.
So, the idea is to identify a few bug candidates
that we can actually make a proof of concept of.
So, we have something to show.
And they can have a few levels of interest like triggering a denial of service.
Yeah. Okay. Not bad. But disclosure of sensitive information. Mm-hmm.
And then complete host compromise. Yeah. Even better.
So, the first thing we did was set up a solid development environment because,
well, if you have a good environment to work in, your setup for success is what I
learned. 20 years of building things and now switching to breaking things again.
So, we used some excellent tools. QEMU is an excellent emulator.
We had some trouble before with my MacBook because on the HDMI and that's the one I use.
And that's, it's nice to work on.
But when you're gonna research x86 architecture,
then it runs through emulation and it's pretty slow.
So, I did some compilation on the x86 machine and that was
convenient to just juggle those QEMU disk images around.
Kernel debugger, of course.
We also switched to a, we compiled a FreeBSD debug kernel.
But we had to disable asserts because that's what we want to,
that's what we don't want to, we don't want them to stop us.
SSHFS mounts because you don't want to be copying files manually over.
When we built our custom kernel, I mean, it's just, there's no such thing as coincidence.
Yeah. That was funny. Okay. So, the first attempt was to do offline debugging.
So, we try, we mess around, the thing crashes, the host
crashes, and then we check out the crash dump in the debugger.
Unfortunately, that crashed the debugger. So, well, yeah. Right.
So, yeah, we had to figure out something else.
So, we switched to online debugging.
Our second attempt was to switch to online debugging.
That made it a lot easier to set the breakpoints
and to, you know, to inspect what's going on.
Very well explained in the FreeBSD handbook, of course.
The debugging, online debugging uses serial link.
So, we just emulate that over TCP with two virtual machines and then we can get to work.
When you do this, multiple virtual machines, mind your MAC addresses,
because if they collide, then you're entering a world of pain.
And, yeah, try to understand what you're doing.
This was my first time setting up a debugging environment for FreeBSD.
And, yeah, that led to our third attempt.
Don't just build, like, a kernel on one machine and then just
copy it over to the debugger, which is not aware of the right
symbols and you get a mismatch and it just becomes a huge mess.
And I wasted, like, a week or two on that thing that was annoying.
So, what we did, I just set up a debugger, an end-to-end debugger,
and then I just cloned the disk image and used that as the target.
And then you get some nice info, right?
You can set breakpoints, you can see what's going on.
And, hey, there's that interesting comment here at the bottom again that's showing up.
So, I'm going to run through these exploit proof of concepts
pretty quickly because I want to get to the juicy one.
Except for the first one, that was my first kernel exploit ever.
And it's, well, that was just an exploitation of one of the bugs.
Ilya found this unbound malloc. Basically, yeah, here's the code again.
There's this, do I have a mouse? No, I don't.
So, the length from the allocation comes from the user.
And then the kernel happily allocates that amount of memory that the user asks it to do.
And it does that with a M_WAITOK type.
So, that means it just, you know, tries to do it and waits
until it's, yeah, until it can't find enough memory to do so.
So, that ended up in a panic. And, yeah, okay.
Leveregeable for a DDoS attack or something. Not too interesting.
Well, interesting, but we want to do more.
The second exploit, the second proof of concept, same thing.
But just in reverse, instead of getsockopt(), we utilized setsockopt().
Also, just covered by Ilya earlier.
Same thing, we just pass a huge amount of length to this call.
So, it tries to allocate this huge amount of memory.
We also mixed it in with some multi-threading to make it happen
a bit faster and more reliable on machines with more memory.
Same thing. What's this one? Oh, yeah.
This one is the stack smash that Ilya just mentioned.
Yeah, classic thing will come back in our comprehensive exploit.
Ilya mentioned this as well. This is the out-of-bound write.
The attempt to avoid an out-of -bounds write actually backfires and
it causes the kernel to return more memory than it's supposed to do.
It can be, yeah, leveraged to leak a stack cookie, and it will be.
So, here's what that does.
No panic this time, but we leak the stack cookie, which we will need in just a second.
And here's the debugger showing that those two things actually match.
So, the kernel was very happy to give us that.
Now, the comprehensive proof of concept exploit, that's the
one we really wanted to, you know, combine all these things in.
And the goal is to, yeah, completely take over the host from a compromised jail.
So, this leverages exploitation of multiple bugs because we will need those in a second.
And it's a good proof of concept to show what, you know,
what can go wrong when jail assumptions don't hold up.
We had a few constraints to deal with. First of all, stack protection.
So, you can, when you smash the stack in the FreeBSD kernel and you
want to take over the flow of execution, you can't just overwrite
the instruction pointer without also overwriting the stack cookie.
The kernel doesn't have, where we are, doesn't have executable memory, non-executable pages.
So, we cannot inject our own shellcode from user land.
We also cannot tell the kernel to jump to user
land where we have instructions because of SMAP.
and SMEP, Supervisor Mode Access Prevention.
We can't read or write data from user land into user land.
So, it's a bit limited, a few constraints, but we found ways around it.
The first one was stack protection.
So, yeah, like I just said, there's a value
calculated at boot time and it's put on the stack.
And if you smash the stack, you overwrite the stack and you want to override
the instruction pointer, you have to pass by the location where the cookie is.
And when you do that, a system will actually catch that
and it will cause a panic, which will result in game over.
Fortunately, the stack cookie is a global value set at boot.
So, we just leak it and we replace the value during the stack smash.
And the bug number 50 by Ilya for leaking kernel memory was the perfect candidate for that.
So, this is our code that we used to get the stack cookie.
It leverages that carp interface vulnerability and it leaks the stack cookie,
which we then put in our user land buffer that we used to smash the stack.
Now, we still have non-executable kernel memory.
So, we have to find a way around that. We cannot give it our own instructions.
We cannot jump to somewhere in user land. But we can jump to kernel functions.
So, we've figured out, like, why don't we try to jump to
kernel KLD load and we just load our own kernel module.
And then we can, you know, continue the exploitation with C code.
So, that was, the address of that function was pretty easy
to find because of all the determinism in the FreeBSD kernel.
So, we just did some static analysis on the kernel image and it gave us the
address of where the current KLD load function is, so we can jump to it.
And we put it in our buffer that we used to smash the stack.
And we don't want to jump exactly to the start of that
function because there's a bunch of privilege checks there.
It's part of the jail mechanism.
We actually want to jump 69 bytes into the function.
So, yeah, we want to jump right there so that we bypass
the secure level and the privilege check thing there.
Then, so, we are telling the kernel now to call the kernel KLD load function.
And the function has three parameters.
It needs the current thread descriptor of the thread that's actually doing the thing.
It needs a path to the kernel module that you want to load.
And there's a third parameter that's not really of interest to us.
But the problem is we cannot fill that information into
our buffer because the kernel cannot read from our buffer.
Well, sorry, no, it can read from our buffer, but we don't know the information.
Like, we cannot get the address of the thread descriptor. Or can we?
Because Ilya found in a Phrack article of 2002 that there is
this sysctl that you can use from userland and the FreeBSD kernel
will actually happily give you the address of the current thread.
So, we, yeah, from userland, we ask it the address, gets it to
us, we use it, and we put it in our buffer, which the kernel
then copies back into kernel memory, where it can be jumped to.
That was an interesting one. So, that's our code that we used for that.
This is just userland code that gets that value.
And then when we have that value, we put it in
our buffer, and we get access to it in the kernel.
Then the file argument, same thing.
So, we need a string, the path of the kernel module that we want to load, but
there's no way we can get that into the kernel, because the kernel cannot read from
userland, and it doesn't have access to an address pointing to a buffer in userland.
So, what we do is the same thing. We call our good friend sysctl again.
And this actually is a way for a userland to just ask for
the pointer in the kernel, in the kernel memory, the pointer
to the arguments of the program that you just started.
So, it's a free-for-all way to get any kind of string into kernel memory.
ilja: Yeah, but this gives you a reliable kernel pointer to it.
Michael: So, we do just that.
We pass a parameter to our exploits with the path to the kernel module that we want to load.
And, yeah, we get the address for that in kernel memory in userland.
And we put it in our buffer that goes into the kernel.
So, yeah, file ID, that's not really interesting, so we just set that to zero.
So, now we have this huge buffer that we use to smash that stack,
and it's got all the information we need to execute that thing.
The next step is a kernel module.
So, the reason we want that is because we want, you know,
to continue not jumping around all kinds of instructions.
We just want to have some convenience c code now.
The kernel module has full kernel privileges, and we need to
clean up the stack because, yeah, I'll get to that in a second.
And we want to take total control of the host, which we can do with a kernel module.
We also want to tell the userland exploit what's going on,
and escape the jail, and then clean up from the exploit.
Okay, so reporting the progress back to userland.
Ilya found a very clever way to do that.
We just have the exploit map some userland memory, and then we
have the module establish access to that memory, and then we
use an IPC kind of mechanism to just pass some messages around.
This prevents our messing around from showing up in the host console,
which is not something you want when you're trying to be a bit stealthy.
And the exploit then shows those messages to the attacker. So here's that code.
This is what the exploit does. It just maps some memory.
And then in the kernel module, we get access to that memory.
And then we have our custom logging function that just pushes messages over that thing.
And then in userland, the exploit fetches those messages and shows them to the user.
The next step is to escape the jail. That's what we actually want to do.
So we want to unjail two processes, namely the exploit, but also the shell
that initiated the exploit, because then we can actually do something.
And we want to change the root path for the current process and the parent
process, because we don't want to be in that container jail on the file system.
This is some standard code that changes the jail
process and just basically makes it an unjailed process.
Then changing the chroot, that was not that straightforward
because it's protected by a bunch of pointer magic in
the kernel to actually force you to go through an API.
But we write some garbage C code, mess around with some pointers and
dereferencing, and then we can actually bypass that entire thing.
Now, we also have to clean up from the exploit, because you can't
just go around and smashing stack and messing memory up in the kernel.
At some point, you have to clean that up a bit.
So the cleanup is reporting back that we're done to user land.
Then we have to unmap that user land memory.
It's like, basically, someone throws you a pigeon over
the wall of the jail and says, use that for communication.
Well, you have to take that pigeon out again as well if you don't want to rouse suspicion.
Okay.
And then we have to unlock the giant because the kernel locks,
that's the overall kernel lock, but we have to unlock it.
But we had to unlock it twice for some reason.
No idea why. Unlock it once, nothing works.
Michael: Yeah.
Three times it panics, two is the magic number. No idea why.
Michael: And then we took a page from Linux's book, and we just oops out of there
with a thread_exit(), because we're lazy, we don't want to clean up.
And yeah, we don't want to cause a panic, of course.
So yeah, that's how we report we're done to user land.
And then here's the cleanup, so we give that memory back for our
communication, we unlock the giant, and yeah, then we do a thread exit.
There was one interesting thing that happened during the execution of our exploit,
like it worked, but then after like 60 seconds, suddenly there was a panic still.
And it took a while to figure out why that happened, but it was because
our exploit was actually creating an entry in a table somewhere,
and yeah, there was some event happening, an automated cleanup thing
in the kernel that would try to clean that up or process it, and
because it was a corrupt entry, it would actually just cause a panic.
So the solution was simple, we just changed the, yeah, so here's that panic.
The solution was simple, we just changed the command from creating a entry to
updating entry, which doesn't exist, but that doesn't matter, it would just ignore it.
Now, we have all this information going into the buffer that we use to overflow
the stack, but the thing is we didn't know where to put that information
exactly, like for example the pointer to the string of the path of the
kernel module, where exactly in our overflow buffer do we have to put it?
There's some techniques for that, it's basic stack smashing 101, you
just create what's called a cyclic pattern, and then it's a pattern that
doesn't really repeat too soon, and then you can use it, you overflow
everything with that, and then you can actually use it to calculate
the position in the buffer of a certain thing you're interested in.
So we were able to give register values or other interesting values, we
could use that to calculate where in our buffer the kernel would read that.
Different FreeBSD versions and different kernels, they have different
offsets, so, well, we store them in the exploit and we detect them at runtime.
So here's that function, very simple, but the interesting bit was that we
just had ChatGPT do our work for us, so we dumped that function in ChatGPT,
and then we could just tell it, where's the thing here, we just, for example,
that's a register value that we got, and we just told it, like, this is what
we have, that's our pattern function, where is that in our overflow buffer?
And it would just tell us, and that made it very easy
to actually just, yeah, figure out all those offsets.
We could have calculated it manually, but yeah, like, we're lazy, right?
So now we have these, that was Ilya's idea, to put some structs in the
code that would make it, you know, multi-platform, like, for example,
FreeBSD 14.3, 15, different kernels, we can just add more offsets to that.
So the platform is detected by the exploit at
runtime, and then it just chooses the correct offsets.
All right, so demo time, this was hopefully going to be a...
ilja: Yeah, the demo gods were not with us today.
We have it in a live system, but that one wouldn't connect
with HDMI, so we have to do the demo on the pre-recorded.
So, let's see, is that playing? So, hold on. On the right side, you see the jail.
You're inside the shell on the jail, and on the left
side is the host of the jail, the host machine console.
There we go.
So we go into our exploit, we build our kernel module, which will do the work
for us after we've done the exploit, and then we copy that kernel module just one
folder back where the exploit is, because we like to do extra work for no reason.
Okay, we build the exploit, and then we execute it. And there we go.
Is this still... Yeah. For good measure, we'll just reboot the host.
ilja: Sweet.
Michael: Okay.
Fantastic. Sweet.
So, this is the last bit of the presentation, which is sort of the, you know, let's...
What are our observations? What are our conclusions?
Sort of what can be done better?
Observations were, from the audit perspective, we didn't
complete everything, but we did find 40 or something bugs.
Types of stuff was all over the place.
It was out-of-bound reads, and writes, and stack smashes, and heap corruption,
and integer overflows, race conditions, information leaks, unbound allocations.
Just a usual standard set of things you see when you're
doing kernel development in the language which is C.
That is kind of from the, you know, from the audit perspective.
It's all kind of within parameters. It's not great, but it's what it is.
From the exploit perspective, the last time I wrote an exploit was 2019.
The last time I wrote an exploit before that was 2013.
So that's two exploits in 12 years. So I'm very rusty.
This was Michael's first exploit, by the way.
Yeah. The last one was never. But it was way too easy.
Yeah. That's the thing. Yeah. I was super rusty.
He'd never done it before, and this took us a couple of weeks.
It shouldn't be this. It's 2025, almost 2026. It shouldn't be this easy.
It was way, way, way too easy. So what are the conclusions here?
Tried and tested is often confused when it means more secure. It doesn't, right?
The network stack is great.
It's tried and tested, but it's not like it doesn't have any bugs, right?
There's a difference between, you know, being hardened against accidental
bugs that happen on occasion and bugs that never happen by accident
unless somebody really forces them, and the two are not the same, right?
The other thing is a lot of the code that we looked at, especially those
interfaces that are like root only and were created before jails existed, that
dates back to a different time, you know, mid to late 90s, 25, 30 years ago.
Older code tends to have more trivial bugs.
It's just not built with modern security in mind.
We have learned things in the last 30 years about writing safer C code
that we didn't, we thought we knew in the 90s, but of course we didn't.
So yeah, code written with only root cause in mind was sort of lax about that.
And I would say, like people do this on purpose, but sort
of in the back of your head you're going like, oh, only
admin can do it, you know, it's not that big of a deal.
And so yeah, the jail threat model, you know,
wasn't anticipated when this stuff was built, right?
Or when the subsystems and drivers were written.
And yeah, as we said before, exploitation was just too easy, right?
Now I want to give a caveat here, because it may sound
like we're kind of pointing and laughing, but we're not.
Because the reality is that building an operating system is incredibly hard.
It is very, very hard, it takes a lot of hard work.
I mean, just the stuff that we did with the debugger and all
these jumps you have to, these hoops you have to jump through.
Like if you are the guy building that debugger and you don't have a debugger to
debug your debugger, your life is a thousand times worse than ours was, right?
I mean, that's just a fraction of a thing, like an OS is
enormous and it's decades of work layered on top and somehow
someone has to keep it all in their head and get it working.
So I'm not here to point and laugh.
Like they are kind of serious things, but it's very,
very difficult and I understand that it's not easy.
And if anybody thinks this stuff is easy, by the way, go
build your own operating system and see how hard it is.
It is unbelievably painful. Now having said all that, we do have some suggestions.
So one is, yeah, more auditing would be needed.
I think I certainly haven't completed everything and even the
stuff I've completed, I'm sure there are things that I missed.
One thing that jumped out is it was too easy to get the kernel pointers we needed.
It was too easy to get that thread pointer.
It was too easy to get a kernel pointer to an arbitrary piece of memory
that we can reliably use and put whatever content we want in there.
And we realize those interfaces are used by real programs, but
how about we, historically that maybe made sense, but how about
we build better interfaces or different interfaces that give those
programs what they need without giving them kernel pointers, right?
Yes, the pros would need to be modified, but this should
have been done anyway because grabbing stuff out of the
kernel and pointers is by design racy and a terrible idea.
And that was okay in the 1970s and 1980s, but it's 2025. This is not okay anymore.
Module loading, yeah, I mean, module loading is cool. It is very helpful.
It's very useful.
It certainly makes life easier, but it also makes
life easier on someone who's developing exploits.
It is significantly more reliable and powerful and flexible than shellcode.
And so it's one of those, yeah, sometimes you need it, but other
times, like OpenBSD basically has said, no more kernel modules.
You built the kernel and that's it.
I'm not saying that's got to be your only option, but maybe that can be the default.
I don't know. I think there's room for improvement there.
Yeah, exploit mitigations.
As you saw, when we grabbed that pointer to current KLD load, that is the point.
That pointer is the same.
If you're x86, if you're x86 FreeBSD and that version, that address works everywhere.
It's the same, right? That's too easy, right?
I'm not saying ASLR solves all problems, but it
would have at least given us a bad day, right?
We didn't have that bad day.
We just grabbed for the address and we were done, right?
The other thing is the stack cookies are nice, but they're so 2003.
Intel came up with shadow stacks and code flow integrity and all those things.
And in modern x86, that's available.
I want FreeBSD to use it and I want them to use it in the kernel, right?
Yeah, there will be pain points, but the technology is there. Let's use it, right?
Another suggestion I have, and this, depending on who you talk to, may or may
not be controversial, but C is a nice and powerful language, and it has served
us well for the past 50-something years, but in my opinion, it needs to die.
The benefits no longer outweigh the downside. There's just too much risk.
I mentioned Rust because it seems to be the one that has momentum.
It doesn't have to be Rust. There are other languages.
If you want to write Ada, great. But, you know, Rust seems to be a good contender.
And so I think it's time, you know, we're starting to see this in
the Linux kernel, and Windows is doing some of that, and we're seeing
that in some of the browsers and some of the mobile devices and so on.
It's time to start building our new low-level stuff in memory-safe languages like Rust.
I get this is not an easy thing. This is a multi-year effort.
No doubt you'll get religious wars. You have to convince all the right people.
You've got to build the infrastructure to do it, and then you've got to work in stages.
You've got to, you know, new subsystems first, maybe a driver first, maybe then a subsystem.
Eventually, hopefully, at some point, you get to backport some of your code.
We're talking easily five, ten years in the future before
this has a real effect, but it will have real effect, right?
Rust is not a silver bullet, but we know from the data we've seen so far
is that it drastically lowers the number of memory-related issues you have.
By default, Rust is memory-safe.
Of course, when you're doing curl stuff at some point,
you can do unsafe things, and that may still cause issues.
But, I mean, to iterate over a list or something like that, you don't need raw pointers.
Rust can enforce memory there, right?
It can enforce memory boundaries, all those kind of things, right?
So, it will drastically reduce the amount of issues that you would have. Sweet.
Yeah, that's it. That's it. *APPLAUS*
All right. Thanks, Michael and Ilya, for the great talk.
I really like it personally, and now we'll get to the questions.
So, if you have a question, I think you can queue up at the
microphones, but, yeah, I can also just come over there if you want to.
Audience: Okay, so I have a few quick questions.
I'll start with one that I just want to clarify and answer.
Did you say that some of these APIs will just give a kernel pointer
to the userland without you needing to do any kind of out-of-bounds?
ilja: Yep.
Audience: Okay, just want to clarify that point.
ilja: Uh-huh.
Audience: Second question.
Have you already or do you plan to publish a full
write-up of the POC stuff and how you got there?
Yes. So, we have the GitHub thing we're talking about.
We are going to open all of it, publicly available,
including comments from the FreeBSD folks, all these things.
It's currently not public because some of the
bugs aren't fixed, but we also don't want to wait.
So, we have a mirror. The link is actually in there.
It's currently empty, but after this talk, I'm going to start
pulling out the stuff that are already fixed and put it in there.
Everything is going to be made available and publicly available.
Audience: Okay.
ilja: Including the fuck-ups we made.
Audience: Where will we find this?
GitHub, the link. Hold on. Can we? We should be able to go through that link.
Should. Is it at the beginning or the end?
Uh, yeah. It's in the beginning. Oops. Yeah, there we go.
That's the FreeBSD jail security research mirror.
Michael: Nice short URL.
Audience: GitHub, the VS
ilja: So, the private one has mirror out of it, but it's not publicly available yet.
It will be someday when all the bugs are fixed.
All right. One last question.
If I want to ask you more broader questions about your
audit methodology, where might I be able to find you?
The very first slide has our email addresses in it. There we go.
Audience: Right.
ilja: That's us.
Yep. Okay. Thanks a lot.
Okay. I think now we have to put an epilepsy warning in front of the talk.
We'll start with microphone number one.
Audience: If you had Rust in the kernel, would you turn on integer overflow checking in the compiler?
Um, I would. And, um, uh, cause I know there, there's pros and cons.
One, the reason why it's not by default if you write most Rust code is perf.
Uh, but the literature guys basically said, perf isn't that big of a deal there.
And they, in the kernel conflict for Rust, it is
on by default, uh, hasn't caused problems yet.
And that's good enough.
And so if I were to do that in FreeBSD, I would
day one, say, turn on integer overflow checking.
Okay. Thank you.
Host: We'll do microphone number one again.
Hi. Thanks. Thanks for the talk.
There will be three question marks, but it won't be long.
Uh, so first, uh, did you find the NFS, uh, uh, in, uh,
integer comparison, uh, via uh, compiler, uh, warnings?
And two, uh, did you try to find other issues
of that kind with compiler warnings and linters?
And do you think, uh, these kinds of security issues can be easily found?
And did you, did you try to, you know, run, uh, batches of tests around those linters?
ilja: Yeah, those, that's a great question.
The answer is no, I didn't find it with any tooling.
I just read the code and saw that's the problem. Um, I am, I definitely checked.
But I'm convinced that the, you can get the compiler to
find this because it is, it is a, I'm taking unsigned int
and putting it in signed int and there's no explicit cast.
It works, but there should be some warning somewhere and you
can probably turn on some switch and there would be a warning.
Um, in terms of tooling and scaling that out, yes, but there's work because
it'll generate a whole bunch of things where it's not that big of a deal.
And then you got a, you got 5,000 of them.
And somebody has to read them and somebody has to make decision.
And so you want process around that so that somebody isn't
wasting a year scanning through things that aren't very important.
Um, but yes, you could totally do that.
Host: Okay.
So next I would ask the signal angel if there are any questions from the internet.
Um, doesn't, doesn't look like it. Uh, currently not.
Uh, only a question about the link for the GitHub, but.
Oh yeah. We put it in there.
Okay. Then we'll do microphone number two.
Audience: Uh, yeah.
So, uh, for these kinds of attacks, uh, have you investigated if there are mitigations
against, uh, speculative execution vulnerabilities deployed in FreeBSD anywhere?
And have you tried, if yes, using them to read kernel memory?
That's a good question. I did not think about this.
I mean, it's certainly relevant. Um,
Michael: That was too new for us.
ilja: Yeah.
Well, the thing is, cause we had a long list of things we wanted
to, cause we're like, oh, we should do heap smash as well.
Uh, but we'll, then we'll be like, when do we cut in this presentation?
Cause we're already an hour in.
Um, so this is a, we didn't even think about
this, but this is a good one to put on our list.
Um, to be continued, I would say, but we haven't yet.
Audience: Thank you.
Host: Microphone number one again.
Audience: Will you document your debugging environment?
Michael: It's, yeah.
Audience: And write up on how to, what you found work for you when debugging?
Michael: Yeah, we did.
Uh, it's in the, it will be in the GitHub repository as well under doc.
There's a setup guide.
Hey, thank you for your auditing work there. Um, wondering.
Host: Could you move a little closer to the microphone?
Oh, sorry. Thank you for the, lots of the auditing work.
And I'm wondering if you have tried this against hardened BSD
because those guys did a lot of, uh, memory protection stuff.
ilja: Yeah, I actually, so I work with one of the hardened BSD guys.
So we did speak about some of these things.
Um, uh, most of it probably would still work on hardened BSD.
I would imagine the expo would be more work, but we haven't tried it yet.
But I'll go talk to Sean about this because, um, but everything
I've had conversation with Sean over this doesn't indicate
that it would be impossible, but it might be slightly harder.