[ Music ]
>> Ladies and gentlemen, we're going to have an interesting talk
about bugs in the blockchain brought to you by Louis and Gabi.
[ Applause ]
>> Hello everybody.
Yeah, thanks a lot for coming.
It's still early.
I'm glad everyone could make it.
So our talk is called Don't Panic, Bites, Blocks, Bugs.
And I'm Louis.
I'm a security researcher.
I've been working on this topic of finding bugs in blockchain
for almost two years now.
And I also focused a lot in the past two years
on tool development around our audit efforts.
>> Yeah. Good morning everyone.
I'm Gabi.
Thank you for making it this morning.
We know it can be a bit difficult after like three,
four days of camp.
I'm a security researcher.
And currently I'm focusing on finding bugs in blockchains.
>> All right.
So blockchains.
They've been in the news quite a lot
in the past couple of years.
Not always for the good reasons.
A lot of times it's because a lot of money gets stolen
from somewhere or a lot of money gets lost.
And it seems like a lot
of attackers have a big advantage in the field.
So as security researchers we're kind of trying to be proactive.
And we want to share with you today the knowledge
of how we find bugs and which kind of bugs we find
in those blockchain systems.
So that, you know, everybody can learn from our work.
So today we'll be going through five common types of bugs
that we find in a specific framework that we'll talk
about a bit later and how to find those bugs.
So this is kind of a small shocker bug that I want
to quickly present to show you how important this auditing
process can be for blockchain systems.
So this was an Ethereum project that because
of this one line basically crumbled on itself.
This wasn't a hack per se.
It wasn't like an attacker stealing money or something.
It was just one line that made the whole project collapse
on itself and everybody that had put money
into the project just basically lost everything.
And the code was brand new, unaudited.
The developers said, "Guys, please don't put your money in yet.
We want to, you know, finish up stuff."
And people still put way too much money in there
and basically lost everything.
So yeah, just because of this one line, which is an overflow
in a multiplication, that made it so that the developers
or I mean the community lost control over the governance
of the contract and yeah, made it impossible to recover.
Everybody lost funds.
So as an attacker, what can you gain from blockchains?
The first obvious way is just like stealing funds.
If you are able to find some kind of a bug that allows you
to steal funds from one address or a contract or whatever,
you can just take those funds or these assets,
put them on an exchange, exchange them for real money
and then, you know, walk away with your gains.
Then the other way that we'll also be looking at today
is that if you are able to mess with the availability
of the chain or, you know, mess with people's expectations
of what the project should act like,
then you can diminish the overall value of the chain.
And if as an attacker you knew that in advance
and you shorted the token in some way,
then you can also gain some money this way
and this actually might be even harder to trace
than the first way.
There are a few other ways an attacker can gain money
from attacking blockchains.
We are not going to cover them today, but these include,
for example, DeFi attacks, so decentralized finance.
Also, you know, attacking the underlying IT infrastructure,
which, you know, can happen of course.
Social engineering, which has happened a lot
in the past couple of years.
And yeah, also just, you know,
a rock pool from the developers' financial scams.
So yeah, we'll first be giving you an intro
of these blockchains.
Then we'll explain the five types of bugs
that we wanted to outline to you today.
These are not exhaustive, it's just five types
that we find interesting and that we think
that you might enjoy learning about.
We'll be showing you examples of real world bugs
that we found in the past couple of years.
And then we'll show you how we can use fuzzing
to actually find those bugs proactively and effectively.
And yeah, go ahead, Gabi.
Thank you.
So today we are going to focus on the latest blockchain
technology, which is called the third generation.
But let's just quickly go back and check the other two ones.
So the first generation, it's the Bitcoin.
Basically, you all know it.
It allows you to transfer assets from one party to the other.
And compared with the other two generations,
it has quite a simplicity, let's say.
Also, currently there are not too many bugs
that are being found in Bitcoin.
And probably this is also because it's quite simple,
let's say.
Then the second generation came out
and it introduced the concept of smart contracts.
So basically now you are able to execute code
on top of the blockchain.
So you can write your contracts, your agreements,
and they are going to be executed and stored
in the blockchain.
This also introduces the possibility
of creating business logic.
So basically, companies actually started
to develop themselves on the blockchain.
Now, the idea with the second generation
is that you have the smart contracts that are all
being run on the same core.
The third generation comes out and says, OK,
instead of having one core and a bunch of smart contracts,
let's empower the developers to actually create
their own blockchains so they can define the core themselves.
And in this case, basically, we are
going to end up with a bunch of smaller blockchains,
each one of them targeting a specific use case.
But now the question is, how do you make all of them
communicate with each other and actually being an ecosystem?
So the third generation also introduces
the interoperability so all of these smaller blockchains
can communicate with each other.
Also, there are multiple frameworks
that are built on the third generation.
And today, we are actually going to focus on one of them.
And it is called Substrate.
And we are going to focus on it because basically, we have--
our experience is the most with Substrate.
So it's just easier for us to give you real-life examples
of bugs and attacks.
But all the things that we are going to present you here
can be applied to other blockchains as well.
The thing that is building Substrate,
they are also running two blockchains, Polkadot
and Kusama.
And probably a lot of you know more about Polkadot
than the name of Substrate, let's say.
But besides these two projects, in the ecosystem,
there are another 150 projects that are using the framework.
So basically, now we are faced with the research question
on how do we actually make sure that all these 150 projects are
actually secure?
And how can we automate this in a way?
Because at this scale, basically, you
need to automate as much as possible.
Let's have a quick look at the architecture of Substrate.
It has all the usual components like peer-to-peer networking,
RPC, telemetry, and so on.
But today, we are going to focus on the WASM runtime, which
basically, it's the core of the blockchain.
And in Substrate-- basically, this runtime,
it's being built using modules.
These modules in Substrate are being called palettes.
Throughout the presentation, we're
going to try to use the term "modules"
so we don't introduce too many new terms for you.
And the idea is that, basically, when a developer wants
to create a blockchain, they can either use--
like Cherry Pick, actually-- a bunch of these palettes
that are already provided by Substrate.
So you can choose your consensus.
You can choose if your blockchain
will be able to run smart contracts or not,
randomness, and so on.
And on top of this, you are going to come up
with your own modules that are going to bring, basically,
your own business logic.
OK, so now, let's just quickly look, actually--
I suppose all of you came here for seeing
the attacks and the bugs.
So let's just dive into that.
Yeah, let's go.
So we'll be looking at five major classes of bugs
that we've defined today.
So bug type A would be wrongly priced transactions.
So if a transaction doesn't have the right gas associated
with it, then maybe we can exhaust the resources
of the chain, causing some kind of a denial of service.
Bug type B is-- we're going to look at logic bugs
and unsafe arithmetic to manipulate the actual program
flow.
This will hinder, of course, the integrity of the chain,
because you might be able to tweak a number that you weren't
supposed to be able to tweak.
But as we'll see, it can also impact availability,
because those numbers might be associated with availability,
the numbers that you tweak.
Bug type C will be looking at the reachable panics.
This is a very rust-specific bug type
that we'll be taking a look at.
And bug type D, encourage usage of standard patterns,
so basically misconfigurations in the blockchain code,
because we said that there were quite a lot of things
you could import.
And if you import stuff with the wrong configuration,
then you might be vulnerable to any kind of misbehavior.
And bug type E will be looking at storage bloating.
So that's like inserting a whole lot of data
into the blockchain storage and cause some availability
mishaps.
So first, wrongly priced transactions.
The idea here is that operations on those substrate-based
blockchains are benchmarked.
And then the benchmark time basically
is translated into a gas fee that you
have to pay in order to run the operation on the network.
So whenever I want to run an operation A,
it has an associated gas fee of N.
And I have to pay N in order to run A.
And if A was not well benchmarked,
then I might be able to spam it.
And then the node that tries to run the whole block
might just not finish running it in time,
because it estimated how much time it would need beforehand.
And if it wrongly estimated because the benchmark was
wrong, then you might be able to make it basically--
it's not a timeout, but at least you're
going to slow the whole thing down and maybe cause
some people to be blamed for slowing it down.
That were not you.
So I'm going to talk about a very cool bug that
is related to what I just mentioned,
but not exactly what I just mentioned.
Because this bug that we found--
I mean, our fuzzer found it, actually--
it applies not during the running of the actual operation,
but during the computing of the weight of the gas
cost of the function.
So you can see on the left, this is the gas computation
of an operation that's called sudo as.
And this sudo as takes, as an argument, another operation
to run as sudo, to run as root.
And of course, this sudo as can only be run by, for example,
a governance call.
It cannot be run just by anybody.
It's also used for development, for debugging.
But in real life chains, it also exists,
and it can only be used by governance, basically.
But the computation of the gas cost of a sudo as
has to take into account the computation of the gas
cost of the child operation.
And if you look at the way that the computation is done,
it calls the child computation function twice.
And that means that you can actually
get an exponential number of function calls
just by nesting those sudo as calls or any other call that
was vulnerable to this.
And there were a few.
You can nest them.
And then, for example, if you nest them 41 times,
you get more than a trillion leaves.
And basically, the node that is going to try to run that,
it's not even going to go to the logic of the code at any point.
So it's not even going to have time
to check at any point that you're actually
who you're claiming to be.
So you can call this as just anybody.
And it's just going to time out and completely hang
during the weight computation.
So yeah, this was a very cool bug
that we found a couple of years ago.
OK, so if we are going to have a criminal that is looking more
into how to manipulate the flow of the program,
there are two ways of doing this.
The first one is taking advantage
of the usage of unsafe arithmetic.
And as an example, here it's part of a function
that can be called by any user.
Now, this function takes a few arguments, one of them
being a vector of messages.
These messages have to be processed by the blockchain.
So before processing them, the blockchain
is going to calculate the weight of each message,
then sum all of the weights up.
And then it will just check, hey,
is the sum of the weight smaller than the maximum weight
that I'm supposed to be able to calculate?
And in case if it's not, it will return an error.
Now, the problem in this code is that the sum function,
it's being used to calculate this weight.
And the sum function is not using safe arithmetic.
So it can actually overflow.
So in this case, an attacker can come in
and actually execute this function with a large vector.
So basically, when the sum is going to be called,
the weight calculation is going to overflow and wrap around.
And in the end, we are going to end up
with a very small weight for actually a super large amount
of work to be done.
So the node will consider, hey, I can actually
execute this in the amount of time
that I'm supposed to.
But actually, it won't be able, and it will time out.
Now, in order to fix the previous vulnerability,
you just consider, hey, I will just
use safe arithmetic as the checked function.
But that's not always fixing the issue.
Because when you use safe arithmetic,
it will actually return you an option.
So you have to properly handle the error that
is going to be returned in case of an overflow.
In here, the developers are using the checked functions,
which is fine.
But they are not properly handling
the case of an overflow.
So the function that can be called here by a criminal
allows you to control the begin and of the message count
parameters.
So basically, if you are going to call this function
with an end value of U64 max and the other two ones as 0,
in the end, the blockchain will try
to allocate a vector with the capacity of U64 max, which,
of course, is going to crash the node.
And it will end up with availability issues
in the blockchain.
Now, looking into the logic box, in here,
we have an example of a blockchain that basically
a user comes in and says--
the user actually has to validate their identity.
So a user will come in and say, hey, this is my identity.
Then a registrar will pick up the request from the user,
will check it, and it will say, yes, your identity
is legit or not.
Now, the registrar will specifically say, yes,
this user provided a valid identity.
But it won't say that it provided
this specific identity.
It will just say that the user provided an identity.
So in the end, what the hacker can do
is submit a request for a legit identity,
then monitor the network and see when the registrar actually
comes in and says, yes, this identity is legit.
And execute another transaction very quickly
with a spoofed identity.
and the higher tip.
So basically, in the same block, the hacker
will change their identity with a spoofed one,
and the registrar will still consider
that identity as being valid.
All right, now moving on to reachable panic
for availability issues.
Now, these reachable panics, it's a bit more complicated
now.
There are some changes in the code
that make it not so clear cut.
The nodes are not just going to panic in most situations,
but sometimes they do, and we want to include it
because we think it's still very interesting.
So in Rust, because all of these blockchains and the code
snippets you've seen so far are coded in Rust,
if you can't handle some kind of a result that you did,
you have the option to unwrap the result.
And then if the result is valid, then the program flow
continues.
If the result is invalid, you decided as a programmer
that the state of the program was not recoverable,
and you just panic.
And this is very useful in a development scenario
because you just don't care about if the result is
going to be there or not, and you can just
put unwrap everywhere as a developer,
and it's going to speed up your development.
And you can come back to those unwraps later
and maybe add some management of these edge cases.
But sometimes those unwraps get forgotten in the code.
And this was a situation where in some kind
of a decentralized exchange, there
was a conversion function that was
doing a check multiplication.
And the programmers had forgotten
to handle the case where the check multiplication was
returning a non-valid result, like if it was overflowing.
And it was just unwrapping, and then basically the whole node
might just go boom and have to restart its process.
So these are perfect targets for the fuzzers, by the way.
That will be showing up in a bit.
OK, so in the beginning of the presentation,
I was telling you that you can create your own runtime just
by cherry picking some modules that are already
provided by the framework.
But now when you actually create and import these modules,
you are also supposed to set some parameters
for these modules.
And there might be a bug in there,
actually, if you don't set these parameters correctly.
In Substrate, each account has actually--
it's represented by a structure.
And inside, we're going to find the balance of the user,
some internal counters.
And you can also enhance these structures
with more parameters that are specific to your project.
So basically, each account is going
to take some space in the storage of the blockchain.
Now the idea is that you should not
be able to store such data and have such an account if there
is no token in your account.
And for this, there is a parameter
that is called existential deposit.
And basically, this parameter will say, hey,
this account is allowed to exist as long as there is at least
an x amount of tokens in that account.
Now, it often happens that this existential deposit
is being set to zero or to a small value,
such as during the development process,
it's just easier for the developers to test everything.
But if this goes into production,
then a criminal will be able to come in and create
a super large amount of accounts for a very small fee
or also for free.
So basically, in the end, bloating
the storage of the blockchain and making it run slower
and slower in the long run.
All right, now we're going to move to storage bloating
as a class of issues itself.
I mean, Gabi already explained most of the issue already.
The issue with storage bloating is
that if you, as a random person, try
to join the network in the future with a full node
or something, if you have to download terabytes of data
to actually make the node run, it's not going to be easy.
It's not going to--
yeah, it's not going to make it universally available
to everybody if you have to download terabytes of data.
So yeah, so we take this class of bugs
seriously when we review a project.
If the fee for creating some kind of storage item
is way too low, then we report that to the teams.
And here, in this specific situation,
there was a module under construction
that was handling bridges to other chains.
And it was not ready yet, but they wanted still
to publish the code without the actual bridges part
implemented.
So they just had the registration
of bridges open to the public so that you
could pre-register your bridges with the identifications
you wanted.
And then in the future, you would
be able to send out those--
to create those bridges for real.
The thing is here, they had said the required state for--
the required amount that you needed for creating a bridge,
way too low.
Like, this is 1 million, and it basically
means 1 million of thousands of a fraction
of a millionth of a cent.
So in the end, it's like almost nothing in euro terms.
So you could just register--
spam this bridge registering, and it
would allocate quite a big amount of data on the chain.
But most importantly, in the future,
when those bridges would actually be implemented for real,
this amount of data would actually probably
also go up.
So yeah, this is a classic storage bloating issue.
OK, so now that you are aware of what type of bugs
are we looking for in the blockchain,
you might be wondering, how can I actually
start finding some of this?
And there are at least three techniques that you can use.
Two of them are automated, and the third one,
it's the manual code review.
We're going to start with the third one.
So basically, if you are already doing some manual code review,
it should be quite easy for you to move to the blockchain world.
But you have to keep in mind that a different threat model
will be applied here.
Because, for example, in an Android application,
let's say, if you have a crash, OK, that's fine.
The app is going to crash, you just open it again.
But in a blockchain, if you have a crash,
that might actually be a critical vulnerability
in the end.
So for the manual code review, you just
need to change a bit the threat modeling
and look for different stuff.
Then keeping in mind the five classes of bugs
that we've showed, you are kind of on a good track
to actually be able to start finding these bugs as well.
Mostly using fuzzing, and for this
we are also going to give you a demo about fuzzing.
Mostly we are finding a lot of bugs with static analysis
and fuzzing.
Mostly all of them, I would say.
But of course, the manual review,
it's all the time complementary.
All right, so very quickly, what's fuzzing
and how do we apply it to these blockchain projects?
Fuzzing is when you take a program
and you feed it a whole bunch of inputs
and you wait for any kind of misbehavior
to happen or something that you've aligned to trigger.
And in this situation, the program
is the blockchain runtime itself.
So the part of the blockchain that
executes transactions one after the other
and that interacts with the storage and everything else.
And our fuzzers are coverage-guided fuzzers,
and they're going to look at the code coverage
inside of the blockchain application
to see what's interesting and what's not.
And yeah, it's going to find really complex stuff quite fast
and that it really works very well in this situation.
So to make it really quick, the fuzzer
is going to have a bunch of potential inputs.
It's maybe going to apply some mutations to it.
Then the fuzzing harness will decode these inputs.
So take one input and split it into a few transactions,
decode those transactions, and then run them
one after the other, maybe skipping a few blocks here
and there for timing purposes.
And then the target is going to just run the program.
And if we find some kind of an overflow, some kind of a crash,
some kind of a timeout, or any other pre-programmed values
that we've set to trigger, yeah, it's going to report a crash,
and we'll take a manual look.
All right, now time for the demo.
So we launched a fuzzer a bit before the talk.
We hope it found the crash that we wanted it to find.
Just to show you, this is one interesting part
of the runtime code that we're going to fuzz.
This is the template runtime, so the one
that you're supposed to start with if you
want a normal substrate stack.
And these define the different modules
that you're going to have in your chain.
So the interesting one here is the template module,
which is, as it says, a template module, which
is defined right there.
And in this template module, you have two calls.
The first one is called do something.
It's going to take a U32 as an input,
and it's going to put it into a storage item.
The second one is called cause error.
It's to show you how an operation can cause an error
and return an error in a safe way.
And it's going to take the value that you put into storage.
If it was nothing, then it's going to return an error.
If it was something, then it's going to increment it.
Now, the original code for this is a checked addition.
And the code that we actually replaced it
with to find a crash is just new is equal to old plus 1.
So unchecked math, the classic mistake
that we see over and over again.
And then it puts that new value into storage.
Now, let's look at our fuzzer for a bit.
It's like 300 lines of code.
It's open source.
I'll show you the repo in a bit.
And yeah, you basically decode stuff and then run the--
let me show you the running of it.
You just run the extrinsic here, and the rest
is basically stuff that we've added over the years
to make sure that the--
to make sure the situation is perfect for our fuzzers
to find a crash.
So let's look at how the fuzzer has been doing.
It has found a crash.
Perfect.
We are going to be able to show it to you then.
And this crash hopefully is running the do something
extrinsic first, the do something operation with U32
max, and then cause error.
And this should crash the chain because there was an overflow.
Oh, yeah, one thing I forgot to mention
is that the fuzzer has some checks enabled.
For example, this overflow check,
because in a real world scenario,
like the blockchain would just make the value overflow
and you wouldn't see anything except for the value that
has overflowed.
But in our situation, we want to catch those.
So the fuzzer will actually catch
those overflows as crashes.
All right, so this is Ziggy.
It's our in-house manager of fuzzers.
It launches an AFL instance, or a few.
I think there are like 40 of them on this server.
And also, HongFuzz in parallel, and then they
share corpus so that they can learn from each other.
And yeah, let's try to run the crash that we found.
All right, what does it say?
It says that it tried to do something
with a value that looks very much like U32 max,
and then it ran cause error.
And yeah, there it crashed to attempt to add with overflow.
So we found the bug that we had planted there in our template.
And this is a very easy, or quite easy way
to find a lot of bugs in many blockchain systems.
Now, to show you the impact that this have had--
oh, yeah, first.
The harness itself is open source.
Please go check it out in this repo.
The Ziggy, the tool that you saw that we
used to manage the fuzzers is also open source,
and you can find it there.
And now, back to Gabi.
OK, so now you are aware of what kind of bugs we can find them.
Louis just showed you that the fuzzer that we are using,
it's also open source.
So with these two, you are kind of on a good track
to actually start finding bugs by yourself.
And in the past one year and a half,
we analyzed 45 projects in the ecosystem.
And most of the bugs that we found
were actually found in a semi-automated way, most of them
using the fuzzer harness that we've showed before.
So the idea is that you can actually start
and playing around with this.
You can contribute to the ecosystem.
And you can do this out of curiosity, of course.
Or you can even look into some of the projects
in the ecosystem that are running bug bounties programs.
So in the end, while we--
like in the past one year and a half, two years,
we developed our tooling all the time
in order to be able to automate as much as possible
and cover as many projects as we could.
But of course, as I said in the beginning,
there are over 150 projects in the ecosystem.
We covered only 45 so far.
So we are kind of aware that in order to have an impact,
we need the help of the community for this.
And this is why we also open sourced the fuzzer.
And we encourage you to start playing around with that.
You can consider it a new game, a new puzzle, whatever
gets you excited about it.
Also, as I was saying about the manual code review,
the ecosystem itself kind of needs a different threat
modeling than the usual security reviews that we are doing.
So the entire ecosystem would benefit
if both the developers and the security researchers
will start to collaborate more and share this new threat
modeling with everyone.
So with that being said, thank you very much for making it
to our talk this morning.
And we are waiting for your PRs, comments, or anything
on the open source fuzzer.
Thank you.
[APPLAUSE]
We still have some time for a few questions.
If you have a question, please line up at the microphone
over there.
Signal Angel, is the internet already awake?
No, the internet is still asleep.
They do not have any questions.
Anyone else from the audience with a question?
I didn't understand the incentives
for paying $25,000 to use one gigabyte of storage
on the blockchain.
What's the attacker model?
Who would want to do that?
That's a good point.
Yeah, I heard other blockchains right there.
I guess, yeah, you have an incentive
to make your competitor project as hard to get into as
possible.
And it is a lot of money, this $25,000 for one gigabyte.
But if you compare it to other blockchains,
for example, Ethereum, you would need
to pay tens of millions of dollars
to actually put a gigabyte on that blockchain.
Then the idea is that you make it harder for other people
to get into the ecosystem.
And I mean, this issue was probably low,
or maybe medium severity, probably low.
But potentially, you could find something
that scales even better, where you can actually
go hard on the storage.
So don't asynchronous programming paradigms
and things like this usually say that you should embrace panics?
And isn't it a major failure to try and exclude all the panics?
Yeah, that's an interesting idea.
I guess in that scenario, some nodes
are incentivized to publish blocks,
because they will get some reward out of it.
And if an attacker can make them unable to produce them,
they might even get punished for it.
So yeah, I guess in a theoretical way, I agree.
But the incentives are aligned so
that an attacker wanting to make a node panic
will actually cause a bit of mayhem in the process.
Thank you for your talk.
Let's give another round of applause to Louis and Gabi.
[APPLAUSE]
[MUSIC PLAYING]
[Music]