Jade: It's an honor to be here to talk to Congress.
I'm here to tell you about two leaks in the Great Firewall and how they tie
together and you know what we can do to sort of further this sort of research.
So just to give an introduction to myself, I'm Jade, I'm a PhD
student at the University of Massachusetts Amherst at SPIN.
My advisor is Amir Houmansadr, you should definitely go
apply to work with him if you're interested in the stuff.
So I work mostly in internet censorship measurement and internet censorship circumvention.
So just to give a background of, you know, how the Great Firewall works, what it
is, so most people probably know that in China you can't go to a lot of websites.
They're blocked by different mechanisms, HTTP, TLS, DNS, but in
general they don't like you going to websites that they don't like.
So you know the usual thing is something like an HTTP or TLS reset where
you try to access a website and it will send you a bunch of TCP resets
saying, no, this connection is terminated, you're not allowed to go here.
But in this talk we're going to focus on their DNS injector.
So the way the DNS injector works is you send a DNS request for a website,
say google .com, that DNS request will actually go to the real resolver and the
injectors will send back fake responses with bogus IP addresses, totally not real.
And you won't be able to go to the website because TLS
will break and the IPs don't even resolve in China.
They're actually black holed at a national level mostly.
Yeah, so the interesting thing is there's not just one injector in China but actually three.
So when you send a request, you can actually get up to three injections.
And all of these injectors have different characteristics.
And this talk will mainly focus on injector three.
So even with the cases of the multiple injectors, your DNS
packet actually isn't dropped at the final destination.
The real resolver does get that packet.
So the way the bogus IPs work is when you send
the query, you get an IP, it's totally blocked.
But something interesting about these IPs is actually some of them do resolve.
We noticed a little bit over the last couple of years, stopping earlier
this year, that these IPs would respond to a TCP handshake and nothing else.
And some of them actually hosts an adult website.
So some percentage of requests for blocked websites in China will actually lead you to porn.
Another interesting thing about the injector is that it's bi -directional.
So if you, you know, send a request and you get this
injected response from China, you'd expect that, of course.
You can actually do this from America or from Germany, probably.
I didn't test it on this Wi -Fi, but you can try it.
If you send an NS lookup for something that's blocked in China
to most Chinese IPs, you'll actually get an injected response.
So you can play with this injector anywhere.
It makes it really nice for experimentation.
So, yeah, these are the characteristics of the injectors.
As I said, we'll be focusing on injector three.
And these are just some fingerprints.
So you can tell, like, there's different IP and DNS level flags on these packets.
And that's how we tell which injector is which.
On words to the actual subject to the talk, or the main subject, Wallbleed.
So Wallbleed is an academic work. I want to give a shout to the actual authors.
I made a contribution to this work, but definitely did not lead it.
So that's Shencha Fan of GFW Report, Jackson Sippe of UC Boulder,
Sakamoto San of Shinonome Lab, David Fifield, Amir Houmansadr from
UMass Amherst, Elson Wedwards, and Eric Wustrow of UC Boulder.
If you're interested in just reading this paper, it is here.
So just give an intro of, you know, DNS.
The packet structure for DNS is very simple.
So you start with a transaction ID that says, you know, what is the ID of my request?
And you, when you get one back with that same ID, you assume that that's what you asked for.
So this is how the injectors work. You have some flags, the number of questions.
We'll skip those since the answers that we don't need necessarily right now.
But the name length is the interesting part.
So DNS names are segmented into lengths and strings.
So you have, for example, eight, Facebook, three, com, and then a terminator, zero.
And then you have, you know, the type internet, or the type A and class IN for internet.
So what is Wallbleed?
So regularly, here's what the traffic looks like when you send an injected response.
So you'll send a request for a website, and you'll get back one of these bogus IPs.
That IP is Dropbox, by the way. Clearly not Facebook. And that's blocked anyway.
So it's like, why? Wallbleed.
So if you change the length, you notice we had eight Facebook
three com, but now we have eight Facebook zero x 20 com.
It started dumping memory. *laughing*
And that memory is UPNP traffic, which shouldn't be on the public internet.
So we started with this, basically.
And we did some refinements to see, let's get the
maximal amount of memory we can out of these things.
So first, we shortened our domain.
So we found that 3 .tt would actually trigger the injector.
And we could get vulnerable responses with this very short domain.
And so that gives us more space for activities.
It also turns out that you could just remove the QType and QClass at the end.
And it's fine.
And, of course, you know, we could actually take that length and change it to 0xFF.
And as a result, we actually get up to 124 bytes every time we send a Wallbleed query.
So there were some setbacks in the process.
So 3 .tt was the original domain used to trigger the injector.
And they eventually stopped blocking it for whatever reason.
Maybe it fell off the block list. Maybe the website died.
So we just changed it to 4 .tt. *laughing* And that worked.
So we also had an event where in September 2023,
somewhere in that range, they actually patched Wallbleed.
So they had all of the IPs that were responding for Wallbleed
just completely stopped within a couple of weeks or so.
And we assume this is how long it took them to patch it.
So we just made some modifications. And we put the QType and QClass back.
And it worked. And we moved the label a little bit.
That got patched within a couple of weeks as well.
But we gave us more time to do experiments with.
So to show, you know, what leaked, what happened, we did a longitudinal
experiment where we sent probes from UMass to a Tencent cloud VPN.
So it's our VPN. We control it. We're not DDoSing some random person.
And we sent it over UDP with varied ports to port 53.
And we sent 100 packets per second for about two years.
And got 5 .1 billion Wallbleed responses.
So when we looked at this, we were like, this is
real traffic from people in China using the internet.
Including stuff that shouldn't be there.
You'd think that a DNS injector only handles DNS packets. Nope.
It had all kinds of stuff. Like UPNP, for example.
And you might think, oh, you know, this is crazy, unethical.
Are you spying on the Chinese citizens? Well, no. We're not that kind of people.
We did some initial manual inspections to see what we were working with.
But all further analysis we did with regular
expressions, you know, we wanted to keep things ethical.
But one of the interesting things that we noticed is that this could
actually be used for some sort of roughly targeted surveillance.
So I mentioned earlier that if you send basically
any IP in China, you can trigger these injectors.
It turns out if you pick a specific IP, you can get injectors near that location.
So I'm sure some nation state was using this long before we found it.
But that's speculation. Not only my opinion.
So one of the things we notice frequently in this traffic
is SSGP, which shouldn't be on the public internet.
But it appeared a lot. Which is weird.
We also noticed a bunch of UPNP, which is also weird.
That shouldn't be on the public internet.
And so, when we looked at this, we looked at the headers
for the UPNP packets, which had a location in it.
So it was something like, you know, HTTP. All of them were private IP addresses.
So it's like, what is this traffic even? We also looked for packet headers.
So in IPv4, a lot of the packets will start with 0x45, 0x00. And so, we were
able to find, like, full packet stuff, not just, like, application layer traffic.
So when we drilled down to see, like, what the IP protocol field in these packets
that had the 0x4500, we found a ton of TCP, a ton of UDP, and some other stuff.
That's not super common.
Filtering down to just the TCP packets, looking at the source and destination
addresses, is we found that a lot of it is still private traffic, and
only a somewhat small portion is, like, public IP to public IP traffic.
So maybe there's some filters in place, or maybe there's just... It's unclear.
We suspect a lot of internal management traffic for
the Great Firewall went through the same buffer.
So, yeah. So a bunch of internal management traffic, right?
Well, if you remember Wallbleed, you've probably seen a lot of this.
Or not Wallbleed, uh, Heartbleed.
We were able to find various, you know, what people were browsing.
We were able to find cookies, mail, as well as people's passwords, for example.
Lots of very unfortunate privacy leaks happening here.
So something weird we noticed early on when doing these experiments is the first
four bytes of the leaked payloads would have, like, these seemingly random bytes,
but the bytes seemed to depend on the IP and ports of the traffic that was sent.
So we suspect it might have been some sort of, like, identifier
used for, like, tagging the traffic and load balancing it.
But we noticed they changed twice.
So in September 2022, the percentage of traffic that had this dropped significantly.
And then in June 2023, it dropped even more.
So we also noticed in this traffic some x86 -64 stack pointers.
So we're thinking, oh, you know, maybe this is a buffer overflow into some code.
We can figure out, you know, the source code of this injector.
We didn't get that far. It didn't seem to have that.
But we did some weird patterns.
So if in cases where we found these stack pointers, we say s is a stack
pointer, c is a code pointer, just based on, like, usually what addresses
these are in Linux. 124a are some comment bytes we found frequently.
You can check the paper for what exactly those were.
But we just use them as codes here. And then we have a wild card.
So if we graph the patterns of what of these patterns appeared
frequently, we can notice two change points here and here
that actually match what happened of the digest bytes.
So based on this, we were able to figure out when they were patching the great
firewall and use this as a sort of side channel to see what was being changed.
So we also, you know, just to make sure we actually weren't just
seeing internal traffic, we actually did a real experiment on this.
So we sent probe traffic.
So we used the string GFWBleed, which sureley shouldn't existis on anyone else's traffic.
And some tags to figure out, you know, where our experimental traffic came from.
And we sent it from a single source IP to port
53 and we sent it at about 30 packets per second.
We also sent Wall bleed packets at the same time.
So when we send these queries, sending it both at the same time, and
we expect to see Wall bleed responses containing our own tagged traffic.
We did, actually, at a somewhat low rate because this
machine was handling the traffic of basically most of China.
But we noticed that the percentage we saw actually correlated with when people were awake.
So clearly this was representative of real internet traffic.
We also, from a probe in UC Boulder, sent out probes to all over the world.
Just sending these GFW Bleed, doing the same experiment,
but to all these different arbitrary IPs around the world.
And we found that traffic to IPs using, I think, Geolite 2 database
for locating was affected by Wall bleed everywhere, basically.
Anything that routes through China at one point ended up getting leaked somehow.
So this is not just a threat to the users of China
and their privacy, but also users around the world.
So just to give an overall timeline.
So we first discovered Wall bleed in 2021. Eventually
they removed that original domain from the block list.
There were some patches.
And then the first main patch we noticed where this was fixed was November
2023. And then the second fix for wall bleed v2 was 2024. So just mainly
the main dates for that to remember are in around September to November
2023. And around March 2024. So now I'm going to talk about another leak.
So recently, Geedge Networks is a Chinese cyber security company founded
by Fang Binxing, the so -called great father of the great firewall.
And 600 gigabytes of documents, source code, binaries, all
sorts of fun internal junk was leaked from this company.
And you can go grab that right now.
T his was released by an anonymous source via Enlance Hacktivista.
And there was a lot of really great initial analysis
provided by the great firewall export project.
And that includes Amnesty International, the InterSec Lab, Tor Project, Justice for
Myanmar, Paper Trail Media, Der Standard, Follow the Money, and The Globe and Mail.
So definitely go check out that reporting. It's really good.
And so we've been doing some further work because obviously
this is a gold mine for academic censorship researchers.
So we identified as one of the components, so the MESA platform's DNS , parser.
So we strongly suspect that the MESA platform was
what they used for the DPI within Geedge Networks.
And we specifically looked at their DNS parsers.
The idea was, did these guys do Wall bleed? You know, is it their fault?
Can we point fingers at them and make fun of them?
And so looking at the timeline, here's some git commits.
So based on these, we might say that, you know, that looks pretty promising.
And then the patch after that, you know, maybe it was late.
Maybe they hot fixed it. Something like that.
So these would be the comments to look for if that were the source of the leak.
So probably not. We looked at the wording for memory leak.
I use Google Translate for this because I don't know Chinese.
But we think memory leak here implies, like, an actual memory leak.
Like, you know, not freeing stuff. Um, yeah.
So what the patch appears to do here is it will take
the parsing logic and move it earlier in the function.
And so that avoids allocating a session object for the, DNS session.
Something funny I noticed in the source code is they call their malloc, dictator malloc.
Like, that's a little on the nose, guys, don't you think?
Um, but so, you know, looking at the part comments
after that, they didn't have any relevance.
They were just Linux management stuff. Uh, so yeah.
In general, censorship is getting worse across the world.
And what we want to do here is note that, you know, these censors can be attacked.
You don't just have to, you know, measure, figure out
what they're blocking, figure out how they're blocking it.
You can actually just go after them.
And this hurts not just the censor, but also, the users. It's harm, obviously.
Like, people being in a censored country and having some, like, idiots
running this, like, unsecure software just destroys your privacy as a country.
If you're interested in any of this stuff, definitely come talk to me.
And if you're already doing this stuff, come talk to me.
I love getting connections. So, yeah.
If you have any questions, uh, you can ask me now.
Or, uh, you can email the primary author on this paper
who's going to have more knowledge of this topic.
Special thanks to, obviously, all of the authors of the original work, a special GFW Report.
Definitely go check their stuff out.
And then, additionally, all the , analysis of the Geedge Leak . Thank you very much.
*applause*
Thank you. All right. No, that's more time for questions than amazing.
Thank you, Jade, for your talk.
So, if you have questions, we have microphones over here.
You can see them in the room. Please queue up if you have questions already.
Otherwise, there's also the possibility to ask questions on the internet.
On the IRC, it's hackend .org. The room is hashtag 39C3, Hall Zero.
Or on matrix, it's just Hall Zero. I see no questions at the moment.
So, Jade, my first question, you mentioned this dictator alloc.
So, did you have a look into it? What was the special thing?
Was it aligned in any way? Or what's...?
Jade: So, it appeared to be designed specifically for single applications running on a server.
So, like, these are, you know, dedicated appliances.
It seemed to be doing some, like, low -level allocation at the kernel level.
But it didn't seem particularly special.
It's an interesting target for exploitation, though, and I can share
anyone the path in the leak if anyone's interested in analyzing it.
Herald: So, if some people want to talk up to you after the talk, where can they meet you?
Jade: That's, I actually don't know where I'll be during Congress.
I can probably put up my email.
Okay. Then hit up my email.
I see one question over there at microphone number four. Please shoot.
Questioner: Thanks for your amazing work.
It's really funny that we can talk about this topic here in
public, like exploiting some, firewall service and so on.
So, it's actually unusual, I think.
So, it's just like , the only opportunity to hack , a censorship
firewall to not get , driven by court or something like this.
And especially, how do I handle , that you are identified
that the vulnerability was maybe also abused by other
actors, which maybe not have that , good intention in mind?
Absolutely. Yeah.
Thanks. Okay.
Well, thank you for the comment. Any other questions at this point? Signal Angel?
I don't see any. So, let's thank the speaker again. *applause* Oh, there's one.
Oh, sorry. There's one just arrived.
Thank you for your excellent talk. Would you ever consider traveling to China?
*laughing*
Jade: I'm pretty sure, especially now that they know my face, I'm like gigabanned from China.
Okay. So, uh, now they are all arriving. Microphone number one.
Hi. Sorry. I'm Mona.
I was wondering, it's been a few months now since the Geedge leaks.
I'm wondering if , there have been any updates or where we
can learn about what folks have been finding in them or yeah,
has there been anything published , I guess on this front?
Jade: So, as you probably know, a lot of us are scrambling to publish stuff about this.
We're currently have a bunch of stuff that's currently under publication
that I can't exactly talk about right now, but one of the interesting things
we're looking into related to this is actually, can we exploit Geedge?
All of their source code is there. We know how their things work.
We can find vulnerabilities. It's in there. You just have to find it.
And we've been actively into this.
Questioner: Can you describe where is Geedge , deployed or how is the code?
Jade: So it's deployed actually in , regional areas in China.
So the way, we believe it's not actually the Great Firewall China.
It's just sold to local customers.
So Guangzhou, um, there's an island off the coast of Taiwan.
I forget the name of , but it's also deployed internationally.
So Geedge actually , exports some of their censorship software to these other countries
, sort of following along the lines of wherever the Belt and Road countries are.
And this includes Kazakhstan, Pakistan, Myanmar , Ethiopia, and there's a mystery
customer that's like A21 in their code names , that we are not sure what it is
yet, but we're actively trying to figure out who's buying from this company.
Okay. Thank you so much.
Jade: Thank you.
Herald: I see another question of microphone number six.
Uh, yeah. Thank you again for the amazing talk.
Has there been any retaliation against China
researchers, either from, China itself or domestically?
Jade: I'd rather not get into the details for the safety
of the participants, but unfortunately, yes.
Questioner: Thank you.
Okay. Microphone number two.
Questioner: Thank you for your work.
I was wondering, can you say something about the
level of centralization of the Chinese internet?
So , in a sense that you can only see specific phenomena
in one ASN or , you have only one AS in China in general?
Yeah. So it's actually very diverse.
Certain areas like with the Geedge thing actually are deployed only regionally.
Like Geede's software seems to be deployed only to certain customers that want it.
In the case of Wallbleed, we believe it to be the national
level injector because many different IPs stopped responding.
Like I think we had the figure of it being in the millions.
And so that , indicates that this was sort of a national level coordinated response.
So it, it sort of varies.
As I said, there's also the three injectors, all of
which have different behaviors and different block lists.
Um, yeah. Thank you for the talk.
I have one question about the passwords you found.
You mentioned that you found these in HTTP headers or query parameters.
So does that mean that much of the traffic in China is
unencrypted HTTP or did they, did they break TLS too?
I don't believe they broke TLS. But I, I think it's just a lot of HTTP traffic.
They do have specific specialized resolvers and injectors
for other protocols like TLS and QUIC and even, uh, TLS ESNI.
Herald: Number five.
Yeah. Thank you for your talk.
I was wondering if you had done any analysis on what sort of DNS queries
would trigger the blocking and if there was anything surprising in that.
Jade: I've looked at some of the categories.
A lot of it ends up being like gambling or pornography
or, you know, politically sensitive websites.
I can share you places to get some of these big
lists of blocked domains if you want afterwards.
It's sort of what you would expect.
There's nothing particularly exciting about it in my opinion.
Herald: Number four.
Questioner: Thank you for a great talk.
The thing that you mentioned, there are three layers of DNS injectors really fascinated me.
Why there are three layers? Maybe why not four .
What does the purpose each layer has? Do you know, or can you share more about it?
Yeah. I mean, it's not clear what the purpose of these injectors are.
And I think there actually might be some work looking into this.
I suspect it's just, so China has, you know, maybe a national level
firewall that says like, this is deployed at all of the gateways.
You know, these have to happen.
Um, but a lot of censorship is actually the Chinese government saying, Hey, you
ISP, you need to implement censorship right now, or bad things are going to happen.
So, you know, that ISP implement censorship,
and then maybe another ISP implement censorship.
And so if your traffic routes through any of these ISPs,
it will get hit by all of the injectors along that path.
Questioner: Thank you.
Herald: I see the internet has more questions for us.
Signal: Yes.
The internet wants to know if you can tell us more about
the general setup that you use to analyze the traffic.
Jade: Sorry, I missed the question.
Can you come again? Okay. Let's send over to microphone number two for a second.
Questioner: I can also repeat the question from the internet. If I should?
Sorry. Yeah.
Questioner: But that's not my question, right?
So it was, if you can talk about the general setup of, I forgot the last word.
Jade: Uh, Geedge?
Questioner: Yes, I think.
Geedge. Yeah, absolutely.
So something we noticed , and I think I might have forgot to put the slides in it,
but so they have , their main DPI platform known as SAP , and then within SAP they have
a bunch of DNS parsers that appear to have formerly been known as Stellar, but then
they integrated it and it just became part of the platform , called MESA platform.
So there's active work in trying to actually run people's own
copies of that , that's being done by other great researchers.
We don't currently have a full understanding yet , but I'm
sure there's going to be some publications pretty soon.
Questioner: And my question was, why do you think there are different injection layers?
Like there was injection like one, two, and three, and you only did three.
Jade: Yep.
Questioner: Why do you think they are split?
Jade: Yeah.
So as I said , I think, I think I answered this previous question,
but I think it's the ISPs are essentially being told to implement
their own censorship system on top of the national level system.
And so this might not, this might be just for not just
accessing traffic internationally, but also internationally.
And so there's a lot of layers of censorship , not just
internet, but rather like, you know, social media companies
being told to do certain things, self -censorship as a result.
But there, there doesn't appear to be coordination as far
as we can tell, or at least much coordination between Geedge
networks and the actual national great firewall, weirdly.
But that, that's speculation on my part.
We haven't fully confirmed this, but that's the idea
I get from looking through these leaks for weeks.
Questioner: Thank you.
Herald: I've seen there's maybe another internet question.
Signal: Yes.
You mentioned in your talks that the DNS traffic
was changed from Facebook to Dropbox, for example.
The question is, is this part of the great firewall or what's the reason for that?
Jade: So it's just part of the great firewall.
So the IPs, as I , mentioned early on are mostly completely blackholed by the great firewall.
One of the things we noticed with the IPv4 injector for A requests is it will
send all these random IPs from Facebook, Dropbox and some other random stuff.
And we believe these IPs are probably related to websites they really
want to censor, like, you know, maybe human rights organizations.
Vut for the IPv6 resolutions, they actually send , mostly Facebook.
Like, like the IPs will have like FACEBOOC.
But something weird in the IPv6 resolver that, you know, maybe
others might be interested in is they send Torito addresses.
And not just normal Turito addresses, but broken Torito addresses.
So it'll have 2001, a bunch of zeros, and then an IPv4
address at the end, which is supposed to be encoded in Torito.
I don't know why they do this.
I think it's just they want something that doesn't resolve.
They do make mistakes sometimes.
Like I said, they actually redirected users to pornography.
Herald: Another question from the internet.
Yes. Next question.
Have you by any chance also looked at firewalls
from other countries like Iran or something else?
Jade: Iran is actually difficult to get internet censorship vantage points in.
A lot of work has been in China, but we are actively looking into other countries.
Especially now that with the Geedge leak, we've confirmed that ,
China is exporting censorship, not just locally, but also abroad.
Herald: Okay.
So thank you all the people who participated in this Q and
A and thanks again, Jade, for the very interesting talk.