[ Music ]
>> Hello and good evening on day two
of the Chaos Communication Camp 2023.
It's late in the evening.
This is Millie Way stage, in case you're wondering.
And the next talk is going to be about incident responses.
So if you're curious about how do you even get there
to have an incident response, how you could prepare
for an incident response, and how you could support
in your organization the incident response team
in doing their job and trying to fix whatever broke.
Let's put it that way.
We have the right talk for you.
This is stories from the life of an incident --
from incident responders, Harry and Chris.
Please, a very warm round of applause.
[ Applause ]
So good evening.
Thank you for joining us today.
We will tell you a little bit of our life as incident responders.
And I'm Chris.
I did my computer science studies at the University
of Erlangen-Nuremberg.
I do this security stuff for over 10 years now.
So my CV is a little bit longer at the moment.
I'm a detection engineer.
Before that, I was a long time working in DFIR,
so Digital Forensic Incident Response
in different organizations.
And --
>> Yeah, I'm Harry.
I studied electrical and computer engineering
at RWTH Aachen University.
And I played a lot of CTF and did some hacking stuff
at Chaos Computer Club Aachen.
During my master's, I worked at X41 DSEC doing pen testing
and patch analysis.
So I also have some kind of offensive security background.
And for around one year now, I'm working
at GData Advanced Analytics doing digital forensics
and incident handling.
First, Christian will give you a short introduction,
and then he will tell you how a classical ransomware attack
looks like.
And in the second part of the talk,
I will tell you how the incident responders work
and what you can do in advance to make it go as smooth
as possible and support the incident response team.
>> So as Harry told you, I will probably --
we'll talk about ransomware,
because the customers we usually have are small
and medium-sized businesses, universities, and hospitals.
And those are regularly --
unfortunately, regularly hit by ransomware gangs.
The main reason for this -- and that's --
if you heard the last talk, why they may be not that responsive
and not so interested in -- they just lack the resources,
so the manpower to do proper security measurements
and to secure their systems, especially in situations
where you are, for example, in a hospital, have medical devices
where you cannot simply install an AV on or even patch the
system, because you lose the certification
as a medical device then.
But also in companies, manufacturing companies,
on the shop floor, we are talking about systems
that have run times of 25-plus years.
So if you look back now, 2023,
we are talking about XP and older systems.
Fun fact, I was in a ransomware case in a WannaCry in 2017
when I got a call from a person from the shop floor
asking me if we have an NT4 expert that can tell us
if WannaCry is affecting NT4.
Of course, you don't need to be an expert on NT4 in this.
WannaCry is, of course, not affecting NT4 systems.
So due to the time slot, we thought memes are the best way
to tell you those stories.
And we have a lot of them.
So in the first section, I tell you a little bit
about how an attack works.
There are a lot of different possibilities how you can
describe and how to structure how an attack works.
There's the MITRE ATT&CK framework, for example.
There was a talk yesterday by Meika Sauko here on the stage.
There's the original cyber kill chain from Lockheed Martin.
You have stuff from companies like Mandiant.
They're targeting the tech lifecycle.
But that's all, in my opinion, too fine-grained.
That's the reason I just take three simple steps.
Get a foothold in the door, look, move, play around,
and cash out.
Those three I will just go over.
So start with get a foot in the door.
So normally, we see three ways how attackers can get
into the environment in ransomware cases.
You have vulnerabilities in remote or internet-facing systems.
You have the remote services itself.
And you have malware.
Starting with the vulnerabilities.
And I just looked up the last four years.
And maybe somebody remembers Netscaler,
the so-called shit tricks vulnerability in December 2019.
It was released mid of 2019, December 2019.
The first POC, publicly available POC,
was in the beginning of January.
And the patch was available in mid of January.
So there was around one week to one and a half weeks
between a public proof of concept for the vulnerability
and a patch for the vulnerability.
And what we saw during 2020, a lot of companies patched,
but the patch didn't remove the compromise.
So they were already compromised.
And yeah, with the patch, they didn't remove the compromise.
So what we found, what we could provably see
or prove evidence for was a nine-month customer
was breached after nine months using this vulnerability.
And we had other customers where we could see
that the Netscaler was affected after two years.
But we couldn't prove that this compromise
was the reason for the actual ransomware case.
And of course, such vulnerabilities
happen not that often.
Yeah, so 2021 gave us a Hafnium exchange vulnerability.
Also, a similar situation, the patch appeared
as an out-of-band patch from Microsoft
on a Tuesday evening, 10 o'clock in German time.
We saw during our incidents or the assessments we did
that the first exploitation attempts
were seen on Wednesday in the morning at 5 a.m.
So around seven, eight hours later.
I know one guy who could patch
because he was online when the patch was released.
Otherwise, Germany was unable to patch in time.
And of course, we can go on with 2021.
Proxy shell also exchanged vulnerability.
Proxy not shell also exchanged vulnerability.
We have in 2022 VMware Horizon,
the virtual desktop infrastructure from VMware.
Just to name also open source stuff,
Zimbra, a collaboration platform including an email server,
has had a vulnerability.
Actually, the vulnerability was in CPIO from 2015, I think,
which led to a compromise using via email.
So you send an email with a CPIO,
with a specially crafted archive file,
and you could drop a web shell in one of the directories.
Yeah, you have, of course, 40 OS,
which is the 40 gate VPN and firewall operating system.
And if you read the news, we started the beginning again.
Netscaler had some issues several weeks ago.
According to Fox IT, we have 1,900
still unpatched Netscalers worldwide.
How many patched Netscalers exist
that have not been checked for compromise?
We do not know, of course.
So that will be a nice year, probably.
So what can you do against this kind of attack vector?
Patch your systems is one thing.
As you see, that doesn't lead to the --
or what you need to do afterwards in such cases,
you need to check your systems for possible compromise.
That is important.
To reduce this, I highly suggest
put your services behind some VPN
so that only people who already have connection to the VPN
can access your services or the services they need.
And that would reduce the attack surface,
at least to the VPN server.
So, but of course, we can also think about remote services
without vulnerabilities.
There can be configuration mistakes,
so the admin does something wrong.
There can be insecure default configurations like this.
I don't know if you know it, but the local admins
or the administrators on a Windows system
are automatically in the remote desktop users group.
And so we had several cases,
especially in the beginning of the pandemic
when everybody moved to the home offices
and they needed to put people fast in the position
to access the internal systems again.
They just put an RDP server on the internet
and hoped for the best.
Additionally, if you put services on the internet,
of course, brute forcing and credential stuffing
are attacks that are possible.
So, brute forcing,
just trying the username and password combinations,
credential stuffing using already leaked passwords
or credentials from leaks you find on the internet.
What you can do about this kind of attack vector
is just, as I said, use multi-factor authentication
and reduce the attack surface,
as in the vulnerabilities before,
by moving the services behind a VPN
and then use multi-factor authentication on a VPN,
of course.
The last vector that we see normally
that attackers can get in the network is malware.
We all know this about those funny emails
you get with the attachments.
Include that have either Word documents attached,
either zip files with visual basic scripts,
JavaScripts, and what you can get,
ISOs you see a lot these days.
Or what you can also have,
that you can have just a link inside the email
and you download the respective file
from some shady file sharing website.
What we saw over the last year was USB sticks again, funnily.
I am not sure if you have heard about Raspberry Robin,
which is a malware that warms via USB sticks.
But I have not seen it as a vector for ransomware yet
on my own.
But there are people who said that it is an initial access
broker for some of the ransomware gangs.
What can you do about this?
If you think you can, of course,
ban simply some file extensions in your mail server
or you change the file association types
in your operating system,
meaning that you do not open the JavaScript
and visual basic script files using, for example,
the Windows Scripting host, but open it with Notepad.
And that will, of course, some people will be --
some people will think about what this is then
and ask the IT guys,
but it is better than running the script itself.
One thing I do not like to say, but keep your AD updated.
This is one thing, keep it updated and read the logs.
We see a lot of incidents where we see that already days
or weeks before,
we could have seen that there is something going on
in your network.
And if you see malware in your AV logs, then react to it.
Just check it.
You do not know how long this malware has been on your system.
The thing is that just because your AV detected it now,
it might have received an update for its signatures.
And the malware was active for days or weeks before.
When they are inside,
then they usually look, move, and play around a little bit.
When they look around, what they do is they enumerate AD.
They do port scans. They search for vulnerabilities.
They check how they can escalate their privileges.
They try to find credentials.
Kerberoasting, we heard in the talk before,
for example, is this one thing.
They try to identify accounts you have running on your systems.
They can get the credentials from and reuse.
For that reason, one of the most important things,
I think, is that you have a principle
of least privileges in your environment.
Only what an account needs, you should be able to do.
You should use dedicated service accounts
for your services, of course.
Just for your information, a service account is not an account
that has an SVC underscore in front of it
and is otherwise a normal user account.
There are dedicated service accounts in Windows environments.
Use them. Use strong passwords.
Today, I know companies who still use
eight-character-long passwords.
I think that is 20 minutes on a decent graphics card today.
Use strong passwords. Length matters.
Twelve-plus characters is the minimum, in my opinion.
Do not reuse passwords, especially on your systems.
The local administrator,
especially in small and medium-sized businesses,
you see a lot that people use the same password
for all local administrators.
If I have it on one system, I have the whole company.
Do not reuse it.
When they move around in your network
using either password hashes they found,
valid credentials they discovered somewhere,
vulnerabilities are also used to move around in your environment.
They try to establish persistence mechanisms.
We heard also in the talk before about the C2 channels,
command and control channels they use.
They install, in some cases, directly any desk,
a team viewer, or other remote control software.
Or sometimes they also use tunneling softwares like ngrok,
or recently they started using CloudFlare D.
What you need to do is have a proper network segmentation.
With a proper network segmentation,
I do not talk about subnetting.
Subnetting means you just have different subnets.
You need to have a firewall between them,
and you need to have rules between them
that restrict access between your subnets.
One thing is especially important,
please use network segmentation to restrict access
to your backup and your management systems
as far as possible.
We see a lot, especially as I said,
in the small and medium-sized businesses
or in the other organizations we have as customers,
that they tell us in workshops,
"Yes, we have network segmentation.
Every building is one segment."
And the question, "Yes, you can move in a segment.
You can access everything."
"Yes, you can, and between the buildings,
you have also a firewall and you cannot access anything."
"No, you can access everything."
And also your VMware management console,
"Oh, yes, yes, we can, so everybody can access it."
"Yes, of course." And that does not work.
When they play around,
they normally try to gain more privileges.
So privilege escalation is a thing,
local privilege escalations normally,
but also using vulnerabilities or misconfigurations,
insecure default configurations.
My personal favorites are group policy preferences
or passwords in group policy preferences.
This is no longer possible since, I think,
2014 to put passwords in group policy preferences.
However, if you had your password stored
in those preferences before the patch in 2015,
2014, then they're still there.
And yeah, they are AES encrypted,
but the encryption key is on the Microsoft website.
So you can just download it,
just take it and decrypt the keys.
During that phase, they also try to disable
your security measures.
The thing you can do is, of course, patch your systems.
So try to get your vulnerabilities
out of this equation.
Try to configure your system.
systems in a secure way, this is not always possible due to some shitty third-party software.
And keep your AV updated and please, as I said already, check the locks and act accordingly.
So in the last phase, they cash out, that's when they start using your backup service,
so they copy data from your environment using file sharing platforms, for example, Mega
NZ was once a thing, WeTransfer, we had already every other file sharing platform you can
think about as a possible way to exfiltrate data.
They also use their C2 communication channels, so sometimes they just use the possibilities
in any desk or in RDP clients.
Or they use file transfer protocols like SFTP.
We saw, for example, in one case that they tried to install FileZilla on every machine
they had access to, because on the first one it didn't work, on the second it didn't work,
on the third it didn't work, because SFTP was blocked outgoing.
And that is one of the things you can do to prevent exfiltration, block at least protocols
you know that you don't need in your environment, and proper network segmentation, of course,
is a general thing.
So in the last step, that's when they start the encryption.
They're running the ransomware, normally they have domain admins at that point, so they
can run it on all domain-connected systems.
They can also disable, of course, when they are domain admin, they can disable the AV
before they start the ransomware.
Ransomwares today disable services like databases and such things, so that they have the full
power of the machine for the encryption.
If you get lucky, not everything works perfectly, because they use group names, and Windows
is especially picky when you have non-English Windows installed.
For example, in Germany, the group everybody is called jeder, and we had cases where the
server didn't work that well because they couldn't change the permissions of the files
first.
They use different encryption schemas, normally they come with an asymmetric and a symmetric
encryption type, the asymmetric public key cryptography, the public key comes with the
ransomware and is used to encrypt the symmetric keys they generate in your environment.
Depending on the ransomware, they generate one key for each system or even one key for
each file.
It depends a little bit on the ransomware, how it works, but that's the usual thing they
use.
I would never count on the fact that there are possibly, maybe, there could be decryptable
things.
In my world, the ransomware gangs have learned and used the standard Microsoft Windows or
some other publicly available libraries to do their encryption.
They execute it via remote tools like PSX, PowerShell, or some use GPOs, group policies,
to execute the ransomware on every machine connected to the domain.
What can you do about this?
It's hard, but the most important thing is have offline backups.
You see, online backups are great, but not that great.
Offline backups is the most important thing.
Don't have it connected to your environment.
The USB disk on a system is not an offline backup.
In my opinion, if you see that something is still encrypting, I'm always hesitant to say
shut down the system, because you can break the encryption and maybe the file that is
currently under encryption or the files will never be decryptable if you want to buy a
decryptor or get a decryptor through some discussions with the ransomware guys.
If it's a VM, just suspend it, and that's it.
If everything is already encrypted, keep cool and call your incident responder.
So now let's talk about incident response.
What happens when it's already too late, and what can you do to support your incident response
team?
First, the things I'll say in this chapter are for our company and how we work, so other
companies might work a little bit different than that.
First, for some reason, incidents always come on Friday afternoon.
Some customers think it is a good idea to try to solve a case by themselves, maybe until
the end of the week, and if they didn't solve it until the end of the week, they call the
incident response team.
Please don't do that.
It doesn't help your company and it doesn't make your incident response team happy to
have to work on the weekend.
In addition, the longer you wait with calling the incident response team, the longer the
incident response will take and the more complicated forensics will be, because you have lock retention
times while trying to do stuff by yourself, maybe you modify some of the systems, and
it becomes much more harder to do precise forensics.
So what happens on our side when such a new incident ticket arrives?
The first thing we do is team internal coordination.
So we discuss do we have enough people, do we have a person for each role.
In our team we have three roles, incident handling, forensics analyst, and malware analyst.
So first let's talk about incident handling.
Incident handler is responsible for all the tasks that our customer is facing, and the
first point is always get the customer out of that headless chicken mode, like we call
it, because when an incident comes at our customer site, everyone is running around
in circles like headless chicken, doing something, but not doing anything helpful.
So this is always the first task for the incident handler.
Handler structure the customer, do meetings, and then do all the relevant decisions, leading
to a secure emergency operation mode.
That means in this case that you have working core infrastructure, so working domain controller,
maybe a working email server, and whatever you need, or whatever you define as very business
critical systems.
Let's go a little bit more in detail.
Probably the first measure will be to cut off the internet connection, because you just
buy you a lot of time with doing that.
No matter how many back doors the attackers placed in the network, if you cut off the
internet connection, the attackers can't access their back doors anymore.
And then you will start to rebuild your network.
You will define everything in your current infrastructure as red network, and then start
building up a green network with clean systems.
Maybe you will start with some admin workstations so that the administrators can work properly.
And then we go through a prioritized system list and build up the most important systems.
This can be of course, like I said, domain controllers, email servers, but also whatever
is important for your current company and for the business.
This is of course in a hospital something very different from a small business or maybe
a university.
And please have such a prioritized list before your first incident.
Because when you start discussing in an incident which system is the most important, then everyone
in the company will tell you something else.
And everyone will tell you that their system is the most important and has to be migrated
and checked and analyzed at first.
And this isn't really helpful.
So during incident handling, there might also be obstacles.
For example, if you have backups and you encrypted your backups, this is great.
But you should think about where to store that encryption key.
Because when you store it in your password manager and the password manager database
is in a VM, which then gets encrypted by the ransomware, well, you have backups, but you
don't have your key to decrypt the backups.
So this doesn't help you very much.
And this is not only true for password.
This is also true for, like, everything else which you could need during an incident, like
contact lists or network lists and so on.
So working with third parties is also a task for the incident handler.
This could be a data protection agency or the customer's customers, because they also
often have some panic and have questions which the customer maybe can't answer.
So this is also a task for the incident handler.
And of course, also working with law enforcement.
But there's one thing to keep in mind.
Law enforcement wants to do criminal investigation.
Incident response wants to bring you back to business as soon as possible.
This is not necessarily the same goal, but it finally makes sense to allow your incident
responders to share all information with law enforcement, because this is beneficial for
everyone.
So why do I do forensics at all, one could ask.
Well, the less precise the forensic results are, the more conservative the rebuild needs
to be.
So if you don't know anything about the attack, you have to rebuild everything from scratch.
So let's say you have maybe two week old backups and you know the initial access of the attackers
was only one week ago.
Then you could use the two weeks old backups for rebuilding, which is really great.
But if you don't know how long the attackers were in the network, then it's really hard
to say.
A second point why you should do forensics is to estimate the impact, especially in an
early phase of an attack.
It is important to know that the attackers only compromised some user devices or that
they already gained domain administrative privileges, because then the incident handling
is completely different.
And of course it is also relevant to know which attack path the attackers used, so you
can harden your network and improve in the future.
How should forensics be?
It should of course be correct, it should be non-disruptive for the customer, so that
they can concentrate on rebuilding their infrastructure and it should be fast.
What means correct?
You want to have a dedicated forensics analyst, so you want to have incident response team,
which is doing incident response in their day-to-day business.
And you want in your forensics teams one person who can really concentrate on forensics and
is not also doing the incident handling and the malware analysis and so on, because then
you can't really concentrate on doing the technical work.
It also should be non-disruptive, as I said.
That means for our company we do remote triage collection, we use the tool Velociraptor for
it.
This is an unofficial logo of Velociraptor.
It's a really nice tool, because we just send our customers an exe file, the customer just
has to execute the exe file and then it collects all the logs we need, all the information
we need and uploads it to our infrastructure, so we don't need to drive to the customer
and start copying hard disks for weeks, because this doesn't help anyone.
I should also say it should be fast, so we have a lot of automation and tooling to make
the work really fast, automatically pushing the uploaded files to our analysis infrastructure,
automatically doing reporting and so on.
I won't go into detail here, because this is enough for a single talk about it, maybe
on Congress, let's see.
So what happens when we do manual forensic analysis?
There are some questions you normally want to answer.
One is of course about the vectors, you want to find all vectors, the attackers placed
in the network.
Of course you want to have general IOCs, so indicators of compromise, so maybe which
IP addresses did the attackers use, so you can block them in the firewall, or which tools
did they use, so you can scan over the whole IT infrastructure and find further compromised
systems and if you have the IOCs you can also cross correlate between cases or share them
with law enforcement.
Another important thing is of course lettering movement, because you can use lettering movement
to build a timeline of the attack, so the mythology here is to look where the attackers
came from and where the attackers went.
So if you have a system A and you see the attackers came from system B, then you analyse
system B and go back in time, and the same you could do forward in time until you found
all relevant systems which were compromised and have a proper timeline of the case.
Another common thing in the life of an incident responder is to fall in a rabbit hole, because
it can be really, really interesting to do forensics, but you need to check what the
customer really wants to know or what your incident handler really wants to know, and
not just analyse something and fall into the rabbit hole, because this really doesn't help
anyone.
Here also the 80/20 rule applies, so I'd say you get like 80% of the relevant forensics
results in 20% of the time, and often this is enough for the incident handler to do the
right decisions to rebuild the customer's networks as fast as possible.
So let's go into reporting.
Nobody really likes reporting, but producing fancy forensics results, doing fancy technical
stuff is worthless if you can't explain it to a manager.
Nobody pays us to do fancy technical stuff, nobody understands, but we are paid to help
our customers and explain the situation to them and to do the right decisions, so yeah,
nobody likes reporting, but it is a really, really important thing.
So after delivering the report, the incident is over and all work is done, right?
No, not really, because such a report normally contains a lot of recommendations, what to
do for the customers, and as I already said, incident response more or less stops when
the customer is in an emergency operation mode with the core infrastructure and the
most critical systems working, but there's still a lot to do, and of course security
is a process, so there's always something to improve to stay ahead of the attackers.
So now let's go into a quick recap.
To protect against those kind of attacks, we have these topics just as a recap.
Patch your systems, it's important, keep them updated, vulnerabilities come and go, that's
one thing.
Have a sane, privileged account management, so use the correct user accounts for the right
task, use the second two-factor authentication for remote services, have a proper network
segmentation, so put firewalls and access rules between your networks, check your AV
locks regularly and act accordingly, and not really protecting against an attack, but keeping
the symptoms a little bit, not that devastating, have offline backups.
So how to make incident response easier for you, for yourself, and for the incident response
team?
First thing is don't try to solve the stuff by yourself, just call the incident response
team of your trust as soon as you know you have an incident.
The second thing is logging policy, because if you have a very short lock retention, for
example, and you only have locks for the last, like, two days, then it might be hard to find
out what happened, and if you don't lock stuff at all, especially in the cloud environments,
this is a thing where you need to look at, then it might also be hard to get proper forensic
results.
Then, as I said, have offline contact lists, because your main server is down, your accounts
are compromised, so you can't use, like, your Microsoft Teams or whatever you use in your
company, you need an alternative way to communicate to your colleagues and/or employees.
Then as I said, prioritize asset list and network plans, and a disaster recovery plan
in best case, and please all have all that in a secure place in best case on paper.
Well, nobody likes doing stuff with dead trees, I know, but dead trees don't get ransomwared,
right?
So that's it from our side, more or less.
Normally this is the point where you can ask us questions, but we want to spin that around
and give you some questions, maybe someone can guess the right answer.
What do you think is the shortest time to domain admin?
We saw an incident between initial compromise and the attacker scanning domain admin.
Yeah, it's not -- somebody said 17 seconds, two minutes, well, it's not that bad.
Six minutes is what we have seen in the incident.
What do you think is the shortest lock retention on a domain controller?
So how long -- zero?
Ten hours, I hear?
Well, it is 2.5 minutes.
What do you think is the highest number of domain administrator accounts we saw in an
incident?
200?
50?
64?
With 120 people working in IT?
What do you think is the longest dwell time?
So the longest time between being initially compromised and realizing that you are compromised?
60?
No, it's not that bad.
I hear a lot of years.
Well, it is like around two years.
So now that's really it from our side.
No time for questions.
Thank you.
If you have questions, we will be there outside waiting.
Come and join us.
Thank you.
Thank you, Herring.
Chris, a warm round of applause.
Thank you.
[ Applause ]
[ Music ]