[MUSIC]
The next talk is from Jadyn and she will tell us about the magic behind one-click deployments.
I think it's wonderful to just when you do server infrastructure, you do one click and the server is just there.
And you don't have the hassle of managing it by yourself as normal.
We also have a signal angel for your questions later.
You can post with the hashtag #CCCamp23NortX.
Then we will see your questions.
And now have fun for this talk.
Okay, let's just head back to the first slide.
And then we can start.
Well, it's buggy.
Okay, welcome to my talk.
The magic behind auto deployments.
As Leon already said, auto deployments is something that most of us have interacted with before.
Yeah, but let's start.
First thing, I'm Jadyn.
My pronouns are she/her.
I'm 22.
I work in cyber security.
And I do lots of things with computers and hardware and stuff.
You can find me on chaos social, Jadyn @ chaos social.
And on my website, as you've already seen, where you can also find the slides.
What is an auto deployment?
And what is a one click deployment?
We will have about four parts in this talk.
First, what even is it?
Then how does it work?
Then we will try to demonstrate it right here with real hardware and real data centers.
And afterwards, there will be a Q&A session where I can hopefully answer most of your
questions.
And if I can't, just hit me up after the talk or on social media or via email or whatever.
Okay.
One click deployments are fully automated deployments.
What does it mean?
It means that we can spawn anything like VM or container, IP addresses, DNS entries, and
much, much more, like user management, firewall, software configuration, monitoring, and anything
you can imagine.
The whole thing is pretty easy.
But you have to get some key ingredients first.
To allow access for an automated system on all your devices and access to APIs, you usually
need some kind of root access, usually done with FSH.
And you need your deployment sources, which are usually from just a repository where you
have your config files and your scripts.
And then there's your source of truth, which is in my case Netbox.
And then you have the big issue of secret storage.
You have to generate secrets on demand for root passwords, user passwords, and a lot
of other things, databases, everything.
And that is mostly done with a utility called pass, which is just using GPG to store your
credentials and encrypt them so you can just put them in a repository and everything is
fine.
The easiest way to get that is just get a dedicated host.
It doesn't have to be in a data center.
In my case, it is.
And you just plug in a small Yubi key.
Most of you probably know what that is.
It's a small device you plug in via USB into your computer, which can store your encryption
keys, your private keys, in a fashion that doesn't allow them to get off there.
They are on that small USB stick and nobody can get them.
Well, you could, but it's really, really, really hard.
Okay.
Inner workings.
You need to gather some data before we can even start with your deployment.
This data is usually provided by some kind of source of truth.
This is my favorite way of doing that because handling config files is, to be honest, quite
a pain to get them everywhere and writing config files isn't that easy either.
And as I said, we want that to work with one click via a web UI, so writing config files
is not that easy.
And in your source of truth, you usually store things like your host name, network interfaces,
IP addresses, gateways, DNS servers, your hypervisor config like memory, or if you get
your machine from some provider like AWS or something, you need to know how many cores
you need, how much memory and such things.
And you also want to know which services you want to install.
The second step is gathering the secrets.
I've implemented a very easy way which is based on the password utility.
As mentioned before, it uses GPG.
And I've written an Ansible module to dynamically generate secrets and store them in an encrypted
way.
This is just an example.
You have your Ansible secret storage which contains your host secrets which are individually
for each host and also your secrets like databases, super user passwords, and the host password,
of course.
And your static secrets like API tokens, DNS access, and things that you don't want to
generate on demand because the secrets will stay the same in your environment.
Second step is to configure your hypervisor or your VM of some sort.
For that, I'm using Proxmox, but you can use AWS or Hatsnow or whatever.
It's pretty much just an implementation detail.
You need to spawn and configure the hardware which usually is done with a simple API request
to your favorite provider or your own services.
Then you already get the first issue, network configuration.
You need to probably set things from your IP addresses to your MAC addresses and all
of that.
That is usually handled by your provider or by your hypervisor and Netbox just or your
configuration management just tells you which IPs do I want, which IPs are free, which MAC
addresses are free, and you just generate that data on demand.
And you already set your default password.
Usually you either get a default password from your API request or you send one with
it or you send your SSH public key.
After that, your device is spawned, and the first thing is we SSH in using our root user
that we just created.
Then we run things, the usual set up.
Everyone does it probably, like you're installing your packages and your default packages, you're
upgrading, installing firewalls, making firewall rules like allowing SSH and allowing your
services that you want to install, for example, a web server which would need port 80 and
SSL ports, and then you ensure that your root password is matching the one that you have
in your database, in your password database, and then you configure stuff like sudo.
That's the easiest and basic things.
You could stop at this point and then you just have a machine that is running somewhere
on the internet or on your own network, and you have a root password, your SSH keys are
deployed, and you can start from there doing your stuff via SSH.
But then we have things like DNS.
So we gather, for example, our SSH fingerprints and put them in DNS, which is a neat security
feature for those who haven't heard of it.
You can put your SSH fingerprint onto your DNS, and SSH will contact your DNS server
and validate that the host matches those keys, which will just ensure that your DNS isn't
doing something weird or you're connecting to the wrong host or something.
Then of course we set our A and quad A records, our CNAMES if we need them, and everything
else.
For example, we saw that, for example, you can also set the PROMETHERS DNS records, which
are just SOV records that allow your monitoring to learn about new hosts.
We will see that in action soon.
Yeah.
Monitoring records.
And then we get into the nitty-gritty of the whole thing, configuring and installing services.
In my example, I just want to install a Netbox instance, so we need to ensure that we have
a Postgres database so that your service can store all its data.
After the database is created, we have to install all our dependencies, and we just
do that via Ansible.
And after that, we configure our service.
And then we run some other small stuff, and then you bring it up, the whole thing is running,
you connect via your admin software of trace, and then you create your super user, and in
the end, your password database will just have admin and your randomly generated pseudo
user password or admin password for the service you need, and you never have to touch that
host with SSH yourself, everything is just in your password database, which you can integrate
into your password manager, and it's done.
You don't have to connect to the host manually.
Afterwards, we also want monitoring.
Monitoring is a very important but often under looked thing in IT security and also in IT
in general.
I don't only need the service to be up and running, I also need to know if it's well,
and if it's having issues or the drives are getting full and stuff like that.
So you usually have something using Prometheus and Grafana or some variation of that.
And as we've seen before, we just add the SRV record to our Prometheus zone, and Prometheus
will automatically discover our new host, and we don't have to do anything on that part
either.
Then even more overlooked part is what do you want to do when you get rid of the whole
thing?
There are DNS records and monitoring things you have to remove, and also your secret storage
has to be cleaned up.
So we will unregister the whole thing from monitoring, delete the DNS entries, and then
remove the VM or container from our hypervisor and everything is removed, and it's done.
I sped through this part quite a bit because now we're going to see it happen in real time.
Hopefully.
So we start with our Netbox instance.
That is the way I have it configured.
It's really loud over there.
So I've created a new virtual machine with the name CCCCAM23.Jupiter.CafinatedLabs.net.
Jupiter is my hypervisor, and this is the host name of the virtual machine we want to
create.
As you can see here, we have our network interface already.
Very nice!
Do we have broken IPv6?
Maybe.
Yeah.
No IPv6 over here.
Very nice.
Let's check.
Yeah.
Well, no IPv6 for me.
It's a modern standard.
It's way too new to be supported everywhere.
I shouldn't have expected it to work.
I'm sorry.
Okay.
That is really distracting over there.
Well.
Do we have IPv6 now?
Hmm.
Well, that's embarrassing.
I'm on the Wi-Fi right now, so I'm trying to get a connection.
Let me just test this.
Yeah, I've unplugged it.
And I'm on the Wi-Fi, but Wi-Fi doesn't have IPv6 either.
Thanks.
I'll try.
Let me just try.
Let's reconnect.
Hmm.
It's working.
It's working again.
Trying to get off and on again has worked.
So let's see.
Well.
Internet is real fast today.
Okay.
We have our thing.
It has worked.
That's good.
And the SSH connection is also up.
Okay.
We can see our ad on a device with our public IP address and our pre-configured MAC address.
We can also render and be generated on demand, but I've pre-configured it so that it doesn't
change quite as often and the op tables don't get full as much when we...
Yeah, I can...
Yes.
Give me a second.
Is there light theme?
Okay.
Can you see it now?
Okay.
Perfect.
Let's go back to the interfaces tab.
And let me just close this.
We have our MAC address and our ad on a device and our IP address.
Then we can go over here.
We can see this is our host name.
The device is currently offline because we want to pre-configure it.
Then we have selected our Debian 12 as a base image.
Down here are some of the Proxmox features we want to configure.
It should start at the hypervisor boot.
It doesn't have a Proxmox VM ID yet, but we will see that this field will get populated
once we have run our script.
And we have configured on which clusters running, on which node, and all of that.
And we've also configured our virtual CPUs, our 2 gigabytes of RAM, and the 8 gigabytes
of storage.
There's nothing else configured yet.
So everything you see here is the config.
Now we will go over to the SSH session.
And run the whole thing.
I'm currently connected to the host I've described before, which is -- has the UBIK key plugged
into it, so I could even unplug this one if I wanted to.
So let's go to our deployment.
This will happen automatically with a webhook.
But for demonstration, I will show it to you this way.
And also the script will pause sometimes so I can explain what's happened.
We have deploy and then our environment.
And then we can just run deploy.
Okay.
Let's deploy this machine.
First thing is we are decrypting our secrets.
Then we are installing our Python dependencies for the Ansible controller.
Then our Ansible script will talk to Netbox and figure out which machine we want to deploy
and which -- with which settings.
Then we are cloning our secret repository.
And generating our secrets.
As we can see, we have generated our root password, for example.
Our root password has been generated.
And where is it?
Yeah.
Our TC key for DNS challenges has been and DNS access has been generated.
Yeah.
Up here we can see that our secrets_host_root_password has been generated.
Yeah.
Let's continue the deployment.
The next thing is we will ask Netbox about the gateways that are in this network which
DNS servers use and records and other DNS configuration.
Then we can also look at some tasks are skipped.
That is a quite normal thing in Ansible.
We are now trying to configure a physical device.
But as Netbox has told us, it's a virtual machine so we don't need to run physical device configurations.
Okay.
Now we would create an update book.
Because we have selected that it's offline, now it will just tell me it's offline and
it won't deploy anything.
If we change that in our Netbox, change it over to online, let me just show you.
We change that over here to provisioning.
Change or let's just say it's active and save it, our Ansible will now get the new data,
new information, and we'll try to deploy the machine.
Let's switch back to the terminal.
And let's run our script again.
Okay.
Same thing as before.
We are collecting our initial secrets for the Netbox access.
We are installing local dependencies.
We are now talking to our Netbox, figuring out what to do.
loading our secret repository.
And as you can see now, we don't generate it.
The secret has been just loaded.
It was generated before, as we saw in the change log,
and now it's just loaded because it was already
stored in our secret database.
OK, now more stuff is happening.
We are now creating our container right here.
As you know, this part does take quite some while
to spawn a new container.
After that is finished, we will see that our--
yeah, exactly.
Now we update the netbox, which I can show you.
OK, netbox will now show me the new Proxmox ID down here.
It has been created as container 104.
And now our deployment continues.
And we are now upgrading the default packages
and installing the basic packages
we need to have our Debian configured.
[VIDEO PLAYBACK]
[END PLAYBACK]
Well, Debian is a bit slow today.
As I said before, it's real hardware and real data centers.
So probably it's the usual effect of when you show it to people,
it doesn't work.
But I can tell you what's going to happen next
and what this means.
Because we first try to connect with our user
to check if the device has been provisioned before,
which it hasn't.
So we switch over to our root user
because the access was denied with our unprivileged user.
Yeah, we have updated our Debian.
And then we want to install our default packages, which
are, in my case, vim, curl, sudo, and tree, I think.
Just the usual stuff you want to have on every machine.
Yeah.
[VIDEO PLAYBACK]
a little bit.
OK, now we are configuring our sudo.
Now we are configuring our firewall.
In my case, I just use uov.
In my opinion, it's the easiest way
to have a simple firewall on your device.
It will install the package and then automatically configure
our trusted network.
So everything within my IP range is considered trusted.
And everything outside it is considered untrusted.
That is important for monitoring access,
as I don't want my monitoring data
to be available from the internet.
So it's only available from the internal network.
And now we are rebooting our host
because we have configured our firewall,
we have configured our root user,
and we have configured our default packages.
After the reboot, we will continue.
And as you can see, all of this is happening automatically.
The device is rebooting, the device is being configured.
And now we have reached the point
where we can start our custom deployments.
So first of all, we are collecting our fingerprints
for the DNS, which we can see over here.
We have added our quad A record, our three SSH fingerprint
records, and our monitoring record,
which just tells the monitoring to look at this host now.
After that, we will configure our users
with their SSH keys, default passwords,
and their permissions.
You can configure many users.
You could also configure PAM LDAP, for example,
so you don't have to configure the users yourself.
And your Linux is just talking to your LDAP for validation.
Then we want to apply SSR hardening to our instance,
just the default SSH options you want to enable and disable.
And we are also installing Feltiband,
which is just a neat security measure
and helps on getting rid of all the people trying
to connect to your host with bad passwords.
Especially if your host is connected to the internet,
steps like these are highly recommended.
OK, now we have configured our SSH.
Now we don't want to install Proxmox, so we skip that.
We also skip disk encryption.
And those are all the services you could deploy on a machine.
And only the services we have selected in the net box
are actually deployed.
Everything else is skipped.
OK, now we are installing NodeExporter
and creating our SSL certificates
so that we can connect to the device with encrypted
and valid certificates.
I can now show you, as we have configured our SSH--
I'm sorry, our monitoring records for Prometheus.
I can now show you the Prometheus side of things.
Let's zoom in a bit and switch to light mode.
As you can see, our host has appeared.
It is currently considered done because we
haven't finished our deployment.
And once we have finished deploying,
which we can see in the terminal right here, as you can see,
our monitoring stuff has been installed and configured.
And now we are installing a final thing I quite like.
I have a custom message of the day package, which will just
display the last successful Ansible run, its git commit,
and stuff like that.
So you know that when you SSH into your machine,
when the last deployment has happened,
because Ansible doesn't deploy continuously,
it gets run by your webhook or by yourself.
And after that, it just stays like this.
OK, we have finished our deployment.
If we check back at Prometheus, we
can see that our host is now considered up,
because everything has been configured.
And we can SSH into our host as well.
Let's just SSH into this host.
Well, it's not a good day today.
Let's just remove that real quick.
Yes.
And now we can see our custom message of the day, which
will just tell us--
oh, I'm sorry.
OK, now we can see the custom message of the day, which
will just show our--
very good.
Very good, Johanna, very good.
We can see our custom message of the day,
which will show you the OS version, Ansible commits,
and everything else.
Well, that happens if you have two IT administrators, I guess.
OK, now let's just decommission the whole thing.
And I'll try to let the decommissioning happen
in the background.
And then we can switch over to our Q&A.
Yeah, the slides are on my website.
And the code can be found on--
you can find the code on caffeinatedlabs.org.
And in the next coming days, I'll
try to open source as much of this as possible.
So you can take a look for yourself.
Yeah, but I'm now open for Q&A questions.
Yeah.
We have about 12 minutes left for Q&A.
I will go around.
[VIDEO PLAYBACK]
- OK, so how much of this is standard Ansible, or at least
readily available Ansible scripts?
How much is custom Ansible scripts,
and how much is custom Glue code?
- Depends on what you consider Glue code.
I have re-implemented some modules.
The biggest of those are the secret management,
which will hopefully be open sourced soon.
And the Proxmox API, I've written a small wrapper
so you can directly interact, because community
general Proxmox Ansible modules try to abstract very much.
So you can't configure things directly in some cases.
So I've just written a small wrapper around the Python
library Proxmoxer, which just passes through the arguments
to Proxmoxer directly without any logic in between.
- Just as a reminder, we can also
have a Signal Angel, so if you watch on the internet,
you can also ask a question with the hashtag #CCCCam23NOTX.
Now the next question.
- How do you handle backups and restoring them
if something gets broken or something doesn't work anymore?
- OK, I ran a backup before every Ansible deployment
automatically, which means that I can simply restore the container
to the state it was before the deployment.
Those backups are capped for five times or something like that.
So you have your standard backup rotation there.
And Ansible just creates a backup in case something goes wrong.
You can just restore the previous state
and continue from there.
- Thank you for the presentation.
You said you are going to decommission the system now
in the background.
I'm quite curious.
Is it like extra roles you wrote for unregistering the VM
on Proxmox and the DNS entries?
- I have written most playbooks which
deal with deep with provisioning and deep provisioning
to do it themselves.
You have a variable which will tell
it's the netbox status variable, which will just tell you,
is it active, is it deep provisioning, is it offline?
And the playbooks decide on those variables what to do.
- So you have a complete list of all your VMs.
And if one is not in the list but on Proxmox,
it gets deleted.
- No.
I'll just show you.
When we switch over to our netbox,
we just change the status to decommissioning,
which is just this.
And now we save it.
And afterwards, we can switch back to our deployment.
And now the system will detect the change from netbox
and delete the container.
We will see also that it will create a backup
before the deletion, just as it would before any other Ansible
run.
I'll let you continue that.
Is there other questions?
Yeah.
- Sorry.
Just do you have some experience on secrets management
in a more enterprisey environment?
- What do you mean by more enterprisey?
- I mean, with one Yubikey, you just have one.
But you have several people needing to access the manager--
a part of HashiCodeWalt.
- Yeah.
You can scale the way I've done it to a small team.
But the encryption is a serious factor
when deploying hundreds of hosts,
because every operation with the Yubikeys just takes some time.
There are ways around that to use hashing for secret generation,
for example.
I am planning to implement something like that
in the future.
But I haven't implemented it yet.
- Do we have any questions left?
- Hello.
With regards to the scope of the secrets,
is it just for stuff that's stored encrypted outside
the container and inside?
Or is it stored encrypted outside, then decrypted inside?
What's the scope for that?
- Yeah.
It is stored encrypted outside the container
and unencrypted inside, because usually the config files don't
allow for encryption.
If you have a config file that allows for encryption
or hashing, you should do that.
And I do that.
But sometimes, if it's not possible to do it,
it will be stored unencrypted inside the container
and encrypted outside.
- How much time does it take to set up
a whole new service which contains about three or four
new processes which need to be started, a database, and so on?
How much time do you need to set up all these Ansible scripts
and all this stuff if you know how to deploy everything?
- So you mean like if I, for example, have a service that I
don't have a script for, how much time
it takes to write an Ansible playbook to run it?
- Yes, of course.
If you know how to deploy the service, it's fine.
- OK, so yeah.
As most services, they're usually the same structure.
You have your system D units, and you
have your code which has to be installed some way.
That is usually a copy paste job of just taking
another similar process and just changing a few things, right?
So a basic role can be done in about an hour
and a more advanced one in like 10 hours to a few weeks,
depending on the size of the project.
- Correct.
OK, as we can see before, we have now changed our state.
The proxmas ID has been given back to the pool.
The device is offline, and the container has been removed.
Final step here is to remove our DNS records, which
will just remove the old records.
And then the rest won't run because it knows
that the device is now offline.
And one final thing is to remove the entries in your .ssh known
hosts file.
Yeah, those will be removed.
And now you can redeploy your host under the same IP
address without SSH.
Yeah, shitting its pants, to be honest.
OK, are there any more questions?
- You use containers.
This works also with normal ProcFox VMs?
- Yes.
The configuration is a bit different because in VMs,
you usually have to configure the IP addresses inside the VM
and not inside ProcFox.
But that's just a change to your deployment playbooks.
And you can have that work as well.
You could even go as far as to do that with a host.
You just screw it into your rack.
So you just install your base system.
And then from there, you can deploy just
like any other machine.
Any further questions?
- OK, then thank you for your talk.
It was interesting for all of us.
And a huge applause for Jadyn.
[APPLAUSE]
[MUSIC PLAYING]
[ music playing ]