[Music]
 Hello and welcome to another episode of Ask the Teams here at Chaos Communication Camp 2023.
 With me on stage today are two members of the NOC, the Network Operating Center.
 We have Will H. and Nico Duck.
 And we'll talk about what the NOC actually is, what they do, etc. etc.
 To give a little bit of an impression of how much work goes into getting Internet to the camp and to the people here into the tents.
 So let's start with the team.
 What Network Operating Center or Internet Manufaktur?
 Which name is it?
 Kind of both of them. Hello everyone.
 Most of the times are called the NOC, but on the radio you can't really distinguish between NOC, VOC, LOC, POC and everyone else.
 So we prefer to be called Internet Manufaktur at least on the radio.
 And it sounds a bit better and sounds more like what we do here.
 And how many people are we? How big is the team?
 The team is about 30 people actually.
 We mostly work in English because we have a lot of people from different countries, from the Netherlands, from the UK, as well as just Germany.
 So we're a slightly different team, I think, in Chaos Events where we're working really nearly always in English.
 All right. And the size of the team, of course, varies during the event.
 So a few people during build-up and then gradually increasing until...
 Yes, yes. All of the work is really in advance of the event and in build-up and tear down and not less during the event itself.
 Unless something breaks.
 Exactly. And things do break, yes.
 Well, it's the outdoors.
 Yes, yes, exactly.
 So at a Chaos Event, at a Hacker Event, Internet is, I think...
 You could argue, is it more important than water and waste water, but it's one of the higher priority items.
 So how do we get the Internet to the camp? How does that actually work?
 Because we're in the middle of nowhere, basically.
 Yes, so actually this year we actually have two separate uplinks, which is quite nice.
 In previous camps, because it's difficult, we've only had one uplink.
 And in the meantime, fiber has been built a lot more for ordinary users around the countryside.
 So we've been able to get a connection here in the Ziegler Park and also one a little bit further away,
 where we had to run a fiber through some fields manually to reach.
 So you're actually taking the fiber cables and running them through fields, through trenches?
 Yes, exactly. There's a lot of cables being installed around the site, which is actually more challenging.
 Because there's a lot of things here at the Ziegler Park, a lot of train tracks and tents and so on, that are maybe in the way.
 This is also quite important in the planning stage. So a lot of work goes into planning before the event.
 Where do we have to lay fiber? Because we need connectivity there.
 And which routes can we use to get the connectivity there? Are there active train tracks?
 So we have to duck underneath or are there active road crossings and train tracks where we have to build a bridge over it?
 It's similar with the power team. So we cooperate together and have a look at each other's plans
 and where we can do stuff a bit simpler and only do one bridge or only one trench under the train tracks.
 But yeah, this is a lot of work before the event, because we cannot really dig here.
 There's a lot of old brickwork in the ground. So doing longer trenches is kind of impossible and really annoying.
 So when does planning start for all of this?
 I think the first meeting usually is around GPM.
 That would be March, April, May in that area where we get together, meet who is there.
 So it's not like we don't have to be there, but it's always kind of nice to meet friends and have a roughly idea on what we are doing.
 And then most of the stuff is planned on the internet from home, doing kind of a lot of CAD works with a map, planning all the automation configuration.
 Planning for this event is a bit easier because we already know the Ziegler Park having been here twice before already.
 So we know what we're facing with train tracks and buildings and all this stuff. So that's a little bit easier.
 And of course, things like there are some items which are a longer lead time, like arranging the actual uplink is really that work starts perhaps a year in advance, just to be sure.
 So dear project office, please find a new location. Nock is looking for a new challenge.
 Any rodent incidents so far? Because I think the last one or the one before we had a few outages.
 Yes, yes, no rodent incidents. We have had a train run over a fiber and we've had some other damage actually just accidental.
 Well, I guess a rodent eating a fiber is also accidental, but yes, some other damage.
 And we have some more resilience this time as well because of improvements.
 Yes, and also around the site as well. So we don't have to kind of rush to resolve problems immediately.
 I mean, we already touched on it a little bit. So we have two uplinks to the campsite. What's the bandwidth?
 We have a 20 gig plus a 10 gig. So we have 30 gigs available.
 A little bit smaller than at Congress.
 Yes, yes, it is. Yes. But actually the usage is lower, right? It's a different kind of event in some ways.
 And so, you know, our usage is I think like five gig or so around there. So that's perfectly fine.
 So the usual credo use more bandwidth still applies.
 Well, people can, yes, but actually most people should enjoy other things. There's plenty to see and do here.
 Absolutely, absolutely. So once we have internet uplink into some sort of building in the data center, which we'll touch on in a minute.
 How do you distribute this around the campsite? You already touched on the topic of fiber. So you're laying fiber.
 Yes. So it's a lot easier for us because there's no real distance limitation to just run a lot of fiber.
 So every Darden Klo really has a fiber uplink of some sort. So, yes, there's a lot of cables to run all around the site.
 And that's what Nico was talking about with the planning and the work required there because we have to figure out how to run these cables.
 And also a good thing with fiber is this electrical isolating. We have different generators on the sites powering different grids,
 which are not necessarily on the same level. And we cannot connect these with copper cables.
 This is also why we require unshielded ethernet cables this year to be connected into the Darden Klo because we had incidents in 2019
 where we had a coupling between two generators on a shielded ethernet cable. Luckily, no hacker was harmed there.
 But we still wanted to make sure that we can continue the tradition of not harming any hackers with power on network in this year's event.
 So the generators you mentioned, these are the braider nodes, the power generators to produce the electricity that all the hackers and tens and infrastructure needs.
 And so we have fiber all around the campsite. You're digging below. Usually you dig below the train lines. Some accidents happen.
 So you have fiber at several points at the campsite. And I've seen some people are not familiar with the Darden Klo.
 So the fiber ends in the Darden Klo at the switch probably. So could you explain a little bit what is the Darden Klo?
 Where does this come from? How does this work for people who have never heard of it?
 So actually what you need is you need to rent a large number of enclosures, right, that you can put your stuff in that are waterproof and easy to secure.
 And well, it turns out these plastic toilets, you can go to an event rental company and they can figure this out.
 So they're not equipped as actually with toilets. They don't have chemicals and there's not actually any toilet paper in there.
 They come right from the factory, right?
 Yeah, actually these did. Yes. And so, yes, but they're easy to rent and lock up.
 And so it's strange because people arrive and they think, oh, well, there's lots of toilets everywhere. Oh, camp next to the toilet.
 It will be convenient. It's like, yes, it is convenient, but for Wi-Fi and Internet.
 So what's inside one of those plastic toilets?
 At least a switch.
 Some fiber?
 Yeah, some fiber connections, some power like for the local things and often things from from other teams as well.
 We even have a picture left here.
 Yeah.
 So, yeah, you see some fiber spools on the ground.
 So usually our fiber comes on spools and we don't get new fiber for every event on the perfect length.
 So there will be usually more fiber left and these spools are in a dart and clove.
 And then there's a switch on the back. There's a POE injector for the Wi-Fi access point, which is also attached to the pole on the dart and clove on the outside.
 And we see a little black box on the right side with the yellow sticker on it, which is a CWDM splitter.
 We used this technique the first time last year at MCH in the Netherlands and reused that here too.
 And we got a little bit more.
 What's the benefit of using that? What's the functionality of this for someone who's not deep in technical?
 This is difficult to explain without a diagram, actually.
 However, we did a good presentation on this at the MCH event.
 But yes, it really is. It's so we get some flexibility in how we run the fiber and the number of cores of the fiber we need.
 Because, you know, we have like around 65 dart and clove.
 So there's a lot of stuff we want to deploy this efficiently and not have quite so many fibers because that gets very difficult for us to manage.
 But the basic stuff is like with light. There are multiple colors of light and you can do the same thing on fiber.
 You can have different colored laser optics on the fiber and break in and out just a single color.
 So we have four decays in a row and each decay has a different color. So in the end, there's only one fiber cable with two cores in it, powering all four decays.
 And with this technique, we can also do redundancy a bit better because we can break it out from two sides.
 So most of the decays here this time have two times 10 gig uplink to two different sites on the venue here.
 And if one fiber breaks, there's a high chance that the dart and clove is still active on the remaining leg.
 There are some dart and clove where this wasn't feasible in the planning stage.
 So they are single homes to one of our core sites, but it's still the best we could do.
 And those are the ones where the fiber has been damaged, of course.
 So just again for those viewers who have never been to such an event and are still confused.
 The plastic toilet has a switch in it and people then come up with a regular cable and say, I want 10 gig of one gig connection to my tent.
 Do they just open the toilet or what's the process?
 So actually you unroll a cable and leave some slack for us to connect it.
 Well, actually our great NOC helped us volunteers. Our angels are going around with a big round.
 So they walk around the whole site and connect the cables over time.
 So you leave your cable and they will come around, unlock, plug it in and move on to the next one.
 And then the same in reverse, I think, at the end of the event.
 And you mentioned earlier that this year the big change or one of the changes is that you're only allowing unshielded cables for the potential differential.
 Yeah. Well, what some people also do is they use a long shielded cable where the shield is only connected on one end
 and then have like a meter unshielded pigtail in the dart and claw to get rid of that problem,
 but still have the benefit of a shielded cable on the long run, which is beneficial for 10 gig, which we do offer in these dart and claws here.
 So you can get 10 gig to the tent, which I can't get at home, which is nice.
 Yes, actually nearly every port in those dart and claws is 10 gig capable now, 10 gig copper.
 The support in people's laptops or so on is not so great, but it is available. Yes.
 And also it's the same band with up and down. So we don't do like one gig down and 100 meg up or 10 gig down and 10 meg up.
 So no, you're doing the proper Internet. Yeah.
 And if I connect my laptop to the cable, is that directly connected to the Internet or do I still have some firewall or some netting in between or?
 Only on the Wi-Fi. On the Wi-Fi we offer firewall connectivity because on the Internet there's a lot of port scanning going on
 and this is not good to have all these like probing packets on the Wi-Fi and also for the battery life of your smartphone.
 If it gets constantly hammered with port scans, it will quite rapidly decrease.
 So the default on the Wi-Fi is to only allow connection from the camp site, but not from the Internet.
 But you can choose if you would like to accept that on your laptop or even on your phone.
 But all the cabled connectivity is the bare Internet without any filtering, anything.
 So I should harden my laptop before I connect it to that cable.
 Or you should always harden your laptop before you connect to any cable.
 But especially here, it's different from at home. It's the raw stuff.
 You already mentioned Wi-Fi. I mean everything is wireless these days, of course.
 So you have an access point on every Datenclaw. How many?
 So we have actually 146 live right now. What actually happens is you'll notice that's more than the number of Datenclaws we have deployed.
 And what we actually do is we start every Datenclaw has an AP. That's the thing on the pole that sticks out the top that people will see.
 And then we deploy like fill access points, like maybe in villages or in...
 At heaven.
 Yeah, exactly. In the bigger tents and so on.
 And one of the things is we actually keep on building this during the event.
 It's like people report poor coverage in an area and then we go, oh, hey, let's fix that.
 And then we keep on doing this until maybe halfway through the event. And then it becomes pointless.
 And you mentioned the access points are on top of the Datenclaw and then there is also a large lead tube, a colorful lead tube on top of that.
 What is that about? I mean, a lot of people ask me, what is this thing doing on top of the Datenclaw?
 Apart from looking very nice, has it a functionality?
 Yes. Yeah. I think they've been with us since 2013.
 They are basically receiving DMX or ArtNet data via Ethernet, showing the animation.
 So if the animation stops working, it will switch to a dim light.
 So you know that the Internet or the Ethernet is down in this Datenclaw.
 If the power is down, obviously it will be dark.
 But if the power is there, but only the network is down and has an issue, then it will be a dim light.
 We are always kind of playing with the animation.
 So during the day we currently have a flame on. In the night it was a more dim or red or green flame.
 In between we had some green with pink stars in it, kind of sparkling stars, which is a less visible or not less noticeable animation.
 But it's kind of the color scheme of the camp, which is quite nice.
 So the animation can change, but as long as it's blinking, it's fine.
 So do you have one of your angels on top of the hill to watch out if all the Datenclaws are online, if all the lights are blinking?
 Or do you have a more sophisticated solution to monitor these things?
 Yeah, we have some monitoring.
 And one of the big changes we made this year was some implementing some things from other camps.
 A lot of people would have looked at the map, the new map this year.
 And actually we have all the Datenclaw on the map and the icon changes color when the devices are deployed.
 The Datenclaw is live, it goes green. If it then fails for whatever reason it goes red, which is pretty nice actually.
 Because often there's a local power outage or something like that. So we can report it and people can also know.
 Or a train running over your fiber.
 Exactly, yeah.
 Right. So that's how the Internet gets to the tents and to the Wi-Fi devices.
 What about the services that we offer? We offer like Wikis and stuff like that.
 Is that all running through that 20 and 10 gig Internet connection?
 We're very keen on the stuff that is necessary for the operation of the network. This actually processes sensitive personal data.
 Things like your MAC address, your IP address. And we like to collect as little data as possible and to not leave the site.
 So we have the servers that maybe say your DNS resolver and this kind of thing.
 And they're actually all wiped, securely wiped before they leave the site.
 So it's important for us that those services which are essential on the camp are hosted locally and we have a data center container for this.
 And then also we're hosting other servers for some other teams which are good to have on site.
 For instance, like the DAFOC for video streaming and a few other teams as well.
 And do you also provide some sort of co-location? If people bring their own servers?
 Not here. It's too expensive because of the electricity source and the fact it's rather warm here.
 And we can really only provide this for like essential uses. And even then it's quite a bit of effort and cost to provide this.
 I think I've seen it's like one container with the blinking light.
 Yeah. So we started in 2019. We were like, why hide this away? Let's actually make it nice for people to look at.
 So now you can go and look in the window and peer in. And I can see from the fingerprints on the window quite a lot of people are looking in there.
 Maybe we should get a window cleaning agent.
 And of course it's locked up because otherwise people would improve the situation.
 Yes, and we need to keep the door shut because it needs to have air conditioning and so on.
 And also the other teams depend on their services being up and not being well fitted around with event phone is hosting in there.
 The video streaming and recording is running in there. The GSM team has their rack in the colo.
 And that's basically it. It's us and the three others. So we only provide that service to a very close user group.
 But since everyone can have 10 gig at their tent, it also lowers the need of a Yolo colo as we had on the congress earlier.
 We even managed to get a second data center site this time so we can have a bit of redundancy.
 Most of the services are single home in the container.
 But some of the stuff like the Wi-Fi controller, the uplink, the routing is also mirrored to a second site to minimize the outage if something breaks.
 The biggest enemy of network is always those big yellow heavy machines that either dig through the cables or unlock the container over.
 So having some redundancy is always nice, right?
 At least the big yellow machines having on site here are really friendly to our network.
 So we didn't have an incident with these. So they behave so far.
 Mostly it's actually people where we have power outages when it rains a lot and someone's left a connector in a puddle somewhere and then we cannot provide network without power.
 But we're working pretty closely with the power team here who are great. So we're making sure that it's all working.
 Well, I've heard of quite a few demands of delivering power over Wi-Fi. So if you could work on that, that would be awesome.
 Right. And of course, we have seen it at other events in the infrastructure review.
 You also have a dashboard where you collect all the data and the traffic data and numbers galore.
 Yes, everyone loves these numbers and so on. I mean, we're at over 6000 Wi-Fi clients right now.
 And 89 switches deployed. So that's a good amount of hardware and a good number of users.
 There's also good mobile coverage for people who do not want to use the network here now with Telco putting a mast here.
 But you're not controlling their mast, right? No, no. But it provides some alternatives.
 But most people need the network.
 Well, with our network, it's not much fun running around here with access to the map and everything.
 But increasingly, you know, all the other teams, they've discovered they can use computers to manage things and they need internet as well.
 I mean, even say the power team have a QR code you can scan to report issues.
 And there's lots of this ever more dependent on the network for so many other services that are provided for the camp.
 For some reason, the video operating system, video operating center is still planning all these shows on the container walls.
 We will have to convert them. Sometimes this way actually works.
 Most of our knock planning that's on site when we have small jobs and so on, we have a big Kanban board with Post-it notes.
 And it's very satisfying. You take a ticket, you move it from needs doing to in progress and you come back and you put it on there.
 It's done now. And now we have a whole wall of stickers that are done.
 Talking about done, we already touched on the topic of build up tear down.
 So that one thing is laying the cables, digging up trenches or whatever to lay the cables.
 What about the software infrastructure? Is that is that also someone sitting and installing operating systems and stuff?
 Kind of. Yeah. Most of the stuff is reused between the events.
 So there's a big share of people in the Internet manufacturer also doing that on like the MCH last year in the Netherlands and likely again at the next Dutch camp.
 In 2025 and also on EMF camp, there's a big share even Will has his nice T-shirt on.
 So yeah, we try to reuse as much as possible of the stuff.
 So there's quite a lot of automation going on with event infra, which is located in the Netherlands and run by volunteers also from the NOC.
 So a lot of staging is going on there. Then more staging is going on here in an automated process.
 And we now even have it automated so far since at least 2019.
 And I think it started before that also port changes to have a port in a different VLANs are handled automatically.
 So every switch is getting a new configuration every 15 minutes.
 So when the POC or the event phone people come around and ask for a new port for a deck station or if another OC needs something,
 they have a guest or they have access to our Netbox where all the configuration is stored, can do the change and 50 minutes later it's deployed.
 Self-service really. That's what we need because it's too large to have another human in the loop.
 So it's different to like the setup in the most of corporations where you need to fill in a SAP form and then run through 20 iterations and three months later you get the port opening.
 No, there is no three months later.
 We don't have a change advisory board every 14 days.
 But also most of the teams are in a fairly similar constellation every event.
 So we know each other fairly well. They know how we roll. We know how they roll. So that's good for both sides.
 So you're using the knowledge between the different events or friendly events or whatever you want to call them.
 And also I think the hardware and the network ranges, et cetera, that's also shared between the events because I noticed sometimes when I'm here, I'm still located in Amsterdam or near Amsterdam where the MCH was.
 Geolocation breaks with these kind of events.
 Yeah, mostly it's actually due to the access point because the access points move around and they go to a lot of events actually.
 And so you see that and sometimes it's due to the IP addressing as well.
 But yes, yes, we do. We don't have a set of hardware and not use it. It's used by lots of other not for profit events around Europe.
 Yeah. Also Event Infra, which is lending out this equipment, is a non-profit in the Netherlands working with CCCV and all the other events.
 So the events can borrow the hardware from Event Infra in return to a, I would say, competitive price.
 But it's really good for the event because it's from the community for the community.
 So some events, depending on what they do, donate more money. Other donate hardware, which is also fine.
 So there's an increasing big pool of switches, of Wi-Fi access points.
 So I think most, if not even all of the stuff we use this time is coming from Event Infra.
 So if some non-profit event needs some sort of network infrastructure, reach out and we will see and you will see what you can do.
 Yeah, there's a lot of teams, a lot of organizations using this.
 And you have to build the network yourself or find some people to do that. But it's not a full service operation.
 Hardware. If a for-profit wants to get rid of some hardware, which is still good to use,
 so some data centers tend to kick out their old boxes after five years, which is not that long in our terms.
 You can reach out to Event Infra to donate the hardware.
 That's how we were able to offer 10 gig copper to all the areas because an organization was clearing out the data center.
 And we said, thank you very much and took 60 odd switches.
 And otherwise they would probably have to pay to get them disposed properly.
 Yeah.
 OK, I think we touched on most of the points in an event like EMF Camp or MCH or the next Dutch Camp or CCC Camp.
 Is your team filled up? Are you still looking for volunteers during build-up or tear down?
 Or is it too fragile? Do you need so much inside info that it doesn't make sense to offer you help?
 Always during tear down. Every team wants help during tear down.
 And because there's just so much infrastructure that we need to roll up and put away.
 But we do have a quite well organized system for this because we know we have databases and things with all the bits of equipment in.
 So it's not just randomly done, but people can come and help us.
 Please don't go to the Daten Klo and just tear down.
 Exactly.
 And at that point, come to the NOC and ask what tasks you could do.
 Yeah, we have checklists to collect all the items and make sure and then we know and everything is also wiped before it's both in terms of its storage and its physical surface, which can get quite dusty before it's packaged and sent back.
 Since most of our work is before and long before the event, there are always some friendly people who approach us on like day minus one or day zero and want to help us.
 But this time the network was finished building up everything on day minus one.
 So we had to send them away because there was no work to do on these days.
 But on early build-up or then the tear down, there's always more help to do.
 And we always had a lot of angels again this time to help out during the setup.
 So we built teams with, I would say, one experienced member of the NOC and one or two more angels.
 And this worked really well.
 Yes, it has to be.
 You have to have some experience with plugging in fibers and so on.
 And all this work is physical during build-up.
 We're not like sitting at our desks like configuring things because we have a computer for that.
 It's running around the field, it's driving around the field.
 It's a lot of manual work than you would like to think.
 But many of us have desk jobs and so it's great fun to come in the field and do that.
 And after the first tape, the whole body complains.
 That's what ibuprofen is for.
 Many of us are also working in the industries and the colleagues are always asking why are you on vacation and why are you doing basically the same thing.
 But it's not really the same thing.
 And it's with different people.
 So all of us are doing that because we really want to do that.
 And this is a whole other team experience than getting paid to do that.
 You learn a lot in both technical and transferable skills by coming and volunteering for many of the teams here.
 Absolutely. I can totally attest to that.
 I have one last question.
 We didn't touch on this when we talked about the datum clothes.
 I have to circle back a little bit.
 As you said in your explanation, these are plastic toilets.
 They look like plastic toilets.
 The print on them says it's a toilet.
 Do people use them as toilets?
 Unfortunately, what happens is a big pile arrive in a field and maybe we don't get there to log them early enough.
 And someone maybe also really needs to go.
 So they have an accident.
 So unfortunately, every time there's always one that's quarantined off and left for proper disposal.
 Yes, exactly.
 This time the number was fairly low.
 We calculated five spare datum clothes and only needed two of them.
 So that's good.
 And hopefully shows like this will get out the word so maybe we can get the number down.
 Accidents happen.
 The major thing for ordinary users out there to know about most of our infrastructure is that we can't know it's broken sometimes unless you tell us.
 But we will come and fix it if there's a problem.
 Any last words from you on NOC?
 Did we miss something?
 Any important topics you want to get out?
 Messages?
 There's a few things that people, if they want to know more, they can go and look in the entrance of the DC, the data center, which is near the info desk.
 There's quite a lot of information online.
 We're now FAQ as well if you're still having some trouble.
 And the NOC help desk will also help you with the service.
 Is there unshielded cable at the NOC help desk?
 I got that question quite often.
 I think the Spatii is selling Ethernet cables.
 Ah, OK.
 They have some cable, maybe even some cable left.
 I'm not sure.
 But they had some.
 We do ask people to bring their own cables because we can't provide for 6,000 users.
 But who reads documentation before the event, right?
 Yeah, well, we do write it.
 I'm not blaming you.
 I'm just asking.
 Anything else we didn't touch on?
 Anything else you want to touch on?
 No, I think that's most of it.
 We could go on for hours.
 I'm not going into technical details or tech porn.
 A lot of the stuff when we want to explain it is like you need a diagram and some actual slides and things like that.
 And that's more in-depth.
 And to be honest, we've covered that a lot in previous review presentations and things like that, which are all on media.ccc.de.
 And you're doing an infrastructure review as well for this event, right?
 Oh, we better write that then.
 We should start preparing one.
 Some funny photos.
 This one is going to be quite short, like 3 to 5 minutes.
 Probably just some funny photos.
 Most of the stuff is still valid from MCH's review, so we could just share the link.
 Yes, adjust the data.
 All right.
 Well, thank you very much, Will H.
 Nico Tuck, thanks for joining us in the studio.
 Thanks for having us.
 This was another episode of Ask the Teams, this time with the Network Operating Center.
 Have a great day and hopefully see you really soon in another episode.
 Cheers and bye bye.
 [crickets chirping]
 [crickets chirping]