[MUSIC]
Okay, I think we are ready and on our way to dive into the very exciting world of
industrial coffee makers and without further ado I give the stage to you guys.
Well then, welcome to our talk "Unexpected Coffee,
Adivin to Industrial Coffee Machines". I'm Hethi, I'm doing offensive security at daytime
and at night time I'm doing industrial coffee machine hacking and other fun projects.
And that's Datacop. Hi, I'm Clemens or Datacop or Data for short.
And I'm mainly, I'm an electrical engineer but mainly doing networks and embedded Linux systems and
yeah. And that's the car. But if that's fine with you, we will just go on. So first things first,
what's a coffee machine or what's an espresso machine? We have coffee beans, we ground the
beans and then we add basically just hot water under pressure and then we have good coffee if
we did it right. And then what's a vending machine? And there are different types of vending machines.
There are basically two base types and one is there are those instant beverage machines
where you just get a cup and you press a button and you get something that was
made from water more or less. But typically it's just powder. And there are the other vending
machines like stuff in boxes or vending machines for bottles and so on. We won't go into this detail
but that's for the first part and what we actually have here. And then what's an espresso vending
machine? And that's more or less a hybrid of an espresso machine and a typical powder-based
vending machine. And that means we have the best and the worst of both worlds actually.
We have not really a perfect coffee brewery and then we also have those powder things and not
really milk or anything. But they exist and they're quite huge usually and they allow special
combinations like latte macchiato which I will go in a bit in a detail on how this is made actually
here in a few slides. And then basic stuff in this machine we have coffee and we have toppings.
For the coffee we have a brewer that will be on the next slide also and we will open the machine
later too. And that one is designed for more or less a specific high throughput especially for office
like 300 coffees per day. And also for low maintenance. So these machines typically
don't need daily maintenance like big espresso machines but those are more or less I think you
should clean them once a week but that's just a quick cleaning cycle and then the large cleaning
cycle if I can remember it correctly it's done more or less once per month and typically not
by the company where this machine is standing but by a separate company that's also supplying
the coffee and all the ingredients. Those machines typically have fixed water installations that's
why we have a pump here and an external water tank and luckily this machine in the manual has
a note that says if you don't have fixed water by the way the water input valve has a rating so you
can hook up a pump there with 24 volts and 1.5 amps max which was quite nice for the camp. Or you can
also buy an add-on for it there is also add-on with a big tank and you can buy it from the vendor too.
There are also different add-ons you can like put on the side and so on so it's like buying a car
more or less you can have all kinds of customizations and then yeah we have fixed water,
we have coffee beans and milk. It's usually just supplied as milk powder and those are toppings.
So there are different types there are also vegan types and we this machine currently is
filled completely with vegan toppings and to actually mix the powder with the hot water
we have or inside there are some mixer wheels more or less so you can see here on the left
or not really well but we will see later in the machine on the left there is the coffee brewer
more or less. Please don't break any stuff. In the middle we have more or less the closed mixer so
from the top the toppings will drop in and will be mixed inside of this bowl with water and on
the right there is basically a second one but that's just a mixer wheel so that's the opened mixer
and that mixer makes the mixture creamy and mixes the water with the topping so you don't have
crumbs in it and you can also run the motor at different speeds for example for the latte macchiato
I mentioned earlier latte macchiato typically is or for these machines is milk in the middle it's
coffee and then it's milk again and the machine actually makes this differently it does two
pieces of milk and then the coffee on top and this works because the milk is mixed at different
speeds of the mixer so they have different physical properties so that the coffee will actually
be in a layer between the two milk layers although the milk powder is exactly the same.
So that's quite interesting it's how those vendors come up with quite a few hacks on how to do
premium coffee things and so on.
And this is the whole machine how it was also set up at Lavendville inside of the camp area
in the MetaLab dome and it will be later set up again there so if you want a coffee or
some cocoa feel free to go there and use the machine and but you can see here you
or we have three of those white containers where the powder toppings are inside there and those
have a motor that will basically put in the powder into the mixer bowls.
And about this machine this is the company is called Rea Vendors it's a quite large company
and it seems to be initially located in Italy and the machine itself is called La Rea Grande Ti
and then some we don't know what they what the rest actually means but
there are different types of this machine. Last year at MetaLab we got a mail from somebody who
said that they have at their office this machine and it's broken since a year and they want to get
rid of it and if we would be interested in repairing it or doing some other stuff with it
they didn't want to just throw it away but to maybe give it to people who repurpose it or repair it.
And we both saw the mail and wrote to each other like a coffee machine and in the worst case we
can always build a coffee pot out of it a cocktail pot yeah for the Robo Exotica
and the initial error was that the brewer motor failed initializing and for the brewer motor is
if we go to back to the picture if you have the brewer on the left you see that there is a piston
inside and this piston is moved or the whole brewing chamber is moved for like 15 to 20 centimeters
for the whole mechanism to work.
So just before repairing it it was standing around the MetaLab or hex space for like three
or four months and it annoyed quite some members because we don't have so much space
but in the end we were motivated to take our time and repair it and so we're going to show
you now the repair process how we debugged it and how we managed to repair it.
Okay so the company or the people we got it from already said that they think it is probably a
MOSFET power driver problem for the motor and so we were debugging a bit of the electronics and
that's the board on the back of the machine so there are several boards we will go into the
different boards in the slides too in a bit but there is one main board and that's in the back
and there is all the power electronics there and all the motors are connected to this and there
were two or the first you on the right the bigger MOSFETs we suspected at first so
that was all the information we got from the people that they assumed that they are broken
yeah but they they tried to order them the electronics and it's actually this data sheet
there are some automotive MOSFETs and that's a bit interesting most of the electronics is actually
automotive grade we don't know why but well endurance and yeah coffee coffee is critical
infrastructure so that was our first try and they tried to order them but they are not complying
with the arrow hs directive of the eu so they are not lead free so you can't get them at mauser or
any other electronics company anymore but aliexpress for the win yeah that was really like
five seconds of search and we found a bunch of them it turned out those MOSFETs weren't the
problem so we swapped them they're actually new inside there now but those were not the actual
problem so we further debugged the motor itself there is a pcb on side of on top of the motor
and a magnetic wheel so when the motor turns there is a turn signal or there is a tachometer signal
back to the main pcb that tells the
microcontroller how far it has moved and also with current sensing basically the microcontroller
can sense how much pressure the motor is currently applying to the whole brewing unit and we thought
maybe this these sensors are actually at fault so we took a large oscilloscope for no reason and
a power supply but those were actually working quite nice and then since the machine needs water
we need to go to the kitchen we had to set up the whole thing in the in the kitchen yes
including the oscilloscope and annoyed more people
um so what we actually did there is we did a bit of tracing of on the pcb
where the motor lines are actually or the motor connections are actually going to and it turned
out they are they are split from for high and low sides into two chips and one of those chips
yeah so on the left you can see there is this larger chip right next to the blue capacitor
and there are two chips right below it and it's quite interesting those are for completely
different motors and i or we think they did it this way because the upper chip does very well
power sensing so you can get an analog value out of the chip for each output how much
what's basically are going out to the motor and the lower chips are the lower side of the
of the motor controller that basically just
yeah control the negative side of the depending on how you turn the motor and those are can
actually also do current sensing and they're not just standard mosfets they have some kind
of electronics inside of there so you can you can break them a bit more difficult and also on the
input pin you can actually do power sensing so if you supply a logic value to the input pin to actually
to switch through the mosfet or to switch the mosfet and it's and the mosfet goes into power
limiting it will try to lower the voltage on its input pin depending on how much overcurrent it
is currently having so this was not implemented on the board but those are quite interested
quite interesting features of those mosfets so and also again aliexpress and then after after
clements clements did the soldering we started to pray for the first boot again and yeah we had our
first success we had got the first coffee out of the machine but um
it was cafe schwarz and it was really black and we didn't want to risk a subscription for the toilet
because the machine wasn't clean for like one or two years so we started the cleaning process
so we cleaned the whole machine and we brought you some pictures of the cleaning process
and it was quite dirty and after that the machine was like like new and the good thing is the
the company that produced the machines offers
the machine that produced the machine offers a really good manual how to disassemble the parts
and everything so it was quite easy to disassemble it and assemble it again and yes we had a clean
machine and as next we're going to show you some more parts of the electronics
okay um
thanks okay so um there are actually three bigger pcbs inside one is the one we showed before that's
the main pcb in the back that basically has most of the power stuff and then in the front we have
a touch display and that one was really interesting when we opened the machine the first time because
we saw that that's an arm imx6 processor so it's running pretty we were pretty sure it's running
linux and also there is an sd card inside and it has some usb and a gigabit ethernet port that's
everything of this is not used so you can use the usb port to program the machine
but this also seems to be added later or it was added later and the second or the third pcb is
a real power pcb because there is no no typical boiler inside of this machine
it has an induction boiler so it can basically vary
it can basically change the temperature of the water on the fly so if when it's currently running
it's not heating the water if you press a button on or if you want a beverage it's basically
heating the water on the fly and not taking too much energy and also it's quite interesting
you can program different water temperatures for the different outlets and depending when they are
switched over the valves you can get a mixture of different water temperatures which is i think
quite unique for this machine and also for repairing or for for taking it apart those machines are
typically really big so there is a lot of space inside you can more or less take the whole machine
apart you can replace everything you can you can really maintain the machine quite well so
compared to those cheap plastic machines also that are sometimes on on smaller offices
these machines can basically live forever if you have enough supply of a few things that might
break like the valves or so on but they're all standard valves well then let's check out the
software hardware setup so as said glans already said we have two components there there is like
this there is a cpu inside it's like a mic controller 16-bit industrial or coffee processing unit
and there is a gpu it's a imx6 arm linux board and on this linux system there is a qt binary running
with displaying the rear gui on the display the rear gui is a html javascript based web application
and you can run it in your browser actually it doesn't make coffee but it says it does and there
is a rear boot exe file so you have an exe file you have an arm cpu and if linux running what
um let's solve the mystery it's a dot net binary for flashing configuration files and extracting
configurations and it's run with mono on the linux arm processor well it works it it works
and uh when you and you put in a flash drive with a specific folder structure it boots this
this uh this tool to to to put the config on it or to get the config and i must say i still
understand how the boot procedure still works because you have the qt binary and the and the
config binary and i'm still not sure how they know when to start what i tried to find it didn't
manage to find it in the system so um you have also a windows tool to program it it it looks like
this it's uh let's say last century more or less the the red picture down there is also animated
also quite nice i'm sure that was a feature request and but i must say even though the ux is quite
questionable it is super powerful comfortable configuration like this you can really make a
power saving menus you can make happy hours you have a really a lot of configuration options for
the machine it's super nice but the interface is quite interesting let's say like that but let's
get on the program on the machine part so there is a programming menu in the machine itself a touch
screen it looks like this you see different buttons uh it doesn't look very um comfortable
to use and i will come back later to to this part why and what uh let's first also check out the web
app um if you press the manual online button it will actually display the pdf of the manual of
the machine from dsd card on the tft display and you can scroll through it it's quite nice but it's
also a bit annoying to scroll i think there are a few hundred pages um so there is this web app it's
html page and uh when you're on the menu you can choose the beverage in the end and there is like
a page menu html where the beverage is shown and if you choose a beverage you come to a page confirm
page and uh they have like the selector is still like the parameter what what coffee or cacao you
want and if you press the confirmation one it goes to another html page and uh when you confirm it
there is this qt binary as far as i understand understand the process it will check the url with
the parameter and will know what coffee or beverage it will make and when it's done it will change the
url again in the binary that you come to the done page and will switch back to the menu so so the
api between the web page and the uh the qt binary is actually the url parameters so the the qt
binary manage the whole whole html in the back in the end and uh shows it there let's show you some
excerpt from the javascript files that found so that's how you how you how it's implemented on
the javascript the whole thing are like arrays in javascript and they have a name and a picture path
and you can change the names you can change the pictures and it's also quite interesting one
even more interesting is the the javascript for the prices
like every array part is one of the beverage with the price in cents
but i'm not even sure if the machine knows what currency it's running on so seems so that there
is no currency logic behind it but i was digging in the binary the really interesting thing is when
starting it up it's actually uh it updates those files the binary updates those files but i will
show you know what so i was digging at qt binary with githra and i found this function in there
it's called generate javascript price and availability and uh let's see what this means
so there is code that generates the javascript down there from the binary from from text more
or less yeah from sex so the binary generates the javascript file in the end because the config is
on the machine and not on the html files and javascript files um so next uh we also check
the communication between the machine and the front and more or less yeah so between the linux
system in the front and the the 16 bit uh processor in the back uh there is a serial port and uh
we already saw in the linux system that it's opening the tty on the imx um board more or less
and then just found s trace in the system and we're like okay easy task let's try this out
and found that um so you can see uh dark red and uh dark green lines here and one line per or they
um they just uh one is sent uh to the machine or to the controller one comes from the machine the
red line comes from the controller in the back in the back and the green line is sent to the
controller from the front linux system and it's roughly every second and if you don't do anything
here um more or less the binary command that's or the binary message that's sent from the linux
system is more or less always the same um the the red messages are interesting because uh in the
middle um there is this text that the machine displays in the lower half of the machines
or the display displays in the lower half of the machine um so um and it has a quite weird encoding
because uh if you see on the in the top here um there is uh in german for the uh for this trace
uh filled wasa off bitter vaten and between off between the two words off and bitter there is no
space there is no character no anything between there and that was quite weird for us um and also
um there was a weird break on the display too so there there was like a space between the two
words but it was not just a regular white space so more information about it it's a binary protocol
so the first part is always binary and the last part is always a binary code and also the commands
that are sent from the touch display to the back are some binary messages and those texts with a
missing space in the middle when debugging this or let's see uh let's look at it here um when
counting the characters uh we always found that the whole text is 32 characters long and the middle
part where there is no space is always 16 characters or add 16 characters and who did a bit more of
electronics probably already knows where the software comes from we like okay hmm and then
we realized in the end there is this corrector display shout out 20 for the photo box in the
binary kitchen tent and in the end the text is usually shown on such a corrector display
and they in the end only use this display to show the characters instead to implement a proper
software for it because the older machines only have these buttons and a corrector display in the
end so the buttons you saw on that on the touch display for the programming menu probably on a
previous version of this machine um the those were actually real buttons and the texts are sent for
those 16 uh characters uh by two lines so 32 characters displays you have in this in this
really old vending machines and um probably uh the whole back of the machine so thinks that it's some
kind of machine that has physical buttons and no touch display and for a block diagram so those
are the three um pcbs we have the the heater board we have the cpu and power board and the linux board
that's connected over serial and everything that is part of the machine even payment and some and
some kind of modem you can plug in is connected to the 16-bit controller so what is a bit frustrating
because we are hoping to do more hacking from the linux system but we have to deal with the 16 16
bit controller so it's super hard to you need to reverse engineer every option in the end to
implement the same binary protocol to configure the whole machine and work with it it's quite
tedious task in the end and then let's get to our lessons learned um always verify claims if someone
makes them because uh well sometimes they are not uh correct aliexpress sells you automotive ships
that you don't get in europe or in austria especially the only problem in austria is we
have real luck because in austria is a new law that you can't get easy stuff anymore because of
some packaging directive and we are really lucky because it was from the first of january 2023
in charge and we ordered it like in december and got it before that really lucky for us i think
we actually got it in january or in january but still delivered but currently aliexpress doesn't
deliver to austria or not a lot most of aliexpress sellers um as you see industrial equipment is state
of the art legacy it's it's really it was a little bit heartbreaking for us to see the reality that
we deal with and one interesting thing is also the windows configuration software is not publicly
available and we currently doesn't have the latest version and we can't use it for the coffee machine
configuration currently so this machine is built in 2017 um so quite new actually um i think those
those machine types were introduced around 2015 or 16 and the screenshot you showed from we showed
from the windows tool i think was one year before this machine appeared and if you look very long
in the internet and we spent several hours looking for stuff for on these machines you can find it on
some shady download grabbers grabber portals or whatever but we still haven't found a version
where we can actually program this machine it's possible to program it over the the touch display
but uh yeah you can take hours just for a single beverage so there's also some possible future work
in this regards so there's also more possibilities for software analysis analysis we would like to
obtain the programming software so if someone has something lying around you'll be happy to talk
with you or if you know somebody who knows somebody who works at a coffee vending company maybe
um i still want to understand how the boot procedure works because there are two different
binaries and i couldn't trace how they know if the flash drive is plugged in everything it's still a
ongoing um research for me so if you want to help out or interesting more for a machine
talk to us ping us um and we have two more interesting things so um there is rfc 2324 who
knows it who has read it okay quite a lot of people um so i was playing around since a few
years to actually implement it on a coffee machine and then this machine came around and i said hey
finally um a machine that could run it but it turns out it's a bit more difficult and actually
while working on the machine and um looking at the rfc uh the coffee pot control protocol is really
just more or less for coffee pots so classic coffee pots um espresso machines it's difficult
also this machine can do also can make tea with tea powder and i've heard there is a specific type
that it's not too bad um and also this one can actually also do a soup so there is soup powder
um we don't have anything we don't have tea we don't have soup uh but uh because typically
you have to take like 10 kilos of it and we don't want to risk to have 10 kilos of soup
for this machine lying around um and maybe a new rfc extension for the coffee pot control
protocol is needed for coffee pots and not tea pots for those machines too at least we have also
crème brûlée as a powder flavor yeah we have crème brûlée
it's quite tasty if you don't have too much powder in your in your coffee mug
yeah it's uh it was a bit difficult to find the right dosage
um well and there's of course the last question that everyone wants to know does it run doom
and well i can say it can let's check it out
uh
there is uh actually a safety um switch here so that you can't operate the machine
when it's turned on but you have to operate the machine when it's turned on to program it
so there is a anti-safety key here that is here in the machine for maintenance
and it's it actually switches completely switches off the power
and programming the machines there is a there is a button uh inside of here or actually two buttons
i'm not sure they do the same i'm not sure why uh they put in two buttons but i think uh this
plastic presses plastic presses on the first button and if you press it for a second then
it will go into the program mode i'm still waiting until our management wi-fi hotspot uh
turns on it's in the machine inside we added it so i can ssh into the machine
and uh so even though the the serial protocol is quite weird and the setup is quite weird
they i think they're actually quite nice machines um i think it's more the more of a management
issue of the project that it turned out like this um because you could do a lot more with those
electronics and um that's actually the board here is custom but there is a system on a board here
um that's uh from uh very side so that's a vendor that uh supplies you with um linux
arm linux boards where you can just run software and i think it's even running some kind
of debian uh they publish on the site probably on the website probably a few years ago um
you have an ethernet port here and there was no root password or anything um because yeah if
you have the machine open you can do whatever you like and the root fs is completely on the sd card
so um that's actually quite nice for swapping out the logos or trying out things uh you don't have
to flash anything um you want i guess you
so um it has a touch display so it had to be done or it has a display so it had to be done
we're running just a frame buffer doom although the machine itself is running x11
which is fine um and
we already in before we started to
to look into the serial protocol and the software we thought like we could actually integrate
those two parts that you can either switch between coffee mode and doom mode or actually
implement a few things in doom where the coffee you get depends on your doom points
and if you die just at the beginning you will just get hot water
but that's still to do i need to kill the x11 search
so a bit of um information on the machine itself all of those containers you can actually take out
so um that's more or less empty currently for example um you can just take this out
the motor plugs in into the rear here and operates the um the mechanism um for the uh powder and um
it should also fit again
ah yeah
all right our demo failed i'm very sorry
you can come to the village and we'll show it to you we're gonna fix it
okay
thank you very much for your presentation uh we are out of time but since uh you are so
maybe uh tell everyone where exactly uh can can they find you um we are um more or less at the
meta lab or lavendville village that's on the match is the village the village was the largest
or the longest name and it's the silver or white dome basically right next to chaos west i think
chaos west probably is known by everybody thank you very much yeah thank you