[MUSIC]
Welcome back on the Nordic Stage for the next talk.
This talk will be in English.
Unfortunately, there will be no translations, but you will be able to follow, I guess, if you came here.
I hope you have all nice places.
And I welcome Jürgen Papel, who will talk about building a cloud-free digital voice assistant with free and open source software.
Please give him a warm welcome and enjoy the talk.
Thank you.
Okay, so first slide, I guess, is always about who's talking up there.
I'm from Cologne, Cologne.
I work in security management, and I build stuff mostly for my kids, and I break stuff, whatever needs to be broken.
So first we're going to talk about what our voice assistants actually composed of.
Then we're going to do a demo.
You are probably not going to be able to see what's happening on the display, but we'll get audio, so you'll be able to hear what the hell has to say.
Then I'm going to talk about what is it that actually comprises this voice assistant, and then there's what's left to do for release 1.0, and what's on the agenda for beyond that.
And at the end, there will be a Q&A.
So when you're talking about voice assistants, there's really five components that make up the core logic.
There's the wake word detector.
And you presumably know this because it's, okay, Google, Alexa, Siri, whatever.
That's the wake word detector.
Because you don't want whatever voices you have to always just try to make sense of what's being spoken and interpret that as commands.
So the wake word detector activates the assistant.
Then there's speech to text, so it's recording whatever someone has to say, translates that into text.
And the next part is the intent parser, so trying to make sense of what's being said.
Skills, that's the execution of whatever the person requested.
And then text to speech for the feedback.
So that didn't work or whatever.
So let's look at each of these just briefly.
The wake word detector is a presumably easy task.
You can do sort of like pattern matching on the acoustic signal, et cetera.
But the issue here is that you want a really, really high or a very, very low false positive rate.
Because otherwise the assistant will be going like, oh, yeah, what do you want from me every couple minutes?
And that's very annoying.
And of course the environment is, you know, as it is.
We don't know.
And I don't know how it will behave on here.
I haven't tested it, so we'll see.
And of course everybody has a different voice, so that task isn't really that easy.
So as far as free open source implementation, there's Microsoft Precise.
And the true and tried Parkinson's.
And there's others.
You know, I'm not trying to list everything here.
And the selections is really more or less what I played with in my project.
So the next part is the speech to text.
So there's really two parts.
There's the software that actually tries to make sense of whatever the audio signal is.
And then there's the language model.
You need a different language model for each language.
And essentially the language model contains the words that are known to be able to be translated.
So the critical parameter here, just like with the weight word detector, the false positive rate here,
it's the word error rate.
We'll get back to that later on.
And again, two open source implementations, Mozilla Deep Speech, somewhat recent,
and Kali that is really fairly old by today's means, but still a good system.
So the next is the intent parser.
So independent of whether I tell them turn yourself off, execute system shutdown, or go to run level zero,
the intent is the same.
So intent parser derived the meaning from what's being said.
So in this case, it would map to the skill system shutdown with a default parameter now because I didn't tell him otherwise.
Onto the skills.
And that's really completely implementation specific.
There are some that have already a lot of skills included.
Some come at a bare minimum.
And there's pretty much always some API or SDK that you have to implement additional skills.
So text to speech, again, language model, training data.
So of course, the function is to give the user feedback on what has happened.
And two of the rather older ones, the F-Lite and Mary TTS, and very recent, I think last year, was MyCroftMimic 3.
So with that in mind, we'll do the demo.
I'm going to just leave this right here.
I have a USB cable.
That's just power.
So I'm plugging it in right now.
And it'll take a while.
What's coming now?
Oh, yeah.
Disclaimer, I have taken some audio from the movie.
So that's being used here.
That's not part of the project because obviously you can't distribute that.
According to German law, what I do is legal.
So it's booting up for the Linux system boot up.
It's roughly 30 seconds.
And in the time in between, it's showing some animations.
I have the animation on the slide right now.
And what I did is essentially just rip artwork from two different projects and constructed something that would sort of make sense for the boot up phase.
And once it booted up, there'll be the red eye popping up and I should say that I'm fully functional.
And I haven't heard him.
But we can, I guess, switch over to the hell now.
Let me just take a look.
It seems to be-- yep.
It seems so.
Two buttons or rotary encoders on top.
For the volume and turning volume up.
I don't think anybody can see it.
There's like a volume indicator just like a tachometer on the car.
So the wake word is OK hell.
That didn't seem to work.
Let me come around.
OK hell.
OK demos are demos.
So I can trigger this manually.
Probably easier from down there.
One button is the volume control.
And the other button is an onscreen menu.
And I have a menu voice command.
When I push the button, the wake word detector triggered.
So what it's doing now is actually loading the language model right now.
So the first request is like a 10 second delay.
And I want to move that into the boot up phase.
So it doesn't have to be on the first spoken request.
Hello Dave.
OK.
Now it should work right away.
So let's try that again.
What date is it?
It's processing now and rendering the speech output and maybe it will work.
I'm sorry Dave.
I'm afraid I can't do that.
I guess it's still a good demo right?
So I can power this down from telling me to turn yourself off etc.
What happens then is it shuts down the system, shows the animation.
And you probably can't see it right here but it will be in the recording.
So it's the live animations, the live lines that go flat.
And once it's gone flat, it's dead.
So now I'll plug it off.
But I actually have one more demo to show.
So I'm going to plug it back in.
And that's the part, well how do you configure that thing?
So it has a web interface actually that I'm going to connect to in a second.
And what I can do there is essentially configure the whole thing.
I can configure commands that it should listen to, what actions to take etc.
So we'll see whether or not that will work in a minute.
Okay, let's boot it.
So maybe you can see right here, just barely.
Of course it has to be in the style of the movie.
And so the first button is orders.
So we did try what date is today.
So that's this order.
So it listens to the prompts.
What is the date? What is today's date? What is today's date?
And why that is exactly the way it is right there is something we'll talk about later on.
So let's leave it at that for the demo.
And get back to the slides.
Oh, there it is.
So what are the key takeaways from the demo, except for the demo gods weren't that good to me.
There is the speech interface.
Of course that's the main purpose of the voice assistant.
But there's the physical interface, turning the knobs, getting feedback on the display, anything.
And there's also a lot of little things that need attention.
So the eye candy, these boot animations.
Other people did the work, I just cut everything together.
But that needs to be done.
It is actually something where you know what is the device's date, so you can expect what is it going to do when you interact with it.
The configuration and of course error handling and many, many other things.
Why am I listing these? Because that's the core of my project.
The actual voice part, so wake word detector, text to speech, intent parsing, etc.
Is something that I'm using other software for.
But as always, you fiddle with it and then you adapt it a little bit.
So that part is also a little bit of part of my product.
So what is essentially the idea, the goal?
I want a voice assistant that is actually usable.
So response times, boot up time, processing of commands is important so the user doesn't get bored while waiting for the response.
Error rates, you want it to just work.
And also important, everything should be done on the device.
No connectivity required except for maybe skill executions.
Then of course, or at least in this setting, it's open source.
And it should be as easy as possible to build.
And there's a couple of things like 3D printed enclosures. They were designed so that the print models are as easy to print as possible.
So in the entire model, there's just one overhang.
Well, actually, no, the second one is in there.
But I essentially tried to make it as easy to build as possible.
And also use components that are rather readily available.
And yes, pies are hard to get nowadays, but the situations get better.
So we hope that they become available again.
Especially not a project goal was the price.
So the components in here run up to roughly 120 to 200 euro, depending on what you put in there.
So I put also an ID card reader in there.
You know, you don't need it.
Then you don't need that.
So what's in there?
We have the pie zero two.
And that's where the Linux system is running on.
And that's the greenish thing up here.
Below it is the display with a microcontroller.
And below that is essentially a sound board.
It has two microphones and a speaker connector.
And then we have a GPIO extender.
So more pins to connect to.
And in my part of the project, I have a requirement for RFID card because my kids have RFID cards.
They put in a reader and the Peppa Pig or whatever starts playing.
So I wanted that to be part of this project also.
But that's just the bare components.
How do we need to connect them?
So the first part is we connect the pie zero with the hat.
And that's actually, if you could get the camera right here, that's actually a cable in here.
Is that in the picture?
And it just runs down and connects the hat that's down here behind the wire mesh.
So those are connected now.
Then we have next the connection from the pie zero to the display with the microcontroller.
That's just a regular USB cable.
And that provides power and data channel to the microcontroller.
Then I2C connection from the microcontroller to the GPIO board.
Because maybe you can also just get the idea that these displays with microcontrollers, they don't have too many pinouts.
So then I2C connection from the GPIO extender to my RFID reader.
And then, of course, and I already mentioned this, we have the microphones on the board.
Then we have a speaker.
And then we need power.
And now we're pretty much almost ready to go.
Except we didn't talk about the rotary encoders as the user interface.
So they wire up to the GPIO extender.
And you need five connections per rotary encoder.
It has a push button, so that's ten cables.
And then we have quite a lot of cables in a very constrained area.
So that's getting quite a bit fiddly at times.
But wait, there's more.
There's I2C polling.
I2C is a command response protocol.
So there's no way that a device that has an event, so the rotary controller, when you move it, it changes the state on the pins.
I2C has no way to say, okay, hey, here's something that's happening.
So you need to poll this.
The microcontroller polls the states of the GPIOs every, I think, ten milliseconds.
And when something changes, then it knows, okay, the button has been pushed or whatever.
And obviously that's not such a nice solution.
But what is the alternative?
Well, this GPIO extender from Adafruit has two pins that you can actually use to notify the system that something has happened.
Something has changed.
And you connect that down into the song board.
It has an additional pin out board.
It's a growth port.
And then at least you could detect right there, okay, something has happened.
What has happened?
We don't know.
We need to poll that using I2C.
And then you also connect the RFID card reader with that and have then also a means to get events from the card reader.
I stopped there.
I didn't do this.
Yes, it would be the nicer solution.
But with this small space and that many cables in there already, I was like, okay, no, let's leave it at I2C polling and no outside channel that notifies that there is an event just because of already the amount of stuff in there.
So one thing that we also have to consider is when we were to connect these cables, the notification lines, so to say, that some I2C event would be or some GPIO change has occurred so that big poll.
We need to talk about the voltages because the Raspberry Pi zeros are not five volt tolerant.
So the I2C powering system and I2C is both three three or five volts capable or.
Compliant, so you would have to pick a microcontroller with a display that runs on a three three system for for the I2C.
Otherwise, you fry your pie zero or you need to do a level shifter, which adds even more stuff to that package.
So I left it at that minimal set up, so to say.
OK, this is the current state.
There have been some components that were swapped out.
So before the pie zero two came out, I did have a pie three B in there, which, of course, filled up that that that case even more.
So I was glad that the Python
to Pi 0.2 does have enough power, at least with some overclocking, to bring usable performance.
Before the displays were available with embedded controllers on the back side from just regular
shops where you buy that stuff. I did have just regular LCD in there with a display controller,
and also just happy that I got rid of that stuff. And the last change really is I put
in a bigger speaker in the bottom because the speaker I did have in there first was
rather not so loud. So hardware was really, really stable. And now we get to the MCU options.
So essentially the display what you see is behind the dome. So this is the component.
And again for printability and for easy build, you print this, you print this, and this is
the design casing for this microcontroller that I have in here. You have a different
microcontroller, one of the three that I have modeling for. You just swap this and
put it in and you're all good to go. So the first one that I started with is the Roundy
Pi. That's a board with a round display, and that's actually the one that's built in right
there. And it has an RP2040 in there. That's what's on the board known as the Pi Pico.
It has six pinouts, works with 3.3 volts, so that's perfect for us. It has four GPIOs.
Two of those are power and ground, and two of those are ITC capable. So it works just
-- oh wait, no, power and ground I already have up there. We have two additional GPIOs.
But we have ITC and that's all we need. That's what it looks like. It even has an SD card
on the back side. I don't make use of that. The next one is a M5 stack core 2. And that's
probably the most easiest accessible option to build because it is one of those monitor
package thingies that they have with their M5 stack line. It also has a growth connector
so that's just easy to connect. There's one issue with this. This board has a CH9102 chip
for the USB connection for the communication. And it reports always connected. The serial
CDC ACM connection is reported as always status connected. So there's no way for the Linux
system to conclude in what state the processor is. It might not even have a working firmware
on it. I'm connected. So I like this physically but that chip behavior is really not so nice.
And then there's WaveShare also an RP2040. It does also have that CH9102 chip same behavior.
The nice thing about it is it has 30 pinouts. That's what's in here. And they're really,
really tiny because they're half the size of a regular GPIO. So instead of 2.54 millimeters,
they're in 1.7 millimeters. So that's really, really tiny. So let's get to the software side
and I need to hurry up a little bit. On the Linux side we have the speech processing software.
I used Calliope and why that's the case we'll get to later on. That's for essentially all
the voice interaction. And then there's the physical interaction with the buttons and
the display. That's essentially what has been the core of my work for this project. And
the last part is CommitUp. So that's a package that's readily available in Debian essentially
for network configuration. So all the application part components, they are in Python. They
run in micro-VSGI application server with the Python 3 plugin. And if you have never
seen that application server software, I highly recommend it. It is one of the most extensively
implemented application server software I've ever seen. So on the Arduino, the micro-controller,
Arduino is the framework that I use there. And essentially what the thing does is just
I/O handler. It just regularly pulls I2C, is there something that the user triggered
for me. It also displays whatever needs to be displayed on the, shown on the display.
And essentially as long as whatever device you have, as long as it can signal its state
to just some pins on the GPIO extender, then you're all good. You can use it. So the firmware
is in C++ because the animations, all the graphics that are in there take up quite a
lot of space. And the RP 2040 only has 264 kilobytes of RAM. So that's rather constrained.
The ESP32 has loads and loads of memory, so that wouldn't be an issue. But with the RP
2040, Python, micro-Python, circuit Python wouldn't have been an option due to memory
constraint. So the micro-controller is really stateless, once in an up and running state.
And I2C polling, we'll get to this just on the next slide. Because that's also where
software history, let me skip over this. Okay. I moved my slides. Software history. MyCraft
did build, and we've seen these MyCraft components for weight word detector and speech to text,
I think, on the Earth's side. They wanted to do everything on device also. They put
everything they made open source, except for the speech to text because they did rely on
cloud services for that as well. So MyCraft was the initial voice assistant software that
I tried, but then switched over to Calliope. For the weight word detectors, the open source
detectors, they have too high false positive rate to be usable in just everyday situations.
Right now, I actually use the Pico Voice Pocupyne 2, which is not open source, but which is
free in terms of money. You can get a personal license for free. So that's not bad also,
and it works quite nicely. And then we have the speech to text topic. So the open source
information have usually a word error rate of around 10% or more than that, and that's
really unusable for interacting with a device where you just expect to have, you know, you
just want it to work. So the reason why this is so high is because the vocabulary is so
big. You know, they have 50,000 words in that language model, and they don't know what to
expect. So if we could reduce the vocabulary, then we could improve on the word error rate.
And that's what Calliope allowed us to do, because the biggest drawback of Calliope is
that it does no intent parsing. It does essentially string matching. So you have to configure all
variations of the commands that you wanted to react to, and that's rather bothersome.
But because we do that, we have a non-vocabulary. So we can reduce the vocabulary for the speech
to text, and that has the effect that we have a word error rate of way, way, way less than
10%. I have to really try to, you know, under better conditions than this to get it to misunderstand
me. So Calliope also has REST API, and all these components talk to each other, essentially
MQTT for the different processes, USB, I already said, for talking to the Arduino controller.
The web interface for configuration is done using flat dev, so it talks to the flat daemon,
that's IPC, and the browser, of course, talks to the frontend that is the console. So talking
to the microcontroller, we have around 15 commands that need to be sent to the device.
The device needs to send a response. The device needs to send events when some IO event has
been detected, and also just this lock or debug event from the microcontroller to the host.
So various parameter types, so I decided to use JSON as the protocol, and essentially
it's a JSON array. The first element is the command, the second element is a dictionary
of key values for the parameters. So due to timing, I'm going to skip over this real quick.
It's essentially booting up, configuring, I need to configure where the GPIO extender
is on the I2C bus, and then I tell it, okay, start. Why did I put this in here? Because
the I2C polling is done on the second core of the RP 2040 and also on the second core
of the ESP32, because they both have two cores. We can just move the I2C polling issue to
the second core and not worry about that anymore. End of configuration, and then it says I'm
running, and then the Linux system says, okay, show the idle screen, which is just the current
time. When an IO happens, there's an event from the microcontroller to the host, which
essentially just say, okay, I have it on this pin. You told me this pin has the name volume,
and it's an event that you configured as delta plus one, so I get consumable events for the
applications inside the Linux system. And then the response is, okay, well, then show
the overlay for the volume control. Let me skip over that. So anecdotes. Let me talk
about the USB here first. I did plug in the hell into my laptop, and at home I have a
docking station with a USB hub, and once I put this project aside for a couple weeks
and then pull it out of the drawer, start working on it again, I plug it into the USB
hub, and I hit every now and then errors that I'm unable to explain. And then it takes a
while, and then it's like, oh, man, I forgot the USB hub doesn't have enough power for
the thing to operate correctly, so it's really a power issue. And I've run into this many
times, and then it's like, okay, I did forget about that again. And I'm going to skip over
a couple of these. The dome that does really nicely, I can't get it apart, you know, it
sits on the display that does the lighting, does very cool reflection effects. That was
the hardest part to source. I bought like 20 or 30 different domes before I found one
that was really a good fit for this project. And MyCraft, they just went bankrupt, I think,
at the beginning of this year, at least they closed their operations, so their source code
is still online. There have been a couple of commits to it, but, you know, it's probably
more of a done deal now. So release 1.0. I need to finish quite a few bits here and there.
And one important part is when you just, you know, put in the install image on the SD card,
you need to configure the network card. So the idea is to show a QR code on the display
that you scan with your mobile phone that has the connection for the Wi-Fi. You connect
to the AP that's running on there. That's the CommitUp software. Then you configure
and then you're all running. Oh, yeah, and I want to do documentation. So hardware. I
really have no idea if there is anything else to add to it. So hardware is probably stable
unless some Pi 0.3 comes out. Software. I want to take a look at other open source
wakeboard detector implementations. And the other part is Calliope needs in some areas
quite a bit of code refactoring. So that's what's down the road for me on that. And that's
it. So thank you for all the organizers and angels for enabling us to be here. And now
we have time for questions.
[Applause]
So do we have questions?
Hello. Thank you. One thing I, by myself and others, usually observe with these devices
by the usual known vendors with cloud and anything is kind of, hey, play this and that
music and then you have a Spotify or YouTube music account in the background. Do you have
any thoughts about solving this multimedia stuff and casting stuff to TVs or is this
basically something where you have to go while the big windows close and we say we don't
want that?
Okay. What I already have implemented, that's really the other project, why the RFID card
reader is in there. I actually have a Python implementation that's working for connecting
to the Spotify API and playing that. And I don't think I've released that yet. Really
just trivial code changes, but the last time I checked a year ago that didn't work. So that's
a main use case, so to say, for this. In terms of putting that into this whole ecosystem
of software that's running on there, it'll probably be a Kaliope plugin later on. So
expect that sometime down the road. I think there was another question over here somewhere.
How is the microphone performance of the system?
Yeah. So on camp settings, not so good. When you're in a room, like your home, it's quite
good. So three meters is the distance where you can reliably talk to it. So more than
three meters, that's iffy, but three meters or closer is quite good. And I would say if
you're like three meters away, you probably get a word error rate of, I don't know, just
by feeling, I would say maybe 5% when you're up close, because it's right on the wall next
to you or so, then way below 1%.
Hi. How does your work relate to other existing solutions that do the whole thing in one,
like Snips or Jasper or Open Assistant?
Okay. Open Assistant is really like a home automation thing that has a voice interface,
if I know.
But Home Assistant, Open Assistant, that's like Snips or Jasper.
Open Assistant I'm actually not aware of. Snips, I can't play Snips right now. I've
heard I've looked at it, but I can't say right now what it was. So maybe we can talk afterwards.
So all this work that you have done, do you have published it or is this still a plan?
Well there's like a right now, not really working state on GitHub. I have around, I
would say 50 to 70 commits that I haven't pushed to GitHub yet. But I'll do that rather
soonish, so hopefully next week. So probably in a month, I would say, the state on the
GitHub, oh and I didn't put in the URLs for the GitHub. Well, I mean, you can Google it.
But probably in a month, I expect to have a working state in GitHub that you can just
download and install script, etc. So hopefully by then it will be rather easy to deploy.
Right now it's a lot of manual fiddling.
You have the mic.
So the first I didn't get is what's the easiest way, I mean, how easy it is to create new
commands. So for example, is there a web UI for that or do you have to rebuild the whole
thing?
No, no, that was the web UI that I had shown. So this right here, it's not so visible. But
you can edit the commands right here.
But how does that interact with the model? Would you say you have a fixed model of words
that are defined?
So let's say you add a new order and introduce new words, right? What it does is it uses
the Calliope REST API to say, okay, there's a new order, that's the term for it, with
the given prompts, the words. And that is then used by the speech to text plugin on
the request. So when the speech to text request is made, it looks at the working set of the
words in the model, in the Calliope prompt model. So it's always up to date.
Is it offline, that part as well?
Yeah. Unless you have a skill that needs some sort of network connectivity, this thing wouldn't
need anything.
So well, maybe this last question.
Do you have implemented any multi-device management? So I trigger the hot word and the device in
the kitchen and in the living room can hear it. Do they speak to each other, hey, who
will handle the job of us?
It's actually somewhat already implemented, you need to configure it because the Calliope
has HTTP REST and MQTT skills, so to say. So you could interact with whatever bus your
home automation device or system has, like Home Assistant, Open Hub, whatever. It needs
to be configured, but it technically will work.
So unfortunately, we have reached the end of our time, but also most of the questions
were answered, so I guess you are all ready to build your own hell now. And with that,
I think we thank the speaker again.
Thank you.