Tanya Cushman Reviewer:" Elisabeth Buffard Our next speaker, or the first one of today, is Dorothea Tse.
She started Camera on the Linux with Purism, and she thought it's way too difficult to do Camera on the Linux.
There are so many things you have to take care of, denoising, shutter lengths, feedback loops, and many, many more.
And she wanted to make that easier. So she has a motivation, and now we are able to listen to it.
So please have fun with the talk.
Lips obscure, cameras are difficult, and lips obscure is difficult.
It's a friendly camera library. Please enjoy the talk with a round of applause for Dorothea Tse.
Yeah. As it was said, I started Lips Obscura.
I am Dorothea Tse, and I think cameras under Linux are really difficult.
You might think that a camera, what is a camera?
It's like a box with some optics and some light-sensitive stuff inside.
So it's not really that difficult.
There's a couple of components, except we made a mistake of adding computers to it, and that makes it difficult.
But before I get into it, I want to tell you exactly what I mean by a camera library for Linux.
And a camera library that I am making is one that is only concerned with cameras that are actually part of the computer.
And the goal is to abstract them enough so that an application author can write an application that works on many different cameras without worrying too much.
So, for example, cameras like this webcam over here, which is pretty easy to use.
You just set it, and it pretty much just works. You don't have to mess with it.
Or cameras like on the phone, where you can perhaps change some settings. It's more complicated.
It gives you more power.
But the reason why I wrote this library in 2024, I mean, the phones have been there for a decade or even more.
So the reason is Linux has not been concerned with those complicated cameras until very recently, because it started getting popular on smartphones only now.
Before that, it was only USB cameras, which are really amazing in that sense that the USB protocol just makes it...
You don't have to care about anything. You don't have to care about anything.
Meanwhile, on phones, every camera has to be driven differently.
And my story starts indeed with this phone, the Librem 5, where I was responsible, it was a couple of years ago, I was responsible to figure out the camera problem.
So I have to figure out the drivers, I have to figure out the camera application that we're going to ship.
I had to figure out whatever is in between.
And that whatever is the Libre Obscura, which was not present back then.
So my task had to be figuring out what actually exists in the Linux space.
And that's a task for a ten times developer. Yeah, so what do we have? Oh, no.
This is entirely unsupported. No one wrote any drivers for this.
I mean, it's kind of understandable we only just made this phone a couple of months ago.
But the other problem is this doesn't have any hardware processing.
So all the image processing, I would have to write myself. Okay. There are some attempts already.
So let's try and take the user interface from here and start coding.
The user interface from here, processing, I have to write myself. I have to write the supports.
Oh, but there are bindings. That's okay. Take it from here. Take it from there.
And what happened was this camera library that I managed to do with the help of some other people.
And, yeah, it's actually kind of bad.
It's a combination of many different things that don't really match together.
And it only works on this phone. So if we ever change the hardware, we have to change the library.
And no one else in the Linux ecosystem can use it. Maybe I'm not a 10 times developer.
But, yeah, I was kind of thinking maybe it would be easier to write everything from scratch.
But that didn't happen until very recently when I learned about something called Prototype Fund where they give money to open source projects.
And I'm like, yeah, it's easy to write an application. The idea is crazy.
But I might as well just send the application and see what happens.
And Prototype Fund was like, here's the money. I'm like, what?
And this project is extremely ambitious. I'm like, why would you? Well, okay. Might as well do it.
Might as well give it a good try. So what do I want a camera library to look like?
My dream camera library. Oh, that went wrong. Sorry about that. Yeah, where were we?
So my perfect camera library would be, I have this amazing form right here.
Please fill out some capital letters. Who is the most important for the project?
I have a feeling that not every open source project is really focusing on the users.
Like we are developers. We do the thing that we need. And we try to focus on our needs.
But with LibObscura, I wanted to change it.
I wanted to make it easier for users to be able to actually use it.
Because otherwise, what's the point of the library?
So users are the greatest priority for LibObscura and also manufacturers.
So I was starting from this point of, that was the entire story.
I was a part of the manufacturer company and I wanted to make a Linux phone.
I want Linux phones to exist. I want to make it easy to build new Linux phones.
I want to make it cheap. I want to see more of them. So manufacturers are also super important.
Thankfully, those priorities do not collide.
So I can take care of the users and also maintainers are going to benefit from it.
And again, documentation. I want the users to just go in and solve the problem.
So tutorials and if some problem is not clear, then maybe some detailed explanations, API reference in documentation.
I want all of the above. Contributions via mailing list, smoke signals.
I mean, I like how decentralized they are, but I'm not excited about those. So no. GitHub.
I don't trust Microsoft to gatekeep my project. Codeberg.
Users should focus on building ideas, solving puzzles, remembering. No, no.
A user comes here to use my library with a concrete problem.
They want to solve something here already. I should not give them more problems to solve.
I should make it as easy as possible.
Of course, you can't avoid entirely adding complexity to your users because, yeah, libraries are complicated on their own.
Programming is complicated. Mistakes. Yeah, that gets us to mistakes.
Mistakes should not get segfaults.
I'm going to become a little preachy about it because segfaults don't tell you what happened wrong.
They don't even tell you reliably where it happened wrong.
It might be that you broke some property a lot earlier and then it comes by to you a lot in a completely different place of code.
The only worse thing is quietly ignoring errors.
And, like, we've had automatic memory management for the past 60 years as an industry.
We have the time to learn from it.
So, if you impose manual memory management on your users, I'm judging you. Yeah.
So, ideal error messages are those who tell you what went wrong.
And even then, even better, the error message tells you what to do to fix it.
That would be perfect. And that brings us to the choice of language.
C and C++, they don't get the memory management story right.
Python might be a bit too slow to implement all the processing.
So, Rust is, like, the least bad option of the mainstream options. License, proprietor, of course.
No, that's a joke. MIT, I don't like freeloaders.
So, if you contribute to libobscura, also give back the contribution. LGPL. Governance.
I want to have, like, the creative direction at least in the beginning.
But I want maintainers to come over and take it from me.
I'm not going to stay at every project forever, probably.
So, it's nice to have a healthy community. Supported devices. Only video for Linux devices.
So, I don't think IP counts there. But everything else, like phones, Raspberry Pi, I'm open to it.
And I think that should be, really, maybe even the focus.
What's more important, covering every need or staying simple?
I don't like this idea of, like, the need to generalize things.
But if we stay simple and try to spread a little bit from there, I think the results might be better because we just don't try to put everything in our heads at the same time.
So, let's stay simple and see where that leads. And now the big question.
How will you achieve all of this? Yeah.
I have identified a couple of sources of complexity which are on the slides.
And now I have to find the right slide. Yeah. Presentation mode. Thank you.
And the complexity sources come from different... Well, they're different.
They come from different places.
One of the sources is the same regardless of how you take the picture.
It's just the complexity of taking the picture. So, let's go through that.
I introduce to you my assistant. Hello. And... Right. Let's take a picture.
So, what do I do when I take a picture?
I set the aperture and maybe some focus and click what happens here.
Some light goes in from there and goes through the optics and hits the film in case of film camera.
And that happens...
What happens there is a conversion from the spectrum of light to like three different chemicals which respond to different kinds of light, different spectra of light.
Yeah. And now we have the film. This is like the not avoidable thing.
When you have a digital camera, the sensor also has like three different sensitivities to different colors.
So, this is like the first step of conversion between like the actual light and the actual colors.
And then we have the film. So, we go to the darkroom. And in the darkroom... Thank you.
In the darkroom, we project the film onto some paper and we end up with a picture.
But what happens here?
The film has some photosensitive stuff like three different chemicals that we exposed before.
But the paper has different set of chemicals. So, those are not the same. And what happens here?
I think they are like very carefully selected so that they almost match.
But I don't think they match exactly.
And that means there is a color conversion going on between three different kinds of red, green and blue.
And this is also what is happening when you take a picture with a digital camera like this one.
Because the sensor sensitivity or the sensor three different colors they are different than the three different colors that are being displayed.
Those are like pigments or something. And this is light. Those are not the same.
So, there is an actual color conversion again. And yeah. What kind of a picture did I get?
Let's take a look. Oh no. It's blurred. Let's change the focus a little bit closer. Click.
It's overexposed. No. Okay. Let's change the shutter time. A little bit. Click. And there we go.
The photo is perfect. But what did I just do?
I repeated the same sequence, almost the same, a couple of times. I created a feedback loop.
Because first I evaluated the brightness and the sharpness of the pictures.
And then I repeated the same sequence by changing the parameters of taking the picture until I was happy with it.
So, I performed like a looped operation until I was happy with the result. And yeah.
This is something I'm going to have to implement in LibOscura.
Because not every camera actually has a built-in feedback loop. Yeah. But it's easy.
A feedback loop makes a lot of sense. Yeah. Come on. Oh. So, we also know about the Bayer array.
Of course I know about Bayer array.
It's the thing where the sensor on the camera is actually not capturing every color on every pixel.
There are gaps of colors. And you have to fill in those gaps. You have to interpolate that. Nice.
So, you also know about the color conversions that you're going to have to do. Oh. So, yeah.
So, the sensor that's on the camera doesn't perceive the same colors as the human eye does.
And it means that... Oh, no.
You have to go into color theory and find the color matching function with negative colors, negative light.
Oh, you know everything about that. So, and remember, every sensor is slightly different.
But you're also going to know everything about dead pixel detection, downscaling, upscaling, denoising, tone mapping, YUV output, dark frame subtraction, lens shading, chromatic aberration, lens distortion, focus stacking.
What did I get myself into? Yeah. So, that was the easy part. Those algorithms are all well known.
So, you just have to look at the right paper and you can implement it. Easy.
The difficult part is we have no algorithm that definitely works for creating software.
And one part of it is the abstraction.
So, if you take a look at the camera, well, yeah, this phone has actually two cameras.
And the purpose of the library is to abstract and expose the details of the configuration in such a way that you can write an application once and it always does the right thing.
So, we have to make sure that the camera is always selected correctly.
And that one camera has a flash and one has fixed focus and the other doesn't.
And there are phones which have three cameras on one side.
And this laptop has also an infrared camera.
So, let me show you how it looks like from the Linux driver side.
This is the pipeline for the USB camera.
The pictures go from the top here, from the sensor, all the way down to... from the sensor...
Oh, wait. That didn't work. From the sensor here all the way down to the video device.
And there's another video device which is actually connected to this one but you don't see it on this diagram.
So, you have a different camera. And this is actually the one on this laptop.
There is one, two, three sensors. One, two, three, four camera devices.
But this one doesn't have any sensor. And what is this?
This is actually the motor that controls the focus. So, you have to also take care of that.
And this one, this doesn't have any sensors.
You feed the picture into those camera devices and you get pictures out from different camera devices.
So, this is entirely a picture processing pipeline. And let's take a look at this monster. Yeah.
You can connect up to four cameras in any way you want.
And a good library should abstract all of this. Right.
Including the information that is actually not visible on this diagram because there is a bunch.
So, let's go back to... Let's go back to the presentation. Okay.
And there is the last category which is the actual programming.
The abstraction of the video for Linux, the choice of the language, the choice of the architecture, the way that we handle buffers, for example.
Those things are only... Can only be figured out by more coding.
And this is what I've been doing for the past couple of months. And the result is LibObscura demo.
As you can see, I already ran it.
It's going to be running on this camera because I've been busy preparing the presentation instead of adding support for this one.
So, there we go. The moment for initialization. And we have something. So, let's mess with this.
Change the contrast. Yeah. Maybe saturation. That looks crazy. Yep.
So, what you're seeing is the support for controls. Let's crash it, maybe.
No, it doesn't crash anymore. Nice. It does. So, this shows the support for controls.
This also shows the GPU processing that I implemented.
It converts the YUV output into RGB because our displays typically show RGB colors and cameras, for some reason, produce YUV colors.
Color conversion, as I mentioned before. Yeah. So, that's where we are with LibObscura right now.
It only supports... Thank you. Right now, it only supports USB cameras.
And it has some examples and documentation, but I'm under no impression that it's good.
I tried my best, but I'm not the best person to write documentation, but I'm trying hard.
The architecture is actually split into two pieces with a lot of supporting libraries.
VD is the actual camera library. It actually sort of manages everything about video for Linux.
And CrispyCam is the one which contains the GPU processing algorithms. But what do we want next?
I want to add support for the Librem 5, which is my only smartphone.
Also, for this laptop, because I think it's cool to have a laptop with a nice camera.
Better documentations, of course. And a scripting engine. Okay, this is a funny one.
It's kind of an experiment, again, with an experiment. What kind of a scripting...
What do you need to script a camera library for? And, like, what kind of a language would I use?
Any guesses? Lua, Python? No. That's easy. I wanted to try with Prolog.
And why did I try with Prolog? See? That's the thing. I want everything to be declarative.
I really like declarative things.
So, remember all those different configurations of video for Linux? Video for Linux things?
Yeah, this one. I want to... Not have to write all those algorithms myself.
I want the device manufacturer or, like, the distro maintainer, maybe, to describe what each of those devices does.
Or each of those nodes over here does.
And I also want, on the other side, to describe what kind of an output I want. So, for...
In this case, I'm just going to go a little bit deeper. And...
If you see this device right here, it has a certain resolution. It has a certain color properties.
And those nodes might change it in some way.
It might, I don't know, for example, rescale it or something.
And then those devices are also with their own properties, different resolutions.
Maybe they don't support everything that is happening on the top. I want all this...
All this complexity, finding the right pipeline from the top to the bottom, to be automated by language, which is kind of easy to change.
Maybe I want to add some metadata. I don't want to, like, write everything like this in Rust.
Rust was not meant for this. I want to query, what are my options, right?
And this is already implemented in selecting the resolution for a single camera.
It's a very basic concept. I don't know. Maybe it works. Maybe it doesn't. And... And, yes.
That would be... That would be the end of the presentation. But I wanted to...
I wanted to make note that this is just the tip of the iceberg.
And there were many people involved in this.
There were people who provided all the libraries that I forged or I contributed to.
There were people who gave me feedback. There were people who wanted to test their cameras.
If you want to test the cameras, if you want to contribute to the documentation, if you want to report bugs, and if you want to help me find bugs again, like the one that I just found today, or maybe you're arrogant enough to actually give it a try and write some more interesting algorithms, yeah, please talk to me.
Thank you. Thank you very much. It was a really interesting talk.
You now have the chance to go to the microphones and ask questions.
Do we have questions from the Internet? No questions from the Internet. Okay.
Then we go to this microphone here in front, please. Hi.
Thanks for taking on the task to try to implement this from scratch.
My question is, why should I prefer this over something like Video Capture API in OpenCV?
I don't think OpenCV is going to expose everything like this.
I think OpenCV is meant to be extremely simple, and I'm not even sure if you have three cameras on your phone if you can select the one that you prefer.
I expect there is a bit more generality in what I'm doing. Okay. Thank you very much.
Then we take a question from this microphone, please. Hello. Hi. Thanks for the great talk.
Since you said the focus should be on the user, and the user would, at the end of the day, be what exactly?
The developer of something that does something with a camera.
How easy or do you have any simple examples?
How do I use this in a, I don't know, GStreamer pipeline, let's say?
I don't know about GStreamer, but I can show you an example.
I wrote quite a bunch of examples, actually. How do I escape this presentation mode? Really?
Oh, there we go. So I'm going to show you a couple of examples.
And basically the examples, I tried to make them as simple as possible.
So they are organized very well in a separate crate. And here's the very super basic example.
Basically fits on one screen. It's pretty much ten lines of code from here to here.
I don't know if you can see it. I don't know.
I think it would be nice to have a set of bindings to C so you could integrate it into GStreamer.
But I don't think I have the time to actually implement it.
I think there are more important things to do right now.
I was thinking about generating the bindings, but I don't know what the situation exactly is with Rust and C bindings.
I know that there are software for this, but I don't know how good they are. Thank you.
Then the next question from the microphone in front of that side. Yeah, hi.
So again, thank you for the great work, especially the state of the drivers, especially on phone from Qualcomm, the entire USB drivers.
It's very bad. And it's good that you have shown this.
My question would be that the lib cameras tag the people who are, let's say, doing it for Raspberry Pi and Rock chip in the Linux world.
They also had this kind of a problem that either the drivers were not available, let's say, or people wanted to hide the ISP information from some proprietary chip they are using.
So they created a concept called soft ISP.
And this soft ISP enumerates all of these sub-dev V4L2 devices in the user space.
And then they interact with them. And the below info.
So all the processing happens in the user space rather than the kernel.
So maybe you can also, a suggestion would be also look at their part, how they are interacting.
And maybe it can be merged or have a binding with lib cameras so that lib camera apps and all, they already work with FFmpeg and let's say other things like streamer and all.
So that would help you and get this project also mainline and present in normal distros.
Yeah, thanks.
I actually want to implement things separately for the purpose that this is an experiment that tries to explore what is possible and like find better ways of doing the same thing.
So try to not rely on projects that do the same thing already.
And I mean, the moment I start using something else, I'm going to have requests.
And I want to try to avoid it and keep this small and fast so that I can develop in the direction as fast as possible and discover new ideas.
So doing exactly the same as someone else does is maybe educational, but I would not try to do it here.
Thank you. Then the question at the back. Hi.
I heard that your project is funded by this prototype fund, which sounds very interesting.
I've not heard of this fund before, so thanks for that as well.
And my question would be, do you think it is enough for your project to gain enough traction to go on even after this fund ends?
I don't know.
So the point of prototype fund is, well, maybe not the point of prototype fund, but the point of my project in prototype fund is that I experiment.
If this gains enough traction, then it's going to continue.
But if it doesn't gain enough traction, and people decide, yeah, whatever, then I don't want to waste my life on it.
All right. Then we have a question here on this side.
Is there any relation to the Stanford University project? No. I never heard of it.
I want to hear about it. Maybe you have the chance afterwards to talk about it.
Let me take a question here from the front. Hi. Yeah, thanks for the talk.
So my question is quite pragmatic.
So like with the modern smartphones, you get whatever Android, proprietary Android, and do a picture.
And the picture is really good.
And then you try to put Linux there, and you manage to make camera working.
The stuff that you're getting from it is really colors oblique. It's really noisy.
Is it something that is like what sort of extra processing is doing there? Any idea? Yeah.
So one thing that we had, the problem with this phone was that even after we created the library, the application, the colors were kind of crappy.
And the reason for it is we have not created a very good color profile.
I think just creating color profile was too difficult for most of the team.
And we just kind of tried to avoid it, maybe.
But there was also the effort from Sebastian who was also working with me.
He managed to load the videos in.
He was able to load them into raw therapy or something and make them look really nice.
So I think, yes, there is something missing.
I'm not the right person to tell you what exactly is missing.
But I have also here an example of couple things which are not strictly necessary to get a picture out of your camera.
But it's kind of expected. You have to take account into the bad optics of most phone cameras.
And then you have like denoising to just basically make up for the bad sensor, chromatic operation to make up for the bad lenses, focus stacking to do some extra tricks.
So there is plenty of processing that you can do optionally.
But I am currently not an expert at it yet. Thank you. All right.
Then the question at the back, please. First of all, thank you for starting this project.
I think it has a lot of potential.
So especially if it becomes a bit like lip camera, and my first question is, why do you currently only want to support USB cameras?
Do you also plan on supporting my iPad cameras? No, no.
I only want to support video for Linux cameras. And USB cameras are one of those.
I mean, I already kind of support USB cameras, except for this weird crash.
But I want to support any kind of video for Linux camera as long as it's not like extremely complicated and no one uses it.
So that's the scope. I don't want to support like network cameras maybe.
I just want to keep it small for now.
So everything in video for Linux, like MIPI also, like stuff you find on a smartphone. Yeah.
So like this one is not a USB camera. And that's why I even got interested in the idea.
And another point.
So as an assertive goal of the project is to implement the whole image signal processing pipeline and mainly focus on the user space part, right?
So Crispy Image does this via GPU acceleration.
And like the traditional approach of using the fixed function or hardware block elements, which are usually a proprietary mess.
Okay.
So I want to support the hardware blocks when they are there because it's super important for the phone.
If you run like the camera and it drains your battery instantly because you're doing it in an inefficient way, then it's bad.
Therefore, you need to choose the hardware whenever it's available because the hardware is optimized especially for this function.
So this is another source of complexity where the library has to find out what is available and choose the best option for the image processing.
It's totally in scope. Okay. That's great to hear. And another question.
Are you sure about the Prolog approach? Because I think it's an interesting...
I think it's a really interesting approach to this idea of handling the complexity of various configurations.
But I'd like to question if hardware is really as dynamic as you'd assume.
So for example, effectively from what I've understood on normal SOCs, you have a specific camera, a specific ISP pipeline, and you tune it separately for each pipeline and camera sensor combination.
So do you really think this auto-plugging approach is optimal? I don't know if it's optimal.
I just kind of want to find out and I found it cool.
So perhaps the winning solution is indeed to have just a couple of predefined pipelines and also kind of...
That was actually the original idea, but then I kind of got into this Prolog idea because I wanted to let the user choose the resolution that they want to operate at.
And yeah.
This is something that just enumerates the configs and you can put in some Prolog, but I'm not sure if that's...
Okay, maybe I'm not going to show it.
Okay, but this is certainly an interesting research approach. And my final question.
Do you have any kind of ISP tuning application?
So you said that you fixed the images for the Purism Librem 5 phone.
Do you have an application for tuning the sensor parameters? Do you even do that?
What do you mean by sensor parameters? So I'm not really a complete expert on embedded...
I'm not an expert on embedded systems, but from what I've understood is if you have a normal dev board, you have a MyPi camera which you attach to it, then you have to fix function ISP pipeline and you tune various parameters for it.
For example, how fast the autofocus focuses the gains for various flashes or something like that.
Oh, yeah. Do you have an application to tune that for the soft ISP? No, not really.
So the way that we did it was just try what works and we also were implementing all of that in software.
So basically, yeah, we just do whatever we do. There is like the...
I don't know if it's like strictly tuning of the ISP.
There is a tuning of the sensor where you have to do the measurements of like what the dead pixels are, what the exact sensor colors are, what the lens changing looks like, so you have to kind of have those parameters, know those parameters for the sensor, but I don't think we got far enough to actually properly care of that yet, at least at Purism.
So you have some tuning parameters you can define separately. That is in the future. Yes.
Thank you for this deep interest. We have more questions, so please this microphone. Yeah. Hello.
I'm doing this stuff as my daily business. I know the pain.
In summer I was at the Linux plumber conference and there was a working group was founded, the open camera working group.
I don't know if you know from that, but basically people from Google, from Video for Linux,
from Intel and so on want to move the hardware ISP stuff into the user space
because most ISP developers or manufacturers can't open source their stuff
because they buy these hardware algorithms from one or two companies worldwide.
And so the idea is that every ISP manufacturer needs to support
the basic functionality in open source and the special functionality...
Is there a question coming? User space. Yeah, sorry. And the question is,
because you want to support hardware ISP in the future, did you plan something similar
because there is not really anything open source? So as far as I know, the open source question,
the question becomes relevant. I mean, I'm not sure if this is what you mean.
I had problems with adding support for the sensor because a lot of the registers in the sensor
were undocumented. Is that what you meant? No, it's the part after the sensor. Yes.
Between sensor and CPU, basically. So they also have some registers.
I expect this to be taken care of by the kernel. I am not a very good kernel developer,
so I kind of try not to think about it and just do whatever I can do with whatever is available.
So I don't really want to... Well, okay, if the sensor... Okay, what do you mean by open source?
If the driver is open source, then it's fine. If it exposes the right interfaces,
then there is really not much else to do in terms of code support. So we'll just...
The main problem is that most drivers aren't open source. Yes. I mean,
if they expose the right interface, I would not put a special effort into supporting them
if the driver is not open source. But if they expose the normal video
for Linux interface, then why not? Okay, thank you. Okay, we have time for one last question.
This microphone, please. Please do a short one. You mentioned that simplicity
is one of your design goals. Simplicity usually means that you make decisions
not implementing a feature so that you get a simple interface. How do you make the decisions?
Do you have a guideline so that they get consistent? Because you need to make the decision,
I think, at least once a day. Yeah, so for now, there are not many contributors.
So I just go by my own feeling. I don't have to consult with anyone.
And I'm not really going very... in a structured way. But if people start contributing,
I'm going to have to develop some guidelines for myself and for the team. Thank you.