Hi everyone. Welcome to my talk. Thank you for joining me today. And the next about 20 minutes, I'll be speaking about a project we were working on for a couple of months now.
Just a short disclaimer, the project is still going on, so I will not be naming specific things. I will leave out some specific details, but I will be talking about how we did that.
And I think this is the most important and the most interesting part of the whole story.
So this is a real project as I said, and this makes it so interesting for me, I think. I think also for the community and for everyone working on Android.
Understanding the implications of devices being outdated or being updated on regular basis is super important here.
So my name is Igor. I am working with Android for 15 years now, I think.
I was one of the common trainers for Android, one of the original Android distributions for Raspberry Pi.
Many years ago, I think it's eight, nine years ago, and about seven years ago, I started a new company in Syria, where we basically help companies build new products with Android.
It doesn't matter which type of products, automotive, dashboard, IoT, whatever. If there is a screen, if there is some reason to have Android on it, and the company doesn't have the resources to do it themselves, we help.
That's all my thing, and by help, I mean, we're doing initial onboarding, we do audits, we do automatic builds for overday updates, we have an infrastructure for distributing updates for remote management.
For many, many different things. And here, the main idea, as I said, to keep the devices up to date, but this is really a hard thing to do, because many manufacturers are not really very interested in keeping the software up to date, because they make money by selling hardware.
So, if you are a hard manufacturer, and you want to sell as much devices as possible, and also new CPUs and new system on chips, you really don't focus on keeping up with all that speed of the Google pushing the Android releases.
So, you stay low, and you give your customers whatever they want for the initial release, and that's it.
But, obviously, this is very important, right? So, we want the devices to stay secure.
We want to know what we want to be compliant with all the legal obligations. Many of you probably know that, I think last year, the cybersecurity resilience act came into power,
meaning that in the next year's companies were building digital products will have to provide updates to their clients.
So, there are some regulatory and legal implications for that, but also there are economic and environmental benefits of keeping devices up to date instead of throwing them away and building new ones.
Actually, while I was working on this presentation, I learned that building a single smartphone generates something between 30 and 70 kilograms of carbon dioxide, meaning that if we are talking about the fleet of devices of several thousand, we can save what 50,000, like 50 ton of carbon dioxide just by one project.
It's kind of a cool thing to know, I didn't know that when we started this project, but I know now.
So, we were approached by a company who has a fleet of devices. They purchased those devices from a manufacturer directly, and many years ago, and they were working fine and everything was well.
But today, they can be considered outdated. So, the manufacture of those devices basically refused to provide any types of updates, any types of upgrades.
So, the position our clients were in is either to drop them and purchase new ones, somewhere else, or to try to upgrade them to something newer.
And we were talking about several thousands of devices. So, it's not just one or two or three. Maybe this little bit too much work for just keeping your own smartphone up to date.
But I think this is the main story behind lineage. So, a community helping each other to maintain a set of devices up to date.
But here, we were in the position where it was a dedicated custom build device, and the question was, can we actually upgrade it?
So, the project though was, can we move it from Android 9 to something newer, whatever it can be, and by this also reduce the dependency on the original manufacturer.
So, what was the device? We got one sample in our office to take a look at, and we learned that there was no schematic available for us.
Actually, it would be available, but we would need to sign a ton of NDAs, and we were not interested about that. So, we said, no, thank you.
What we knew is that the UART is available, so we can get actually bootbox from the device, and we knew that we could be able to unlock the bootloader and use the fast boot to flash.
That's already a pretty good start, I would say. So, it's better than jail-breaking and routing most of other smartphones on the market.
When we got the device, we learned that it's quite popular SOC from I think 2018, which was used quite widely in the market for different types of smartphones as well.
The device we had had also a lot of peripherals and many different cool things in it, and it was a pretty cool device to be honest.
So, I kind of like the idea of making sure it can run something newer than Android 9.
By the way, can you hear me well in the back of the room? Thank you.
So, this is the hardware part, then we took a look into the software part, and well, it was Android 9, you already know that.
So, we learned that it was using Linux kernel version 4.9, which was okay back then.
But, and we also, we were told that we will get access to the BSP, and then we got access, partially at least, and it was basically a single zip file.
No history, no repair manifests, no information about what was changed and why.
No support from ODM, so the original device manufacturer said, nah, to whatever you want, but we are not helping you.
But, for us, it was super important that we got an expert on the site, like on the site of our client, who was working with this device for many years,
who knew a lot about the kernel, who knew a lot about the hardware, so that was a huge win for us.
Otherwise, it would take much, much longer, I think.
So, we said yes, we said yes, we would like to try it, and we started the research.
We started to, you know, by understanding what can we do, and is there anything on the market on the internet available?
And obviously, there is a lot of BSPs for different smartphones, for different tablets, many open source projects.
And we started looking through all of them, like everything we knew about for pin compatible or compatible socks and for similar kernel versions.
Basically, we were scanning all Wikipedia pages, we were reading all articles, we were crawling through the forums and looking for unofficial and official builds for this type of the CPU.
And at the end, we have compiled a list, we have compiled a pretty long list of everything we found.
So, it was the list contained, like, a name of the device, which is, you know, the dedicated name of the project.
The SOC, it was using the kernel Android versions, and whether it was actually maintained or not, if it was outdated already.
And then, while we were evaluating those results and scrolling through this document, we noticed one of them, which was a perfect match for us.
It basically had exactly the same kernel version, it had Android 14, which when we started the project was the most recent one, I would say.
I had similar peripherals, small S, not really exact, but pretty good.
Match, and it was officially maintained by a lineage community, and it was actively, actively underdevelopment.
So, if we were successful on this project, it would mean that, rightfully, in future, we could upgrade even further and go beyond Android 14.
So, we downloaded the BSP from lineage, we made sure that we can compile it and everything works, and then we thought, like, how do we approach this?
So, there are many, many ways. I will present some of them. Obviously, there are more possibilities, and some of them will work in different projects.
I will share some of our experiences, why would it some things? But, if you will be in this position, I'll be in future, I'll be happy to hear your opinions, I'll be happy to hear how you approach such things.
If you already did that in the past, just to learn how other people approached this.
So, for us, the very first step was, let's make sure that we can build a kernel. So, we had the kernel from the original BSP, and we knew that it was compatible with original BSP, but it was not compatible with lineage.
The problem was that in lineage, they had different defines, they had different make files, they had different tool chains, and there were two ways for us to go.
Either we can replace the lineage kernel with a kernel of the original BSP, or we can keep the kernel which was used in lineage, but then patch all the device to information, all the configuration parameters, everything into that kernel to make sure it works.
We tried to do the first one, we failed terribly to be honest, so it was really, really difficult, because inside the lineage kernel, we found many dozens, I would say, of commits, regarding tool chains, regarding specific things, which lineage was using.
So, backporting all of them into the original kernel was kind of stupid, so we did it the other way around.
We migrated the device tree, we migrated the kernel config, we migrated some of the configuration, not all of it, because our strategy was, let's keep the kernel as small as possible.
Let's remove all the stuff we don't need for the initial boot, and make sure that we can get the system to visualize somehow.
So, we disabled audio, we disabled touch, we disabled networking, we disabled everything we could disable, hoping that it will give us a smooth boot for the first time.
We needed to add some Android specific stuff, like you probably know some of you probably know there is this firmware, entry in the device tree where you can specify the amounts, the early amounts for the participants, so we had to add that, and once it was compiling, we thought of moving to the next step.
And here in this step, I would say that without the expert, without the guy I was mentioning in the beginning, working for the company who hired us, without knowing all the details of the kernel, I think we would need months more work.
So it was, we did that, and I say, I'd say maybe one month of work, I think without him, we would need at least five.
If you don't know the kernel, if you don't know the hardware, if you don't know anything about the device you're working with, it's really, really hard to go like this.
But we managed that part, and we thought, okay, so the kernel is compatible, we don't know yet if it boots, but it's compatible, so let's go to the next step, and the next goal is to actually make Android boot into the system UI.
And here we did the similar approach, so the first step was, we went into the device 3, like the device config of Android, and we tried to make it as similar as in the original BSP, right?
We made the partition layout compatible, we removed all the features we didn't need, we changed the configuration for the, for the halls, similar, and added many, many different tweaks to basically reduce the size of things which will be, which will be compiled and used by the boot during the boot, in order to avoid many of those things we are not interested in the beginning.
So our main goal was to make it boot, and this was standard debugging cycle, right? So compile, flash, boot, check the logs, figure out what is, what is crashing, so if there is a freeze or boot loop or something, figure out what is the part which is not working, and then figure out why.
And usually the reason was that the configuration inside the lunch was different from the configuration inside the original BSP, so our goal, I would say, or our move was to go back to see if we can change something, we can reuse something from the original BSP, understand the difference, and this is how we proceed.
In the end, so for the booting device, for the very first time from the kernel until the UI, we had to swap two things, the first was the graphical stack, so there was some property drivers, some property blocks, we needed to reuse from the original BSP, and there was a USB gadget configuration thing, which was crashing the whole system.
I don't know why, but then again, we swapped back with what we had from the original BSP and the devices booted up.
I know it was pretty cool, experience I would say, it was the first time we did that, this type of project, so that was pretty cool.
But we knew it's also more, let's just the beginning, right? Because now we are inside Android, so Android is booting we see the UI, but as I said, we disabled all the drivers, so nothing was working properly.
Here we started to be adding things back from the beginning, so we started with touch, because if you have a smartphone or a device, it's easier to work with if you can just scroll around or touch it.
The USB ADB was not working, as I said, we disabled the whole in the beginning, so we were going step by step, and every single time it's okay, check the logs, try to find an arrow in this specific configuration, try to understand what Linux has is using in this specific thing, figure out what was the configuration original BSP.
Remember that we disabled some kernel drivers, we did not port some device three parts, so we have migrated them as well, step by step for every single feature step by step.
And when the device was, I would say in a good condition, so most of the things we are working, we started finalizing and cleaning up stuff.
Enabling as Linux again, try to figure out how to make Android-35 boot work, remove all the things we didn't need, all the clients didn't need, it's not us.
Figure out how to sign it with a client keys to be compatible to the old device, to make sure that we can roll out this whole thing as an over-day update and future, integrate all over-day updates, logic and clients we have.
And that's basically the main story until the device had Android 9 and the device had Android 14.
I don't have too many things left, I would rather prefer a little bit of a discussion at the end, I have one more slide.
I would consider this project a success, but it's far from being complete now.
As I said, there's a whole bunch of peripherals we're still working on, so the device had multiple cameras, the device had a fingerprint sensor, and we are migrating those whole step by step.
We will continue working with Linux, it was a fantastic support.
I think without Linux it would not be possible for us to do this at least in this time, so maybe it would be possible if we would spend the next couple of years on that.
But this was not going to happen.
And I can say that there is a huge dependency on knowledge and availability of source code for projects like this for keeping devices up to date and maintainable.
I think the availability of source code is a huge criterion for device longevity.
So if you want to make sure the devices are actually usable, many years after they were manufactured, we have to open source the most important part of the device.
It was difficult without the support of the original device manufacturer, it was really difficult.
As I said, we had a lot of help from the expert.
Without it, I'm not sure if it would be successful in that time, maybe it would take much longer, probably it would take much longer.
I think if the device is persistent, if you have a goal, if you know what you want, you can still do it.
I mean there are examples of jail breaking smartphones and keeping smartphones up to date, which we are not meant to work like that.
For developers, we didn't do a lot of reusing existing USB but had to implement everything from scratch.
So this is my call to action to everyone here and online and in the community, please try to contribute.
We are contributing, we are a commercial company but we are contributing where we are possible.
We are not violating any in the A's and we are not violating any other agreements with our clients.
We will be doing so in the future as well, we will continue to do so.
And I would love to see more stories like this to learn how people are actually upgrading devices, how we can keep the community rolling.
And I wonder how many million devices could be actually saved from going to waste and being replaced if there would be an open source implementation, if there would be an repository where you can just download your code to some adjustments and push it.
So that's the story for me. Thank you very much for your attention and I'm helping for questions.
Two questions one is just for this project.
We did it.
So how many people worked on that as a whole time?
We are a small company. We only have, thank you. So I'm being asked to repeat the questions.
So the question was how many people are working for us and on this project.
So we are a small company.
We have, I would say, ten people in total but in many different areas, backhand fronthand apps on the A's P level, there are just three people who can work full time including myself.
But since I'm also involved in the organizational part, I'm usually not working a lot on the low level stuff.
So I would say it's, it's the expert I was talking about. He was doing a lot of work also hands on development work.
It's one full time employee on our side and maybe one or two people jumping on and off all the time.
And the second question is, do you expect to change?
So with the upcoming policies that the vendor can obtain more bets, your problem, knife, or that even has to do this stuff.
The question is, if I expect any change in this policies or vendors where they say,
we will not provide any updates and the question was if I can expect any change in that.
I hope so, I hope so very much.
I still see a lot of product builders who say security is super important for us but we have no budget for it.
And the policy is that doesn't matter.
Yeah, but they say we will take care of that when it's there.
But if I mean what people don't understand today a lot is that if you don't embed some kind of capability of installing updates later on,
there will be no way of updating it even if it will be necessary.
So we try to educate our clients as well.
So using our clients comes to us when it's either too late or something is going really wrong.
Not all of them.
The companies who think five years in advance, that's really nice to work with them.
But by far the most come to us and say, oh, we actually didn't think about this part about security over the other things.
Can we fix it now and then they ask us how can you retrofit such things and that's not possible.
So I hope Europe is moving into the right direction or I think Europe is moving into the right direction.
I hope Asian manufacturers will follow, I hope they will be forced by that.
But it's it's a difficult position to negotiate, I think.
So because if you have product builder, a small product builder in Europe and you go to Asia, China, Taiwan, India and you say, I want updates.
How the manufacturer will say, well, yeah, maybe maybe it will cost you a million, maybe two, maybe five, we don't know today.
So this is the position where they are and I think building up tools, building up tools like for automatic compilation,
automatic rollout will support that a lot.
And forcing standards as well.
Yes, please.
The question was, if porting the drivers can be supported by enforcing GPL,
this is something I would like to believe in, but the reality is you ask the manufacturers and they say,
yeah, we know about this but we are not doing this.
So and if you go to China and try to sue them, you will not be successful.
So I think the importance here is to do it in advance.
So when you sign a contract for a new product,
we have to think about such things as advance and get the access to the BSP early on, so they won't tell it to you later on.
Yes, please.
Excuse me?
Not the vendor, but our client, the company who hired us, they had a copy of the BSP, they had a copy of the device three.
It was almost complete, I would say.
So there was a lot of stuff in it.
They opened sourcing part of it, yes.
I tried to summarize the question.
If our time is up, I will be available here if you have any further questions.
So we can continue to discussion later on.
The last question was, if what we will do in future,
so if the road is closed for us now, what we did already and it will not be open source fully,
I can say that the goal was to reuse the lineage BSP, so there are used to lineage device things,
and we will try to contribute there as much as possible to be compatible with our device.
This is the goal, so we can rebuild it in future from there.
For the other projects, again, I think it's a question of getting the BSP as early as possible.
The problem is that the device three source is not in the air.
Yeah.
The device three source is not GPL, that's one of the biggest problems, yes.
Thank you very much.
Can I take another question?
Yes.
Can I take another question?
No, it's not for phone calls, I'm not sure I can talk about this, but it's not for no.
So there is no real, so it's a B2B device, so it's a device used by a person for specific task,
but not for consumer grade device.
Thank you very much.