All right, can everyone hear me?
Well, thank you so much for coming.
I know this is the last talk of this day,
or like at least within this bedroom.
So I imagine some of you may be quite tired,
so just to get their spirits out for the last time today.
Can we get a big yay from the audience?
There we go.
That's what we're looking for.
Well, with this said, there might be a small surprise for some of you.
You might have explained a slightly different title.
But today we're here to talk about
Fluorite, the console grade game engine in Flutter.
I'm here on behalf of Toyota Connected,
presenting this presenting this talk.
My name is Jamie Kerber.
I'm the consulting senior engineer at very good ventures
and lead engineer on the Fluorite project.
This talk has been prepared in coordination with Joel Winnarske
at the principal engineer's three at Toyota Connected North America
and lead and founder of the Fluorite project.
He's very sad he's going to be here with us today,
so he asked me to share a short video message for you all.
Well, thank you for attending our talk.
My name is Joel Winnarske.
Can you hear that?
I work for Toyota Connected North America
and I have a lot to do with that.
I have a lot to do with that.
I'm not able to...
Hello.
Welcome.
What's the name?
I...
One, two, one.
It's the thing you're making.
Yes, what do you mix up?
And thank you for attending our talk.
My name is Joel Winnarske.
I work for Toyota Connected North America.
And I am responsible for embedded Flutter.
Hello.
Welcome.
And thank you for attending our talk.
My name is Joel Winnarske.
I work for Toyota Connected North America.
And I am responsible for embedded Flutter Innovations to Toyota.
You may have noticed it changed the name of the talk.
Unfortunately, I am not able to attend Phasam this year.
Instead of canceling, I made the decision to share another innovation we are working on.
Let me introduce Jamie Kerber.
My colleague from Sweden.
Jamie?
Take it away.
Well, thank you very much, Joel.
There's a wonderful introduction.
With this said, let's just get into it.
So, where are we today?
What is Toyota Connected State of Flutter on embedded platforms at this point in time?
Well, Toyota is responsible for the development of the IVA home screen,
which is the Flutter Embedder for embedded targets running Linux with a wheel and interface,
which is a huge part of the Meta Flutter project on the Octo Linux,
which enables all kinds of embedded and automotive Linux solutions.
As a real-life production use case, we have Toyota RAV for 2026 model,
which has been announced very recently, and it just so happens to be running this precise stack.
So, that's production great for you.
We're all very excited for some of you to try this product very soon.
If not, well, perhaps next time.
The downside of that solution, however, is that just like most other Flutter applications,
only supports 2D static interfaces, which kind of limits the expressability when it comes to designing UI and UI interfaces.
So, we were kind of thinking, how do we want to solve this problem?
And we realized that the next obvious step for digital cockpit innovation are game engines.
So, sounds great, rather, just like take a big old game engine, slap it in and see what happens, right?
There's a lot of very fun use cases for it.
For example, let's say that in your new product, you might want to integrate an interactive user manual,
which might show, for example, how to use certain features of the vehicle,
in a step-by-step basis, providing very clear instructions using 3D illustrations to your users.
Another one could be hardware-state visualization.
So, let's say that you can see on your screen a 3D model of the vehicle you find yourself in,
and let's say that you might want to notice whenever one of the lights is broken.
With this, we will be able to pinpoint exactly which one it is, making it very easy to enable,
for example, self-service scenarios, pointing the user exactly to where the light bulb is,
and again, tying back to the previous step-by-step tutorial feature,
how to take care of such a user's case.
Another one is environmental mapping.
Let's say that you find yourself in a particularly tricky environment,
with a lot of obstacles, and you want to be able to avoid them swiftly.
With this, we're able to, if we have a 3D mapping available of the Obsident
environment, projected on your screen inside, allowing you to maneuver more accurately in avoiding those obstacles.
On top of that, we can enable way more natural controls for, let's say, things such as heating,
or like wipers, or any kind of functionality in a vehicle, by providing 3D objects that looks similar to the real-life counterpart,
making it a way more intuitive experience to control such settings.
Last but not least, there are already a number of other manufacturers on the market that do provide similar solutions.
Thanks to Fluorite Integrated and Flutter, we will be able to enable all of you to compete on that market by providing equal,
if not even better solutions.
However, please note that all of the above are just technical speculation in my part,
and there's not any way we'll sort of reflect the roadmap of any existing product.
With this said, what would be the challenges of integrating a game engine in a Flutter application in an embedded scenario?
Well, we actually tried that, and it turns out that all of the existing game engines that we tried were not adequate for this type of solution.
Based on our internal investigation, the main problems with Hinti and Oran Real were that they are a close source software.
They require proprietary blobs to be shipped as part of that living distribution,
which unfortunately means that we would lose on young to compatibility, which is a massive chunk of our use cases.
Not only that, but whenever you integrate a native view of this type in a Flutter application,
for every single one of them, you have to spin up a separate instance of an entire engine,
which is very, very heavy on resources, and in our experiments has led to very low frame rate and poor performance.
On top of that, count have the licensing fees that might go as high as multiple millions of dollars per year.
That was a bit of a non-starter, so we tried with something more promising, a real app and riser,
that being good though, an open source game engine, that has lately seen quite a bit of adoption.
In that case, when testing on the Raspberry Pi 5, which is one of the more groundbreaking and better platforms,
we notice an unfortunately long start of time of 20 seconds in above.
That's how it was unacceptable.
But in further testing, we found it was also resource heavy in a similar way to Hinti and Real, at least for our use cases.
So we tried to go with something a bit closer to the Flutter Core.
Using the impeller render and the Flutter GPU package, one should be able to integrate some sort of 3D visualization in your Flutter app.
However, we found that while it is stable on iOS, it unfortunately is unstable on Android and flat out unavailable on Linux, macOS and Windows.
So with now being able to learn to Linux, that's on our solution that we have found unavailable to us.
And also to that you have to count the immature API of that solution that is in constant improvement, but was just not moving fast enough for us.
So what is the solution that we came up with?
Let's start from the technology we already had on hand.
First and foremost, we have Flutter running on embedded platforms.
Thanks to the IBA home screen and Meta Flutter project.
Second, we're working on a Flutter SDL3 embedded, which enables cross-platform IO across multiple types of embedded devices and platforms, as well as operating systems.
To that, the Google's filament 3D rendering engine, which allows you to render really high quality 3D environments using physically-based rendering and anything that entails.
So how do we put those 3 together?
We got Flurite.
Flurite is a game engine fully integrated in Flutter, because it's simply because it is a package that you integrate in your existing Flutter applications, meaning that you can greatly reduce the complexity of merging a game application and a Flutter UI application, since you code both in the same language using Dart, as well as any other tooling.com with Flutter.
Thanks to this, you can share code between UI and game logic.
This is because Flurite view is just an order of Flutter widget that you can integrate anywhere in your applications.
This way it benefits from the Flutter layout system.
You can place multiple of them at the same time for multibus scenarios.
And this also means that they can share state between each other.
You can have game objects and UI widgets being part of the same context.
Can use any of the existing Flutter solutions for a state management, such as provider, river pod, or block to talk to each other in real time.
Let's go through a short showcase of features that we have already developed for Flurite.
Well, the main one is the high-performance ECS core.
In order to stay within the realm of the well-known style of game engine APIs, we decided to base everything on top of our own entity component system.
With this, we implemented the main core in C++, which allows us to drive really many new optimizations for low-end and embedded hardware by controlling memory and processing in a very very tightly integrated way.
However, all of this is presented to you.
The developer has a very clean and high-level dart API, which is very closely mimics.
The same APIs you might already know and love from other game engines such as Unity, Unreal, GoDale, and many others.
As a part of that, we provide a hierarchical scene graph just like in all of those other game engines,
allowing you to provide complex and nested objects in your scenes.
This means that all of the knowledge and skills that you have acquired while developing four older game engines is 100% transferable.
Plus and minus some API minutia, but here in this picture, you can see a working example taken from one of the applications you're going to see later in the presentation.
It showcases a simple bouncing ball.
So we have an AD, we have a name for it, and we have a list of components, which might sound familiar.
You have a transform with position scale rotation, you have a collider, you have a model, a renderable, and a behavior script with such a functions as uncreate on a blade frame and similar.
One feature, particularly proud of, are the model defined touch trigger zones.
This feature enables your 3D artists and your game developers to work in parallel by letting the artists define which areas of the model are clickable,
and for the developer to later connect that event to some sort of interaction with your application.
In this case, you can see the wheel being tapped in that increasing the tire pressure.
At the same time, though, you can see that there is a synchronization between the game state and the flutter application,
because you can change the sliders and that regulates the size of the various tire pressure indicators on the set car.
What you're seeing right here is an application that we have developed in flow right.
It showcases a simple 3D scene with a lot of high quality assets, as well as the aforementioned model defined that trigger zones.
And of course, we want to present console grade 3D rendering.
Google's filament enables us to provide high quality workflows, which are optimized for embedded patterns.
We also allow you to use a variety of high quality, physically-based assets, and lighting systems.
But of course, that is just one of the many use cases.
One interesting we provide is a customizable shader and rendering pipeline, which can allow you to, excuse me,
to design any sort of more stylized performance in accordance with your designer's own vision.
And of course, it wouldn't be flutter if we didn't have hot reload.
As you can see in this video, I'm simply typing a different value for, like the orbit distance for the camera and pressing save,
and in the matter of less than a fraction of a second, it's reloading the scene as we're looking at it,
which is orders of magnitude faster than all our solutions like you can get out.
With this, where is the fluoride game engine going next?
What is the roadmap look like?
Well, the very next item in our agenda is implementing the geode physics integration, which would allow us to define any sort of realistic and deterministic interactions between game objects in our simulated world.
This will be available as just an error component attached to your entities, probably called physics, which will allow you to define a variety of collider settings, either rigid or soft bodies and many more.
Next up, we're going to enable a higher amount of creative workflows.
We're working on delivering CLI and GUI tooling to support both designers and developers in mutually developing rich applications for all of your use cases.
As of today, we already support many open formats.
We have Blender compatibility.
We support GLTF and GLB models, KTX, HDR as well as other formats for textures, and the shaders are programmed using a superset of GLSL, which is the same shader language you might be using on most other platforms.
All of this is supported out of the box.
Thanks to this, you can very easily in a matter of minutes if not seconds, port assets from your existing game projects in those other engines.
And of course, this wouldn't be an embedded automotive talk if we didn't mention full cross platform support.
Thanks to Flutter SDL3, we'll have full support for embedded platforms, mainly Linux, which includes the Octo.
Mobile and iOS and Android.
Desktop on all the major, on all the major desktop OS systems, as well as game consoles.
This is all thanks to SDL3 that not only provides us with all of the tooling necessary, but thanks to the platform agnostic nature of both Flutter and filament, which has very easily allowed us to experiment with that.
And we'll help us in delivering those experiences to you in the near future.
And of course, it would be kind of sad if we kept all of that to ourselves, which is why we're also going to release an SDL3 API in Dart.
Available as a package to include within your Flurite applications.
Well, but how do we get to all of those next step? You know the same teamwork means the framework, which is why we're seeking collaboration with engineering teams.
Mainly with we're looking after is the commitment of development resources to the Flutter project for the purpose of establishing and delivering on a common roadmap as well as feature development for the purpose of this engine.
There is a website available right now. It has a very neat looking coming soon view.
But in the next couple days, we're going to update it to include all of the key information. You have seen in those slides.
But with this said, I'll actually, I'm going to give you a couple seconds, too.
Can I go that QR code?
I have a question about, do you want to create an SDL3 API?
We're right.
Sorry.
Sorry, I didn't know in the microphone.
My question is about accessibility. You know, Flutter integrates nice to with native OS accessibility frameworks.
What about the UI and stuff that is internationally?
It also possible to, for example, write my game components or like the free objects with like something like no accessibility with it.
And it will be exposed to the someone who is, for example, blind.
Right.
Yeah, we're still working on the API for that. However, it is something we are actively considering.
We are fortunate enough that Flutter does have a rich accessibility API thanks to its cemetery.
So the plan would be something adjacent to incorporating and incorporating a semantics node as part of the semantic subject, which would be then provided by Flutter right view widget.
Right.
This is the question section.
Oh, no worries, no worries at all.
So any more questions?
Thank you. Yeah, you mentioned cross platform support across all the native platforms that are supported by Flutter today.
Oh, sorry.
You mentioned that you are supporting all of the native platforms supported by Flutter today.
I was wondering if you also explored using say representally and web GPU.
If it's possible at all to also make this available for Flutter web apps.
We have yet to consider like a broader range of those.
Right now we're mainly focused on targeting just the latest versions of when those some of the Linux distributions and macOS.
But please to bring it up whenever we open the GitHub repository.
We'll be happy to take a look at your issue.
Thank you.
Any more questions?
Anything at all?
Then I have one question.
Do you render all the stuff on CPU or on GPU?
Oh, no.
It's fully GPU accelerated.
We're using, we're using Vulkan to drive the Flutter right through the renderer.
So we basically get a default GPU acceleration of whatever platform you find yourself on.
Right, thank you.
I'm seeing one more question.
Bring them on.
Last one.
Do you when you building this infotainment system for that car?
Do you use, do you have some UI testing process that says that the feature you're building actually works as intended?
Could you repeat that?
Do you use any UI testing solution?
As in when you're building this infotainment system for the car.
Do you actually, how do you verify that, you know, this works?
Do you do any UI testing that's the question?
I don't think I can answer any of that.
I'm mainly focusing on developing the software around it, which you're asking is probably under the responsibility of a different team.
But again, I'll be happy to look into that a bit more later and see if I can provide you with an answer.
But I'm sure that whatever we are working on will never be shipped into a real vehicle unless it passes to you as rigorous quality standards that are already being used in any of the current products.
So if this is available to a port or will be available to port to different boards, is there or outside of a car?
Or maybe that's the cars like vehicles or the focus.
I work with a lot of embedded systems in robotics.
I do a lot of visualization.
I'm wondering if there's any plan for integrating sensor data or to be able to read sensor data and like render that in flooring?
I mean, that would be kind of cool to be able to, you know, point clouds light on our images, things like that.
I mean, yeah, that's definitely a very cool use case scenario.
And while I'm not sure whether we're going to experiment with that ourselves, it's definitely sort of the type of experience we want to enable.
In terms of board compatibility, like what we're going to do in general is tested on a very small subset of target devices.
However, in general, assuming we've done everything right, this should run basically on anything that runs Linux,
supports certain hardware requirements, and has a version of Vulkan of 1.1 and above.
If I have it correct, right now we're definitely testing against Raspberry Pi 4, so like that's sort of like one of the devices we're looking at.
Sorry, there's very Pi 5, my bad.
This probably is some more, but that's the one I can definitely tell you about right now.
We have time for any other questions.
That's it.
All right, thank you so much for coming.
If you have any other questions about strategy and digital direction of the project, as well as collaboration.
Please email Joel.
Joel binarski at tutorconnected.com.
However, I'm available myself to answer any and all technical questions.
So, let me know.
Thank you for coming.