Thank you very much to everybody for attending this talk at the organisers of the first
and conference and this track.
I'm Chen Nakasanova, she's my lacanal.
We both present in this talk about getting more juice out from the rest of the PIPU.
A brief introduction.
We work together in the graphics thing at Igalia.
We have been working in the graphic stack for the Raspberry Pi for last year.
I'm mainly working on the user's per side related to Mesa and she has been working on the
current side of the project, supported the different parts.
So well, we are talking about and better this Raspberry Pi.
We're going to focus on the Raspberry Pi 5, of course, rounds in 2020, 23 in around October,
I think.
And it's used a next generation of the video core from Broadcom architecture, that is the
number 7.
It's a next step where the Raspberry Pi 4, into the several improvements, many ready to
support up to weight when the targets that allow us to get into some kind of support
for OpenGL, desktop 3.1, improves several operations, there's more part of this
ization.
We have more availability to change the weight that we handle the registers.
And well, since the moment we, the hardware was available in the market, the code BIP, there
was available the source code in the kernel side and in the Mesa side, completely upstream.
So I'm going to give a brief introduction about how is the graphic stack in the Raspberry
Pi generations.
Because sometimes it's complex to identify with server, I are reducing.
If you're using a Raspberry Pi 1, 2, 3, that is based on video core for the generation.
In the kernel side, we have a BC4 driver, that is the one that handles the display and the
rendering.
In the suggested space size, we have a BC4 driver that supports OpenGLS 2.0.
When we change to the new version of the generation, we have a 4 or 5 that have the video
core 6 and 7.
We maintain the BC4 name, but it's only for the display, the things that you put on the
screen at the end.
On the render side, there is a different drive that is B3D.
That has the same name that the user space driver, that is B3D that we use for OpenGL
or OpenGLS.
We also develop the book and drive that is B3D for book.
That's the complete names of different models and drivers.
Now I started in the focus in the part that we thought that it was more interesting
for this presentation, we are talking about performance and I'm going to talk about the
part, ready to use a space, we use the B3D driver and the B3DB, are a base of messa.
So part of the common infrastructure that we have available for all the drivers in Open
Storage are there.
In our case, the OpenGL drivers supports enough performance version of OpenGLC.1, because
the presentation of the hardware and there are some, they are emulated in some way, and
we are OpenGLS, that is the version of OpenGL, adapt the foreign devils, performance
in the Raspberry Pi 5 was launched, it was the same also in the Raspberry Pi 4.
So for the Vulcan site, when the product was launched, we supported it at that time Vulcan
1.2, and last so goes we get the conformance for the new version from Vulcan 1.3.
If you are interested in about what we support in the API, we have a presentation in previous
XC1.3R, we went into the data about the new extension, what you can do with the B3D APIs.
Today we would like to focus on performance, last year we focused working on some scenarios
that are the ones that we have a CPU, GPU limited scenario working with high resolution,
that implies that the limiting factor is the GPU, and we haven't been working with the
DFX bands, that is a common, based on the industry for analyzing the performance of your mobile
phone, and we get a performance from last year, well, 2DF December 35 of December 2023 to the end
of last year, 24 of over an average, 100% of performance increase.
That is, we did this analysis on Android 15, because the LS bands is not available for
arm limbs, so we needed to use this version of Android, so I'm missing work that Michael
had from, so we can try the useless space driver in Android, and then get the real results
from the commit from the end of last year to just one month ago, so we see that there
are different demos, and we are getting in some cases of performance improvement from
this, better based on a slice show to get in 13 frames per second in some cases, so
well, now we're going to go into detail into the what where the TMS issues will have been
doing, we need to understand a bit how the tile-basser render was, that is the kind of
present that we found in the Broadcon GPUs. In our case, when you prepare a job to submit
to the GPU, there are two stages, the third one is you prepare a GPU line, being
a bit job that is the one that is in charge of doing the next thing, you will have multiple
protocols, so it's going to analyze the geometry, identify with the parts of the frame
buffer generating, it's affecting that local, so at the end it creates a list of this tile,
that is the small piece of the image, it's affected by this local, with this list, we go to
the render job, that we see stuff list, and start loading, it's tile, it really knows, with
protocols, we're affecting that tile, it only executes that part, so with load one, once the tile,
we do all the protocols operation, and then we store the result of that, so the mess way of getting
more performances avoiding this load and store, because if you're going to do more laws and
stores, you'll split the things in different jobs, so the first demonstration that we have with
great performance increase, that it goes around 40% in average, was one that we discovered maybe
by chance, it was an extension that was tested by here, that was working in the drive and a
ceiling implementation, because the drive was waiting, if you were writing to a texture,
and then we're going to sample it, there was a job finishing to store that and for a lot that,
but if the combination of rim buffer was the same, you could reuse, you had access to the texture
in the in the in the cache of the of the GPU, so that the results would be already available,
remove it at that and see, get us really nice results, we can see here at a different steam,
on the right side is the current version, on the left side we can see that there is,
you can fill the difference from 16 to 24 frames per second, we also did a lot of compiler
optimization, and maybe there are like 15 per quest, improving, reducing the styles of the
improving the scheduling of this traction, reducing the number and with that work we'll remove the
number of instructions to run up, not almost a 5% but we get a performance about 3.57
in average, a lot of work and well, it should work on not so amazing with the previous one,
it was just one, come in and identify in the issue, we also take advantage, we were together myself
that, you have the ability in a title, in a title error, ketit tour, if you don't need the
results of the hint at the render that it usually happens when you have death or stencil buffers,
you need it to render, but in some cases we can avoid storing them a lot of them, so applying
some realistic, we could improve that behavior and we had another 1% of improvement, this first
sample has some use, not in this demo, but in the case of Google Chrome, you will use this
operation, that improves the results, and our interesting improvement was the work we are
ready to improve in the early fragment test optimization that is supported by the hardware,
but there are some situations that you cannot use it, one of that is, you are using a
draw call, a draw call has a discard operation, but just say, well, you do not, you cannot write anything
we are happy operating to the frame buffers, in that case, you need to disable the optimization,
but there was a situation, I am a scenario that allows you to do that, if death writes, where to disable
what, in that case, so with that we get a 14% of performance improvement, that everything I
selling is accumulating in this potential way, 40% in one case, over that, it is not linear,
the last, I would like to comment this one, in some kind of jobs, usually happening in
a situation that in a scenario that we have transfer feedback, that we are interesting in
now in the results of the geometry, the application can disable the restoration, so you do not
need to execute at the end the frame and say that, because you are not interested in the
result of the end, so in the case of the Manhattan, this is happening a lot, so every transfer
feedback operation, you do not enable this operation for all the protocols, none of that uses
this restoration, you can disable the load and the store of the buffers, and maybe you have
five calls of transfer feedback that would imply for each frame, five loads from a store that
you are not going to use, so take it in that data account, improve a lot of the performance,
the Manhattan name is one of the most, you can see on the left side, before all the optimizations
have commented today, on the right side, the last result with in this case is at 230% of performance,
this results are done with traces over the execution of DLSVNC standard, because it is easier to
compare the same trace and not doing the execution on the, that we do not have the counter
of having the same friends draw, and we can see here on the different comments on the right
and the performance improvement over then in the in the different moments, we can see that
in the case of Manhattan, the programming is huge, we can see that with the previous slide
that we are almost near the 300% of performance from the original, at the end of 2023,
with it also a lot of work on performance tools, and currently that might be going to
explain, I can do the flow. Okay, so the thing about studying performance is that we also need
tools to help us to measure the performance, the current performance, and see scenarios that we
would like to work to have a better performance. So first thing that we did was CPU jobs and time
stamp queries, last fours then, Chairman I talked about how we implemented CPU jobs, because there
are some vocal comments that we cannot perform in the GPU alone, so we had CPU jobs,
and we moved the CPU jobs from the user space to the kernel space in order to avoid GPU
flushes and CPU stores. And in 2023, we landed time stamp queries and the CPU jobs in the
kernel, in the vukondriver only. But the last year, we were able to support time stamp queries
in Mesa as well for the DL driver. And using type stamp carries is really useful for us when
analyzing performance, because it helped us to identify jobs that are taking longer, and if the
job is taking longer, we can analyze it and think about new ways to improve that job. That is
probably a scenario that is also happening in other applications. And with time stamp queries,
we can have very accurately synchronized time stamps that are perfectly synchronized in the
graphics pipeline, so this is really helpful to help evaluate the time of the jobs.
And we also implemented Fefator Support, Fefator is an open stores stack for performance
instruments, it means that we can access system level information and also Apple level
traces to help us analyze basically data from all your system. And we also have Mesa data sources
in Fefator, which means that now we can also add producers to GPU information, such as frequency,
visualization, performance counters, and this help us to have a unified timeline to work on performance
debugging, performance tuning, debugging. So you can see in this slide that you can have a very
system-wide view in a timeline, which is very useful because in the top you have like CPU information
to CPU frequency and more that I mean it couldn't fit here. But I only opened the CPU
information and the DRM fans because we use fans to synchronize stamps in the kernel. And you
can see all the fences that are being used in the jobs. And right here we have information
from Mesa that is coming from GL Mark II, which is the famous Mesa application. And we can see
you know when the job was submitted and if you see the fences you can understand when the
fence was signally by the job end, you can see like operations that when we are waiting for the
fence in the user space. So this is amazing because it can have a very system-wide view and
understand places that we can start thinking about the forms. And now jumping to the kernel work.
Last year we started enabling a feature in V3D that was historically used. V3D the V3D GPU
has support for four kilobyte pages, 64 kilobyte pages that are called big pages and one
megabyte pages that are called super pages. To enable them it looks very simple. You just need to have
a continuous block memory block with you know one megabyte for example and add the page table entries.
And the Linux driver didn't have to support for it. And I mean you can think why it's beneficial
because it's just like the CPU you know. We can improve the performance using a huge pages
by reducing the MIMU infectious, especially when we have memory intensive applications. Nowadays
shaders have like large buffer objects. So this is important. And but we had a very important
issue. We couldn't have a continuous block of memory using shaman like by default in the DRM,
which is the GPU subsystem in the kernel. So we had to think about a solution for it.
By default, tempFS and shaman allocate memory in page size chunks. This means that if your
page size is 4 kilobytes, we are going to allocate 4 kilobytes chunks. But we needed a continuous
block of memory bigger than one page, right? So we decided to create a tempFS mount point with the
huge equal within size option. What this means it means that we enable transparent
huge pages support in that mount point. Transparent huge pages is something that exists in
the kernel for a while. And it's basically an abstraction that help us to utilize huge pages
without really using huge pages in the virtual memory. It's an abstraction that let
basically the applications understand that page as a continuous block of memory. And we just
don't need to understand what's going down. It's a bit different than a huge page for example.
With that continuous block of memory, it's just a matter of placing the page table entries
in the right places, setting the bits, and then it's done. We also work on reducing the virtual
alignment, virtual address alignment to four kilobytes. This helps us to reduce the memory pressure
in the Raspberry Pi because you know, memory is very limited in embedded devices. And we were using
128 virtual address alignment. And this was basically utilizing some addresses in our virtual address space.
So we did reduce it and reducing memory pressure was really useful. We had an average
improvement of 1.33%, which is not that impressive. But we had a significant performance boost
in some emulation cases. Just remember that when we are using an embedded device, it's important to
set a met device when using transparent huge pages otherwise it can be used a lot of memory
that we don't really want to avoid. This is a demo running in PS2 emulation emulator with burnout
tree. And you can see the difference, you know, it's just this small feature in the kernel and you
can see a huge difference in these applications that utilize big buffers. Apart from that,
I mean, this is really important to use a huge pages, but we had an issue with huge pages that
huge pages by default, by default, use a THP by default, use huge pages of the size of
PMD. In ARM 64 this means two megabytes. And as you can see, our interest is just 4 kilobytes,
64 kilobytes, and 1 megabyte. So we didn't really need to use two megabyte pages. This was
leading to some unnecessary fragmentation in our system. So we decided to use Moot size THP, which
allows us to use huge pages only for 64 kilobytes to 1 megabyte. So we are just selecting this range
of pages that we want to have support. And MTHP is something that exists in the kernel and
help us to have the ability to allocate memory in blocks that are bigger than one page than a
page size, but is smaller than the traditional PMD size. And we created the true kind of parameters
to help us set the policies that we wanted for the pages. Because transparent, which pages
multi THP, in Schmann, had an issue that we also had to configure the THP with CSFS. And as
you know CSFS, every time that we reboot, it will go back to the default. And this is not great.
When we have this, and we want to create a product for client, I mean, we need to have the
configuration set for the client with the best performance. So we created this kernel command
lines where you can just set the policy for the transparent huge pages, just as you do for
tempFest, for example, but this is for Schmann. And you can use different policies for different
pages sizes. And then you can just configure like 16 kilobytes to 64 kilobytes is going to have
the policy always THP. And this is really useful, even if you have an application of Schmann.
Our case is that we use Schmann to back our buffer objects. So it's really useful for
each piece, but it can have other applications.
So that's all. Questions?
APPLAUSE
Hi, did you have to make any trade-offs while writing the compiler optimizations say,
like, longer compilation times or higher register pressure or something of the sort?
Well, there are a lot of different evistics there. It depends if you can't even optimize the
compiler to be faster compiling and not do different strategies, or you get the best performance
in some cases. But the default is, you try to get the maximum number of threads working, and
the system already has a cache. So once they say that it is built, you have to write the cache
version, so to kind of take them and do in that.
So the question for the habit write and the love it by writing is the 60 kilobytes and about
the GP benchmark. And I mean, you know, even the CPU changes, CPU division changes, and from eight
and all, and the only eight gigabyte type, so changes the default type, and more, and you know,
the omiti and use of a box. Have you tried and register 60 GB? 60 GB of advice?
Yeah, we haven't tried that, the product.
I think it is a difference. I think it is a difference, and all eight gigabytes, because of the memory
activity in private, and omiti are you with broke?
Yes, maybe, might I do not know about that, because we know that there is a difference in the memory
hand in there, and there is a work on that, I don't know with this already available.
Hi, how much support, if anything, you get from Broadcom, or the Raspberry 5 Foundation,
to implement this? We are working for the Raspberry 5 at the end, so.
Broadcom provides the documentation. We can read the specs.
Hello, thank you, nice presentation. If you compare the drivers that Broadcom give you
on their customers, their proprietary customers, compared to our their proprietary drivers,
compared to the open source, the drivers and the messa that you use here, do you see
are they the same, and do you see a big difference in performance?
We cannot make the check the difference, because the drivers are not for the same kernel are different,
things, in that case. We don't have the comparatives. Are they on par, or are they,
you don't know at all? We don't know, we know that there is room for improvement from their numbers,
but we cannot do the same rang in both of the accommodation, because we don't have access to the other
one, and we need to work on the same platform. One more question, do you have a way to
to see, to ask for the GPU driver, what processes, per process ID?
What their resources are that they use in the GPU. I tell you the use cases, you have an application
manager on the system. We landed, I think, one two years ago, the GPU stops, so you can get
information from FD info. 6.8. 6.8 is available upstream, and downstream in passory OS is already available.
You can use GPU top, I think. Yeah, it shows the information.
That two are starting. Thank you.