All right, so next up is Mark.
We'll be telling you about Python packaging.
Indeed.
Thank you, Bjorn.
Yes, my name is Mark Ryan.
I work for Revolce and we're going to be spending the next 30 minutes talking about
my favorite subjects.
Python package installation on risk 5.
I don't know how many of you have actually tried using Python on this 5.
Those of you who have have probably enjoyed a bit of a mixed experience.
For those of you who haven't, I'm going to demonstrate to just how frustrating and experience
it can be.
I'm going to do this using a little video I pre-prepared.
This is a video taken of me typing away on a risk 5.
You can see that it is a risk 5.
The first thing I'm going to do is start Python.
As you can see, Python works nicely.
You can start it up, you can run Python programs, and better still, you can install Python
packages.
In this case, I'm installing something called requests, which is a very popular Python
package for sending HTTP requests, as you might imagine.
It is installing without problem, it's also installing a lot of extra dependencies, which
install without problem.
To prove it works, I'm going to start the interpreter, import the package, and try and send
a request.
I apologize about the slow typing here, but we're sending it requests to the false
and website, and we should get some text back soon.
Where it works, okay.
Python sort of works, will it does work, but let's try and install a different package.
This time we're going to install something called sentence piece.
Now, sentence piece is a word tokenizer.
It's used in a lot of AI and machine learning workloads.
You're going to notice it's going to take a little bit longer to install the installation
script looks a little bit different, it's downloading a tarot.gz file, it's saying something
scary, installing build dependencies.
I'm going to let you into a little secret, this is not going to work, and it's going
to fail rather spectacularly in maybe five seconds, and there we go, you've got a lovely
error message, that is not a great user experience, I think you can all agree.
So why is it that we can install some Python packages on this file without problem
and others we can not install?
Well, it turns out there's two different types of Python packages, there are pure Python
packages, those written purely in Python, and there are binary packages, which are written
Python and a mixture of C, sometimes C++, and increasingly Rust.
You can download pure Python packages from Python, which is a central repository that stores
Python packages on this file without problems, but you cannot download binary packages,
and the reason is that Python will allow you to upload binary wheels for risk by 64, and
a wheel is the file format for distribution Python packages, and it's not just sentence
piece that's affected, a lot of the very popular Python packages, it's important Python
packages of binary packages, if you think about things like NumPy, Cyphy, Cycup Learn,
Pandas, Pillow, TensorFlow, PyTorch, PyRow, all of these things are binary packages,
so they cannot be easily installed on RIS5, 64 devices at the moment.
So just stop and think for a little bit about what a serious problem that is for the vendors
of RIS5, 64 devices, they're going to have customers, their customers are likely to have
RIS Python workloads, and those workloads are going to depend probably on one of
more of these packages either directly or indirectly.
If you look at the bottom of any sufficiently complicated Python stack, you're going to find
NumPy at the bottom, and it's not possible to easily install NumPy at the moment.
On RIS5, 64 devices.
Now, you might be thinking that I'm over at dramaticizing this, and there is a solution,
you could just install from source and pip allows you to do this.
So if you remember, when we were trying to install sentence piece, what pip did, is it
looked to see if there was a RIS5 binary that was and so it downloaded a source distribution
and tried to install that, the installation fail, because I didn't have the right, build dependencies
on my VM, but you might think that is a possible solution to your problems, but allow me
to just abuse you of that notion.
Building binary packages is very, very difficult.
It's complicated process and it's very, very difficult to get right, and I'm going to demonstrate
how difficult it is using some VMs that I have pre-created.
So it's time for the live demo, which I really hope this works.
What I have here is I have two VMs.
On the right hand side, yes, it says noble.
So that is Ubuntu 2404, and on the left hand side, I have jammy, that is Ubuntu 2204.
I hope I said that right, noble is 2404, jammy is 2204.
I'm going to refer them to them as noble and jammy from now on, because it's really difficult
to say those version names.
These are running on directly on my Mac, so there are AR64 VMs, I'm using QMU to run these.
And they both have a shared folder, which is goal, falls them, well, you can see it there.
It falls them 2505, so I'm going to use that share folder to copy files between the two VMs.
Now the first thing I'm going to do is I'm just going to try and build send it to some source.
I'm going to build it on noble, I want to build it on noble, because that's the more recent
distribution, so it has a more recent tool chain.
So I'll get all the laser stock to my stations.
So I've got a little script to build it, actually.
Let me just show you what that script is doing, it's not doing anything magic, it's just
to save me from messing up on the typing, it's activating a virtual environment in which
I'm going to preinstalled all the dependencies you need to build send it to space and then
it just builds it with it.
So let's build it, this should take about 10 seconds, I think, so we'll pause for a minute
while it builds, it's got a faster laptop, yes, it's finished, okay, and if you see it's
created this file here, it's great, this wheel file here.
Now what I'm going to do is I'm going to copy this file, and I'm going to copy into my shared
folder, oops, and we're going to switch over to our jammy, I should have done this while
it was building.
We're going to switch over to our jammy VM and we're going to try and install it, and
actually I'm going to just create a version environment in which to install it.
So let's install our wheel, and it fails and it fails straight away with a rather confusing
error message.
It says the wheel is not a supported wheel on this platform.
Now to understand what this means, you can look at the file name, the file name actually
gives you a clue.
The file name is composed of a number of different parts, we have the name of the package,
we have the version number, and then we have these two tags here, and these indicate the minimum
version of Python required to run this package, and also the Python ABI against which
the package was compiled, and they both say Python 3.12, and that's because on a bunch
of noble, we have Python 3.12, but on a bunch of jammy, we have Python 3.10, and so
this wheel is not compatible with the bunch of jammy in its default state.
So that's the first kind of complication you're going to meet.
When you build a Python wheel for distribution, you don't just build one wheel, you have
to build multiple wheels.
One, at least for each version of Python, your customers are going to use, and it's
actually a lot more complicated than that, as we should see in a minute.
So we can try and get around this problem by, well actually I anticipated this problem,
and I pre-installed Python 3.12 on jammy, so we should be able to create another virtual
environment that uses Python 3.12, let me delete the old one, let's create, and then
we should be able to install it.
Okay, let's try that, and this time it works, let's see does it actually work?
You can see it says Python 3.12 now, we're going to try and import it, and we can
import it, it fails, and it fails with another obscure error message.
And if you read the error message, at least it's not too pages long, this time it's complaining
about G-lipsy.
And the problem here is that sentence piece, we compiled it on noble, and the noble has
G-lipsy 2.39, and jammy only has G-lipsy 2.35, and when we compiled this on noble, it
took advantage of symbol that was introduced in G-lipsy 2.38, and that symbol obviously
isn't present on jammy, and so the package won't work.
So our goal of trying to compile on the latest distribution to get the latest version of G-c
sort of failed, but ultimately that's really what you want to do, but at the same time as
we just see, you also need to compile on an old distribution with an old G-lipsy, so your
wheel that you create will work on a wide range of distributions, and that's a sort
of a contradiction that we shall return to later on to see how it solved up stream.
And for now what I'm going to do is I'm just going to build sentence piece directly on jammy
and try and install it in noble, just to prove that that is the problem.
So again, this is going to take a few seconds to build, and while it's building, I am
going to set up my virtual environment.
I'll just build it wrong.
Okay, it is built, let me delete the old wheel as here, and build copy in our new wheel.
Let's try and install it.
And it works, so we built on jammy, we can take that wheel and install it on noble,
that's good.
So the second thing you need to understand is when building wheels, you need to try and build
on an old distribution as possible against the old G-lipsy possible.
But sentence piece is actually a pretty simple package, it only contains, it's not, it doesn't
contain a huge amount of code, and it doesn't have any dependencies.
So let's try building something a bit more complicated.
We're going to try and build numpy.
Here's the script I'm using to build numpy, this is a pretty much how numpy is built up
stream.
Again, I'm building it inside a virtual environment with all the dependencies that are required.
Okay, yeah, this is actually going to take about 50 seconds on this VM.
So while it's building, allow me to just demonstrate that sentence piece actually does work,
I have a small test here for sentence piece, so let's just run that.
This is just tokenizing a sentence using sentence piece, it says hello false demo, and so
we've tokenized this, and you can see that it is working fine.
So this should be ready in a minute.
We are building on jammy, because we've learned our lesson that we want to build using
an old version of you, and we're going to try and install it on noble when it's finished.
Okay, it's finished, so let's copy the package that we built into our folder, and let's
try and pick installing it.
It has installed, but does it work?
Anybody guess?
No, it doesn't work, and we get another huge page of error message.
If you read through the error message, you eventually will come to this line here where it says
the original error was, and it's complaining about a missing shared library, and so the problem
here is that numpy, like a lot of other Python packages, have dependencies on shared libraries
in addition to the G-lip C, and that shared library, open blasers not installed, locally
on this noble VM, so when I try to use numpy, try to import numpy, can't find the library
and it fails.
Now what you could do is you could just try installing open blasers on this noble VM, and
then re-importing numpy, and it will probably work, but it might work exactly correct.
And the reason for this is when upstream packages like numpy use dependencies, they often
rely on a very specific version of those dependencies, and they often expect them to be built
in a very specific way, and sometimes to be patched.
So if you just use the distro version of numpy, you're not necessarily going to get the same
behavior as people who download numpy from pi pi r.
So the way that this is achieved upstream is that when numpy builds their packages, they use
a tool called audit wheel, and what audit wheel does is that it analyzes all of the binary
components of a wheel, and it finds all of their dependencies, and it checks those dependencies
to see if they're all a wireless, and if they're not on a wireless, it copies those dependencies
directly into the wheel, and so when the wheel is distributed on pi pi, the wheel doesn't
just contain numpy, but it also contains all of numpy binary dependencies, so things
like open-blas and lib g4 from there package inside the wheel.
And so we can just look at that working here, so what I'm going to do is I'm going to use
audit wheel to repair my numpy wheel with this little script here, and it's written
the wheel to a different directory, so let's have a look.
You can see also that the name of the wheel has changed previously, the tank here was Linux,
and now it says many Linux 2.35.
And what that means, this many Linux 2.35 is a sort of a guarantee that this particular
wheel will work on any Linux distribution that has a G-Libc 2.35 of greater, and the way
that's achieved is that it's linked against G-Libc 2.35, any dependencies that it uses
that aren't on this white list are copied into the wheel itself, and any dependencies
that are on the white list, it makes sure that the package doesn't use any symbols
and those dependencies that aren't present in a Linux distribution that ships G-Libc 2.35.
So let us copy our new wheel to our shared folder, and yeah, oh, let's install the
version of them by that list installed E-version, and it's working, important numpy numpy
one, and it works, okay.
So that is the third thing you have to consider, when you're distributing wheels, you
need to repair them to make sure that they contain all the dependencies packaged directly
within the wheel to guarantee that they're going to work on your end user's machines.
Okay, so that's the end of the demo, wasn't too bad.
Let's return to the presentation, can I do that?
And we'll just recap what we just learned.
So when you're distributing binary wheels in Python, you need to build multiple wheels.
You need to build one wheel for each version of Python, your customer is going to use.
But actually, if you want to support multiple versions of G-Libc and multiple versions
of multiple types of C standard library, you want to support G-Libc and Muzzle Linux, then
Muzzle, you need to build another set of wheels, if you want to support Mac OS and Windows,
you need to build another set of wheels.
If you want to support X86, AR64, RSI, PowerPC, you need to build another set of wheels.
And if you multiply all these together, you can end up having to build like 30 or 40 or 50 wheels.
I looked on the PIPI repository for NumPy, and they actually build 54 different wheels for
a friend NumPy.
And also, there's different Python interpreters, right?
So there's a PIPI and that needs its own own special wheel.
So that's the first tip of the building Python packages industry.
You need to build lots and lots of different wheels.
The second issue that we have come across is ideally you want to build with a brand new G-Libc.
Sorry, G-C-C, and you can imagine that's important for RSI 5, or we want to build with G-C-C 14,
so we get a vector support.
But you also want to build with an old G-Libc, so you can widely distribute your wheels.
And finally, we have this issue of dependencies.
The issue of dependencies is actually a lot trickier than I've made it appear.
It's not just a matter of running order wheel, and the reason it's not just a matter of running
order wheel, is when you're distributing your package, you have to find out what
version of the dependency that package wants, how it wants it built, how it wants it patched,
and then you have to build it, and then you can repair your wheel.
And often building the dependencies for a binary Python package is much more difficult
than building the packages itself.
OK, so this is all pretty tricky, and this is tricky on all platforms and all architectures.
And so the sort of open source community has come up with a lot of projects to make this
easier, to make the sort of the creation and distribution of Python packages easier.
And so we talked a little bit about distribution before, this is a central repository
for distributing Python packages.
This is called Python.
And that's when you just do pip install numpy, that's where people go through by default to find
its packages.
This problem of solving, of wanting to build with new GCC and old G libsy is sold by a project called
many Linux. So many Linux is a project that publishes container images which are
specifically designed to build Python packages.
They ship with an oldest G libsy, so the newest many Linux container images many Linux 2.3.4,
which ships with G libsy 2.3.4, but it takes advantage of a red hat project called GCC Toolset,
which allows you to install the latest version of GCC without updating your G libsy.
And so many Linux 2.3.4 has G libsy 2.3.4, but it also has GCC 14, which is a really nice combination.
The mellin Linux images also contain the latest versions of the build tools,
stuff like Git and C makes, so you don't have an old version of these tools, which is good,
that might prevent you building things. And they also come with about 10 different pre-built versions
of Python. And that's important because when you're building multiple wheels, you need separate
versions of Python for each different wheel you want to build.
We've talked a little bit about binary dependencies, and we also looked at audit wheel,
so you need to sort of vendor these binary dependencies into your wheel, and that is done by audit
wheel, and it can also be done by a tool called mature and matureness, tools designed to build
hybrid Python rust packages, and it has this repair facility.
We also talked about how you might have to build about 50 of these different wheels,
and for each of these wheels you have to build, you'll have to identify the correct Docker
image, the many Linux image to build them in. You've actually got to build a thing,
you've got to repair the wheel, and then you've got to run all your tests, and you've got to do it
50 times. So to prevent people from having to do this in every single upstream project, there's a
program called CI Build Wheel, which automates this process for you.
Once the wheel is built, you're going to need to test it in your CI, so you're going to need,
and this is, yeah, you're not going to want to do this locally, you're going to want to do this in CI,
so you're going to need some runners to do this. You're going to run a lot of tests to make
sure you will work, and then finally once it's built and tested, you can upload it to the
package registry, and there's a bunch of tools you can use to upload and install here.
So trying a mature end can be used to upload packages to registries like pipeline, and pip,
and UV can be used to install again packages. Okay, so to summarize, building Python packages
is difficult, but on the more popular architectures like ARM and RSSX86, there's a whole pile of
packages that are a whole pile of projects that are aimed to make this make the problem easier.
But if you look at RSSX64, you will see that most of these packages are most of these projects
don't actually support RSSX64. The only ones that do are audit wheel and matureren, and the upload
packages, so trying mature and pip and UV all support RSSX64. So the problem is, on RSSX64,
not only do we have this really difficult problem to solve, but a lot of the tools that are
normally used to solve these problems don't currently support RSSX64. Okay, so but don't get too
disheartened. If we were to have looked at this slide this time last year, there would be
everything would be read in the right hand column. So progress has been made in the last year
between getting RSSX64 support into some of these projects, but it's still not there, and so for the
time being the problem still exists on RSSX64, and it's a big problem, because it's difficult
to install these very useful Python packages. So recognizing this, RSSX64, which is an industry
consortium, designed to improve the RSSX64 system, has created a project, an open source project
called WillBuilder. WillBuilder, and the WillBuilder project is, well, what it does is it builds
and distributes a small set of binary Python packages that we think will be useful.
We're currently building about 30 packages, including things like NumPy, CyPy, Paners, MapLotLive,
maturen, and we're planning to build more packages in the future. Not only do we build them,
but we also make sure that they're kept up to date. So, you know, we don't just build like NumPy,
1.26.3, and then leave it there for two years. You know, we've been building newer versions
of NumPy as well. They all the things we're building are many Linux 2.3.5, RSSX64 images, and they're
built with RV64GCs, so they should run on pretty much any device. And we go to great lengths to
build these wheels in the same way that they build upstream. So they, the behavior of the wheels we
build on wheelbuilder should match the behavior of the wheels you download for other architectures
on Python. So, for example, if we go back to the NumPy dependency problem, when we build and package
open-blasin side-iron NumPy wheels, we make sure we're building and packaging the exact same
version of open-blasin upstream NumPy does. So, you should see the same behavior. And the wheels
are tested, mostly, I say here, because sometimes it's not possible to run all the tests.
It can be because there's a risk-wise specific bug, but generally it's because the test will
require a dependency that doesn't exist on RSSX64. And so in those cases, we disable the tests.
But in most cases, the wheels are tested in the CI before they're uploaded as well as we
can test them by running the projects normal, normal tests. So, when you download wheels from
wheelbuilder, the wheels should work. So, you're wondering how do I download the wheels from
wheelbuilder? Well, it's pretty simple. The way this actually works is that the wheelbuilder
project is in GitLab and GitLab actually allows projects to create their own Python package
registries. So, we build the wheels in GitLab and we upload them to the package registry associated
with that project. It has a quite a catcher URL here. But to install the packages, all you need to
do is upgrade PIP. You need PIP 2.24.1 or greater for RSSX support. So, if you just have an earlier
version, you'll have to upgrade PIP for us. Then you need to tell PIP to use the GitLab registry
instead of Python. And you do that by setting this environment variable, or you can actually
specify this on the command line using the minus minus index URL option. And then you just install the
package. And so, I've got a little video here just to demonstrate this working.
So, as you can see, I'm back on a wrist 5vm. I only have PIP 24.0 here. So, I'm going to upgrade PIP
first. Otherwise, I will get an error when I try and install packages from wheelbuilder.
So, the first thing we're doing, we're updating PIP. Now we're going to install, now we're
going to export our index URL. So, PIP knows where to find the packages. We're going to
try and install sendersbeaks. Now, remember, this failed spectacularly right at the start of the
talk. But, now we specified the right index URL. It downloads the package and installed it,
and it installs it really quickly. Let's try and install NumPy. Just to give you another example.
And NumPy is a bit of a bigger package. It's got all those dependencies. So, it takes a little longer
to install, but it should be installed pretty soon. Wow, that's slow. Yes, it's installed.
And now let's just start Python. An important NumPy just to demonstrate it works.
And indeed, it does. Now, there's just one more thing that I want to show you. I'm going to
try and install another package. This time, I'm going to install something called twine. We mentioned
twine earlier on. Twine is a package for uploading tenants. Oh, it's a package for uploading
other wheels to package registry. So, the reason twine is interesting is because it's a
pure Python package, but it has dependencies that are binary dependencies. And so, I'm able to
install twine here. Even though I've told Pip to go and look at the wheel builder, get lab
package registry, and not to go to pipeline. And we don't have twine in the wheel builder project,
because it's a pure Python package. There's no point in this building it. And yet, this still works.
And the reason it works is that when you send a request to get lab, the package will get lab
padded red package registry. It will first check to see if it knows anything about that package.
And if it doesn't have any wheels for that package, and it doesn't know anything about that package,
it forwards the request to pipeline itself. And so, this allows us to just use one index
URL to install a mixture of pure Python binary Python packages, which is quite nice.
So, that is pretty much it for the talk. What I might just do is pop up the wheel builder project,
so you guys can see it. So, it's here on GitLab. I guess I have the link in the talk at the end,
so let me just show you quickly the project. Here's the list of the packages we're building,
and you can see we're building multiple versions of some packages, so the CMake,
markup save, map lotlib, OB tree, lots of different packages, pandas, tornado, TL parts,
and you can just download those using that link and install them directly. And let me see, yes.
Yep, and that is the end of the talk.
Right, where a couple of minutes for questions, then it takes.
Thank you, great project. What would it take to adapt the wheel builder for other architectures,
like long arch? Sorry, could you repeat the question please?
Like, what would it take to adapt the wheel builder for other architectures, like long arch?
Oh, I see. Ah, yes, that's an excellent question, and you reminded me of some two things I meant
that I didn't say. So, the way we're building these wheels. I remember when I was talking earlier on about
how all these infrastructure projects are needed to build wheels for Python and how a lot of those
are missing for risk by 64. And the two important ones were many Linux and CI build wheel.
Because they don't exist for risk by 64, we have had to patch them ourselves.
So we have created our own many Linux container based on the Ubuntu 20204
and our own patch version of CI build wheel. And we use those internally to build the wheels.
So to do this for another architecture that is not supported by many Linux or CI build wheel,
you would need to do the same thing. And you would also need to make sure audit wheel
supports long arch. And I'm not sure it does. I would have to check the policy files for audit wheel.
And that is the tool you need to repair the wheels. So, if audit wheel doesn't do that,
then you would need a patch version of audit wheel as well. So it's definitely doable,
but you would need to patch these projects that we have to patch ourselves.
Anymore?
Is that going to mean to be many dials? Or let's say those are 13 per unit, or the
twice sunshine, or something like this, which means that we are going to be many off-back edges.
So that's a very good question. At the moment, we're building everything with RV64GC, okay.
It is possible, it should be possible now to build NumPy. If we were to build it with GCC14,
so we'd need to create a new many-lander container, or two-to-three, nine container,
we should be able to build NumPy, that has some vector support in it, right? And that vector
support would come through open blasts. And because open blasts does runtime detection of lots
extensions are available, that will work on all part forms. So it will work on a, you know,
something that doesn't have vector and something that does. But more generally speaking, there is no
real way in Python at the moment, and this isn't just a risk by problem. There's no way to
tank your wheels with information about what extensions those wheels expect. And this is a problem
across all the architect, because it hasn't been noticed until very recently, because the x86 wheels
have been built on distros that targets x86, v1. But that has just changed recently with album
in x2.35, and now targeting AMD v2. And so some wheels built with many legs 2.34 have been
crashing on really, really old machines, and there's no way really at the moment to label those
labels those wheels. And so, yes, so that is sort of an open issue in the Python sort of packaging
ecosystem, and it doesn't just apply to respect it applies to all the architectures, but it's
really good that it's just come up in x86 because they're going to have to try and solve it,
and we can get in on the act. I should also mention, actually, sorry, there's one other thing I
want to mention, we don't want to do this forever. So our goal is just to build these wheels until
such a time as all those infrastructure projects and start supporting risk five, and all the
upstream projects start uploading risk five wheels to wheelbuilder. And when that happens, we'll
start to wind the project down. So it's not something we're going to do forever.
All right, thank you more.