I'm going to give you a presentation about torchemlay hour, or mainly I give you an introduction
into torchemlay hour.
I'm Marius, I'm with AMD, and the talk will be about, well, first of all, as it is the
first MLAR talk today, I give you at least a brief idea about what MLAR is, so that you
hurt the kind of the fundamentals, tell you what torchemlay are is, what the torch
dialect is, and mainly if you, yeah, as it is supposed to be an introduction, tell you
how can you build a project, how can you use it, what can you do with it, and also how
to get involved if you want to.
So, what's MLAR, if you look at the homepage, so MLAR is essentially a sub project of
the LNVM foundation, if you look at the homepage, MLAR.lvm.org, it tells you it's a novel approach
to build a reusable and extensible compiler infrastructure, I put novel into you, yeah,
it raises here because it's not that novel anymore, to be honest, so it was like spread
to the world in 2019, and in 2020 it made its way into the LLVM project, it provides
an extensible way to create and design intermediate representation, so it's all SSA based,
format essentially looks like like shown here, you have a result, you have your operation
with inputs, you can define custom attributes, custom pipes, all that stuff, and what it
is about is like normally if you design, sub-shame, intermediate representation, that's
called dialect, within the words of MLAR, and it's not like you need to go from one dialect
completely into another dialect, you can also mix dialects, means like for example, in
upstream there is an SCF dialect structured control flow, but that can be mixed with other
dialects and you can nicely combine this and dialect just more or less loosely scope, different
kind of things, so like linear operations are in the linear dialect structured control
flow has operations for do structured control flow, so and with that said, touch MLAR actually
aims to provide to be a first class compiler support between the PyTorch ecosystem and
the MLAR ecosystem, it's participating in the LLVM incubator process, so even here I'm here
on behalf of AMD, that's not an AMD project, it's an LLVM project, but we heavily contribute
to it and use it, it's do a license under Apache license with LLVM exceptions, but also under
BSD star license, I think that's due to historical reasons, it wasn't that clear back in
the time, if it either makes it into more like MLAR or LLVM or go small down the route to PyTorch,
and with that said, actually the main component, it looks like following, so if you have
PyTorch which is in the PyTorch repository, PyTorch itself comes which touch of X and what
we have in the touch MLAR project is an FX importer and that FX importer allows you to go
to rate down, to import, charge it back into the touch dialect, so one of the main things in
touch MLAR is the touch dialect, and furthermore what we have in the project are conversions
to other dialects, to other MLAR dialects, so these conversion also live in the touch MLAR repository,
and allow you to convert to TOSA, to Linux, but also to our upstream dialects living in the main
MLAR project, but you can also there's also conversions down to stable HLO, which is more used
or comes from the tells of low ecosystem and lives in the open XLA, organization, so that's
all what is in the touch MLAR project, and the main thing we have is the touch dialect, so that's
the central MLAR abstraction to represent a touch model essentially, mostly of this is auto-generated
based on the PyTorch GDR operator registry, but we also have some manually implemented ops
to make it short, we have those for operations, which cannot be automatically implemented
among other things for example, because they are directly supported in a GDR operator
without having a corresponding operation in the registry itself, so that's why we cannot
auto-generate it, and we also have a touch operator operation to represent ops,
which we haven't been auto-generated for what that is used for, I will show you in a minute,
but that's the main thing, and in MLAR, this all lives, so if you look to upstream MLAR, it would
be include MLAR, here we have include touch MLAR dialect, touch IR touch ops, that's where all
ops are actually implemented, and in line 26 you see an include, that's actually where we have some
scripts that generate those auto-generated operations, that's got included here in the definition.
If you want to build a project, well you need of course to clone it, and there are several
sub-modules which you need to initialize before you can build a project, so the one thing is
touch MLAR comes with one of the git sub-modules is actually LLBM, because we need to
to pick the right version of MLAR, you can also use a separately checked out MLAR, which you
compile an install, that might work, or it might not work, because you need to pick the right
commit, and that's what he is serialized, we're pinning the correct sap module, if you specified
that by one that gives you a shallow clone, so you don't clone the entire LLBM, just to let you know.
It's suggested to set up a Python virtual environment, and install your requirements,
you need to upgrade pip if you want to play around with touch vision, because all the
pip versions normally don't fail on the dependency resolution, and you cannot install touch vision.
In the requirements file, you also get the correct version of touch, which is compatible
with touch MLAR, so that's all in the requirements file and should be pretty straightforward.
If you want to build the project, we don't go through all this CMake magic here,
I just want to highlight two things. You can either do a monolithic bolt, which means
we are building LLBM and MLAR, and touch MLAR in one single bolt, and to do this,
you need to enable the project MLAR, of course, which is a subject of LLBM.
In addition to that, you need to specify that there is an LLBM external project,
with this case, touch MLAR, point to the source here, where to find the source, which is quite
obvious, we are right now in the root of the repository, so it's just here, and LLBM essentially
is cloned into external LLBM project LLBM, that is where it gets, I get some modules, initialized.
If you want to look more into the specifics, I gave a talk two years ago here at the LLBM,
they're through about how to build your own MLAR dialect, that covers some of these
aspects as well, gives you a little bit more background, or you just look into the MLAR standalone example,
which also shows you the different ways, how to build it, but it's also, of course, in the
touch MLAR documentation. The other way is to do a component build, so we build MLAR independent
of touch MLAR first, so that's essentially just building MLAR, without touch MLAR at all,
so it's stripped down configuration set of what we have seen before, and we install it to build
LLBM here, for example. Then it allows you to build touch MLAR against this pre-built version of
MLAR, and the important flex here is you need to specify the MLAR where to find the project,
the LLBM, where to find the project, and that's the thing which is often missed,
and you'll fail then right at the end of compiling it all, you need to point to
various the LLBM that can be found, and that's not installed, it's still in the built here,
you just need to be aware, pass it, and you'll be fine. And with that, you can actually build
the target, for example, check touch MLAR. I think this target specifically mainly works with
the monolithic build, for the component build, it seemed to have some trouble when I last tried it,
so normally people do the monolithic build, but that's the options you actually have.
If you just want to play a round with the project, there's a way easier way, so we got pre-built
Python wheels, which gives you the Python RP, as well as some of the command line tools.
That would be the torch version, which is right now pinned down in the requirements file,
so you probably want to install that one by yourself. Here we point to the 90 CPU wheels,
so if you just pip install torch, that pulls in all the CUDA dependencies, there's also a
rocker version available on PyTorch.org, but to be honest, if you just want to play around with
it a little bit, and don't need to use to upload anything on a GPU, go with the CPU versions,
as well as those are way smaller, and they don't load your display that much,
we do it in our CI specifically all the time for the small tests. You need to install on
next, if you also want to import on next models to torch MLR, we get to that in a second,
and then you can just install torch MLR from essentially, you find this link in the command
in LLVM's less torch MLR release as well, and then you can play around with it.
So a simple torch model could look like the following, so here we have just
a torch in an module with an in and forward path, and what you simply can do to get it
translated or imported into torch MLR is, use the torch MLR FX, use export in import,
call it in your model, say what you want to have for an output, here we say torch to get into the
torch dialect, and that's it, then we are in torch. For onics, it's a little bit different, so you
can just for example download a model which is pre-trained on the end of the start of set,
and then you can use the command line tool to torch MLR import onics, which comes also with
the Python wheels, past the onics file to it, specify an operator set version, what the
does is it raises the operators that are in the model to a specific version you specify that
can be quite advantage if you want to later on compile your model, opposite 17 is what we are normally
using, and output the mness MLR file, so in such an MLR file looks like the following,
I cut it down to the most important parts, there are constant in there like all the weights
are in the MLR file, we don't want to look at that, but essentially here you have an onics add
and onics re-lew and an onics max pool, but that's only names, so the actual operation is torch.operator,
and this torch that operator needs to be lowered somehow, and that is where we implemented
or what is implemented, and it's just like paving the name for this up and later on we need to
process it and decompose it into other other other ops, so for example this could look like the
following, in MLR or in torch MLR normally files are structured as follows, so you have lift conversion
and then we have torch onics to torch, which is the conversion path to lowers this torch.operator
ops which then have onics ops in it to torch ops, for an app's up it's quite easy, so first of all
you have the name apps, one is the operator set, so you can also for some operations, there are
different operators that support it, or if you need to check why is my onics operation not
supported the way I think it should be, that might be one of the reasons that the operator set
version is not what you expected, there's some something technically triggered there, but essentially
we see the revirator and that has a replace up with new op, and we just replace this torch operator
op with in torch 8 in apps op, and that's it, so for that simple ops it's quite straightforward,
or we get then from onics to torch, for other ops it's a little bit different, one of the
this is for example average pool, and that's the thing you might notice that the onics op
in general is supported, but several attributes are not, and here for example there's an attribute
which is called auto pad, but that's not supported, so there are many operations which are
supported in general for a specific offset, but it might not be that the options are fully supported,
so that's need to be something where you should be aware of, but that's also slightly a good way
to get yourself into torch MLA if you want to contribute. There's also a decompose complex operation
path, so for some things like for here it's hard shrink, we replace this by way easier ops,
so essentially you end up with some simple letter than scalar op, a greater than scalar op,
and some other ops, so that is also a way, where this place it's now in a lit dialect torch
transforms, and that's because we are not leaving the dialect, so we are working on the dialect,
replacing ops of the dialect with other ops of the same dialect, that's why it's not
normally called a transform and by you find far there. Pipelines, Pipelines is another important
thing of course, so when we have single conversion, you normally don't want to stick with a single
conversion, you're doing multiple conversions, and if you do multiple conversions, you can
put them together to build up a pipeline, Zeus are the pipelines that are available right now in
torch MLAR, and the most important ones are probably the torch backends one,
either you can look into torch on extra torch back and pipeline, which shows you how to get from
torch on extra torch, but as I mentioned, and if you want to run a model, you don't want to end
in the torch dialect, so the torch project itself depends a little bit from where you come from,
if you look at the full stack, we are having machine learning model and want to execute it,
torch somewhere in the middle, from a compiler perspective, torch is more like in the front,
so it's a front and dialect, but it's not to be used as standalone, so we use it in our
compilers as a front and mainly to get a pie torch model or on next models into the compiler,
but then we go all the way down to the like on tensors, and then our compiler takes over,
does more magic, lowered this all to LB and CPU, for example, but if you just want to use torch
MLAR that's possible, there's a rough backend, which is not included in the Python wheels,
so you need to be aware that it's right now you would just copy over the files from the repository
or from source, so that's the torch MLAR E2E test suite, which is there, and essentially here
our output type is not torch anymore, it's linear contenders, and then you can just say, okay,
please backend, compile it, load it, and there we go, so how to get involved, so a good thing is
maybe you want to import and lower a model by using torch MLAR OPT, it's worth to also look into
what do the pipelines do, if I just take one of the conversions like convert torch on x to on x,
what's the next step, how do I lower it further, like torch decompose complex ops could be an
option, how do I need to, when should I use the canonicalizer, all the stuff is set up in the
pipeline, but to get familiar with it and what it does, it's quite good to just use the command line
to, and heck your way, heck your way into it, and just try to see what it does, compare I are before
and after, if you have issues, fill issues, please, if you have questions, you can also do this by
directing the repository in the issue tracker, but it's most often the better way to just join
discussions on the LVM discord, and what we normally did was new status and what's quite
nice is, we let new status implement new ops, which were unsupported, so right now many of
the ops are already supported, but as I mentioned, not every attribute is supported, so if you want
to get a hands on really, try to see what attributes are missing, hopefully there's a to do there,
but maybe you also notice when you lower your model, that something is not working as you expect,
and then try to implement or add support for one of these attributes, which belongs to an
onex operator, which is also specified in the onex documentation, so this is also a pretty good one
to look into what is the op should be suspected or you try to implement an onex op with a different
ops set and try to add support for newer attributes essentially, so yeah, and that's my brief
introduction into torch MLAR, peace, I was asked, or the question was, in the pet lines
overview, we have the torch simplification pipeline, which sounds quite interesting, if I can
get a comment on, if I could comment on what it does or what's in there, and to be honest, no,
I have no idea, no, I would need to look it up to be honest, so it might be that there are some
shape computations, which could be the thing, we definitely have some shape refinement pipelines,
also simplification might be that, it might also be that it is what I mentioned before,
like this decompose complex op, that that is part of the specific pipeline, but I don't know
that one in particular, what passes that in, I'm sorry, question about the kind of motivation
of this project, like, or the next can do most of these things, and why you might having to be
other, so the question was about the motivation of the project and that annex can do most of the
things as well, can you, well, I think you're referring to the ONIX MLAR project then, so
you're right, that for ONIX, that's for sure true, but for torch, I think it's not, so if you really
want to implement, or if you convert a model from torch to ONIX, that's often painful,
that's at least what people tell me, who are more on the model side, and that's why it's mainly
intended to be to provide a native way to get torch into MLAR, that's why it's the torch MLAR project,
from a historical perspective, it started as NPCOM, which was like more numpy, front and specific,
then switched gears a little bit, and it's for torch, and I think that we have ONIX, it's just
more or less like, you can just wait, work all the way down, with the same stack to not pull in even
more.
One, last, if I have time, go for it.
You should, for example, of a rewrite as a sort of piston-assumatic sequence plus, do you have a
DSL project, sometimes, when you say something like this, as a special reason, rewrite it
or something like we need, and we'll tape it down and send this up.
You mean, for here, for that one, are?
Yeah, but you're, is that a dark, dark rewrite?
Yeah, now I think we don't do it in table jump, but I cannot tell you why we don't do it,
or why it's done that way, maybe because people don't really like to write table jump.
All right, I don't that, though, all right, thank you.