So, yeah, to shout out, we'll be talking about, you know, what does it actually cost in
terms of energy and, you know, maybe even other resources to run this models on GPUs, not
in consequential subject, but, you know, typically gets overlooked in all the excitement
of actually, you know, running something.
So, how many people are interested in optimizing AI workloads, how many people are interested
in optimizing energy efficiency in AI workloads?
Okay, then it's for you and my aim for this very quick session is to increase the number
of hands that we are seeing today.
So, by the way, my name is Tushara and I'm an assistant professor at Canadian University.
So, let's start with a one more question.
So, how many months or how many years, we can power a and typical household, American
household with the power that we used to plane a train GPT-3.
And it guesses, you can shout out how many years or how many months, I mean, how many years,
you can power a typical American household or American house that the same power that
we used to train GPT-3.
Well, some while guess is, but it's 120 years.
So, what this is what we used and if you are surprised by that number, that GPT-4 was 40 times
bigger than that.
So, well, so, and this plot is basically showing that what kind of training the sources
in terms of electricity we are consuming to train these large models.
And this is another depiction that shows that what kind of prediction we are making in terms
of energy consumption of these AI training or inference models.
You can see that it's a very tiny number in the beginning of 2023.
And we are predicting 2030, it's speaking like, and look at the scale, this is three tell
about our, that we are talking and that's 653 tell about, of course, it's an estimation
but still given take.
So, what we can do about it?
So, sustainable AI or DNA is an emerging field and that where we don't really want to
lose all the benefits, we of course, want to reap all the benefits that AI technologies
and people like you are developing, but we want to optimize it.
So, that's the premise of sustainable AI or the DNA and the first important point here
is that we, how we can measure how much energy your AI model is consuming.
So, I mean, we can measure there are various hardware, you know, meters and software
energy meters, I mean some of them, maybe known to you, I mean we have seen some commands
people are trying with SMI, so there are there, but the problem with these is that they
are good for, for computing energy consumption at the system level.
So, you can, you can measure energy, what the entire system, the entire rack is taking,
and if you're interested in, you know, figuring out how much energy my one single API call is
taking, that is not that easy to figure it out and that we sometimes back, we wrote a paper
about that, how we can instrument source code and insert appropriate, you know, checks,
because accuracy is another thing, I mean we can measure energy, but it should be accurate.
So, how we can ensure that it is accurately calculated and log all the details.
So, that's, and this instrument instrumentation is done automatically.
We, in fact, extended that tool, I mean by the way, there was a tool in the previous paper also,
it opened source, and we recently extended that tool code green tool, which is also open source
that you can, you guys can use, that has even further increase the precision of accuracy
in energy consumption. And as a community, we know many, many techniques that we can apply
in various phases of AI workloads. So, I'm just summarizing some of them here.
Now, one very last question that I want to ask is that, you know, at application level,
there are various techniques, as you can see, but what about at a GPU level or GPU optimization
level. So, you, some of you may say that well, we can maximize as a occupancy.
And that is valid guess, but here also, I would like to highlight something, which is very
critical that we recently figured it out, that not only the number of threads that counts,
but also the third block configuration also counts. So, just to highlight that, that all of these
configuration that you see in the highlighted bar, they are using the same number of threads,
but their block configuration is changing, and that is changing the energy consumption for that specific task.
And, and again, if you are interested in, in this, instead of the work, this is, we, we
a predict energy consumption by instrumenting or not only instrumenting the source, but
using the ptx format that some of you might be knowing. So, that is it. If you are interested
in this kind, this line of work, these are the all the papers that we recently published. So,
feel free to take a look. Thank you.