Welcome welcome done, Bishai. I'm not biting.
Let's keep it as a secret.
Hi everyone, thanks for staying up late. I know, it's like how fast 3 so it's already quite
a time and yeah, thanks for being here. So my name is Alex. I mean, probably you can try
to spell my full name in surname, but it doesn't matter. Just call me Alex. That's fine.
I'm from OpenEvilSystems and today we're going to talk about building the A factories
with open source tools, right? So AI, AI, AI, AI, that was a little bit different story,
but yeah, AI. So I'd like to say thank you to EPSYSIS project which helps us to develop
cool things, bring the cool things and yeah, we're glad to be part of it. So thank you.
What are we going to talk exactly today? So that will be the problem and concern. So
why I'm even here telling something to you, right? What's the problem? How we can solve
these problems, right? And we'll talk about what's the ray appliances and then I'll try
to show a demo, you know, and well, think happens, but trust me, I'll try to make it as
smooth as possible. All right. So what are the key problems right here? Now, if you'd
like, like, airlines, AI, and all that thing, it's cool, it's fine, but it actually takes
a little bit of time and effort from you to spin it somewhere, you know, especially if we
go beyond the local host, right? That runs on my laptop. So if we talk about something bigger,
then that's quite an effort to build, right? And then you can fall back to something like
the managed or, you know, AI as or LLM as a service, which is fun, you know, sunshine and rainbows
until you start to bang for that. So then it becomes, well, quite expensive. Now, another thing
is now the configuration, right? So every public cloud vendor that supplies you with this kind of
stuff has its own way and thinking how you should configure, right? So if you migrate from
one to another, then well, learn from scratch, ah, pain in the ass, excuse me. So, right? So if you'd
like to go a little bit further and you'd like to have something unified, that would be the problem.
So how we can solve it with the help of the openabular, right? First of all, we've
probed the idea of AI as a service to your data center. And by data center, I don't mean
data center like the one that's on-prem, but anything, right? So it could be public cloud that you're
using and managing with the help of openabular. Yeah, why not? You can run it there as well. So you
get this flexibility of moving that stuff from on-prem to public cloud, from one public cloud,
to another public cloud, and do it the way you wish, maintaining the same configuration approach,
right? So again, it doesn't matter where you start it, it's going to work. Now, another thing is that,
well, yeah, you would like to use GPUs, right? And in openabular, we developed it the way that you can
actually enable the GPU password and allow the virtual machine that's going to run a specific
appliance to benefit from the GPU password. So again, sunshine rainbows, right? Another thing is
that, well, you would like to deploy it, and to deploy it, you would like it to be fast, you would
like it to be simple because, well, I'm not the AI expert, right? And if I go deep into that,
that probably spinning a real thing in the real server would take me a couple of days first,
maybe weeks, and depends how deep I would like to go. So if you're not the super expert,
but you still out and build something that is not, you know, public to available, or you don't want to
use the CHAPGPT by OpenAI or anything, you know, even more public, then it's quite painful. So again,
with the help of Openabular, you can kind of deploy it in three to four clicks. Of course,
supplying the specific things that you need, like, you know, password, API key and stuff.
And well, where you get those models from hugging phase, right? So another open source to
kicks in, and you just easily deploy that. So and aside from just what I've talked, what we also
offer is working with other clients, right? So you deploy it that LLM, you start to use that,
you start to develop something that works with that LLM, but then it's not enough just deploy.
You need to move it, you need to administer it, you need to make backups, you know, in all the
drill. So for that, again, Openabular provides a full set of tools that you can use, you can leverage
to make sure that you have your operations continuously, you can migrate it to closer to consumer,
you may, you know, run a multiple instances of the same of the same clients. And what's my favorite
one because I love to be to care about the customers or about the end users. So the whole
that thing makes the end user experience smooth as a butter, right? So this is the ready to use
a clients called Ray. So we're using Ray for that, and it's my, but what I mean by ready to use,
I mean ready to use ready to use, right? So you import that on your system, do small fine tuning,
to make sure that it suits your needs, and that's it. So as everything in our marketplace,
it is ready to use. So again, another cool thing. It has a couple of variables that you need
to configure in order to make sure that it works as expected. So the first one would be the API
port, right? So if you'd like it to listen to a different port or can you go with default,
you can increase the temperature and find the temperature to be a more consistent or to be
more creative, then you need your hug and face API key. So and also the model ID. So you need to
select the model that you would like to use. So not every model is supported, there are a list of
the ones that are supported, but these are the most used ones, and you will find the one that
actually you like. So there are some requirements, like at least eight gigs of memories required.
So again, if you would like, so the thing can run on your local host. So if you have a powerful
machine with you, you can start it on your Linux laptop without any issues. But at least,
must dedicate the eight gigs of memory. I would suggest to give a little bit more, like,
I'm starting with 16 to have it. Then what you also can do, you can upload your own
Python script, and this Python script will work instead of our script. So our script is
kind of to showcase the capabilities, basically that would be the basic chatbot, but it's very generic,
so if you'd like it to be more fine tuned, you can check what we did, so take our code,
revamp it a little bit, and pipe it down to your to-app lines. So, and by default, just a
sign note, if you will start with that, the this size that's dedicated is eight gigs, you would
like to increase it, because what we would like to do is to be very tiny, so you can
fastly download and import it into your open and build environment, but then you need to
increase it to make sure that it's around 80 to 100 gigs to facilitate your needs.
Now, as I mentioned before, aside from just giving you all that capabilities, we are simplifying
with this the LLN deployment. Well, more or less, reducing the operational costs, right, because
instead of just paying to a cloud provider to basically use the same thing, you can use it yourself.
We provide a native support for the GPUs, which again is a win-win, so it can run on CPU only as
well, but you know, with GPU, you get much more performance. Now, the robust multi-tenancy,
so you can actually provide different quotas and make sure that some of the users have
different level of access and make sure they're separated and not interfere with each other.
And as I mentioned before, it's the hybrid cloud approach, so it means that you can deploy this LLN
where you want, so you're not tied to a specific vendor, it's a specific public cloud provider,
it could be your local host, it could be on-prem, it could be in a public cloud. So again,
you don't need to obey the rules and laws of and pricing of that public cloud provider.
If you don't like it, you say like, okay, I'm moving to another cool cloud provider.
Right, so what's going to be next in the upcoming release, so in the upcoming releases,
we are going to support the VLLMs, we will integrate, we'll work on open AI API,
and we will also extend the list of supported LLMs, right, from the hugging phase plus,
some recommendations on sizes, fine tunings, and so on so forth. So right now, it is ready to use,
but of course that's the first release, like first public release, so comments, comments,
are welcome. Now, in order to do a small demonstration,
right, so let's have a look where to find that awesome appliance, right, you go there to the apps,
which is basically a list of all services that you can install, and my dear service,
mini-io, WordPress, and so on, so forth, but you would like to search for rate, right,
and you would like to obviously export that to your open Nabila, open Nabila environment.
Now, that's going to appear as the VM template. That's what I'd like to say.
We have images, we have templates. Image is a snapshot of a disk.
Template dictates the way what you'll get in the end, how much RAM, how much disk space,
what's the version network, how to even name the appliance, or name the virtual machine.
So that's why we always working with the virtual machine templates. And virtual machine templates are
customizable. So from the memory that you are going to provide to this awesome virtual machine,
down to the CPUs, down to costs. So with it, it's AI as a service, which may be AI as a service
for yourself to make your life easier, or it might be a little bit further for your customers,
your subscribers, or even in your bigger organizations, you know, organizations sometimes charge
each other within one team, charge another to make sure that the money moves. So you can actually
calculate how much a specific user or a group of users will owe you after they are going to deploy
that virtual machine and going to use it for a while. So all that is customizable. So then that's
where what I've mentioned before, it comes as ADICs good to quickly download, but you must extend it.
So extend it to at least 80. Give it some kind of a network, because it must download the
appliance from the hugging phase. And basically that's it. You don't need to edit anything else.
Remember, give it a little bit of power, extend the disk, and give it a network. That's it.
So then you obviously would like to run it. Now, yeah, one thing I forgot to mention, excuse me,
is that for now, this is going to be launched as a CPU dependent. So I do have an instance with GPUs.
I will showcase the GPU, but yeah, so if you need to attach the PCI device, so aka GPU in this case,
then you also need to edit that at a template level. So some of these things will be available
when you try to deploy the virtual machines. Some of these are not going to be available. So you would
like to make your life as easy as possible and pre-bake a lot of things at a template level.
Now, going to template, click instantiate. So again, verify that all things look as you wish.
Next, now here's the fun begins. Now, you need to supply with some of the variables,
and some of these are just the endpoint for your API. So basically for your chatbot,
the API port, again, this should be in sync with the Python script that you are going to provide
if you go with a custom thing, or leave it as is if you just go with whatever we supply as the
example. Now, find you in the right model, the temperature, so give it a little bit more of
creativeness or give it as straight as possible. Then select the LLM model that you would like
to run, I'll go with Metaslama, but yeah, you can use others. And again, more to come. So it's
just a first iteration of the supply. Don't forget the API token. That is very important.
And before you start to blame us, check your hugging face account. Are you actually able to
download this LLM, or maybe you need to fill a form and wait for like five to ten minutes, so you
will be approved to use that model. So that is just a hint from a truly yours. And I need to copy
and paste that. Yeah, please prepare your cameras, so you can reuse my hugging face API key.
Oh, no, sorry. I'll just get it. Gotcha. So, and then the one app ray model prompt. So something
that we are going to pre-pand along with your chat. So what will be there? Well, it's up to you.
Depends on how you'd like to find to it. So one of my examples is that I'm trying to
think about learning Spanish, but I'm very bad at it. So I need a sparring partner. And I don't
want to abuse my friends with my very bad Spanish. So I can ask the chatGPT, so your chatGPT excuse me,
the LLM model that, hey, you are the Spanish instructor. So please assist me in learning Spanish.
So something like that. Or may go with a little bit different use case and say that you are,
you know, the Python professional. And you would like to assist me with the code.
Now, so again, this could be the prompt that we are going to place before the actual chat.
Next, verify the network and storage again. So this just to make sure that you may want
to attach a different network. Whatever the case may be. And that's it. So the appliance is
going to be deployed in a while. Again, if you see the green light here that doesn't mean that
actually it's up and running. So it means that the virtual machine is up and running, but there's
the internal process that happens. So we need to download the modular itself. We need to
configure the rate to actually serve that. And to achieve that, we have, well, you can have the
command line access to that and you can run some nice commands to make sure that your appliance
is working as expected. So for example, come on, serve status is going to show you what is
the current state of the appliance. So it might be deploying. If there will be any error,
you are going to get it here. So if it fails for whatever reasons, for example, that DNS or maybe,
well, you have a very, very bad connectivity that also can happen. But yeah, in general, all your
can issues or all your bells and whistles are going to be there. Now, what you would like to get is
this. You can think of a big completed, which basically means that it's up and running and it's healthy.
So now you can start to talk to your chatbot. Now, we supply the client.py, so it's a simple
Python application that just works as the chat interface. Well, whatever your imagination, you can
use whatever you wish. So what you need to supply here is the IP address, the port and the API endpoint
aka chat. In this case, it's, well, in my case, it's 12701 because I've connected to the
ray appliance directly. But again, if you're, if you're routing allows, then you can do it from your
computer so you can download this Python thingy from the one apps repository. So it's available
as the open source stuff. Basically, hi there. So that is your helpful assistant and you can say
something like basic Python header. It is going to take some time to come up with the response.
So that really depends on the amount of resources that you supply to this LLM and, well,
to certain extent, it may also crash if it's up for for too long and you're not interacting with
that. So they're kind of looking why the thing happens. But yeah, so sometimes it unfortunately happens.
Now, let's give it a while. I will return back to this to tell you, so we can see the actual
basic Python header. But yeah, before that, I would like to answer your awesome questions
that you most likely have. So, come on, where's the mouse? I've lost power. Yeah, here
this. So, meanwhile, questions and I'm ready to answer them while this thing responds to me
with a prompt. Do we have any? Well, rate self. So it's, do we have any log messages?
Yes, we do, but it's not, we as the open envelope. So rate appliance itself writes a ton of logs.
You can find them in the et cetera, ray, then current session, logs, and there will be a plenty.
So you can enjoy. Yeah, but again, it's not a troubleshooting session. So, I don't want to
go into the shallows of the depths of troubleshooting the appliance. So that's going to take
too much time. Yes, please. Any other questions? Thanks for the question, by the way.
Yes. So, this service way is more intended for the burdens, as opposed to coming in place
to the platform, which is intended for sharing AI models and stuff. Is there any developments
on the, are there any open source solutions for something like having a space home?
Well, is there any solution for the hugging face hub? Sorry, I'm not in the position to answer that.
So in this case, yes, we kind of give you the platform to use it, but yeah, it probably need to call
hugging face, and I'll spend this question. But thank you, noted.
Excuse me, but the mic connection just hung to that, because it's a remote data center.
But if you are really interested into that, I'm available here, so you can just jump
and I'll show it on my computer. Thank you, any other questions?
Thank you.
Thanks for the action.