So our first talk today is going to be Ben Sully. He's going to be taking us through the
August time series toolkit for Rust and take it away.
Thanks very much.
Thank you. Welcome to the Rust Room, exciting.
I have a bit of post-end before and now here I am, first on stage.
So thanks for coming. I'm going to be talking about orders, which is a time series toolkit for Rust.
It's more important in time series analysis and it also has Python and JavaScript bindings.
And the last of which is being used already and refined as content, which is pretty cool.
I'll talk a little bit about that in a bit.
So who am I?
Oh, sorry. Is it too loud?
Okay. Any better?
Even higher.
This is as much as we can do.
I'll try and speak louder.
So who am I, first of all?
I'm a software engineer at Carfine Labs and based on the UK.
My spare time. I like to do boldering and running ultramarins, which is a normal thing to do.
I've been at Carfine Labs for about four years.
I tend to use Rust for personal projects and hackathons as much as I can.
But recently it's been kind of squeezed into very different bits of Carfine Labs, which is exciting.
My background is kind of in statistics and machine learning.
Oh.
What do you think we need to do?
I can lower it and you're just going to have to project.
Okay.
Yeah, we're going to do a lower.
We still need the microphone for the audio of the video.
But he's going to try to project.
I can do my best.
Is that any better? Can you hear me?
Some more slightly?
I have a back of this one.
I have work.
That one's working.
Okay.
There's a lot of echo.
Okay. Well, I'll try and speak as loud as possible.
So yeah, we're going to do.
This is a quick summary of the talk.
Hopefully technical issues will be fine.
I think the talk about what always is.
What can you do?
And then the second part is going to be lessons that I've learned while translating various
different ML algorithms to Rust from different languages.
Some of them things like C++ or Fortran,
are Python.
They could have been written 30 or 40 years ago.
So there's lots of things involved in even finding decent source implementations.
Well, it takes me time for questions.
And then as a bonus, this was meant to be.
And I mean, talk about I ran out of time.
And if you want to check it out, feel free to download the slides and have a look.
There's some content there on things I ran into and trade-offs that had to be made,
exposing kind of a JavaScript interface using WebAssembly.
So first of all, what's to deal with the name?
So to Orger is kind of a verb or now a meaning to predict.
So I think this might be one of the few times in my life.
I've actually named something well because it means the like thing.
But I don't know, hopefully I'll get lucky when I have a kid.
I'm not sure.
Maybe on name them using the same method I use here,
which was domain name during development.
Basically, look to a source for a word that ends in R.
I hope that the dot RS domain is available.
And then write the project later.
So I've got to summarize what time series is.
Probably most people know this, but it's pretty straightforward.
It's a measurement taken repeatedly.
Generally at the same interval.
Usually it's something numeric.
Either a counter, a floating point number.
So they're really computer's level working with them.
Because they're just numbers.
It's, it's, it's, it's a nice.
Fun optimizations you can do both with compression and storage.
Whether that's in memory or on one disk.
And also the process in them is, it's fun.
You've got lots of optimizations you can do.
In the real world, there's some examples of where you can see them.
They're kind of all over the place.
You bit quit us.
Here's my heart rate plotted over a day.
Like environmental software.
You'll see them everywhere.
So things sort of things we do.
We visualize them, right?
I mean, the farmer definitely uses visualizing things.
Whether one dashboards or in the kind of explore view,
you can see lots of time series there.
We like to set thresholds for them and get alerted
when things exceed certain boundaries.
For example, disk usage going above 90%.
Or maybe you want your service to auto scale
if requests go per part go so high, something like that.
And you can do that dynamically,
a standard threshold or using something a little bit more advanced,
like a normally detection or outlier detection.
And this is kind of where all of this comes in.
So all of this is, as I mentioned, a time series toolkit for Rust.
Right?
It's designed to help you with all of these previous tasks.
More specifically, someone on Reddit pointed out quite early.
It was a time series analysis toolkit.
We don't do the kind of that all of the things you might imagine
in time series libraries to do.
We don't do things like resampling yet or storage.
Instead, we implement various different machine learning algorithms.
So forecasting is the most obvious one.
That's kind of, you want to predict the future.
Or you even want to predict now.
And you often want to do that with confidence into all of the prediction intervals,
so that you know how accurate your predictions are.
Clustering, you want to group lots of series together.
You have hundreds of series and you want to find the groups within those.
Those series that are behaving similarly.
Outline detection is a bit of a confusing one.
But this, we refer to this as when you have lots of different series.
You expect them to all behave the same.
And you want to identify the ones that aren't behaving similarly to the group.
And the change point protection is what you kind of looking across time.
And you want to see where your time series, your behavior, your time series changes.
Whether that changes in magnitude or changes in variance, various different properties
that you can detect there.
Just a little bit of examples of each of those.
Forecasting some of the most obvious one.
You just want to be prediction, right?
You maybe want to account for things like seasonality.
You have daily seasonality or weekly seasonality.
You depend on things happening on weekends.
And you might want to forecast if I say or is it or capacity.
So that's, yeah, that's very straightforward.
In August, we have three algorithms for this.
Very, like, kind of left to right.
They go from a simpler or complex and faster to slower.
Extremely reductive.
And they support, as you get more advanced, they can support things like holidays.
We have, like, maybe you want to model COVID separately to the rest of the kind of the normal world.
And things like Christmas, if you wanted to model that in your time series.
I can't put that in your predictions.
In our detection, this is used to identify when one or more series is behaving differently.
So, as I said, this example here is my, you can see it very clearly sadly.
But it's my pie hole home blocking ads.
And you kind of expect many of it remains to be the same.
But actually, whatever it is, beacons.gvt2.com is absolutely miles and away higher than anything else.
It's a nightmare.
You can also imagine pods in a Kubernetes deployment.
You expect them to have the same CPU usage, if they're being low balanced correctly.
And you want to flag in case that isn't the case.
Two algorithms that we use, medium absolute deviation.
So, this is when series are roughly expected to be constant, but the same.
And it's really simple, you just kind of flag whenever the medium is, if each series is different from the group.
And DB scan, you can use when series have more complex patterns, like seasonality.
But you still expect them to move similarly and move in the same way.
And you want to, so it's more complex algorithm, a little bit slower.
But it can handle these more complicated cases.
The reason you might use medium absolute deviation, I guess, is because of its more intuitive easy to explain.
Clustering.
You might use clustering if you want to identify a creature series.
So here in the screen show, we've got one big band across the top and then just clearly like two separate bands.
A little bit lower down.
This useful if you want to kind of group these together.
So in this case, we would flag those little groups, which is useful in quite a lot of instances.
A little bit about the way that works.
It's quite a fun algorithm, and also it's probably the coolest name of any of the algorithms I've seen.
The dynamic time warping.
Why is it called that?
I don't know, it's awesome.
The gift kind of gives it a way here, but you're calculating different distances between each of your series.
Each pair of your series in your data set.
And as the kind of naïve where you would use it, like you clearly in distance where you can pair the same time stamp.
And dynamic time warping instead, you're optimizing to find pairs of values that minimize the overall distance.
And kind of account for shift in time of those series.
And I evenly, that would be really, really slow, because you've got to do like N squared comparisons.
And there's a top right shows.
But there's lots of little optimizations you can do to speed that up by limiting the window that you are allowing us to differ by.
So that's quite fun to implement.
And there's lots of options you can tweak in August to do that.
So the way that works after you've got your, what you use a dynamic time warping to find distances between each pair of time series.
And after that, you feed that into an algorithm called DB scan.
Here's a distance matrix.
In this case, we've got one series at the top, and then you can see some of the distances are a little bit higher.
And then by feeding that into DB scan, you just get a simple marker of like whether a series is in one cluster or another.
So the API is a pretty simple and easy to use.
And close them into our colors.
Some stuff that August doesn't do.
But I was trying to useful to know when I shouldn't shouldn't use something.
Some of them might be obvious, but maybe not.
So we don't be plotting, plotting isn't really like time series in certain necessarily.
Like, there are, there are libraries in the rest, so plotters is really good.
And you can use U-plot from JavaScript, or there's loads of options like map.
We don't live in Python.
We don't do time series data structures.
I kind of touched on that earlier.
But it's more, we tend to work with just simple like Vex or floats.
Things are quite straightforward.
And I don't know if there is one of those right now.
So there's scope to add it to August or to other libraries later.
And storage and compression, not really are concern either.
There's not a database, it's not really designed for database usage.
So you wouldn't use it for storing these things instead you'd use.
A time series database like Vex or influx, something like that.
You should use it, my opinion.
First of all, you need to have some time series.
That's very straightforward.
But like the code is clean and it's relatively fresh.
I mean, it's been idiomatically converted from original live algorithm and languages,
which is nice, easy to contribute to.
It's rust only, it's nice and portable.
You can use it wherever you want to compile to.
And we have jobs written by some bindings.
And as you might expect, it's relatively quick.
Like it's, I haven't done extensive benchmarking against other languages,
but I've made sure to profile things.
I make sure that it's not like crazily slow.
So it is generally faster than other implementations.
Even compared to sort of NumPy,
which you would expect to be fast to have things very highly optimized.
I think largely because of the extra control you get in rust
and lazy operations like iterators and compiling down
into very optimized code.
You get fast implementations by default.
Okay, so that was a very high level overview of all this.
There's a lot of things I didn't have time together.
Please do check out the docs and the demo and the code.
And I'll touch on those later.
So next up section two is,
I want to talk about the process of actually conversing in LML algorithm
from another language to rust.
And there's a lot of lessons that kind of came there.
I'd like to pass on.
So this is the kind of process that you might expect to go through.
If you're converting an algorithm right that you've found
in a different language.
It looks fairly sensible. It's quite nice, right?
It's probably what the working with legacy code book says.
I'm not sure.
However, things don't usually go to plan.
And especially here.
So when you're looking for source implementations,
there'll be meant.
Especially if it's a popular algorithm,
you'll find several in different languages.
Each of them will have a different trade-offs
and you have to kind of choose which one to go for.
You'll be very lucky if you find tests for these things.
That will be lovely, wouldn't it?
But they don't really exist.
People will tend to maybe write examples and blog posts,
but you don't tend to get tests.
These algorithms are often written by researchers or scientists
who have better things to do.
Your implementations that you found will all disagree.
Which you won't realize until a little bit later.
That'll be fun to figure out as well.
As you're translating, you might kind of think,
this is awful.
What are they doing all of this manual indexing for?
We should definitely rewrite that a little bit better.
And as you're refaturing, you will think,
yeah, we should do this much sooner.
Let's shift this earlier in the stage.
And then you'll end up with this kind of half translated half,
like Frankenstein's monster of a code base.
And who doesn't love optimizing that?
That will come ways sooner.
You'll definitely start that in 20% through.
There's no way you're waiting to be end.
So the end reality looks a little bit more like this.
You find a lot of indentations.
You'll be swatted back and forth between them.
You'll translate some functions line by line.
Others will be done.
It will be a hot mess.
I'm going to be asked.
Get familiar with the debugger.
But fortunately, there are some things that you can do to improve this process.
So I'm going to go through and give you a little bit of advice.
In case this is something you plan on doing.
So first of all, finding the source implementations.
Nothing's going to be perfect.
The whole point in you doing this is that it isn't perfect.
It's not a raster, right?
So there's no way.
But also, it's going to have problems.
It's going to have just going to be slow things in there.
You're going to have to accept that and make it better as you go.
Why are we trying things with published papers?
It feels obvious if you see it.
It's really useful to have a paper or a book or something concrete
that you can then refer to for motivations.
Extra comments, that kind of thing.
Ideally, you want something that uses as few languages as possible.
I mean, even NumPy drops down as I see you.
So you're probably going to be struggling.
For example, an R library that I translated that ETS implementation
which we use in August for forecasting is written in R.
But it has some, R has some built-in Fortran functions.
So you have to go and figure out what they're doing.
And then it also has a manual C++ implementation of an optimizer.
So we had to go and figure out what that was doing too.
It's not recommended.
Ideally, you will find something on Crate.
So that would be the dream.
And often that is the case.
You need it to be open source.
Ideally, it would have tests.
But as I said, that's unlikely.
And if you have a responsive author or a recent commit,
that's great because you can talk to them and ask some questions.
So do you be getting tests?
As I mentioned, you're not going to find much.
But tell where you can get.
So if you see any reproducible examples,
whether it's in papers, blog posts, books,
you can turn those into examples,
integration tests, spent maps in your ROS Crate.
This is going to come to virtual,
but you're probably going to want to test some of the implementation details.
Like, as you go, there's a lot to do.
It's low-level things that you're not going to be confident in.
So write test for them.
If you have to throw them away, that's fine.
That's not a big deal.
You're going to want to be, this is a bit facetious,
but you need a big screen,
because you're going to want those two things side by side,
and be able to step through each line by line.
So yeah, ROS, as I mentioned,
the test screen won't make this all very nice.
I tend to make everything that I find, as an example,
into an implemented integration test and benchmark.
And exercise your public API to make sure it makes sense,
when it's been converted.
Assertions are great.
Make sure you're using debugger set.
They're helpful.
So when you're translating, when you get to the translating bit,
my advice would be to start with iterator and esitors,
and just the standard library, and work with these as much as you can.
Low dependency count is like the new hot thing.
So if you can get by without adding any of these kind of array dependencies,
then that'll be great.
And also, if you learn the adapters and kind of work with the functional style,
there's often enough in there to reproduce even like a really obscure number by functions.
And I'll show you an example in a minute,
but the other advantage is that this gets really heavily optimized by ROSC.
So you end up with really fast code without even realizing that you've done it.
And you might even get things like SIMD for free.
It's fantastic.
As an example at here, as much we can see there.
But the left example is the NumPy code.
And the right is iterator and esitors in ROSC.
This is exactly the same algorithm and it's literally like taking out of the code bases.
There's exactly two allocations in the ROSC version.
God knows what's going on in the left side.
Performance wise, there's a lot of manual indexing.
There's like loads of copying going on.
So you get like a lot of power just by doing these standard iterator adapters and using those as much as possible.
There's another example of that in the optimization section.
However, if you end up working with 2D arrays and metric matrices and anything that has more dimensions,
things get problematic.
Vex of Vex are inefficient.
They're not great for various reasons.
You have to do two loads of interactions to get to the actual value.
And they really don't play well with kind of auto-dref of ROSC functions.
So especially if you're using them as arguments to your public functions, it's not ideal.
So in this example we have a function that takes a 1D array.
It takes a slice of floats.
That's great.
You can pass a reference to a vector to everything.
We'll get also de-reft, really, and everything works.
As soon as we have the 2D function, we want to take a slice because we don't need to own that value.
But now we can't pass a Vex.
We can't.
If you try and pass that, then everything blows up.
Because the inner Vex can't be de-reft.
And the array has got you covered here.
This has really efficient representations of 2D and multiple dimensional array.
So it's really, really useful.
It's basically, it's got a lot of the functionality that NumPy does.
And you can, they're even a good, like, NumPy.
And the array for NumPy uses this doc that you can use there.
So yeah, the function works.
It's great.
So what if you run into a huge dependency?
And actually, as I was thinking on the training about this,
it doesn't really just apply to ML.
If you're rewriting a code base in a different language,
and it has a huge dependency, what can you do like that?
You don't really have time to rewrite it as too much to expect.
So this happened in August when we were translating the profit algorithm.
Profit basically does a bunch of data manipulation,
and then hands everything off to a library called Stamp,
which is a Bayesian framework for doing Bayesian analysis basically.
Rewriting standard would be impossible.
It's like, a best in class framework.
It's written by experts in the field.
It's like, how a combination of APIs of C++ with, like,
17,000 commits.
And the way that it's called is like,
you compile a binary that represents your model
and class data to it in files.
So, that's not going to be rewritten.
That's going to be too hard.
Instead, we kind of thought, well, how can we still use this?
And we want to use it in the browser as well.
Well, how are we going to be able to do it?
We thought, maybe we could compile stand to WebAssembly, right?
That's a bit of a wild idea, like, some really old,
ten-year-old C++ code bases.
Will it compile to WebAssembly?
Surely there's dependencies there, like, system calls,
and various different APIs that WebAssembly doesn't support.
And also, how are you going to pass the data,
like, WebAssembly only has numbers,
and we need to pass much more complicated things it.
Fortunately, the WebAssembly system interface now exists,
and the component model.
It's a bit nascent, but these things handle exactly this.
Exactly these use cases.
I'm not going to go into too much detail because I'm not
time, but basically, the model is you write an idea
that represents the kind of work that you want to do,
and you do that in a language called WIT,
which broadly represents a prototype,
it's like a kind of idea of a standard idea.
Well, there's an example in a second.
And then you write a tiny bit, promise,
it's not much, tiny bit of C++ to implement that,
using the stand libraries and calling in to stand as a WIT.
And then you can compile that to a WebAssembly component,
and that's basically a WebAssembly module,
self-contained, portable, and can be run using any,
in theory, any WebAssembly runtime.
So the idea looks a little bit like this.
Sorry for the dark mode.
You have records and stocks and all of these things.
When you, the component model's tooling has a bunch of,
but a generator to convert that into idiomatic code for your language.
It doesn't have to be rust.
So this will get started into idiomatic rust, idiomatic go,
or type script, anything you can't need.
It also automatically handles conversion of those types.
So if we've got that, we then need to use it from rust.
So we're using, there's a variant that comes along with the component model of bind gen,
which basically takes the path to your idl file,
and generates you a bunch of stocks and traits and everything for your code,
and you can then call that from inside rust as if it were a stock,
but it is a stock and has functions and methods,
and you can pass everything in there as if it were something you didn't.
In this example, like we were embedding the WebAssembly that we just compiled.
So that's in the binary, there's no runtime dependencies, whatever.
There's a bunch of, setting up the WebAssembly runtime is just a bunch of machinery really.
And then we can call the optimized function just like it was a regular function with regular stocks.
And it runs inside WebAssembly at near native speed.
We don't have any, like, build time dependencies on any C compiler,
or any C++ compiler, and everything is purely embedded in the binary.
So that means we can use it from, we can use it in WebAssembly if we need to.
A few caveats, like I said, it is new, it is, there's a lot of playing around.
This is some of the repose in the bytecode lines.
Org, a huge respect to the bytecode lines.
There's just so much going on there, I don't know how they produce so many things.
It's early days, and you have to do a lot of hacking around, but it's fun.
There's a lot of tools to learn about that kind of thing.
Another downside is that WebAssembly doesn't support certain features, like exceptions,
and we just have to hard abort in those cases, which is not the end of the world,
but it's not necessarily that pretty.
And the minute only, wasm times supports this, but that's not, that's fine.
Wasm times put it in rust, so we can easily just embed that.
Okay, so we're moving on to refactoring now, and I've really got a little bit of advice here,
because I had to cut that out.
But when it comes to writing idiomatic code, you have loads of options.
This is about using the type of system responsibly.
And what I mean by that is, you can do some really clever things with rust type system.
It's probably better than the source language that you're coming from.
So you can use things like typestick to avoid things being misused,
where you want to embed, like a state machine in your type system,
to make sure that people can't, for example, fit a model twice,
or predict when using an unfitted model.
And that's really powerful when you combine it with that type,
so things are combined, and that state machine flows really nicely.
The downside is that your users will probably want some way of doing this at one time.
And if they want to do it at one time, then they have to store these two different structs anyway,
with the different type, they're different types.
So they're going to have to use an either way, like they can't,
unless they store these things in two separate options, which is a poor way of doing things.
So you might want to, you will notice this when you're writing to your bindings,
if you write some of the Python JavaScript, because you are basically your user at that point.
And anything can change when you're writing when you're writing Python JavaScript.
So here's an example, like in machine learning that often you have a concept of an unfit model,
where it's just been created with some hyperparameters, and then you pass it in data,
and it turns into an actual model that you can then make predictions with.
So the type state where you're doing this might be to have an unfit model, which has a fit method,
and then a fit model, which has a predict method, and you can't miss use that.
However, when your user wants to have a button that turns one into the other,
they're going to have to have some way of representing that, and they're going to need to have some kind of tag,
which ends up being probably this enum that we have with an unfit and a fit model.
So you can consider that when you're writing your APIs, but how they're going to be used,
you may just want to offer this instead of the type state model instead.
Obviously, this turns one error into runtime errors, so everything now returns a result.
It's just the way it is, unfortunately.
Finally, a little section on optimization.
So as I mentioned, we want to use real-life examples as benchmarks.
Make them as realistic as possible, if you have your own data, then that's better.
And split them up into the sort of things that your users are likely to do.
In ML, that's going to be things like pre-processing, fitting, predicting, or clustering, that kind of side of things.
Use criterion, read about the timing loop, as well.
So there's no way of passing your data into criterion,
or that's itter, it's a batch, itter batch, or itter batch, or itter.
These differ based on the way, how you have to use the data that you pass into the benchmark,
whether you have to take it by reference or multiple reference or open as an object.
For profiling, so I kind of made this slide for myself as much as everything else,
because the internet recommends various profilers, and I always forget which one that it is that works best.
As an example, it's sampling, it's always sampling, I always forget that.
It's great, it's perfect.
The invocation is tricky because you're calling sampling, which then needs to call cargo bench,
which needs its own arguments, which then needs to call criterion, which has got its own arguments.
So there's a lot of kind of manipulation.
Basically, I would recommend copying this and using it everywhere.
So by default, this uses the five-box profiler,
pops it open in a browser, and then you can use it to kind of debug things immediately.
It's really fantastic, it has everything you would expect from a profiler,
with call three using the four flame graphs, source code, and line counts, that kind of thing.
Including the assembly itself.
You can even share it with other people, and upload it into, kind of, get your issue up here already easily.
In the notes, for this, if you want to download this slide, there are more tips on performance,
specifically written by Nicholas and other group, who just lost performance work is great.
A few more hints on optimization, so specifically,
your slowness is likely to be in allegations.
Generally, you wouldn't want to pre-allocate with capacity.
If that doesn't help, and you're finding a lot of more allegations than you expect, still.
We're not using the same backing buffer a lot.
This sounds obvious when you say it, but a lot of these original algorithms, as you're translating them,
will, they won't have the same level of control that rust offers.
So you're, as you're translating it, you might not notice that you're actually reallocating every single time in a loop.
If you can reuse those same backing buffers, try and keep it out of your public APIs.
You can use a lot of jemilee species up by nicely.
And be aware of how vexed from itter allocates.
The docs for this are actually under from iterator, the implementation of from iterator for vex,
which is, I would say, about two levels away from where you would expect to be, perhaps.
But it underlies the collect method of iterator, using a lot more than you might think,
it has some nuances around how and when it allocates and preallocates things.
This should get you quite far, like with these tips you'll probably be okay.
But if you want to squeeze everything out, you may want to do some more useful things.
So one thing I found was really helpful is to replace any explicit indexing that you're doing with iterator zip or iterator tools,
either. So if you're iterating over a lot of things and you need to assign to an output array or anything like that,
rather than indexing, even using, so on the left here, this old example you used get unchecked.
So you can use unsafe get unchecked and pass it in index.
But then you're adding on safe into your code, which ideally you would avoid doing.
And you can avoid doing by using zip.
And the compiler knows exactly what's going on there.
It can optimize things, it can eliminate the bounds checks, anyway.
And you get by the cache locality because it knows how to load everything in cache lines.
Another one that we found quite often, division and square root is surprisingly slow.
So you need to avoid doing it in loops.
Often you can store a scaling factor instead, so in this case we were doing some multiplication and division on every single call to iterators.
There's an iterator next implementation, right?
We can actually store the scale factor instead, and then we only have to do the most of the divisions once,
and we just do multiplications on every iteration.
Okay, another example of this, by the way, was square root.
So in the implementation of the dynamic time warping that we talked about earlier,
we calculated that you could be in distance.
That involves swearing something and then taking the square root for every calculation.
Instead of that, you can just square everything, sum all the squares,
and then do one division at one square root at the end.
It's pretty obvious.
I was wrong when you look at outside of the code, but it helps loose, and it's a huge optimization we did.
Okay, that was a lot.
I hope there's some useful tips there.
I think we're going to take questions.
Let me do some other questions.
But including questions.
Okay.
I'll do questions first, because I don't know if I'll get through everything otherwise.
So, yeah, there's a little summary.
Please do try August.
Then it's only had time series, and you want to try analyzing it or working with it at all.
Give it a go.
We have, as I said, Python bindings, and JavaScript bindings on MPM.
It's being used in the fastest front end, which is exciting.
We use it for outlight detection already, and there's also APIs for doing forecasting,
and change point section there.
Give it a go.
Give, porting sum and algorithms.
It's fun.
You can have a lot of fun optimizing things and speeding things up,
and yeah, writing some really nice APIs at this piece of piece of piece of piece.
And I would say, explore WebAssembly, both of the, like,
the complex dependencies side, using maybe using WebAssembly components, maybe not.
And for the JavaScript bindings, there is a whole section of this talk,
which I've probably not got time to cover, on the different trade-offs that you can make when using things like
Wasm Pack, and Wasm Vine Gen to generate your Wasm bindings.
And that's usable from anywhere, whether it's in the browser or different languages in one time.
So, give that a go.
I'm going to take some questions.
If anybody has any.
And then we'll, I thought you're not going to have time to do anything else.
Yeah, sure.
Question the mic.
I kind of already did check online, but the NDRA doesn't support like mixed mix types of columns.
So, it's like 2D array or 3D array, but not like 2 arrays of different types.
Do you have happened to have any recommendations of such mixed type?
Yeah, libraries.
But I think the question is about mixed type arrays.
I would suggest using something like, it's more of a data frame approach.
So, in Python, you would traditionally use pandas or something like that.
Pollas is the kind of rust version of pandas.
And it does have rust APIs.
I have to say I haven't played with them much.
And I don't know how well documented they are, because that's how I checked.
It was mainly the Python bit that was documented.
I imagine that would be the way to go is to use something like pollas.
Yeah.
Hello, thank you for the talk.
I'm Chris, how do you exchange large amounts of data with your vast and looking
bonus?
Basically, how do you transfer the large quantities of data in and out of it?
Do you use any kind of civilization for a mad like patchy arrow or something like that?
I can see what you were saying about the microphone.
It's not been great now.
I think the question about passing things in and out of web assembly.
Yeah.
So, in both the case of when you're using wasm bind gen and using the component model,
they basically use the float 64 array type in the JavaScript side.
And it gets converted into a linear memory block, which is then mem copied into a vex.
So, it's not done using any kind of chunking or any kind of library like arrow.
I haven't noticed any performance bottlenecks in that side of things whatsoever.
It's pretty much a memory I think.
So, yeah.
As long as everything's size is green, you can do that efficiently already.
Thank you for the talk.
On the topic of optimizations, did you try speeding up bottlenecks using Cindy?
And if so, what's your story doing that in stable rust?
I have an open issue to try and do some of these things using Cindy.
But I haven't got around to it.
I think the outlier detection one would be a really nice candidate.
Because, actually, if you look at the NumPy code, it already is kind of vectorized.
It's written in a vectorized way.
So, we could try doing that.
I haven't checked if that has, maybe the compiler's already added those Cindy optimizations in there.
And I would love someone to come along and show me how to do it because I've never actually worked with it.
But when I looked into using the portable simmed in the rust libraries, it's still nightly only, I think.
So, I was never quite sure how to do it using stable rust.
I'm not sure that's such a big concern for this because, well, it would be for rust users.
But for the Python and the JavaScript bindings, we can happily just use a nightly compiler to create those bindings in the first place.
So, that would be useful.
So, if there's no more questions, can we thank Ben?
Thank you.