Okay, hi everyone.
Well, this is now the exact moment of this open research bedroom, and thank you for being here.
But first, let me introduce myself and my team.
Well, actually, I'm not alone on this slide.
Because I didn't want to take credit for a tool that I didn't develop myself or at least, well,
I'm not a regional computer, I'd say.
Because the main developer of exam is Yom Kiddereo, Guillem Plit.
And sadly, he couldn't be here today, so instead you have me.
There's a lot of research engineers.
And you also have a bunch of more active in our manager and also a very enthusiastic user of them.
And we all work as research engineers in the Media Lab, which is a sociology lab.
And you'll see later why this is important, because actually exam was developed in this specific context.
So, what is exam?
It's actually a common line tool that you can use, so in your terminal, two manipulates CSV data.
And the idea is not ours, because exam has actually been fought from another project called XSV.
That is not maintained anymore.
And it was created in Rust by burnt sushi.
If you don't live up in Rust, know that burnt sushi developed many half of the current Rust ecosystem.
So, he's really a celebrity.
He was apparently decided that XSV was not his priority anymore.
And so now, exam was completely rewritten from that baseline to fit our needs.
So, why do we use exam?
And this is going to be the outline of my presentation.
We use then because first we love the CSV format.
Second, because we love the terminal, of course.
Third, for sure.
Because then is easy to use and fourth, because exam is fast.
So, why we love the CSV formats?
I underlined the we, because apparently this is an unpopular opinion.
And well, there are many reasons why we should love the CSV formats.
Again, we're ready to love letter to the CSV format that you can find on night.
And it's an unpopular opinion, because, well, this love letter started a flame war on Hacker News.
With hundreds of people like telling how much they hate the CSV formats.
So, well, let me focus maybe on one point why we love it.
It's mainly because, actually, we don't work with engineers.
We work with social scientists.
And these are normal people.
They like.
They like reading text and being able to edit text in a text show format.
And since CSV is that simple and CSV is text, this is the first reason.
The second reason is because social scientists are used to table a data.
Well, I have to include myself in the lot after 60 years working with those people.
So, we as social scientists, we often think it's in terms of variables, so columns.
Spread across a population of individuals, rose.
And well, we actually have lots of different tools dealing with table a data.
So, it's an Excel, a data, or pandas, you name it.
And so, CSV is actually the formats to interpret those varieties of tools.
So, why do we love the terminal?
Well, maybe, maybe you could say we love it because we are old.
But, let me focus on some figures.
We love it because we do repetitive tasks.
And so, the terminal is actually a short cut.
And well, thank you Paul for sending the video.
It was perfect.
Second, because it's actually an archive of our previous activities.
And third, because the family doesn't change much, or at least hadn't changed much in the last decades.
So, actually, we can continue doing our repetitive tasks always the same way.
And we like it.
So, now that you understand why we love the CSV format and why we use the seminal,
let me explain why you should and actually we do manipulate our CSV data in the terminal with that.
And we do it because exam is really easy to use.
And well, first reason why exam is easy to use.
It's actually because you don't need to know a lot of exam to get started and actually to adopt it very fast.
So, actually, you just need to know maybe two, three, four commands exam to actually benefit from it.
And maybe you will have the rest of your data management pipeline in another cutting language, maybe in Python, maybe in R who knows.
So, just don't feed the doc with the quick tour, test it.
And you will probably adopt it, maybe just for those quick use cases.
So, what commands are the most useful?
To answer that question, I asked five colleagues to search for their exam history and send me the result.
So, let me present this highly representative survey.
Well, the most important command, of course, is Xanview, alias Xanvi.
And it's actually really the only command that you should know if you should only know one.
It only shows a tabular file in a nice format, of course, with the colors depending on the type, et cetera.
But also, it's the last command that you will use in the end of your exam pipeline to show the results of what you did.
Then, there is Xan headers, or Xan H, and it shows, well, the headers of your file header.
So, the syntax here is also simple, Xan headers, my file.
And then, well, if they are duplicates in the name of the columns, they will be shown in red.
Actually, I was surprised by this one, being the third one, because it's not the easiest.
Because, actually, Xan Map is applying the same operation to all rows in your CSV file.
But, of course, you have to then know a bit about the syntax of the operations that you can apply.
So, maybe it's not for today.
Maybe today, you can focus on Xan's search, which is really easy to use.
You just do Xan's search, and then a phrase.
Of course, you can also use regax, apply it on your file.
And then, you can do it to visualize the results.
Here, I selected one specific column.
And you get all the lines that contain your search.
And the last, for today, but, of course, not the last Xan, it's Xan frequency that apparently everyone calls Xan Frek.
And it comes to the frequency of a modality in a column.
So, for example, here, apparently most lines were created in 2022 October.
The second reason why Xan is really easy to use is, actually, because it takes CSV in, and it gives CSV out.
So, of course, you can write the results of your pipeline in a new CSV file.
And also, it means that Xan is very modular.
Since you can pipe commands into one another.
And here, we have the perfect example of a user who wrote this on our GitHub three days ago, I think.
Well, I had to note what's the meaning of ETDF.
It's a pretty empirical, cumulative distribution function.
And, well, that's his use case, and he's using Xan to do it.
So, he's piping Xan progress into Xan map, then doing Xan group, et cetera, et cetera, and being with Xan plots.
And, actually, don't make fun of T-Bosh title.
Because you might end up being doing the same, or many worse, because actually, that's the way you programming Xan.
Well, also, Xan is easy to use, because it's full of useful tricks.
And, well, I mean, I will name a few of them that I use a lot, but I probably forgot a lot of them.
The one I really like is the ability to use the wildcard operator when you select columns, which means that you don't have to type the entire name,
of your columns. So, here, for example, I select all columns that contain sine, sorry, in my file,
but it also means that if I don't want to type a site address, I could only write a site A and wildcard and get my column.
Another useful trick is Xan Hizk, that's a plot histogram, but if you use it with temporal data, you can use the dates parameter,
and it will actually add empty rows for missing dates in the data, so that you can have a correct temporal histogram.
Another one is the ability to highlight some phrase when you've flattened the data, so here you have your data that is Xan Hizk flattened,
and you use the highlight here to highlight a report of your data that contains this phrase.
And very quick one, Xan Paral Cut, let you concatenate several CSV files together.
And since it does it in parallel manner, then you can pre-process each file, and so, while this step is not done, after concatenating the data, but before.
So here, for example, you have like a bunch of CSV data from 2022, so we are Xan Paral Cut to those CSV files, and then pre-processing it by searching for parliament,
and then, of course, highlighting the parliament in the result so that we get this data.
Then, of course, let's be honest, the main reason why we use Xan is because it is fast.
So, how fast is Xan? It's the question I will try to answer with the quick demo.
So, actually, I created three scripts, one that is computing frequency using the Python standard library.
One that is computing in using pandas, and one that is computing it using Polars.
So, if you are new to that ecosystem, pandas is actually using a site to accelerate the process, and Polars is actually using Rust to accelerate it.
So, you would expect that the Python standard library is the slow list, we will see that it's not always the case.
Pandas is a bit faster, Polars is a bit faster, and well, of course, the first step of the demo, but let's see that.
So, here, you have some CSV file, few tweets, and many tweets are the two files we will see.
So, let's apply a frequency on the few tweets.
XanFrag, the user up, XanFrag.
Okay, so this is the kind of data that we get, so these are the most retreated user in my dataset, in the small dataset.
Now, let's benchmark it on the peak data set.
Well, actually, you have to know that native data set contains 90 gigabytes.
And so, it does it in 9 seconds.
Okay, then, let's do the same in Python.
Well, do we really have the time for the...
Okay, well, let's keep the two of them.
Well, pandas is slow because it has to load everything in RAM.
So, it takes actually the same time as the Python standard library.
Well, let's see, let's see, polar, which is written in the rest.
So, the same language as, as Xan.
And well, actually, let's use H-top to see also how Polars used the RAM.
So, Python, frequency, Polars.
And then I'm getting it the many tweets file.
And well, actually, I don't want to see up.
Oh, no, no, no, no, no, no, no, no.
Like...
Both, both, both, both by our Xibet.
For us, then.
No!
Oh, did I name it?
Ahahaha.
Clack, like, like, like.
Thank you.
Okay, so, but I don't know if I can see Polaris, many tweets into slash live slash
And let's start it.
Three seconds, so you will tell me, okay, Polaris is actually faster than that.
This is true, but it's also because if I start it again, you will see that it's actually using all
processes as my of my computer.
And actually, I can do the same with Zen.
So if I do Zen threats here, but I use the dash p, then I start it in a parallelized manner,
and then it only takes one second.
So actually using all the processes of my computer, then is actually faster than Polaris.
I think this will be the end of my presentation.
Thank you very much for hearing me, and I'm open for questions.
You are fast to have any questions?
Nope, one.
There are loads of imports, loads of rights, excuse me, for instance.
I'm just curious how you would compare to Zen in terms of speed and so on, even if the interface is going to be different,
because you are out to a guy on the basis.
Yeah, well, well, yeah, sorry, I really, which is a question.
So the question is, have you heard of that TV and how do you compare it to Zen?
And so actually I haven't tested.
But well, the philosophy is quite different, because I guess you need to index your data in a way, or like,
like create your database in a way, and then like send you queries to that database,
which is actually not the case in Zen, because actually you won't work on like the base file, which is just your CSV file.
And there is no, like, first step where you send your data or index your data in that database,
that's since I don't know that TV, I don't know how it works, so maybe it's very fast.
I think in the example that we saw from on GitHub, it was using pipes, others' batch pipes.
Yes?
Is there anything with Zen to keep track of some of the context where you are?
I guess it's like almost catching an explainability of like,
but with 15 things together, how do I know what the fifth step is doing?
Okay, so the question is,
you are piping, exam commands, one into another,
and so do you have any way of seeing in what context something happens?
I guess in order to debug or, well, actually, exam errors, like,
tell you at what exam commands they have failed, so that's the first point,
but maybe you had something more specific in mind?
Yeah, I guess I'm coming from the perspective of the other huge data set,
100 gigs on pro-sets, and you made them stay halfway through.
Is there any way to design, know about the previous steps or the next one to be able to catch anything?
Or is it just standard classified thing?
No, it doesn't know where you come from.
It can tell you just like the line number of what it has previously seen, but that's all.
Let's say those steps.
Yes.
So the question is, is it similar to SQL?
And I won't be able to answer that question because I don't know what SQL is, but maybe you can explain.
SQL.
Okay, so the question is, is it similar to SQL?
Well, in a way, yes, of course it's a way of preparing some data and getting some results.
Well, as I was saying, like using SQL is also like dependent on like using some kind of database.
And so like putting your data in that database and most often like indexing it in a way.
So that's the first thing that you should have in mind before processing your data.
Well, here, really the idea is I can apply example to any file in my, any CSV file in my computer without like having to ask myself, do I have to index it?
Do I have to put it in my database?
Oh, yes, it does.
It's it's mainly for, well, it depends, but it does.
Okay, kind of type inference at the visualization step, so that you can like visualize differently numbers from dates from etc.
And then, well, at the, at the operational step, for example, if you have some comments that are specific to numbers, for example,
well, it will be able to deal with them specifically.
So, yes, yes.
Yes.
Well, for now, like people are just putting issues.
Sorry.
Yes, so do you plan to have a system to have people add their own comments to well.
And the answer is, for now, people are just like putting issues on the GitHub repo and we will do them or not.
I don't think there is a plan to let people have like create that would be aliases in a way like create a pipeline and you name it in a specific way that could be an option.
We didn't discuss it so far, but it's interesting actually.
And well, always there is always the possibility to find the project and add your own comments, but that's a bit heavy.
Yes.
Sorry.
Well, the funder script takes 1 minute to run 1 minute 15 something 15.
Sorry.
And it takes, well, actually, I added a restriction to the columns that are red, so it only reads one column, so it doesn't take all the memory, but it blows the entire column into memory.
And so of course it takes a lot of memory and of course, if you wanted to read the entire the entire file, you would.
And actually the patent standard library, it takes actually same time like 1 minute 15 and doesn't take memory.
Well, well, how do social scientists manage to like keep the comments, they produce really used and all the content.
How do social scientists manage to like keep the comments, they produce really used and all the preprocessing they did, since everything is in the terminal, it's not ideal because you don't have a trace of what you did.
So, many of us are using batch files, so you actually write down your pipeline in that SH file and run it and well, of course, we don't do it for very quick pipelines, but when you're starting to have a very large preprocessing pipeline that's what you do.
So, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know,
you know, you know, you know, when you know, when you know that the school teacher school will answer examination, you know, like you know, we know, at all.
I'm going to tell you that two of them are not going to do it.
I'm not going to do it because of you.
I'm going to do it because of you.
Okay, thank you for the question.
I will have a hard time.
We pity it entirely, but I'm trying to like interpret it.
So the question was, or maybe a remark or was of it about the fact that Xan is also kind of a low-tech technology,
actually you use very little resources and also you are not using an LLM to just visualize your data or plot your data.
Well, I have to say it completely agree.
Well, that's a philosophy or because of you as we were saying that we try to apply at the lab in the tools that we use and we develop.
Of course, like we try to make tools that are robust but also that choose as low resources as possible.
Also, because we have to say that social science researchers, researchers not always have many resources.
Many of them are poor.
They only have their computer that doesn't have many resources.
And so it means that they need to have tools that can be run in their own terminal and not on a huge server with GPU cards or anything like that.
And so, well, the tools we develop try also to answer to that requirement.