My next speaker in the section of today, which is called using Go.
And using Go to write a shell and programming language, looks so interesting.
I could not reject these proposals to come talk about how to build a language in a programming language.
Which ran to our class.
Hello.
I guess you can hear me.
How is everybody doing?
I'm obviously, let's all try to stay awake.
All right.
So I'm going to talk about developing a modern shell and programming language with Go.
My name is Qi.
As introduced.
Right.
This thing is called Elvish.
The website has a domain name.
I'm very proud of.
So first thing first.
So I think a lot of people don't know how to show different from a terminal.
Because a lot of the time you kind of use them together.
They kind of blend together in a lot of people's minds.
So the way I think about this is that like the shell is just a program that runs in the terminal.
It's usually the first program.
So you can think of it as kind of back when people still set home pages in the browser.
It's kind of like the homepage browser.
It's like just another web page.
But just you go to it first.
So shell is a program that runs your terminal by you.
You tend to start it first.
Actually don't have to.
You can say you're terminated to start them or anything like that.
Right.
So Elvish is a shell.
That means it's kind of like bash, zesh, and fish and other things.
So there's a bit more modern.
And by modern I mean two things.
One is that it has more powerful interactive features.
There also has a full-fledged programming language.
And we'll talk a bit about both aspects a bit.
And there are other modern shells.
Here are just three.
And there are more if you look up.
So why make a shell?
So the way I think about this is that like in our profession as programmers.
We're actually very similar to like back blacksmiths back in the olden days.
In that we are uniquely qualified to make our own tools.
That's something a lot of other professors don't have.
I think shell is a way to make your own tool.
Something that you can use.
And it is something that shells then can be also used to make other tools in the form of shell scripts of course.
So I think of like the rationale for making a shell in this two kind of recursive aspects.
Right.
Let's talk a bit about what does the full-fledged programming language look like.
I think a lot of people think that advanced programming features and shell script are kind of incompatible.
I don't think that is true.
And here's like a very contrived example to try to convince you in a minute.
So so this is kind of it's not.
It's a very real workflow to have to do all the time in that.
I have two different servers.
Unfortunately one of them runs and dead in another runs.
And I want to like update them.
And I want to do this in parallel because updating ones that we shouldn't need to wait upon the other server.
In a traditional shell language this can be awkward.
But in Elvish like you can model this in a pretty natural way.
So for instance, Elvish has nested a data structure.
So you can write a list of maps and each map describes a server and a server has a name and has a command.
And then Elvish has this peach command means parallel each.
It takes this list and it takes a lambda.
And the lambda says what you need to do with each element of the list.
So that kind of comes very naturally in Elvish.
And I think you would it would be a little bit awkward to write it in like many other languages.
Like if you write in a traditional shell.
It's a bit awkward because they don't have like nested data structure.
So you have to kind of mix data and logic.
But if you write it in like another program language like Python or even go.
It doesn't allow you to like just say SSH.
You have to say something like exact command SSH something like that.
But in the shell this is all very natural.
And Elvish also still has all the familiar shell features.
So you can just run commands.
That's what you do in the shell at the time.
And it has like things like wildcards.
So for instance you can count number of lines in your go files.
This is something actually actually to quite often.
So you can do.
But Elvish also extends this pattern in that it has recursive wildcards.
So you can count all the go files not just directly in the directory.
So this is also something I do a lot of time.
And unfortunately Elvish has more than 70,000 lines of code now.
And you can also use Elvish as a wild write scripts obviously.
And I think in many ways it's kind of complimentary to go.
And one example I always like to give is that Elvish has a continuous deployment pipeline,
which doesn't use any existing solution.
It's just some.
It's a go program and a JavaScript that I just put together.
So I think that like go is very good to write a web server with like all the build thing.
It should be module and stuff.
So if you want something simple, you don't need anything besides the standard module.
But when it comes to actually building things like invoking a lot of commands,
you'll get a little bit unwieldly.
And I compliment that by just writing the meat of the CD logic in an Elvish script.
It's actually in the Elvish report.
Like part of it is in the Elvish report.
But another part of it is in the different report called Elvish up.
Let me show that.
Okay, so it's for how to find files.
Give up.
So I think like I think this is a very nice pattern.
Like you have a slightly more low level language to handle things like speed serving
and have a more high level language to define your actual CRCD workflow.
Much better than I am.
Right, interactive features.
So let's just do some demo.
You have things like Elvish has it's in that highlighting.
And with more than like a highlight things like strings, comments,
has completion.
Like pretty standard stuff.
And it also has a directory history.
I have shown that a little bit if you press Ctrl L.
It gives you a list of all the directories of being through.
So most of the time you don't, you shouldn't need CD.
Like, this is where my slides are.
I just start typing SLI and I see.
Okay, I'm not currently in the slides directory.
So it doesn't work.
If I just type SL, I can see slides.
I go to slides.
And then it has command history with Ctrl R.
So this is kind of the same binding as in Bash or Zesh.
But instead of like looking at one command at the time,
it gives you a list of all the commands.
And you can do searching like, like, this is FF,
and pay command.
I wouldn't ever know how to be a phone scratch.
But it's in my command history.
So nice.
Right.
And it also has a built-in file system navigator.
It's press Ctrl L and M for navigation.
And this allows you to like, preview files and like,
change directories and that kind of stuff.
Right.
I wrote this soon.
The entire UI will be programmed over the new toy framework.
And that's a bit ambitious.
It hasn't happened yet.
Oh, I skipped this part.
So I think the good thing with having introductory features
and a good programming language is that I think they
should have very good synergy together.
So one thing when it comes to like customizing prompts,
I guess people nowadays use like pre-pagged things like
Starship and stuff, but in traditional like in Bash,
after to learn this little language that has like
percent of the H is hostname.
percent you is using it that kind of stuff.
In average, there's none of that.
Instead, the prompt is just a lambda that you can assign to.
And you can just output stuff in your prompt.
So this is something you can do to get something similar,
but not exactly the same as the default prompt.
It's more similar to the Bash's default prompt,
I think.
And I hope this works.
Please work.
I think, oh, okay.
I haven't actually installed Starfish, but if you install
Starfish, the way you switch to Starfish is just,
it's just called Starfish in your prompt.
A Starship, sorry, not Starfish.
Well, that would be a nice name for a project.
All right.
It's a go talk, so let's also talk a bit about my experience building
I'll wish with go.
So one thing I kind of showed a little bit earlier is
which is data type support.
I showed lists and maps, but they're also like pretty
center things like Boolean strings, which are just
those Boolean, like this is what you get.
If you implement a language using another language,
you kind of steal the whole language's data type.
So you can do things like this is true.
This is false.
This is a string.
This is just go through false and strings.
And you can do.
And also I still go syntax for strings, that kind of stuff.
And you also have numbers.
So you have goes primitive number types.
Interflow 64.
And go standard library also has those big numbers.
In L, which is kind of a blend together.
So they're like numbers auto promote to the appropriate numbers type.
So you can compute 40 factorial just by doing this.
And it works.
And if you write something with slashes, it becomes a rational number.
It becomes a big rat.
I don't want to say that, but you can do something like this.
And then the list and mapping limitations are kind of special.
In that they're actually modeled after closure.
They're actually mutable.
But we don't have time to get too much into that.
It's quite interesting.
If you even know closure, then good for you.
If you don't know closure, you should look it up.
And standard library.
That's another thing.
I still from go, like I still a lot of goes standard library.
So you can use the math module.
Like, basically, it goes math module.
So it has most stuff in here.
I don't think it has everything, but it has probably most things you will need.
Let's try this.
It has pi.
So, right, it should be zero, but it's a very small number.
And strings.
So one of the biggest gripes I have with traditional shelves is that.
The string manipulations are so painful.
There's an operator for trimming the prefix and the operative for trimming the suffix.
And one of them is percent.
Another is hash.
And then remember which is which.
So in LVs, I just still go standard string library.
So you can do has prefix.
You can do train prefix.
And everything's fine.
And regular expressions as well.
Just to prove that it doesn't always return true.
All right.
Okay, that's actually just another colour for showing more of a lavish.
But let's also talk a little bit about, kind of, to a slight deep dive.
I guess a shallow dive into the implementation.
I think the most interesting, like semantic part about shelves is.
It's a pipeline semantics.
So it's a thing that doesn't really exist in many other languages.
But if you look at this.
I mean, this is a trivial example.
So I'm echoing the PIDM.
Accounting how many characters they are in the PID,
which is not very useful.
But you can imagine, like, like, this being a much more complex command.
Like, the thing with the pipeline is that they always run in parallel.
So if this is a very complex amount that probably needs to make natural requests and stuff,
the WC command will run in parallel.
So, but, and this turned out to be, I think, that's actually very suited for, and go to implement.
So let's kind of speedrun through the interpreter implementation.
So this start with stuff on the string, and it gets parsed, those are standard stuff.
And then, Elvich has these kind of compilation step,
but there's no way too much about those.
But let's just keep in mind that this gets compared to a pipeline of, like, up-for operation,
which has two phonops from kind of roughly means command.
And one thing you realize is that, like, in shell, those commands really, like, have different standard input and standard output.
So you have to represent that.
So, obviously, you represent that using just a structure that has this thing.
This is not a real definition, but good enough for this purpose.
And this pipeline all, you can execute it with the, within the context,
and the phonob, you can also execute it with the context.
So that's kind of the setup of the pipeline semantics.
So, implementing the pipeline semantics.
I kind of, this is, really, this is, this is, again, more real code,
by it's kind of the meat of the real code.
And you can see, we actually use a lot of kind of, those booting stuff in here.
So, for instance, like, this is a, let's assume for simplicity that there,
we just, like, consider two forms,
because if you can do two, you can probably do three and four and five.
So we have two of them, and those two commands,
there, the, the output of the first command is connected to the input of the second command.
So, go give you just, you just, OS, OS, or pipe, and you get, like, two parts.
And then, Elvish also has this idea where you can actually pass go values through
to different commands.
So, that's an Elvish specific thing, but that's just a go channel.
And then, you can create the context for the first command,
which I didn't show the inclination of this, but you can probably guess how your works.
So, you just replace, it's a center output with the, the writing end of the pipe and the channel.
And for the second form, you replace this center input with the reading end of the pipe and the channel.
And you can now execute them parallel.
Again, this uses, like, kind of, pretty center go, and idioms.
You use, you, like, a spin-up, a wake-roop.
You add two, and then you spin-up two and go routines.
One of them is executing the first command, and then wake-roop down.
You can look at the real code.
It's, I mean, it's more complex than this, but by necessity, but, like, the meat idea is kind of like that.
You can see those kind of, this is a go, here's a WG though down.
Because it needs to handle, like, end command, things stuff, two commands, right?
So, go is actually a pretty decent language to write a showing.
This is a recap of the current section.
So, like, you get a lot of data types, and the standard library for free.
And it also has very good primitives for pipeline semantics.
And another thing people don't always appreciate is that, because go is a garbage collector language.
You might think, oh, this means your language must have proper formats.
But, actually, it gives your language garbage collection for free as well.
So, like, if you want to implement a language, like, in Rust,
and your language is garbage collector, you have to write a garbage collector from scratch.
But, go already has a garbage collector.
So, if it just makes sure that language values are go values,
and the language values are unreachable when go, like, when their go values are also unreachable,
like, you can kind of pick it back on the, on the go garbage collector.
So, that's quite nice.
All right, let's also talk a bit about testing strategy, which I think is,
if you are, like, into testing stuff, it's very interesting.
If I'm not into testing stuff, hopefully I can convince you a little bit more.
So, I think testing is very important, because give us a confidence about the correctness of the code.
Like, if you never test your code, like, what did you write it?
And, how do you know this correct?
And, but, also, especially when you're changing your code.
Like, when you're first write a code, I mean, like, the running ones, I think, is fine.
But, you come back three months later, you have no idea what your code is doing.
And, you have no idea if, like, changing a piece of code, we'll change another part, like.
Right. So, I think the most important thing about the testing strategy is that you should make it very, very easy to create and maintain test,
because there are already lazy. So, if you want to write more test, there must be very easy to write.
So, easy to write test leads to more test leads to higher test coverage.
So, I always has 92% test coverage. I don't think it's high enough, but it's probably higher than most projects.
And, now, let's talk about the interpreter's API, like, because most of one non-most, like, about half of the average is probably rated through the interpreter.
So, interpreter is actually one of the easiest things to test. So, because the, this API is very simple.
So, it's input is just code, and this output is some text, and in the case of average, also output is some values.
So, if you, like, this is your code, echo, hollow world, and this output is hollow world.
And in average case, you can also output values, but it has a special command called a put. So, you can put hollow world for bar and in outputs to values.
Right. So, this is how most of the average test used to be written.
Like, this is a pretty standard in go community, I think, table driven tests. So, we just analyzed the API of the interpreter.
So, you kind of encode that using a struct, right, and then you'll be the table of those structs.
So, in the average case, the struct looks like this, this is code, and then you want some values, and then you want some text.
And then you start writing these kind of things, and then you test function, you iterate through this slice, and then you, like, assume that this encapsulates the entirety of the interpreter, and then you just compare things.
So, this is, this was fine for a while, but then you realize it's actually quite repetitive.
So, every time you need to add a new test case, so what do you do? Let's look at that.
So, step one is that you must have implemented something to new, otherwise, why you're writing tests, or maybe you're writing tests for existing stuff.
Anyway, and then the second step, usually what I do is that, because the average is a shell, and we just test manually in a terminal.
So, let's assume that I'm going to need to join the string drawing, which you must be familiar with, like, those strings are drawing.
And then I would just do this in the terminal, and I say, oh, this is good, and then I manually convert the interaction into a test case.
So, let's look at step three, like, I already did step two, I already tested it actually, I just tested it manually, not writing a real test.
So, but shouldn't the computer be able to derive this test case from a manual testing session, like, and computers are really good at repetitive tasks.
So, that's what I did. So, instead of defining strikes for inputs and outputs, let's just, let's just, like, copy-paste the terminal transcription into the test, and let's, and this thing just gets passed into the struct we have seen earlier, and then everything else is the same.
So, then again, let's, so, let's kind of demonstrate that. So, let's assume this is how we do today, as to our drawing, A, B, right?
Then, okay, I'm not using the default prompt. So, anyway, the kind of the, the test is kind of assumed that using the default prompt.
So, let's paste that here, and then let's, let's actually run the test, and it passes, just to show, sure that it doesn't always pass, like, this introduced a manual type on here, and it tells you, okay, it didn't pass.
And this was fine for a while, until I realized that I don't even want to do the copy-paste thing. So, what should we do instead, and we can just, like, if I need to do this in the terminal, then I have to copy-paste.
If I don't want to copy-paste, I have to do this in the editor. So, what I did was I wrote an editor extension, and in this case, the as code. So, in the editor, so, this is how this works now.
So, in the editor, I pretend I'm actually using L, which is not nice, but it's not too bad, and then there's a key, I can press, in my case, out, enter, and then it just, like, runs L, which for me, and shows the output.
And then I press save, and this is my test case. There's no copy-paste thing involved. I just type, I kind of, this is also my manual testing session.
So, my manual testing session results in an automatic testing written. So, I think this is a very good kind of property in that, we can eliminate test writing as a separate step to a development.
So, because you will always do some sort of manual testing. If you can combine your manual testing with automatic testing, and then, probably you're kind of, kind of, offset for adding more test cases.
All right, so the interpreter was very easy to test, because that's very simple API. Now, let's consider something slightly more complex. So, so, L, which is also a terminal application, as I've shown you before, like, it has all those, like, fancy bells and whistles.
So, so, I also want the terminal application to be tested. So, and just to give you kind of, like, this is a very rough approximation of how it works under the hood.
It's made up of a bunch of widgets, and there's a widget called text area. And for instance, one test case who may want to write is, if I press X and Y, it inserts X and Y, right? If I press left, it moves the cursor to the left.
And if I press backspace at this point, it leads X instead of Y. So, this is one possible test case I want to write. So, this API is now much more complex than the simple interpreter API. It's no longer like, text in text and some values out anymore.
It's kind of taking turns. Like, I do something, I want to see something, I want to see something.
All right, and the solution to that is, we already have this framework of running testing the editor, why don't we just, like, make use of that.
So, which I just create, I wish bindings for each of those widgets, and this binding comes in in the form of two commands.
There's a send command, and there's a render command. The send command emulates some keyboard events.
The render commands just don't set up the terminal. Again, another demo.
I'm so okay. I chose the Sally different, has got it. So, in this case, that's right, ECO.
And then I also just pressed Alt back enter as I did it before. So, and then I press left.
Oh, it didn't work.
No.
Okay. I think it's the test feature.
Anyway, this is more complicated than I try to make it look so, and that's probably why.
Let me look at my test. I think, oh, text area, text demo fixer. Okay, sure.
Right, because the way is implemented, by default, the left command actually doesn't do anything.
I need to inject a specific kind of handler to it.
Okay, this a little bit added a little bit more stuff in here.
Ah, it still didn't work.
Anyway, let's not worry too much about that. Let's choose a Sally simpler test case.
So, let's know left keys anymore. We don't support that.
Let's just do this thing where, like, I'm emulating kind of this.
Like, when I'm taking type in the echo command, I start by having ECA H, right.
And it's red because the command doesn't exist. And as a type, oh, it becomes green.
So, a type ECA H and it's red and that type, oh, it's green.
So, you may have noticed that there's kind of this single, and there's this kind of a DSL thing here,
in that, like, the first line. Again, let me turn off syntax highlighting.
Just to, because it's a bit distracting now, plain text.
So, okay, this doesn't fix it.
And let me look at this file in the terminal.
Okay, so this is how it actually looks like, kind of, in pure text form.
So, this is kind of a DSL in that the first line just represents the content of the file.
And the second line represents the style of each character above it.
So, by inventing this DSL, I'm able to encode the terminal state within pure text.
Like, R says it's red and G says it's green, right.
And then there's also this little thing that represents the cursor.
It's too bad that I cannot show, like, it goes to the left,
but it actually can go to the left, and it will overlap with those RNG characters.
Because it's actually done using Unicode, kind of, direct critics.
And it looks like a barrel.
Anyway, I don't know if you've got this from, like, my box demo, but that's kind of the idea.
So, right, so if you want to make testing, your test code, you need to make testing easy.
And you should embrace DSLs, embrace text.
And if DSLs don't solve your problem, that means you're not using enough of it.
And this kind of similar idea of, like, just testing input outputs.
Actually, it has a lot of presidents.
I think the earliest example is materials tests.
They actually write shell scripts.
Elvis just set up his slide even more complicated.
Like, this is an actual material test.
Like, this is a shell session.
All right, so I talked a lot about what Elvis can currently do.
But I can also talk a bit about, kind of, my vision for, like, for the future of the shell.
That kind of goes back to why I want to write a shell.
Like, I don't think Elvis in this current form is, like, all I want it to be.
I want it to be, like, a little more.
Because I think shell languages are really, like, the language.
If you think about it for, like, real personal computing.
Like, today we are just using apps, like, packages that are developed by the companies,
which I don't think is really ideal.
I really want people to kind of build their own tools.
And Elvis is kind of, like, my, my means to that.
And, like, in my vision, we should be using, like, shells to manage your personal service.
And to build simple CRCD pipelines, which I did instead of YAMO.
And using shell, like, using shell to build, like, later UIs,
that I tailor to you need instead of, like,
I don't know, like, paying $200 to buy boutique and macOS apps.
And I think shell should break free from the terminal at some point,
but it's not happened yet.
And if you want to learn more, if you want to learn more about interpreters,
if that's kind of an interesting thing.
There's a pretty, like, I think it's pretty standard these days to recommend an incredibly
interpreters, but just in case you haven't heard it.
And if you want to use a learn elvish, go to this website, and you can get elvish.
And it has a one-line installation script that doesn't involve curl into bash.
Because elvish is one binary thanks to go.
So it's curl into tar instead of bash.
So.
Right. And it goes back to what I mentioned before, like, a shell is really just another program you
don't need to think of, like, if I want to try elvish, you have to give up bash or zesh.
It's just another program. So if you want to try it, you don't have to do, like,
or nothing, you can just download it and then use it as another program.
Every now and then see if you like it, if you really like it.
And there is some guys on, like, how to use elvish as a default shell.
You can also try it in the browser.
Okay, I'm surprised every time this is still up, it hasn't been detoxed yet.
So. But you can go here and there's an elvish session directly in the browser.
You can do things.
And if you want to hack an elvish, if I'm get up.
And there are developer docs, which I hope is pretty good.
There's architectural interview, process of contributing and that kind of stuff.
Also testing changes covers some of the stuff I talk about in testing strategy.
Do I still have time for questions?
Sadly not. However, we've got a great hallway track over there.
We even have a bar right next to our devroom. I just discovered this.
I mean, I'm being locked since 9 a.m. So I just discovered that.
So if you want to go grab a drink and talk about an amazing shell language,
this is the guy.