Brilliant, okay. Well, hi everyone. It's nice to see you. This is my first
fostum, so I'm learning a few things. It's very, very friendly so far, so please don't
change that. This torque represents a very similar torque that was given at
Jupiter-Con. I don't know if anyone here has been to Jupiter-Con before, but if
you were there and you saw that torque and you think it was better the first time
around, I'm really sorry, don't tell me. So I have a few questions for the
audience before we start, just for my benefit. Who has heard of a project called Jupiter-
Book? Interesting, okay, cool. Who's ever been interested in designing web apps?
Okay, also cool. Who's a content author or a researcher or anyone who might think
they could fall under that bracket? Okay, that's, well done, that's exactly the
I was hoping for. So last question, who believes it was a really bad idea to
wear a turtleneck before giving a fostum torque in a rim of this temperature? That
would be me, but we're here now, so. Right, so hi, I'm Angus. I've been in open
source in some lurking capacity for over 15 years. After finishing my physics
PhD, I moved into software development full-time with the research software
engineer, although at the time I didn't know that's what it was called, and I'm now
in applications and infrastructure engineer as an organization called 2ITC. So I
won't be speaking much about 2ITC today, but if you've heard of my binder or Jupiter
Hub, 2ITC are a non-profit that are heavily involved in these projects, and we as
part of our work with communities, we do a lot of upstream work, and our job is to
provide the communities that we work with, a digital home. So that's all I'm going to
say about Jupiter Book. Sorry, no, that's not all I'm going to say about Jupiter Book.
That's 2ITC, and like finally, because I use Nix OS, I have to declare that
publicly. They tell you that when you start using it. So there we go. This is why I
offer my demos, bro. Okay, there we go. Right, cool. So at a high level, Jupiter
Book is a tool that enables you to take collections of documents like notebooks or
markdown files and build them into a nice user-facing book-like experience.
We kind of say it in the long form, which is that it lets you create enriched and
interactive multi-document computational narratives, which is an absolute mouthful.
So I'm going to talk to you a little bit about the current iteration of Jupiter
Book, and we'll give you a bit of a sense of the journey that it's taken, and we'll
get into more of the details as we go. So here is an example of a Jupiter Book with a
community that we work with called Project Pithia. I really encourage you to look at it.
It's a really interesting project, and I'm not just saying that. But crucially, Jupiter
Book lets you take a collection of documents of documentation, which is what we have here,
and assemble it into something that has a more natural reading experience.
Okay, cool. So Jupiter Book's quite old now. It's been around for a good number of years,
and it's gone through quite a few different evolutions. It started off with a tool that
involved using Jackal in a pretty hacky-templating system to produce the content, and it
went through a really big redesign, which was a grant-funded project, where it was built
as a basically a distribution of sphinx, where there's an ecosystem of sphinx extensions,
and at the end of it you would get what we used to now call Jupiter Books. And most Jupiter
Books in the wild up till now have been these. There are still quite a lot today that are
not using the latest version of the stack, which I'll talk to you about in a minute. But what
I'm here to talk to you about today, if I try and switch sides, is this bit? So Jupiter Book
2 was released in October, and we talked about it at JupiterCon. It's a whole new text stack.
We don't use sphinx anymore. Lots of things are similar, something's a different, and
I'll touch on a few of those today. Okay, get my step count in.
What?
If you off-camera, if you cross the line, the lines of the thought.
Say again, sorry.
The part of the room that's in camera, the camera over there, is in the, so if I want to
make real gestures, it's over there, and then okay, cool. So one of the last things to
say is we've gone through quite a big social change as well, and it's really important
to think about the social parts of the project, and I'm really glad that the last speaker
was basically talking quite a lot about that. We've gone from different organizational structures,
different team members, and I will touch upon that later, but just wanted to call it out
that that is not just the tech that's changed along the way.
So next, describing the motivation reasons for how we got here, the one of the big gaps
that we felt in Jupiter Book and in the wider ecosystem is the gap between the exploratory
and the learning phase of doing research and of being a scientist or other sort of affiliated
profession, and getting to the point where you're going to communicate that with your peers,
whether that's sort of in an informal sense that you're publishing in an informal manner,
or whether they're actually publishing to a journal, and anyone who's tried to publish to a journal
using Sphinx before knows how painful that is, and so that's part of the technical and social
reasons why we're here today. Finally, if anyone finds us, I'm going too fast because I've
got really excited. Please raise your hand and I'll take it as a hint to slow down.
So in order to talk about this topic, I'm going to break down the sort of space of personas
into three different personas that we might imagine when we're thinking about publishing,
about content, about science. So firstly, we have the learners and the learning communities,
and these are people who need to gain new insights, to gain new skills, to learn the tools of
their trade, and usually we think about undergraduates, but this is a very restrictive view of
what is a very wide persona. Once these people have gained these skills, they've got their
networks, they've started to integrate themselves into the researching sort of portfolio.
We then think about research communities, and these are people who are becoming explorers and
researchers and collaborators. They have people that they work with, they have science that they're doing,
and they are building upon the skills they build from this previous section.
Typically, these personas are coming up with their own experiments and their own ideas.
There's a lot of idea generation happening in this middle phase. And then finally, we have
the scientific communication phase. You've created something polished or not as most of my
publication would have been, and you want to share it externally or formally with your peers.
You want to communicate it crucially using the same tools. Anyone who's done publications before
will know that you might have a tool chain that you use all the way up to the middle,
and then suddenly you have to learn later, like the final year of your PhD, depending on who you are.
So, we really want to recognize that that is a big jump in the technical stack that people are
using, and we want to try and, as a Jupyter book project, make that easier. So, we want to
single tool chain. If I can come up with a catchphrase that can realize the goal of having an
entire knowledge creation life cycle captured within its remit. So, oh, look at that, Mr is an ecosystem
of tools and standards designed to meet. These needs, here we go. So, the idea is that the
missed project or the missed ecosystem comprises tools and standards of composable building blocks
that come together to serve the different parts of that life cycle that I just showed you.
If anyone was in the, which talk was it, there was a talk yesterday that was on Dock Speck
or something like that that talked about Dock and ASTs. If you happen to have been there,
it's very relevant to this, and I'm keen to meet up with the author later, but for now,
there's an interesting aside. So, Dock book crucially does three things when you run a build on a
Jupyter book project. It parses in resolves all of your documents, so if you have input documents
like mark down files or Jupyter notebooks, it takes them, it reads them and ingest them, and then
produces an internal representation called the missed AST. Now, that's always the way that these
sorts of tools work. What's different about Jupyter book, too, is that the missed AST is considered
an asset that we care about. It's versioned, it's schemered, it's shared, and it's a structured representation
of your documents. I think one of my jobs in this talk is make everyone hear care about AST,
which I think is an uphill battle, but I'll do my best. And then finally, once we have this AST
across all of the different documents in your project, so you might have different pages in your site,
for example, we then want to render and export them out into your target format. That might be
a website that's interactive to new users. It might be a PDF that's being published in a journal,
and you can imagine, from those two statements alone, there's quite a lot of things to think about
when you go from the AST into the rendering step. The benefit of having these separate phases
amongst other things is that we have different extensibility points, so you can change how the
documents are passed and you can add new features there. You can modify what happened when we have
these documents and do interesting things. So for example, Jim from the cheering way, who's one of
the organizers of this session, he don't think he's here. He added a plugin that he had a plugin that
makes it possible to check, do your images have alt texts for an accessibility checker. And those are
the kind of things you can do on a structural level by having an AST that you sort of share and build
tooling for. It's the advice to say, you can also extend the final part. That's sort of a given,
you can have different templates, you can have different PDF themes, and I'll talk more about that in a
minute. So to build a new stack here with the guiding principles, we wanted to make something
that was simple to extend and easy to use. We wanted the content to be canonical and machine
readable. I love the word canonical, but ask me what it means, and I'll change my answer depending
on the time of day. It does have a meaning, and then content should be modular and composable,
which sort of connects to number two and will come on to that very shortly, and then computation
should be a first-class citizen. I come from physics, and I'm sure most people have a connection
with computation given that we're here, so computation is a first-class citizen. It's not
something you do sort of as a bolt on or on a side. And then finally and most importantly, and I'm
not saying that because of the last speaker, it should be governed by an open community.
So these are our goals, feel free to ask me whether we've achieved any of them, and this is the
first one. So Mr. Superpower should be this easy to use and simple for users to extend.
The goal is to be simple and approachable for someone without a ton of technical expertise,
but with the capabilities to extend into those for new use cases and bespoke needs.
Easy to use. Document format. Everyone goes to mark down. It's ubiquitous. It's used everywhere.
It's not used by the same dialect everywhere, which is really annoying, and I'm not going
to tell you that we didn't create a new format of mark down for Geewed to Book because we did.
And that's a good thing. But we start there, right? So here's regular common mark mark down,
and it has mentioning features. You have title, italic text, bullet points, block quote, math, etc.
And it's supported all over the place, but when you try to do more than this, it gets a bit tricky.
So we can render this in HTML and look pretty nice. But imagine that you want to, for example, add
a figure. So here's some common mark that represents a scientific paper, and you've got a figure
one that corresponds to a graphic with a caption. To actually do this, you need to take what is
a semantic concept of a figure and render out into mark down yourself. So you can't say that this is
a figure with a caption what you actually often do is create an image node, and then below it,
you put some italicized text with a figure one written there. And that works really well, but there
are some challenges with it. Firstly, if you add a second figure above it, now you've got to remember
to redo all the figure numbers, which I never did. And secondly, there's no real
tool-level information here that what this is is a figure, and that's a more nuanced point that
we'll get on to in a minute. So I'm just going to make my speaker note to not size 105. There we go.
I think I've covered everything. Yep, so mainly this is fine, but it's not good enough.
So mist mark down, we'll have a note. Mist mark down is a evolution of common mark. It's a
super set of common mark, which means that we do all of that, but we also have some additional
features. And one of the ones that I'm going to point to you is, oh, we have in line math,
brilliant. That's not very exciting. What is exciting is we have block-level math as well. This is
a directive node. If you've heard of MDX, it's kind of like an MDX component. If you've heard
of programming, it's kind of like a function. And so here what we're doing is describing block-level
content, and it has a name, it has an option that's been passed in called label, and it has a body,
which in this case is a math expression. But it's not limited to to that. So that you can
see this as one way that we can build extensible reusable building blocks. And we have quite a few
things here, so we've got a math directive. Now here's the figure that I was referring to. So
figure, we have a name, an image, some formatting information, and an exception. I can then reference
that figure in my prose later on, and it will do interesting things I'm going to show you. So
having that semantic information from that's a figure, and it's got a label, it's already enabling
us to do interesting things. And we actually use the same syntax for citations. You can see over here,
here's a citation in a bibliography, and that's using the same lexical approach to build that citation.
There we go. So because this AST is enriched, if I have a link to a citation, I can hover over it,
and if I have a link to a figure, I can hover over it, and it brings the content, that's not
very useful. It brings the content into the foreground, here we go. So I'm talking about something here,
and usually the context of where I'm talking about something is really important. If you have
a hyperlink, I don't want you to jump away and go and read that page, because then you'll
lose the context of where you've come from, and usually I'm embedding a conversation about
figure in some wider prose. So the useful thing with these hover references is that they are
local, so you don't have to move away from where you currently are to learn more about the figure.
So another super powerful myth is that it's a canonical and machine readable. So every missed page
has adjacent equivalent, and you can go onto that with curl or whatever your favorite utility is,
and you can look at the structure of the page. So we have lots of metadata here, like your title,
and your authoring, and your licensing, and we also have important stuff like what is the content
of this page in a structured way, and that's really excited from a reuse and sharing standpoint.
I've already said I'm going to make you care about AST. The first step to doing that is that once
you have this AST, you can then do interesting things with it, so you can export to different targets.
You could build through a PDF or Word document or a late-tech document. You could also build
to type it. You could also publish a website, like I've just shown you, or author to publish to a
journal. There are lots of different ways that we can reuse this, and it's because we have an AST
at the center. There are lots of templates that could mean the members of author to target different
journals and different formatting styles, and that list is getting longer by the day.
So here's an example from a company called Curve Note. They are members of the
Jupiter organization, and they are heavily involved in the early development of missed. They
worked with Cypai in the last couple of years to digitize all of, oh, not digitize, but to massage
all of their proceedings into this format. If you go to Cypai proceedings, oops, that's too quick.
If you go to Cypai proceedings, you can see. Yes, we've got all of the, I think 18 years, I'll say
we, they have got all of the 18 years of proceedings in this format. I think it's over 1000 articles,
and they will serve under this one, they are all served under one front-hand interface.
So you can browse through, and because these are sort of digital artifacts, they can embed
interesting things like figures and plots that you can actually drill down into and learn more about.
This is kind of only scratching the surface, but you can already see how like having a medium
that you're intentionally publishing for is really powerful from a exploratory perspective.
Here's the Cypai proceedings template. I think it's very pretty objectively, and this has
made pot of having a template that we've built to from the missed AST, and I'll talk more about
the detail in a minute, but if you're targeting typist, which I think this template was,
it takes one order of milliseconds to build that PDF rather than the minutes you're familiar with
latex makers, etc. No shame on latex make, by the way, got me through my PhD. So this super
path number two is that it's modular and composable, so content modularity is kind of nuanced,
but the idea is that if you have cross references and hover references, we can hover over them and
bring them into the local content and contexts, sorry, to explore them. So for example, here are
cables that are highly detailed, and then you can also have citations that refer to your bibliography.
And these are all internal references. What's also more exciting is that same mechanism will work
for external references. So you can imagine that if different people from different communities
publish their sites as a bit missed sites, as Jupiter book sites, then it is possible with
these into this new tooling to embed or hover over references to that content in your own.
And the goal here is to think about, there we go, the goal is to think about federating content.
So you can talk about distribution content across multiple different providers,
and instead of having to copy and paste a figure into your paper to say, oh, this person did something
cool, you just embed it in all of the citation and attribution metadata should be there.
Now that is a goal, we haven't got all the attribution done yet, but for now you can certainly
embed and hover preview things some other people's resources. And what this video was showing is that
there's a, I think this is a kind of project-pithy cookbooks, no, this one isn't, but effectively you
can have, I'll show you the proper one, you know, you go, project-pithy here.
Here is, they had project-pithy had these tutorials called cookbooks, and the idea is that you can do
sort of specific analyses here and share them. But all of these analyses are kind of doing the same
thing in certain context, they all need to run on a Jupiter hard, and they always do x, y, and z.
And so what you can do is the authors of project-pithy here can basically publish a guide,
and then that guide's material can be embedded and referenced in these cookbooks when they're
providing guidance for how do I set up a virtual environment or how do x, y, and z. And that's
a moving target we'd love to do more on. I'm rushing through it because I'm aware I'm probably
short for time, but please do asking about that at the end if you're interested.
Computation being a first class citizen is something that we're thinking about all the way through
the Jupiter stack. So this includes Jupiter lab as an editor. We want to try and blur the lines
between code cells and mark-down cells, for example, so that you can have a mark-down cell that defines
some code that's coming from a code cell. So I think down here we have a widget in-line in some
text and we have a computed expression that comes down here later on. The idea is just to sort of
play around and see what happens if you think less about a linear thing of blocks and more about
a document that can live and breathe. Jupiter lab missed needs a bit more development work in the
last couple of months, and that's one of my projects, but the goal here is to think about how we
can reduce the barrier to interactive and rich experiences. Here's an example from microscopy,
static figures aren't great. If you can drill down to them and you can actually zoom in and switch
the column apps, etc. You can learn a lot more than a printing seven figures out statically in a
PDF scan, and so it's really worth thinking about how you can lean into embedded and rich experiences
and publishing for this kind of target. Some people have deployed compute resources like
jute hubs and binder hubs and project pithia is an example, and project pithia cookbooks, for example,
some of them can run on a binder hub, so here if we take the power button it's going to connect and
spin up a binder instance, it's going to run the jute decernal for the current document,
it's going to compute all of the cells and provide you with any rich interactive figures that
might be there, and this is really useful from the idea of reproducible analyses that people
can actually test out and share, and there are lots of nuances here again, I'm going to skip over them,
but suffice is that it's really fun being as press a play button on the website, and that's the
main reason we all do this, so there we go. Another mystery part is that it's super easy to use for
developers to extend, so here's some very quick examples. I'll see you in turn, I think with
curve note, created as ammo bar on the right hand side, every time you click on a figure,
it has this trailing list of figures on the right hand side because you might argue that the
figures are the foreground of the paper that you're reading. This was done by the fact that
missed is built in two ways, I'm kind of talking about here, but it's got the JSON state that's
the documents, and then it has this renderer that's currently a React front end, and that renderer
can consume that state and render a different context, so you can either modify the renderer
to do more things, or you can bring it into another context, so here, for example, is a website
time that someone has embedded some section of missed in because they're just pulled in the
React components and they're rendering the document there, and there's all kinds of things from
CDNs and galleries that you can imagine doing with the ability to pull in parts of a published
duplicate book site somewhere else. Sometimes you want to pull in bespoke visualization, so here, for
example, is a sequence of visualizer. This is the kind of thing where you might want to embed
a little widget that people can share and reference, and we're working on ways to make it possible
to embed little of absolute things in your missed sites to be able to do this, and this is an
example where we just hacked our own code to make it work, but missed is permissive enough to let
you do that. I'm going to skip this one, which is not super important, let's keep going.
You don't get to see that demo, sorry. Right, finally, and most importantly, I don't want
to rush through it, led by an open community. So for duplicate work and for missed,
communities, the most important thing, we want to be an open community that is governed by
a collection of peers, a collection of organizations, and individuals that all work together.
So we have a team compass, and in most duplicate projects, have these, and it describes the
internal structure and organizational practices of how we work and what drives us forward. And we have
it here so that people can understand the way that we work and the way that we operate, and
see a path for them that they can start to consider how they might contribute.
Whether that's reviewing pull requests, whether that's adding new issues, whether that's
triaging things, whether it's checking for typos that I've made in the docs. If we have an open view
to how we work, this should make things easier for people to engage for the first time.
So this is the core team. I'm here. That was not apparently my partner says it's a terrible
photo of me, but I've put it in anyway. So we have two ITC curve note, a black-cenship data science
amongst other collaborators here, and the list is growing. Crucially, we have a much larger group
of frequent contributors outside of the core team. So there are lots of different organizations and
people driving the project forward, and that's one of the really exciting features that we have
as part of Juvedberg. So you can learn more by looking at some of these links. If it sounds like
something you'd like to be involved with, I'm already drowning in GitHub notification so a few more
won't hurt. Please feel free to ping me, and I'd really love to hear from you. I don't know how much
time I have left. Five minutes. Do I have time for a short demo? Always a time for questions.
Okay, great. So I've got two tiny demos. So if I can figure it out, there we go.
It's not, you're not supposed to be saying anything, don't worry.
Okay, let's try this. Whoops. La-ba-ba-ba. Cool. Right, say. Cool. So here's a Juvedberg website. This is what
you might see when you start up right in some content. It's about fake birds, and then on
how to the fake bird conspiracy. Cool. I'll show you more later. This is a Juvedberg page. It's got
lots of interesting markup. You've got this of definition lists. These which previews that I told you
about, sorry, Mr Bird, out of the way, and then also I have some code in here. That's a really
not good look of my face that I took earlier. So you can embed all kinds of stuff in here. You
can do all kinds of interesting things, and the source code for this just because I want to show you
was if I open up my file, there we go. So we have some information about what's running the code,
some abbreviations, my name, because I'd like to see that everywhere. The title, some code that gets
then embedded in a subsequent expression, an interesting figure that wasn't very interesting,
and it's a finally a code cell on the bottom. You can see how actually the amount of markup there,
although probably I'm familiar to some people, is quite thin. What's captured in there is more
interesting is the AST, and that's the bit that I was spending all this talk talking about,
but I've run out of time to go any further. I'll skip the other conspiracy theories demo,
and I'll hand over to questions if anyone has any.
Yes. Yes. So there is support for Jupiter light already. Computations that you really
know, there's quite a lot of interesting parts to it, so there are projects like end script and
forge, which we don't support yet that I would like to, because that makes baking actually complex
environments a lot easier. There's also three kinds of ways you can execute with Jupiter book.
You can execute locally when you're running the build. You can execute in browser using the
binder mechanism that I showed you, and then there's a third one that I've temporarily forgotten,
but suffice to say there are different ways you can think about computation, and Jupiter light is
pertinent to the in browser form. We also support passing in a Jupiter server URL, so if you can
start a Jupiter server on our remote binder or on Jupiter gateway, you can run it in CI, but use
a binder hub to actually build the book and do the complex computation, which is something that
too ITC worked on recently. So there's lots of interesting places to take this, and it's quite
nascent, so I think there was a lot of interesting runaway we could explore, so thank you for your
question. You know what, last point. Oh, sorry.
The link I will go back to, and then can I ask you off?
Is there anything special?
Well, the digital.
a markdown in the east or to provide a pipeline of documents on this case and for the best
parties of the talk before we communicate.
So my understanding is about an editor is something we'd like to have.
We haven't done a lot of work there yet, but members of the Dubb Book community have worked
on making like a rich whip what you see is what you get editor for this.
Injusting a dock ex, like a word file, and then using it as an input file, we don't support that yet.
But from having seen other talks this week, I don't think that's incredibly difficult.
And it's something we probably could do.
The difficult part is figuring out what subset of word documents you want to support,
and the more restricted you can be kind of the easier it gets, really.
Does that answer your question?
Let's talk after this.
Thank you.