All right, I'm going to start talking so everybody needs to be quiet because I got a lot
of slides and you got to look at the slides because I'm not reading them to you.
You need to look at the slides and listen to me and don't get all sugar crashed because
I know you've probably all been eating waffles like me.
That's me.
This is my talk this year.
I will be, everybody's been talking about email.
I will not be talking about email.
I was asked to talk about something else.
I will be talking about the 1968 Masterpiece Destroy All Monsters by Ishirohanda.
I assume everyone has done the viewing and seen this beforehand.
I'm specifically talking about the J-Map edit of this film.
Much less often seen in Western translations, so I'm going to run you through the basics.
Last time I was here at Fossham 2 years ago I did my talk in the terror of IMAP.
The translation we used then was called J-Map, the absolute minimum.
You hadn't seen this talk, you should probably watch it right now so you have the
context you need.
Oh wait.
No.
Okay, let's keep going.
That's how I talked about this guy who knows who this is.
Very good.
I would also take Godzilla, much harder this guy who here represents SMTP submission.
When you wrote an email and you're sending it for your ISP to send it out, not when ISP
is talking to each other.
That's Rodean, not talking about it today.
There was something that was going to help us out with this problem of IMTP, I'm
App and SMTP and that is J-Map.
J-Map is awesome, it's really good.
Other people were talking about it and they were talking about it because it's so good.
If you have not been following lots of hydrogen movies, I will try and explain, especially
if you want to hear from my talk last time, Godzilla and King Caesar were horrible monsters
that are beneath the earth underlying everything we do and they're useful, they serve
a purpose in nature, but they're like worst, so that's, I'm App and SMTP are these horrible
things underneath everything we do and they're really useful and we're glad we have
them but they are the worst and so, mega Godzilla was sent to Earth from space to defeat
the monsters.
Much like J-Map was sent to Earth from fast mail to defeat all the protocols.
Thank you for attending my talk.
But the thing is, there's not just those two monsters, right?
If you watch these movies, there's like a screening of them, this is a freeze frame from
destroy all monsters and if you haven't watched the movie recently, right, we talked
about IMAP and POP, you can also see active saying dynamic DNS, but today, I'm only
going to be talking about this guy, who knows this guy?
That's Geterra, King Geterra, put respect on his name, but first, IMAP.
So this is a picture of IMAP and what I've done is I've kind of highlighted all the bits
of IMAP.
If you are going to write an IMAP client or server, you need to understand the bits.
There's like response codes, there's things called tags, there's lists, these are lists
of atoms, there's responses, and unsolicited commands, it sucks.
I mean, all protocols you have to learn how to deal with the protocol, but you want to
know as few as possible, because every extra protocol is using up space in your brain
you could be using to remember the names of monsters.
So there's one really important property of IMAP, mostly you know, like I'm up as my
mail, and I read my mail, that's mostly what you need to know.
The really important thing is, this is called quick racing, and the top you see, it says
quick racing, state 123, how do you know which one is the state, well, it's like a list
of five items in the third one is always state, it's terrible, it's IMAP.
So what happens is your client is saying, I've been online before, I've synchronized my
mail, I got to state 123, what's good, what's new?
And the server says, I'm at state 130, if you apply all these changes to your cash, you
will be at 130, and then we will agree.
This is super useful, it's why you can read your mail offline, it's why when you connect your
mail client, it comes online really quickly, but it's really, really good.
So the good stuff is great, but the bad stuff, but all these, let's go back.
All this crap, it's a buzzkill, I don't want to deal with it.
So we've got JNAF now, and JNAF gets rid of all of this.
I'm not going to teach you how to use JNAF today, I only have 25 minutes, I'm already
going real fast, I just want to show you how easy it is and how worthwhile it is to learn.
So this is JNAF, start off, look at that, it's HTTP, that's good, that's like one protocol
to rule them all, you know, that one, you're good, this take everything we have, stick
at an HTTP, they're putting DNS and HTTP now, it's pretty good, okay.
So this is just posting our requests, we say, I want to get some emails, which emails
one, two and three, there's some more stuff in these little ellipses, and my previous talk,
I talked about the AR, but there's no tricks, I promise it's not weird, it's just some
stuff I couldn't fit on one slide and you'd be able to read it in the back.
So you say, get me those emails, what do you get back, you get those emails, all right?
It's just like, yeah, you said get emails, here they are, email one says, hi Brussels, you
know, and then look at who's it from, it's a little bit of JSON, totally easy, no
tricks, just works really good.
Later you can say, hey, are there changes, the last thing I saw was state 616, this
is quick recent again, right?
I want to re-synchronize my offline cache, I had 616, and the server said, yeah, you bet, buddy,
you had 616, if you re-fetched these items, you could be at state 717, and now you synchronize
your offline cache, just as good as I'm up, I might say better.
But doing it in this way, it makes things really, really simple.
There's no remember in which element of which is what, there's no writing a custom protocol
layer, it's just some JSON and HTTP.
Okay.
JMAP is really easy, all you really need is JSON and HTTP to get all your data, great.
Offline sync is not trivial, but it's really easy.
So then you've understand JMAP, at least the basics you need, for this, let's go back
to Monster 0, this guy, this guy, this name, can you get a, you got to remember this,
this guy is important.
Now, the most, what's the most notable thing about this monster?
Three heads, that's right.
And what are the names of his heads?
KARDDAV, CALDAV, and DAVDAV.
Okay.
These are the three things we need to slay to get more legacy protocols out of our lives.
Let's look inside and see what these protocols look like.
So this is us getting my calendars.
It's just HTTP, right?
Oh, that's fine.
It's HTTP, Rick said HTTP would make everything good, right?
What the hell is that?
Like, okay, it's HTTP with the HTTP extensions for distributed application, versioning,
or whatever, it's called, nobody knows how these work.
I work with like the experts on this stuff, they don't know how it works.
It's, it's trash.
But let's find.
Okay.
Let's say it's not trash.
You read the RFC, what better way to spend your time than reading RFCs, you move on
with your life.
I understand all these new verbs, there's prop find, there's report, whatever.
Look, what's in the body, right?
What's going to be in the body of this response?
Oh, okay.
XML, oh, first off, let's look at this, HTTP response code 207.
I don't expect you to know all of them, but you probably don't know 207, multi-status
with XML in it.
What's this going to be?
Let's look.
Okay, here we go.
It says, the multi-status, one response for this URL is a 200, okay.
And this other one is, you're getting a bunch of responses inside a response wrapped in XML.
This is why we need to destroy this monster.
Okay.
And you're like, okay, whatever, man, it's XML.
I grew up coding in 2002, XML is my life's blood.
So let's keep looking at this XML and what else we find in it?
See how it stops like going in and out?
It's because there's a C data in there.
And what's inside the C data is this stuff?
What is this?
Let's zoom in.
Enhanced.
So this is eye calendar.
It's like the little prize inside your box of XML.
And here it looks like a bunch of, you know, like a bunch of hex or something.
But you can break it apart.
Yeah, somebody shouted out key value.
First there's like, there's bracketing beginning and ending.
And then there's these key value things.
So inside my alarm, there's an alarm ID, an event ID, a trigger, and you can even break
that down because I was a part of a key, but it's not really a part of the key.
It's this value date time.
And here's what we call these.
The whole thing's a component.
There's property names, property values, parameter names, and parameter values.
And if you think you'd never get confused between which is a property in which is a parameter
you're wrong.
Anyway, it's one more thing you have to remember.
And by the way, you're probably thinking, but at least there's not a bunch of weird
value encodings of different ways to encode the value based on the parameters that are
attached to your property.
So you're wrong, it's a really fun format.
Okay.
So card dev is eye calendar, this is weird of format, embedded in XML, synchronized
over dev, which is basically HTTP with a crayon up its nose.
And that's calendars and contacts with the same thing.
It's just using V card, which is a very similar format.
And you work with this long enough, and you just want to destroy all monsters.
So there is one more thing.
I said, there's one good thing about, I'm not by one to talk about, there's one good
thing about these dev protocols I want to talk about.
That's this.
This is your sync token.
You get all your calendars, your contacts, you're writing disk.
And that later, you say, I've got the sync token, please give me a report new HTTP method
by the way.
Give me a report, which is a sync report with the sync token that I used to use.
And what happens, you get told just the stuff that you have to update.
Nice, efficient, re-synchronization, that's what we want, don't throw that away.
So you only get back what changed since last time, but of course, it's all wrapped in XML
because that's the world we live in now.
OK, so we go back to these guys, how are we going to fix this problem?
Well, we're going to have JNAP for calendars.
JNAP for contacts, and JNAP for file storage.
These are the three weapons we will use against, can get around.
So we can go back to JNAP.
This was JNAP, right?
We saw this one earlier.
I want to get some email.
And instead of saying, I want to get my email, we're going to say, I want to get my contact
card.
Absolutely the same.
When we got our email, it looked like this.
Here's the list of emails you wanted.
This one has a subject.
You get, here's the contact, Karz, you want it.
This one has a name, it's a email address.
Like, that's it.
There's no weird parser, it's just like life is easy.
You can write a 20 line little program in a cron job for whatever you want, dealing with
this stuff without learning weird libraries because it's so easy.
But changes, right?
You want to synchronize your offline cache exactly the same.
I want to synchronize my offline cache.
They're not running at different ports, which are different paths.
You can send, like, synchronize all my offline cache and you get all the responses back
in one HTTP round trip.
I'm not going to get into it, but it's awesome.
Okay.
So now that you know, JNAP for mail, you already know everything else.
There's no protocol really to learn.
All you really need to focus on learning is the data types.
Those are different.
Different properties.
They have properties.
If somebody told you, here's a calendar event, and the keyword list on it, it's an unseen
calendar event.
I don't know, maybe he's actually all work.
Okay.
I'll keep moving.
There's just new data types.
So in contacts, there's a data type for address books and cards.
In calendars, there's a data type for calendars, events, and participant identities.
Let's not talk about that one.
If you're used to dev, this is the mapping.
Order devs, collections, become address books, Caldav collections, become calendars,
V-cards, become contact cards, V-events, become calendar events.
That's it.
This all maps.
There's been an enormous amount of work to mean that you can use your JNAP stuff, on your
old DAV data, and you can just port over without having to export everything and re-import
it somewhere else.
So let's look at the data types.
Here's a contact card.
It's an ID, and then like a name, addresses, phones, emails.
I said I don't like having things hidden on the slides with these ellipsies.
I want to show you what's really in that data.
So let's look at that.
Here's a name that we might have on our contact card.
A name is an object.
It's called a name object.
It's got a full name sometimes.
Here we've got one, King Gidara, and components.
And what are the components of his name?
He's got a title, and he's got a given name, which is useful to know.
It tells us that it's not, you know, King of the Gidara House of Gidara.
His first name is Gidara.
I don't know his lineage.
I'm sorry.
Those things, by the way, come from Unicode.
They're from Unicode Technical Report 35.
This is interesting, or useful, because it means that if you're going to work with these,
you don't have to figure out how do I take these and combine them in a culturally appropriate
way for the language that I'm working in.
There's libraries to do it.
When you're working with contacts, when you're working with calendar events, you are
dealing with horrible human weirdness to begin with.
Calendars are weird.
You can't fix that.
The French tried it didn't work.
Contacts are weird.
Adresses are weird.
You have to represent that complexity.
And so anything you can do to not implement another way of fixing it is better.
OK, keep going.
We can do this guy, nicknames.
He's got a nickname, how many?
That one nickname.
It's not actually a counter, which is the identifier for his nickname.
And localizations, this one is really nice.
It says, if you're trying to read the full part of his name in the Japanese language,
use this text instead.
You can localize almost everything you need to.
And you start thinking, well, I don't like this slash in a key.
Why is there a path?
It is a little weird, but it's a core concept in J-Map.
Anywhere in J-Map, you might be saying, I want to take this one part of an object
and replace it.
That's how you updates.
They're partial, not complete.
So this is just saying, if you're going to work with this card in Japanese, apply this
update before you start doing stuff.
Really, really elegant, and uses all the stuff you have to do for the basics of J-Map.
Here's his address.
So we've got one address in here.
It's got a work context.
It's got a preference.
This is probably his most favorite address for you to use in contacting him at the office.
And here's the actual address.
Similarly, what kind of thing?
That's a street address called the name of the address.
And a locality, he's based in Tokyo.
Not actually truly can get her as based in outer space, but I
had to make this slide quickly.
So there you go.
That's how an address looks.
Very easy to work with.
Yeah, I did want to talk about this.
So here's his addresses.
And let's say that we want to fix his address.
He's moved.
He's changed jobs.
And what do we want to do?
We want to update one of these.
Remove an address.
And we're going to add a new address.
Now, if we were doing this in Card Dev, what would happen is we'd read the entire Card
Dev document, parse it into memory to some kind of data structure.
Update the structure, change in the addresses, serialize it back into V card, and send
that to the server.
Now, it seems pretty simple, but the problem is everybody has their own weird little
ways of reading the card into memory and writing cards back out into memory.
So either you're loading it to memory and remembering all the literal parts, you did
not need to update and deconstruct so that you could write them back precisely, which
is what you should do.
Or you load the whole data structure, you change the parts you want, you rewrite everything
in your own particular idiom, and you post back to the server, and you just hope.
You did not accidentally change anything that some other client was relying on.
And you will.
You will break this stuff.
If you do this in JMAP, you compute the difference you want to make, and you tell the
server, please apply this patch to the object on the server, and don't touch nothing else.
And if it does touch something else, because you have a side effect, it tells you, I applied
your patch, which also made this thing change, and your catcher means up to date.
And if you need to, you can say, apply this patch, but only if nothing has changed in the
meantime from when I started working.
This stuff is going to save you a huge amount of time.
And when I patch it to the address, you want to add one, change one, remove one, it looks
like this.
Change the preference, and the address known as address one, delete the address known as
address XYZ, and add a new address, which is this whole address that you're adding.
That's it.
Really, really easy.
Nothing weird to learn here, just some data structures to stick together.
I have worked a lot of pain.
Okay.
That was contacts, right?
What were the new things introduced by contacts, contact cards, and address books?
I didn't talk about address books, and address book is like a name, and an ID, and is it
your default?
That's almost about it.
There's nothing to say, contacts are obviously big and complicated, but it's all just
the basic complexity of contacts, nothing about JS contact, the format for contacts and
JMAP, nothing there is weird.
It's just the complexity of the human weirdness.
Calendars are also human weirdness, and we're going to talk about calendar events.
Again, it's basically the whole source of complexity.
Here's an event.
We're going to on yesterday and two years, two days ago, but in two years, destroy Tokyo.
Sounds great.
Some people will be coming.
There are going to be recurrence of this event.
We're going to destroy Tokyo more than once, and some of these recurrences have overrides.
That means not every event is exactly the same.
Some of them have little tweaks.
A common tweak you could imagine is Bob said he's not coming to lunch next week, but
he will be at all the other lunches you have to override next week's plans.
Again, don't like ellipses.
Let's blow these guys up.
Not Tokyo, but also Tokyo, but it's examples.
recurrence rules are very simple, but my example here is quite simple.
What's the recurrence yearly happens every year, July 20th, January 29th, we destroy Tokyo.
Together ones are a little more complicated.
Participants looks like this.
Get that one key.
It's just the participant here known as participant one.
Means nothing.
You just need it to address things.
Just like we did with addresses in the contact.
It's got some roles.
This person will be attending.
The mayor of Tokyo should be attending.
But his status is actually delegated.
He's asked someone else to come.
You could also see Needs RSVP, things like that.
Very, very simple.
Again, boil down to about as simple as you could make that.
Currence overall, it's also very simple.
This is a set of patches against each recurrence.
You say which occurrences being changed and how?
Well, 2029, we're skipping it.
2029, you've just been your first time rebuilding Tokyo.
We're going to wait one more year.
And then way off in 2038, we're going to have a long, long morning, sleep in a little
bit.
And we're going to destroy Tokyo five hours later than usual.
These are recurrence overalls.
Hunger patches, pointing at a particular event, can't get much easier.
recurrence rules themselves are huge pain.
I don't want to get too far into it.
Just imagine this work through really briefly.
You've gotten the event.
It happens on the first Saturday in December every year,
but also the day before that, which might be in December,
but it might be in November.
OK, this actually, that makes sense.
It's easy.
It looks like that.
And in JS calendar, it looks like that.
And you're going to chase on.
I'm saved.
No.
All the complexity is still there.
It's still sucks.
Someone earlier said, when you use JNAP, it's
sticking a bunch of complexity into the server.
And that's true.
And it's good.
It means your client doesn't have to know anything.
So if I don't want to think about this, and I don't,
when I ask for an event, I think,
called a query query is like, you saw, get or changes.
Queries of search.
This is search in my email.
Find all the mail from me that's been flagged.
And you get back a list of emails, right?
Oh, these emails are the ones you want.
And then you can either get them.
If you need to, or read them from your cache.
Same thing for calendar events.
Query in my events between these dates,
anything that says destroy in it, and you're
going to get back that one event, the destroy Tokyo event.
And then you have to take the recurrence rules
and figure out all the ones that actually
occur in here in your life sucks.
But if you say expand recurrences, the server
will compute all the recurrences for you.
And it will tell you, here are magical IDs
that you can fetch and get the pre-patched
all the overrides in it properties of event,
and that you can use to update and override
things onto those events.
And then your client can be stupid.
And since most of the clients I write are like 100 line
crong jobs or maintenance things,
you want to be able to write stupid clients very good.
OK.
So these are the things that I've talked about.
They are RFC 9610, which is where you find contacts.
And JMAP counters 26 is a draft.
It's basically done.
But we've realized some things that we should probably
fix before we really ship it.
But it's real close to done.
The other thing I can talk about is file storage.
File storage is version 0 of a draft.
So this gives you a sense of where it is.
It's at a very early stage of development.
But that's been using file storage for years.
I'm going to guess like eight years or something via JMAP.
Pretty good.
Like it sits there.
It works.
Maybe we've changed some bits of the code that
implemented.
I don't think we've changed the API more than once or twice
in those eight years.
Like it just sits there and works.
So we have some goals about how to make it robust enough
to make it an internet standard.
But basically, you could follow along and watch the mailing
list and see where it goes.
Mostly it's sitting there right now,
while we're traveling around at conferences.
Here's what it looks like.
You get a file node object.
In a file node, it just represents a directory entity,
if you want to think about it in Unix terms.
It's got an ID.
It might have a parent ID because it's in a directory
or it has a parent directory.
It's got a blob ID.
A blob ID is a whole bunch of bytes.
If you think about an email, here in JMAP land,
we try not to think about the RFC 5322 encoding
of the email into a sequence of bytes
that get center on the internet because they suck.
We just think about these JSON objects
that have like subject and from and parts
because then your life is good.
But under the hood, there is a pile of bytes
that is this email document.
That's the blob.
Everything has a blob, including file nodes,
which are mostly blob.
It's got a blob ID.
It's got a name, it's our file name.
And some other properties like,
is this an executable file?
You can't ask JMAP to run this for you,
but you can't ask it whether it could be run
where it on your file system.
If you want to put one of these onto your server,
you do a JMAP upload, which is basically
post to a URL that your told is where you send uploads.
And then you make one of these.
You say file node, set, create this blob,
this name, this parent, there you go.
You got a file.
OK, right, winding up.
This talk has mostly been really basic high level
overview of this stuff.
Like I said at the beginning,
there's no way I was going to really teach you
to use JMAP in 25 minutes.
But I wanted to show you all the stuff that you can do with it.
Why it's good?
And we can just start with, because it's so,
so much simpler than Card Dev, Cal Dev, and the Dev Dev.
Because it gets sort of the weird HTTP,
it gets sort of the XML, it gets sort of the V card,
and I calendar formats.
And it brings you back to working with stuff
that you already know how to use.
And by the way, all the APIs that you're
probably using in your day-to-day life
for like talking to this automated scale
that pulls your weight down every morning,
those are HTTP and JSON.
That's the kind of simplicity you should have
when dealing with the most important data in your life.
You're to do list, your calendars, your contacts, your email.
That's where the easiness should be.
And that's where JMAP is putting it.
Anyway, we talked about how it's so simple.
It does work at scale.
It's not just a little toy protocol.
It is a very good protocol for doing all the big scale
stuff you want.
Fastmail runs almost entirely on JMAP internally.
These are the documents.
I'm not going to stop here, just here they are.
You can find them in the slides later.
You should expect to see some changes coming
to JMAP for contacts.
We've made some realizations after shipping it.
A little stuff.
These are the calendars ones.
They will get standardized.
Again, some little changes are coming.
This is the file storage one.
I'm sure there will be changes coming,
and I have no idea what they're going to be.
So everybody wants to use JMAP now.
How do you use it?
Well, you've heard about a bunch of different JMAP servers today.
You can ask me about them later.
You can ask other people who have talked about JMAP.
Unsurprisingly, I'm going to say the easiest way
is to be a fastmail customer, and go to our dev page
and say I want to API token.
It's there.
It's good.
We're going to be having contacts up for everybody to use
in the next week or two.
We're already on our beta platform.
That's it.
Any questions?
APPLAUSE
So thank you, Rick.
I think we have a strong contender for the candidate
talk for a fighting most monsters at first time.
But this is everybody winning and breaking up
to challenge with our old professionals.
I've defeated them all.
Yeah.
Is there any real world map clients that are trying to implement JMAP?
Ben is raising his hand, Parulo, which was talked about earlier
implement this.
You can run fastmail on your desktop and an electron app,
which runs very nicely on Mac OS.
There's also one called Swiftmail, which I think
might be out of development.
There's a Swift, a JMAP client called MailTemi,
which runs on iOS.
There is.
Yep.
I saw you earlier and it left my head.
You've got to go.
I don't think that you can run on desktop yet.
Yeah.
So there are some coming.
Yeah.
It's good to see with our clients.
Yeah.
Thank you.
Yes.
What's the recommended JSON parser for this?
I don't care.
I feel like if your protocol has a recommended
JSON parser, you have made a mistake.
I'm sorry.
What about CB?
CB, well, if you want to find out what is the most reliable
JSON parser for the language you're working in,
I am not the person to ask.
I am not an expert in particular libraries across these things.
I can tell you that we use Json internally
in our C implementation.
Our Perl implementation uses JSON excess.
And that's right.
We use a lot of Perl.
I'm sure there are some other JSON implementations
in our ecosystem.
But that's, I'm just not the expert for that.
How does it become a way to encryption of data inside the JSON?
Because if you can kill us, determination, and so on.
So the question is, how does this work
if you want to encrypt your mail, especially end-to-end?
Is that all like specific headers?
Yeah.
So I would say the brief answer is, if you want fully
or heavily encrypted communication, email
is the wrong place to start.
So you can encrypt your email.
There are extensions for JMAP for dealing with encrypted email.
They are largely for server-terminated encryption.
If you want to have the email be fully end-to-end encrypted,
the server can't process things like, give me the from,
give me the subject.
There are things for server protection,
which means this particular header has been encrypted.
So the extensions can handle that.
I will tell you the vague advice.
I would give someone implementing more end-to-end encryption
would be synchronized the blobs,
using entirely blobs synchronization,
and run a JMAP proxy layer inside your browser,
which, by the way, is how fast-mails offline client works.
It runs a JMAP client and server in the browser,
talking to itself for the sake of synchronization
and offline work.
I think I'm totally out of time.
So we have to keep on procrastinating.
Yeah, anybody else?
Yeah.
When will we release JMAP push?
This is a man who knows the question to make me flee.
So JMAP, not mentioned in this talk.
JMAP has two mechanisms for doing server-side real-time streaming.
One is you can get an event source,
which is a long running HTTP connection
that just keeps telling you, you could re-synchronize this type,
you could re-synchronize this type,
and it stays open, works really well,
and you can reconnect.
This is great if you're running app in the browser.
The other is RFC 8030-style web push,
where you tell the server, here is a mailbox
to what you can post updates.
That's what should go on to the know about.
Fastmail doesn't implement that yet.
I don't know the answer.
We haven't working.
There's just a lot of little bit of edge cases.
So I would love to ship that as soon as possible,
but I hope it will be this calendar year,
and I don't think it's going to be before the middle of the year.
This is what everybody has.
I know, everybody wants it.
All right.
Thank you.