All right, here we go.
Yeah, for those that have been here in the start, or not been here, my name is Franz
York.
I'm from Austria, and I will talk a little bit about mostly, but not necessarily completely
a client related topic, but it is sort of crosscutting, so it's not about a very specific
client, but it's about a forthcoming new standard way of improving email, which for the
moment, or probably for the time being, will be called structured email, and I will walk you
through a little bit about what it is about, what you can do with it, what is around in terms
of bits and pieces, and especially if you are an inner client developer or an email software
developer, what kind of actions you could take, or what kind of help or feedback you could
give here at the current stage, I'm going to switch by the way, oh, no, sorry, okay, well,
another good back, I thought it's a, I thought it's arrow keys, but it doesn't work
because it arrow keys, okay, this way, okay, sorry, here we go, that's the motivation slide,
so we already know, email is still a quite unique technology with very unique properties that
not so many other technologies have, so if you say the web, in a way, is a building block
of the current internet in terms of something where you dynamically interact or synchronously
interact with websites, email, so to a certain extent, it's a most standard, asynchronous
technology are currently using most of the time, and it's also very unique in terms of
the addressing, we are email addresses between, as I said, in the beginning, exchanging
between your private and the public space, so it's actually a very versatile technology,
and most of you will know that because you use even for all sorts of things, right, so it's
like a little bit Swiss army knife of IT sometimes, which is to the good and to the bad
to the certain extent. However, email, since it is widespread, it's challenged all the time
with new features, so we saw in Benstock already also, there is a lot of movement in terms
of private chats, like in some messaging, which I think has captured a lot of use cases,
which historically some people were using by email, and there's also in the copyright world,
you have things like Slack and team, and all these kind of things, which again, overlapped
to a certain extent with email, but still email is actually growing worldwide, so which is part
of the reason you to, you know, IT grows worldwide anyway, but also, you know, when you look
into your mailbox, email to nowadays is a lot of transactional, so a lot of males, which you have
are basically your orders, your foster registration, your whole registration, your energy bills,
these kind of things, and so there's a lot of email, which is actually generated by not pay
humans, center you, but generated by systems, and you know, templates and stuff, but in a way,
your email client is sort of dumb, so it cannot help you a lot with that, it can help you
somehow with calendar invitations, but it cannot help you with all the other stuff, most of the time.
But, you know, if you think about this email, it's a little bit like your personal API,
like you receive posts in the form of days, and you know, your bills and stuff, and put it in your
in your wardrobe, now it's, you find it in a folder probably, or something like that,
but still it's a very manual process, so the whole motivation behind is how can this
done more efficiently actionable, interoperable, and maybe even enjoyable, yeah, so it's
forcing in a row, let's see how far we can get here. So what's a basic idea, so the idea is,
you know, there is machine readable data on the web already for a long time, so many people
of you might be aware of RAF, in general, as a standard by the W3C, for knowledge representation,
which is around for quite a while, and it's also used very widespread on the web by the way,
so if you do a web search nowadays, and you see these rich search snippets, which tell you this
pizzeria, has system this rating, and this is this phone number, it's very often actually
fueled by a structured data annotated by the website hostels, owner of the website, so you have
some 10 million websites, any new sites you look at, any cooking site, any music site, yeah,
you look into the metadata of the site, and it would typically have so full metadata of content
in the metadata of the site, which is just silently used by the search engines so far, but which
could be leveraged for much more, and in ways the idea for email is sort of the same, right? So
you have historically different representations for email, so email started text-based, asky-based,
and there's still asky-based, to the round we have seen in the very first talk, which is great,
but many email nowadays is also HTML-based, yeah, but email already was so flexible,
that typically you can combine both of these things, because you have these multi-part alternative
body parts, so many of the emails you receive today have this feature where you can look at
them at the command line, but you can also look at them in a fully-fledged visual client,
and the idea here is just to add a third part of it, part of it, which is a machine-reader part,
so if you have an email client that can understand that and can act upon that kind of machine-reader
but data is fine, if not, you can just use email as you used it before, so it's actually quite a
quite a nice thing, which can be added into the email without disturbing the workflow of people using
email to that extent, and actually a lot of people do that already, so to a certain extent,
the happening, if you scan your mailbox and I'll show you a tool later where you can actually
try that out if you want, you will find that for instance, many airlines already put this kind
of structure data in their emails, they mainly do that because Gmail is rendering the airline
case within Gmail, if you opted into that, but they will also send it out typically to your
non-Gmail account, but just your typically email client will just not make use of it, yeah, and
yeah, so but of course if the email client knows that it could help you much more, you know, adding
this to the calendar, probably re-bubbing the flight and whatever you want to do with that kind of
data, very short, maybe because some people might know, okay, making something smart or so there
is a icing around, right, like everyone is talking about, just for those who get a little bit
confused, how this relates to the other, right, so with AI typically you don't have necessarily
structure data, but you have text-based data in the AI twice makes sense of that, that's why
you try to find other cases, might be a flight from here to there, whereas the idea of structure
data in that sense here comes from the source, like the sender will already, you know, provide
well-defined structure data for the certain use case because typically the sender will know already,
so in an idea where these two things are complementary, yeah, so obviously it's a little bit
closely to have structure data for everything, so you will do it if you have the data already,
anyway, so the case for AI is always like, you know, there is not yet that prescribed data,
but if the sender anyway has that structure data already, it typically makes much more sense
to already include it, so there is no AI that has to guess on that or something like that,
but these two things are also used to work together to a certain extent in, you know,
providing additional things for AI assistance and so on, so it's not a thing that is, you know,
totally different, but it comes from a different direction that's said like that, yeah, so what
use cases can you use is for, so our retention sets are is to standard seeing like orders,
reservation, the flight reservation, the train reservation, the hotel reservation, there is also
things you can do in instant messaging, right, so for instance in instant messaging we had already
an example by bandwidth pulse or locations, right, if you do that right now in email it's not
really possible in an interoperable way and having some some markup here which would tell the email
clients this is actually a location or this is a poll, it could be very beneficial in making
that interoperable, similar things with things like articles, recipes and songs, I've been told,
I've been telling you that a lot of websites you already have submitted data, yeah,
so if you if you for instance go on on a on your mobile on a web browser and share
a Spotify or whatever song with like your WhatsApp or something like that, it will basically show
up a link preview at least, yeah, so you see like the song album cover or something like that,
do that to an email client and it you will just see the URL, yeah, so you have stuck to that kind of
Spotify thing, we structure data since for instance on Spotify the structure data is already in the
website we can even add additionally the structure data so your client doesn't just have a link
or just an album cover but the client actually knows this is a song and this is the artist
and could for instance is a map set to another service if you're not a Spotify subscriber or
your personal np3 archive which is you know why it's your thing so that also gives you a level of
you know portability and you know less beings stuck to a certain service provider here
and as he is case might be something like new slators or meeting info so you're of all been receiving
emails from the first them organizers like you know this is a venue here is the train station
here is probably hotels or restaurants something like that and it's really always a link in the email,
yeah so your email client has no clue what this is about so what about first time could you know
annotates this kind of information in that newsletter which is anyway you know crafted very nicely so
that would be a very helpful if you could very easily as an import that to another app or where
you manage your trip with or something like that and finally there's also email specific use cases
for instance like you might know what vacation notices which tell you somebody is absent for a
certain point of time it's also something you need to read by yourself understand by yourself
and actually there is a particular draft ITF draft for this particular use case you know
leveraging structure data also in that case so your email client can basically understand
that's remote persons absent and there is also you know
and everything more on some more use cases there is a particular ITF draft for this interested
and certainly this is also a call to the community to you know think about use cases provide feedback
on what people might find useful and so on so how does it work for many implementation side
because you know we have a chicken act problem here email is a very um you know
widespread longstanding ecosystem with a lot of actors and it's very hard to get all of
some you know on one boat to make this happen and obviously you for this to be useful you
need senders to do it and clients to use it as well and the clients will say the senders should
put that in first otherwise why should I care about the senders will say oh no client is supporting
that for sure I do yeah so yeah that's classic you check in act problem good thing is I told you
some senders already are using it through certain extended list also for senders at least for
automated senders it's to a certain extent probably easy because as I say if I already
build the email out of a database already maybe even using a certain system a lot of people are
using it might be easy to just put in some mechanism in the template and then have a real
part impact on on the data involved also senders especially put especially a professional
senders I interested in that in a sense because if it would enable user better user experience
on the receiving side um they are interested of course in you know providing this kind of
add-on work because it would help serviceability in the inbox it's a similar mechanism somehow
here for the web yeah so people started annotating that on websites because basically they had
a benefit in Google search or in other search engines um there's another way also of course
that would be useful if you could buy yourself if you could if you would draft an email if you
could send certain structured emails I mean nobody would necessarily require you to write like
RTF in the email explicitly but you could think about you know when you share a link for
instance it could try to pull the structure data out somewhere and there could also be 10
plates for something like a poll or something like that yeah so there might even be support
uh in email clients for this kind of use case on the receiver side um you might think about
adding this on the MTA level like if the mail comes in you order process it already uh and write
it somewhere to restore or to some system this could at least something not 100% in focus but
that could obviously be done um currently for for uh implementations it's more on the user agent
levels from the email client level where it's directly visible to the user but these two things
could easily um work together of course and maybe just to explain a little bit it's it's actually
very similar so this is just the screenshot of a meeting invite many people of you know so there
is RFCs for this particular use case and it's basically working very similar like so also in
calendar invite sometimes you have servers acting about that already sometimes you do it in a client
so you have a very same situation of how to deal with this kind of data um and also what's
interesting is you have a typical way of presenting that kind of information adjacent to the actual
mail or even fully replacing the email yeah so this is a user experience which is very similar
of which people are used already for invitations or these calendar invitations and many tools
do it in a similar fashion I would say yeah so this is sort of a little bit of an inspiration
um so in a way you can think about all this structured email is a sort of a generalization of
that yeah so um you have this thing happening for calendars but you probably wouldn't write like
500 more RFCs for very specific topics but the idea is to have a more um general framework here
um so how does it work generally so um this is now the use case on the client side so we you
really have something like parula or son of a bird or something like that um so the first question is
when should you analyze the email in that case yeah so um um obviously when the mail is incoming
to the user that is not on the client side typically because the client would need to be active so
that would be the NGA case which we don't look at right now here but one thing could be once the
email client is running you look in the background and new mail is coming in you process it and you
look is a structure data inside which you might want to act on you know the other option is to
use or explicitly open the email and you look then into what is in that kind of email yeah
in terms of extraction I told you already some airlines um doing it for instance in emails
embedding this structure data they do it a little bit in and let's say not so clean way so it's
basically put in the text HTML part of the message and actually not many of the windows really like it
because it's a little bit uh yeah I would not say they dirty approach and so on so the idea with
in the um ITF or the probably consensus that will be is every reach is to really do it as a separate
a multi-part alternative type um so somehow maybe at least for some time when this is rolled out
you might have situations that both of these things are around um but you would have a library
probably looking for all of that um the interesting thing is you could also do some kind of bootstraping
beyond that yeah so even if you think about uh having not directly at the F in the email and
there is already a lot of emails um that have some other structure data um like okay ICS files
if they are not a meeting invite to the user just to play an attachment I mean some clients might
render them um you have this pick-up past tickets that Apple for instance sends around for events
many of you might know um also there is some approaches or e-in voices as a very recent
thing is a u especially um and there is also things around like you can even extract from
textual emails certain stuff so the k etinary project which was presenting here at first and
last year in the track um they have basically some regular expressions um where they do support certain
airlines you know extracting the flight data even if the airlines do not yet include the structure data
just based on um on the textual content that of course not a very nice and sustainable thing
but it's probably a bootstraping approach to help the things get running right so then how does
the interaction work yeah so once you did all the parsing and everything what can you actually do
is it yeah and um so one idea is um to do it a little bit like also you do it is the instant
messengers or select also also like you have sort of a cut-based rendering where you do some
have some human readable information of what's a structure data is about so this is something
you could do quite for a lot of data without being too specific because it's probably like an
image and the title and then we come to the actions um but for some sort of use cases you might
also have very specific templates like for a poll such a card would not be sufficient but you
would need a more specific you know rendering of a poll with options and submissions things
another thing you could probably do is that some simple forms yeah where you could probably
enter some information think about your electricity provider asking you for the you know look
up so electricity meter in your backyard typically you go to a portal a little bit just to
enter just a very single number so that could be theoretically something you could also do
with that kind of approach and yeah the engine is of course actions yeah because rendering is nice
but you want to do something with it yeah but the good thing is once you know which kind of structure
data that is like for instance here we know it's a location yeah we can for instance trigger our
launches in our open street map or whatever kind of thing we want to have we might even decide
some email client might even want to decide to render that in a map yeah so because the location is
known think about my case before we are first them since you're list of restaurant recommendations
or something like that you know the email client is covering that might offer you to render that
within the email client somehow in a view on a nice map or something like that yeah so
this is what I call you like platform dependence or there might be also add ons in your email
clients that act on certain kind of sea structure data it's obviously not that's the email client itself
will be doing most of that but yeah you could have a cookbook app for instance where you have
recipes in right and the recipe you get sent in by email might be directly wired into that
the second category of actions might be structured data responses where you say you directly send
a structured data back to the to the ones that was sending to you good example might be the answer
to the poll for instance yeah where you just choose this and or something like you could sing about
a vacation request yeah I wonder if vacation leave next week for two days yes or no
and that's could also be done by that yeah very shortly security and trust it's probably
something maybe also for the discussion if somebody's interested this is also an ongoing discussion
on you know should one require S-mine signing or PGP signing for this kind of content to avoid abuse
here so there is an active draft as well and some discussion on the idea of mailing list for those
people interested in that facet that's a little bit summarizing you know the current flow of how
things work or get changed through certain extent a very short note only an interesting question
is also like you have the message store where the actual messages are but what do you do with
this structured data yeah that might not just be a one time thing you might want to have
somehow within your email client or your desktop some possibility to store certain snippets yeah
to store restaurant recommendations or something like that for which the message store probably
is not the right way to do but this is more like a implementation decision or feel see also
of clients but at a certain point that might also get interesting to maybe make more into
operable between different clients but this is maybe more of a future thing here so in terms of
implementations very shortly so came mail the supports is kind of approach at least to certain extent
since the while already again there was a talk last year here at host them you have some support
in next lot mail the very recently launched an outlook at an actually sorry for that but it was
obviously also a community you want to write in the boat here it's actually specific to
electronic invoices by now but underneath it's enabled or the whole mechanism is already like
built for all of this structured email as well so this will be enabled in a second step here we have
a prototype for K9 mail I actually wanted to have said ready for host them but we will probably
have it in the next weeks published so you can try that out and we have also support for
sort of a number of redmailers so we have a run queue project even on GitHub it's not the most
recent conversion yet so we will also have an update on that one but some people might be also interested
in having a look at that and this for instance is how it looks like if I sent you a restaurant
in the K9 implementation so you see you can directly call the restaurant or have an
navigation to the restaurant here there is some libraries you can use so that was one goes that we
can provide a client developers with libraries for all that extraction and also for the rendering
so in order to share information here so everybody has doesn't have to reinvent the wheel
and I told you there is already also some additional testing code so if you want to scan your email
account to know which kind of structured data by the way do I have already in my emails historically
there is this little project on GitHub where you can do that there is also another project where you
see some examples of structured email if you want to might want to play around with that and
yeah for the top project you can also send structured email to somebody or to yourself in case
you want to play around with that actually due to the fact that the ITF standardization is still
ongoing so probably I don't know if I'm somewhere here to the end of the years I might be in
RFC ready for for some last call probably there is different options now still how to
include it in the email but still on the reading side you can do a lot of that and yeah that concludes
the talk here so hopefully structured email will be a way to make make email better and make
email more modern and allow for interesting new use cases for those that want to have that in
their clients it's incremental which is nice so that's really also a very nice feature of email
one aspect that might also be interesting here is accessibility so for people that have maybe
difficulties in reading or something like that structured data might also be interesting to give
them alternative ways of accessing the content here is a link to the ITF work so join ITF
mailing list it's free if you're interested in that join the discussion there is an interim call in
two weeks I think and that's all thank you very much looking forward to your question
all right let's see
yeah all right thank you
sorry yes I know okay okay so sorry it was a little bit hard to hear so the question is there
project it's a Google project basically so you can only use it in Gmail there is actually
yes mostly I mean there's Yandex I think or something you're who theoretically but I didn't get it
to work actually but that's another story there's also actual messages for Microsoft which is very similar
to the amp same just for reference and Gmail actually does pose Gmail support some structure data
and also in a unidirectional way so you cannot send it you can just receive restaurants and
flights sorry parcels flights and reservations you need to sign up for that so it's not really open
and decentralized and it is ant in parallel so how does it differ yeah so the structured email part
of email is really Gmail can understand this is a restaurant yeah and this is a flight but with
amp it's basically an HTML email to zero yeah so you can do a form an email where you can send
me data but there is no idea of what is this from about or what is inside here yeah which is which
is probably also the case yeah so in that sense even in Gmail they coexist yeah so it's two different
things there might be the question so for something like suppose or the location sharing if you
want to do live location sharing you might borrow the feature of updating information in an email
or parts of the structure data at least also for the structured email approach yeah so there might
be a little bit of a synergy which again as you say is something one should do very carefully but
this is the current state of affairs here yeah does it happy thank you for the questions I see
I just scanning the rooms suddenly so I have to start okay one of the biggest problem I have
the agreement but on the demo about how to type the data exactly yeah you mentioned it quite
yeah can you it above it yes so you're absolutely leaning the point so I think the main thing here
will also be about you know how to most clients would agree on certain schema for certain use cases
I think one there is multiple answers to that but keeping it short I think one is there is already
vocabulary around in RDF like for this search engine kind of annotations thing on the web
a schema org I think has some 900 classes already for certain things so this is already quite
widespread use it's not very top quality I mean we can never long this pattern on that but this is
at least something you can leverage um second thing is there will be certain things probably even
standardized on the ITF level like vacation notice where it makes sense where it's a very super
clients specific and then there might be something like community work so at least this is one of the
things not yet 100% clearly figured out but the ideas probably to have some community working on
that and sharing some kind of templates and use cases make sense and you have to think about
typically attachments you see to email yes so in a way this is related to that so not email
clients also don't agree on which kind of attachments you can add it's basically the operating
systems or add-ins to the email clients which say okay I can read a PDF and you could think about that
in a similar way yeah like you could have as a way of having add-ins registering for certain
structure data and in that sense you might not strictly need that kind of consensus between all
the email clients it would just that in order to understand a certain one you would need to
have a suitable item here but maybe you can continue this question often I've one more question probably
and then we unfortunately already over thing yeah this is a good question so I would
think so that for instance like people like Gmail also do the expected I know from some ISPs
that already also do analysis and say basically have a combined approach of using structured data
and expected data and putting that into a model so in that sense yes but I'm not aware of a
pure client already so that's what made it up thank you very much again