All right, folks. We're going to get started, or my glasses. This is a new problem in my life.
So some of you will understand when you get to a certain age. Anyway, this next session
is a case study in critical infrastructure and what happens after a crisis. Please welcome
John Erickson. Hello. So my name is John Erickson. I am currently the community's manager
for OpenSSL Foundation. In the past, I was a community manager at Stack Overflow in a few other
places and today I'm going to talk about what might have been the worst day in the history
of the OpenSSL community. And perhaps also the very best day. I like to start my talks with
the OpenSSL mission. I'll just read it. We believe everyone should have access to security and
privacy tools wherever they are, whoever they are, wherever they are, and whatever their personal
beliefs are as a fundamental human right. And this formulation is a relatively new for the
mission, but the belief system, the idea that we believe everyone, wherever they are, whoever
they are, should have access to these tools. It goes back even before OpenSSL was thought of.
So let's go back to the Mesolithic era of the Internet when the World Wide Web was invented.
And the World Wide Web depended on the Internet protocol, and you're not protocol, you pass
packets back and forth from one server to another. And anyone along the path can open the
the packet and see what's inside, right. And there's nothing to prevent anyone from seeing
the plain text. And this is actually sort of a feature of the system because it's open by default.
The idea is if you have the URL, you can get the information. We want information to be free,
right. We want everyone to be able to find what they're looking for. And that's great. And it
is started by research institutes who really want to share our information. But what if you want
to sell stuff on the Internet? I know it's crazy, but imagine if you sold books by going to a
web page and then the books will be delivered to your house. How would that work? Well one problem
you'd have is when people send your credit card to you, anybody along the chain could read your
credit card information and that would be bad. And then things like they would be able to find
your home address because how do you get the books to the home or the office? That's also bad.
So there's a lot of places where you just don't want your information to be completely free
and available for anyone. You actually want it to be very limited to certain people. And so this
was a problem that was actually solved by a company called NetScape back in 1995. They developed
an early browser and an early web server and they invented something called SSL. Doesn't matter
what SSL stands for. None of the SSL is for secure and it meant that all of your packets are
passed back and forth would be encrypted. So if someone grabbed your packets they couldn't read it
and figure out what your home address is or the very private email that you're about to send.
And considering that I was thinking about my I have a new laptop and I thought about all the things
I had to install and I was like okay so I'm going to install Firefox. Okay that's it.
I mean obviously a few other things home-brew and such that they used for developing.
There's a little problem or there was a problem in the early 1990s and that's that the US government
said this is a military technology and you cannot export it outside of the US or Canada.
So think about our mission. Where ever you are. This is what the world looked like in 1995.
There's a story encryption for everybody in you know look into that section up there
and everyone else had sort of weak encryption and by week I don't mean just that the NSA could figure out what you are sending.
Week meaning like someone with a regular desktop computer and a few out or a few days could
extract your information. So unless your secret is very time constrained you know you have a time limit on it.
That's not very storm. And by the way this isn't even explained all the problem because if your browser
that you purchased is in the US but you're speaking to a server that's outside of the US
while you're getting weak encryption. You're not going to get strong encryption. That's a that's a big problem.
And so someone decide to solve it. In fact two someone who have been to live in Australia
and these guys Tim Hudson and Eric A Young invented a system called SSLAY
and made open source and this is a very common thing in an open source project right you see something you want
and you can't have it either because it doesn't exist or in this case you can't have it because you don't live in that
certain geographic region and you build it yourself. And then instead of just saying oh I've got this thing for myself
you say well I built this and I want some help building it and doing more and so I'm going to make the project open source
and other people can contribute and you kind of conquer. You've ever played the game risk you know Australia strategy
which anyway it doesn't it doesn't actually work in the game risk but it worked great for SSLAY
and so it just spread everywhere because if you know weak encryption or storm encryption which
would you prefer okay I agree storm encryption it is. And then because SSLAY was used in other projects like
Apache and Pearl and all sorts of other things it actually went and you know kind of conquered the US too
right because if you're building a free and open source project would you rather use
SSLAY which is open source or would you rather use net escape SSLA which you had to pay for?
A couple of caveats in this there was still a lot of us uh net escape SSLAY right because if you had
purchased a net escape you the server the browser for some reason you know like I did what I was in
the 90s then you'd use SSLA from net escape and it just is a natural thing. Fortunately they
talked to each other the great thing I mentioned I didn't mention before but net escape and made the
standard available they'd publish their standard so that other people could communicate to their SSLAY
and then the founders of SSLAY left to join RSA so now you've got a great open source project
and the maintainers have gone off and done something else and and so what do you do I mean this is
there's a problem right well since I wasn't really much of a problem because there's a community
on the SSLAY users list and so that community said hey we can just fork the SSLAY code
and then we can scrape off SSLAY from all the files that have it and put open open SSLAY back on it
and they released it this is actually take two the email for take two that happened in January of
2009 or 1999 and you can see they nicely credited the founders of SSLAY and and they actually
made the license a little bit nicer it's in the patchy cell license so in other words open SSLAY was
able to just sort of take it over where SSLAY left off it's everywhere it does all the things it's
not just the web anymore you can use it for email you can use it for just anything you can think of
on where you're transmitting data for one place to another I talk to a guy who is working on
serial port communication which was great I was so excited about it here like this when I was an
intern at the National Weather Service and and he has an open SSLAY connection so that you can
actually connect to a serial port over the internet which I was like wow I wish I had that back in the 90s
so that's the end of my talk right because open source one open SSLAY has conquered the world
yay team but there's a problem this is what I consider the worst day in the community
and this is a front page of the New York Times not the top story but still in the front page
is this the story about this problem with this these websites right what is this it actually
says it later in the article but not in this section the headline it's heart lead how many of you have
heard of heart lead yay heart lead is amazing because it means that all sorts of people know
about open SSLAY all of a sudden right so the downside is I worked at I said I mean I was working
at Stack overflow at the time and our our system administrators were like oh no what are we going
to do because we had a lot of servers and almost all those servers had open SSLAY and an
all of need to be patched and then you need to make sure that your patch actually was
was fixing the problem that you had actually done all the steps and you know there's new reports
about how people should change their passwords and how they'd lost you know potentially lost
all their data that actually probably didn't happen there probably was a huge amount of data
we can't know for sure because there's no no way to know because of the nature of the bug but
but it cost a lot of time and it gave open SSLAY sort of a bad name in you know among the general
public even we're not talking about tech people we're talking about people who read the New York times
so yeah I mean it could be both right but I see what you mean so here's my diagnosis of why
that happened the Open SSLAY community was not keeping up with the popularity of the project itself
Open SSLAY was used everywhere by all sorts of people and the community this is the number of people
contributed of some sort of commitment each month and I've got all the way back to 1999 and the
pink line is the 12 month average and you can see it's around five people you know it goes up and down
five people for this project but that's not even telling the full truth because the number of
people who are actively maintaining it was closer to two or maybe like one and a half so it was
not a lot of people who are actively maintaining it and the community was sort of stagnant
like you're not seeing anywhere near the growth compared to the use of trade okay so let's ask
the next question why is that? I think this picture a couple weeks ago it says Mojavecaca
community I thought it was amusing because this is not looking like a very healthy community
it's all prickly and dry and unless you know what you're looking for you can't even see the
Mojavecaca they're actually these two sort of stump things they're supposed to have these big green
leaves they look amazing but these do not look amazing they look prickly and I thought this is a
good representation of what the community was like before Hartley. Cryptography is intimidating
people you know think about what happens if I start working on this project and something breaks
what happens if I start working on this project and I create something that's not secure
oh my goodness that's a lot of responsibility and so not everyone feels like they can do it
not everyone feels like they can contribute to openness to self and that's true even today
and not everyone can contribute code either and this is maybe even a little bit harder for
a project like open SSL which is mostly written in C. C is a fantastic language but again it's
intimidating not a lot of people you know a lot of people know it but there's even more people are
excited about other languages you know whilst my train of thought but but it's C is not exactly the
easiest language to work with for a lot of people and then there's Pearl I love Pearl Pearl is amazing
a lot of people don't like Pearl right Pearl is nice of them C right well not for some people
and then it's all there's a sort of a caveat which is that the Pearl that's used an open SSL
is used to generate assembly code so if you're excited about Pearl I'm sorry you're not actually
writing Pearl you're writing a Pearl generator for assembly code and that's breaking my brain just talking
about it imagine trying to work on this project and so these are intimidating things how many
people could do this right and then the other thing that's going on is the people who are
maintaining the project are really into it right they're really excited about it are they thinking
about how do I bring new people into this project no they're thinking about how do I avoid having
people try to commit some tell me this is how you should do it right that's that's a bother it's taking
away from the code and then the other thing is the community happened on mailing lists again I love
mailing lists there's email to you know that that was the way things worked in the 90s but mailing
lists have a problem which is that you kind of have to to get involved in it you kind of have to
read the mailing list a while otherwise people are yelling at you and then you're submitting patches
over email you know all that sort of thing it's it's not that it's impossible it's not that you
can't do it it's just another barrier that makes it slightly harder for people to get into the project
so this is actually the picture I took I cropped it down to just the fun part but this is the bigger
picture which is you can see there's more variety there's more stuff going on you can actually see
Mojavec with the green things sticking out I love this picture some of you may not love the desert as
much as I do but I come from Southern California and the desert is my home and this is is a great
picture and what I like about it from a metaphorical point of view is that it's not just the Mojavec
community is not just Mojavec and the open SSL community is not just the maintainers and developers of
code of open SSL there's actually a lot more going on here and then the other thing is Mojavec
I won the ways that they are able to thrive and reproduce is they depend on fire so when fire comes
through it makes it easier for them to spread their the seed around and grow and expand
sometimes there's an necessity for creative destruction in life so this is a change
anyone know where a heart bleed bug happened on this graph it actually happened a little bit
after when people started becoming more active I haven't really figured out exactly why but you can
see that before heart bleed you would go literal years without anyone new joining the community
and since then it's been going up and up and up and that's great right this is how a healthy
community survives people like order so this is actually a different picture in near Nevada and
there's a Joshua tree and there's an art installation it may not be possible to see all the way in the back
but there's this cool art installation of someone has put up stones that make these towers
and a bunch of people have come out to the desert to see this I recommend it
that's excellent price too this is what the community looked like in 2015 so still mailing lists
but now there's a wiki and it's on GitHub and I honestly think GitHub makes a big difference
for a lot of people because instead of having to go to a mailing list learn how to make your patch
follow the rules of this individual community get up stream lines a lot of that right
we all know how to create an issue and GitHub we all know how to create a PR on GitHub
there's still some nuance like from project to project but the interface is has actually helped
it creates a sense of order in the community that means that people are more likely to join
another thing we did was we started adding tests so I don't know if you can see it really well
but that top line is the code increases over time is actually a logarithmic graph so that's not
a weird shape and then there's tests which is the green or test which is the pink line
and after heart bleed there was a real focus on creating a testing framework and creating some tests
and that actually helps contributors a lot because you don't have to know the entire code base
to make a change if I make a change in one part and I test my part and it seems to be working
it's doing what I think it should and then I run the test suite and it didn't break something else
somewhere else I have a higher sense of confidence that my change was actually a good change
and not only that but the maintainers can look at that test case and the test suite and say
okay it didn't break any of our tests and they can feel a little bit more confident that they
that the code is worth bringing in heart bleed was good for the community communities that don't
grow are in the process of dying and so this is my favorite graph because it's showing that the
openness to the community is growing and there's just a million different ways that that's happening
right here's some things that we've done the past year that has happened in the past year
I started a survey to find out where openness to sell is being used and we had a conference
in October that was amazing more recently we had an AI the community that does AI find a bunch of
vulnerabilities in our code and one of the things I like is that they say that it's hard to find
it's extraordinarily difficult to find vulnerabilities in open as sell and why is that because the
community is gone through and fixed a lot of things that doesn't mean heart bleed isn't possible
in the future it means it's less likely this is a picture I took just today I don't know if you can
see it home brew has their thing and the example that they gave is brew and sell open as a sell
I don't know why I think that's so cool but I think it's awesome I want to say thank you to our
supporters we are the open as sell foundation we depend on the supporters the supporters
allow me to come to this conference which is not cheap because I come from California
yeah the conference itself is cheap but the becoming isn't and I'm not ready for questions but
why I'm answering questions please scan my QR code so that you can get access to the newsletter
that I'm definitely going to be writing tomorrow so I can get it out this week this is a brand new
thing that the open as sell foundation is doing and you know I'm writing it so you don't have to
worry about it being spam because I'm lazy and I'm going to be pushing every month or every
couple of months to push one out all right ready for questions
Thank you very much.
Hi one thing I noticed about the heart blade bug was that the code was committed at like 1130
pm on New Year's Eve yes do you think that was affected well I was there so I can't say for sure
but I don't think that was a I don't think that was a positive thing and you know it's just when
there's a small number of developers and you're getting some code coming in and this was this
code came in was actually part of the heartbeat feature which is a good thing right you want to
bring that in it wasn't like someone was dumping some dumb thing in but it makes it harder when
there's only a small number of people and they have to frantically look through the code
or let's let someone down right so I think having a very small number of maintainers is a
potential problem for projects and and so the maintainers themselves should be incentivized
to grow the community so that's a bigger group of people looking at each of these things yeah
that's really interesting and good to hear the story told from something really knows it
well tying into some things we heard on the main track this morning how do you protect
the mental health of your contributors because you go to the Wikipedia page the guy who put in
the commit that broke things is named I mean that's really rather scary I'm going to commit
something there on the world sort of villain how do you look after your community to protect them
from that frankly unsafe that's probably putting it by being treated yeah you didn't pay for it
that's a great question so one thing for me personally is I don't know the name of the person
I mean I looked at it once and I was like I'm going to forget this because I do agree that there's
a sense of shame right I think the other thing is there are more people in the loop so there's
two reviewers for the code and so it's actually there's more people who have a sense of
responsibility you kind of diffuse their responsibility a little bit yeah I don't know I mean there's
I think there's sort of safety and numbers so right the more people that you haven't
involved it's helpful and then I personally just would never want to make a big deal about
this poor person because yeah it's a potential very shameful thing so time for one more
in the back
thank you for your involvement and as someone who worked at a university you got to deal with this
a large scale it was very interesting I'm wondering to what extent you would measure the health
of a community based on the percentage of tests that are in your product percentage of tests
yeah I mean so I add this this is still not enough tests I'll tell you that so we're at about 70
percent tests there's sort of a point where you diminishing returns where like you're testing things
in lines of code that are just not getting run very often it's more work to create the test
and very low impact on it so there is definitely a place where you you just reach
diminishing returns when I work close to that right and and I think the other thing you know
I didn't mention too much on this graph but the documentation is actually perhaps a bigger problem
right now because you know the test helped you verify that the code is working but from a community
perspective we want people to contribute code and if the documentation has not reached the level
where people are going to contribute code fairly quickly then then we have a problem and I think
I've heard this a lot the OpenSL documentation is hard to use and I agree I'm fairly new to the
project and reading it it's just hard to get into it does what the developers need it to do which is
great so once you're inside the community it's fantastic but it's hard to get in through that
so I mean I didn't really answer your question with it with a number it's above 70 less than a hundred
right and and I think I don't know my 90 would be great 80 something would be would be pretty good too
yeah thank you were at time okay great thanks John
So