Now, we have gates to it, the vice president of the Penable Embedded Systems at the Linux Foundation.
She has several years of experience in the software industry and has been instrumental in
the success and launch of several projects that you may have heard of like Zephyr, SPDX,
Kales and more. Today, she will be talking to us about how Zephyr applied best practices
from the Linux kernel community to achieve more than 100,000 comments with focus on safety and
security certifications. Peace, welcome, Kate.
Thank you. Thank you very much. Just for, let me meet the room since we're in person,
which I adore. How many people have heard of Zephyr before this talk?
Okay, so we've got some new people who haven't heard of Zephyr, so I've got a little bit of background,
but I won't spend a lot of time on it and you're more than welcome to come and ask me more questions
afterwards. Who am I? For those who don't know me, my focus of my career has pretty much been embedded
open source and the current challenges I'm really focusing on is how can we make sure that open source
when it's being used and safety critical applications can be analyzed and made sure that we can
get a safe system working at the end of the day. So you'll see me in Zephyr and you'll see me
in the Elisa project and we're starting working on a space-grade Linux project there and
remontere to with me and so he's a one wearing a penguin with a spaceship top. So we've got
a fair amount of things that we're sort of caring about in this space because open source is very high quality.
So amount of review and everything else is there, but it's not done in a way that the
traditional safety analysis process is going to deal with it. So part of the challenge is going to be
how do we actually make this stuff accessible and how do we actually work it from this perspective?
So these are sort of the projects you'll probably see my name associated with and I've been volunteering
on the SPDX front since it started in 2009. We've been working on S-bombs and you will see evidence of S-bombs
in CS-bombs for those who are in the last talk. We've been generating seed-based S-bombs in
Zephyr for the last four years as we've been trying to do best practices in the industry all the way along.
And in terms of hobbies for me, I like photography, I like traveling and I like taking
separate types of interesting places now. Unfortunately last March they would not let me fly that
kite down there because the bird flu had started and the first signs of the bird flu had made
it down to Antarctica. So I could carry it on and hold it up, but I couldn't fly it. I was
dreaming of being able to fly the kite down in Antarctica. However, word of mouth, so it's not confirmed.
I don't have details. They are using Gen2 penguins with separate-based trackers for monitoring
global warming and seeing the changing patterns of, you know more about that detail? No, okay,
because I've gotten a little bit of hints of this from a couple people, but I don't know the details
and I'd love to tell that story. Anyhow, contact me anytime. There's my, you know, there's my linking,
but for those who don't know Zephyr, it's a real time operating system. And we started the
project off of safety and security in mind. The projects launched formally in 2016 about this
time in 2016, so we're about nine years old now. And we've asked a bit by bit, we've grown the
community and when we looked at it and looked at security at the start, we needed to be different
than what was out there already. Back in 2015, for those who might remember that period, the joke was
security as the SNIOT, applying it wasn't there. And so we really wanted to make sure we could
do security best practices with this project, pretty much all the way through. And so I'll go through
some of that, but it's very modular. Anyone who's familiar with embedded Linux will find it
pretty familiar. It's not Linux code base, but it's structured like Linux. So it's got K config,
it's got device tree, and this lets you be very flexible about just putting in what you actually need.
So the lose kernel doesn't get smaller than about three meg these days, when you really
put it down. The first starts at 10k and goes up, depending on what you want to do with it.
So it was structured really differently for the embedded space, and we have people who have a
long amount of embedded experience who are very focused on keeping it small and not letting things
creep into it. We've also learned a lot from Linux, and I'm going to some more of these lessons,
but things like having an LTS, so that people who are making products don't have to keep up with
if they do not need to. Or they may keep up with tip for a while, and then things aren't changing,
so they just worry about security fixes. The LTS is good for that. And we are actually going after
a formal 61, 508 certification, this effort, for a subset of the kernel, and I'll talk a little bit
about that. And as I'm talking, if, like I say, if something's really burning you and you want to
ask a question, go for it, and those who ask a question get a kite. Okay, and otherwise we'll
just have to put you in the end, whatever's the easiest for people. So where is effort today? What's
going on? Well, we just finished doing some analysis, and thanks to Benjamin and Susan and me,
we've actually had 1100 contributors last year into this project. 50% were first-time contributors,
so the community is growing fairly strongly. And there's a blog post out on our site that
goes into more details. The 150 boards were added to this repo. There's over depending on how you
count it, and where you're counting it. There's probably around 700 and more boards sitting there.
So if you've got a board sitting at home, there's a pretty good chance as a zephyr port available
for it. So you can go and play with and just serve experiment. And we have a lot of samples,
and I can show you some examples of that later. And just last year, Susan, who's in the audience,
started a meet-up program for us, so that developers and regions can get together.
Like, fossil-happens once a year embedded in it, world happens in November, you know, once a year,
and then there's effort to follow for some it. This wasn't all we needed, and people getting
together and talking about things and sharing ideas and having a community is what helps a community,
that helps a community grow to be able to have the interaction in person. And the meet-ups have
definitely been part of helping to accelerate zephyr in the last year, my mind. So to put things
in perspective, where zephyr is these days, the lens foundation has hundreds of projects.
And the CNCF folk, because they're, um, see, because they're going after Linux,
so these are two logarithmic scales, okay? Just to put in the context, this is logarithmic scales,
it's commits on the bottom, and then it's interactions on the side, okay? And then the circles
of the contributor base. Um, Linux is by far still the most active in the successful project.
However, Kubernetes has been coming up with a closer, like, inching up closer.
This is why that community looks at this type of metric. And as a sign effect,
zephyr is now showed up, zephyr starts showing up, lower and lower, but now we're
at the fifth most active project of the Linux foundation, okay? So we're seeing, like, there's
that 1100 developers serve showing up. And so the project didn't measure this another project
measured it. So, lies down lies in benchmarks, but at least it was a neutral source.
And then something that's maybe really excited this last year is, okay, it's one thing to be
a parvalist foundation, but there's a much wider open source ecosystem out there. And so they
also look and see where Kubernetes is in that ecosystem. And they just look at it for the top 30.
And they serve graph that out. Well, does this year zephyr's in the top 30 list? In fact,
we're number 25 in terms of velocity. And the fact that we're sitting up there with things like
mix OS and so forth is like, I was just blown away personally. It's something that has been
served like, okay, when we served, sorry, evidence that we were down here at the bottom, but when
they just finished measuring it last month, we're sitting at 25. And so how does it all play
into an open source ecosystem? The effort here is this first line. Okay, community controls
its commits, the license is Apache. And you'll see now there's almost, well, we'll be heading
to 2,500 contributors, I think, within the next month or two. But compared to the other projects
and the total commits and the commits each month, you know, we're three or four times.
So actually, this is how we do the calculations for that chart you saw. I used to get up,
and I just go and actually mine the data for all those numbers. So you can go and do this
year on analysis and prove me wrong or find out what the did stats are today because these were
from yesterday. But the methodology is fairly straight, but I was starting looking at this back in
2017, 2018, because at that point in time, it's ever wasn't anywhere near this level.
And I wanted to see, okay, when did we cross the commits, when did we cross the contributors?
And, you know, being able to measure things has been one of those features as you're trying
to get growth in your community. So you know what's working, what's not working, but don't expect
it to change like in a month's time or something like that. You have to be able to be willing
to look over years. Look at trend lines over years, and that's what you can tell after you're
actually making a difference or not with some of the things you're trying. So obviously,
this current is right, it's up there. I was never comparing. Well, last year, there was a
six-eight kernel came out, corrected the calculations. They had like, you know, for the one
released, they had, you know, 1900 contributors, but they were doing about nine changes an hour.
And then you saw in the last slide, there was a first two point, two changes an hour. So it's
about a quarter of the rate of the kernel, and so forth, you know, a nine-year-old project,
and pretty happy with that. But I was around when when it started, okay? And when it started
in 1991, it was a very fragmented ecosystem. So we had, you know, Linux, BSD, all these commercial
versions of Linux, and so forth, out there, which kind of resembled this landscape, when's
effort started? Why do we need another R toss? And so Linux was successful, and so when we were
starting to think about creating this project and working with this project, you know, we
sort of looked at what worked for Linux, and what could we apply? And so, Linux today, it's pretty much
dominating most of the markets, it's not including a bet it, but there are places that are embedded,
where Linux will never fit. So the zephyr is designed to go into those spaces, okay? But in 2016, 2017,
the Linux kernel report, which is available for anyone to go back and read, focused on what was
working for that community. So we looked at that, we did our homework, we looked at what was working,
and we figured, okay, when we set the governance for this project, and we, you know, what can we
apply? Things like short release cycles have worked before, you know, how many people were involved
with Linux before the two six stuff, and it was varying a lot with this really cycles,
but once I went to a regular cadence, that was a start of it accelerating. It pretty much is plateau,
but it's plateaued in pretty high level right now, so we'll have to see what's going to change
it and shift it eventually. But, you know, you need a hierarchical model for your developers,
because one person can't be the funnel, you have to delegate trust, and the hierarchical model is
something that was working for Linux, and so we, as we were thinking about it, we wanted to make
sure we could do something like that, too. Another thing is tools matter, you want to keep your
developer experience as easy as possible. After all, Venus came up with Git, make sure we actually
had a flow, right? And so, when things are frustrating, I'm working it that way, so remembering
that tools matter, and then remembering also that a consensus oriented approach tends to get you
farthest and more stability over time. And, you know, the kernel itself had a very strong,
no regression rule. If something was causing a bug, causing regression, it just got pulled out,
until things got fixed. And this led things move forward faster over time. And then, another
factor that, you know, the new kernel has is the corporate participation in the funding,
it's part of what made the kernel possible, okay, in a stable fashion. Now, when we do the analysis,
and it's primitive, but basically if you look at some of the, the stats from LWN and so forth,
the estimate is about 15 to 20 percent of people working on the kernel of these days are doing it as a hobby.
The rest are getting paid by companies using it in products. And as we set up a virtuous cycle,
this means that people are getting jobs and people are working on things. So, we want something
where people can actually have careers, some degree of independence, and there shouldn't be boundaries,
like you should be able to commit to a laptop. Maybe able to commit to the memory subsystem one day,
and then go commit to another driver in a different day. You should not have to have strict boundaries
of these other places you have to go into. It's open for any contributions anywhere in the kernel.
So, these were the key lessons we took out of that report, and some other ones that we saw along the way
is, you know, keeping things vendor neutral as part of the reasons the links kernel succeeded,
okay. You want to have a mix of companies scratching their own inches. You want to streamline
things and make the easy for developers with the DCO, and, you know, do public reviews.
People are going to review things and sign off on it. That's an indicator of quality,
because their name is associated with it. And so, public reviews and sign-offs help build trust.
And so, this concentrated model in the kernel is doing an email and in person summits.
And then the heart of the adult model signed off by is what's indicating that to some extent.
The intro boundaries we've talked about, tools matter, and so forth. So, these are sort of the lessons
learned. I was focusing on when we were setting up governance. And always, I've been keeping it
throughout my career. When developers get frustrated and irritated, they get really good about creating solutions.
So, keeping it so that the creativity works for your project, not against it. It's important.
So, how did we apply these lessons? Okay. Our vision start, and still is, we're striving to develop
the best and class art class for connected resource constrained devices and build to be secure and safe.
That's what we're aiming for. And we've been pretty true to that vision, and you'll see most of these things fall in line with that and
executing to that. So, one of the things you'll notice in this effort project is the developers'
design directions. RTSC has an elected chair, and the chairs elected every two years. So, it's
affirmation by peers, and the developers look at what they're working with, and if they don't like
something, they have a way of changing it, and we've changed it a few times. You know, when we started
it, we were using cake and cake build, we changed to cake and frequency make. We had basically an
NN on micro-corneal, and we went to a unified current. So, a pretty significant change in the early days.
Initially, we were started on Garrett and Jira, and the developers wanted to go over to get
have an issues for communication and transparency. And they basically determined we were going to have a
code of conduct. We put that in place, and then all the APIs, the house, we've architecture groups.
So, we've got places where people can analyze and talk to each other as part of the project,
but also we make sure we have events and things like that too. So, these are the directions
that we're serving decided. So, now I want to look at these lessons I was talking about.
Vendor-neutral decision making. Yep, we've got multiple companies, and I can show you a list
of the end here of our members, and we've got a lot more than our members right now,
who are actually participating in this tree. The next one was the companies and the individual
team participate. As you can see from this, across the years from 21 to now, we're still
seeing steady contributions from some. We're seeing others increase their contribution like an
XP's been doing. And then we see this other, which is, you know, it could be basically obvious.
From the developer surveys we run, we're finding that about 10 to 15 percent of people who fill
in the developer survey are obvious as well. So, it's not just a corporate environment. There are
people who are actually interested in it for their own hobbies. And, like, the code base.
Peacown, for instance, is one of the ones that is being done by someone who is not being paid
to work on it. If you really want to have a tiny bit library, that's aligned with this
things. If you want to have it common across a couple of our classes. So, if you work with Zefer
on this, we've got people doing home automation systems, home gardening systems,
and using Zefer, and we've had that pretty much since the start. As you can sort of see,
we've got a fair number of companies, some of which are members, some of which are not,
doesn't matter, in this way, but we've got the input. And so, as we're sort of changing things
around and evolving, we do have that, and we have our TSC has people from companies as well as
community participants in there, making sure that the community view is calved. For instance,
as part of the discussion. Contribution guidelines, we've been pretty public about it.
We have really haven't changed that much. As you can sort of see here, if you pay attention to
the details, this file hasn't changed for six years, and it's still pretty accurate.
So, it's been working for us from that side. So, we've streamlined that upstream process.
It seems to be working. One of the things I'm going to highlight, which is a personal pet,
well, it's a pet. So, box of mine, I guess, is the best way to put it. Developers certificate
of origin is a very lightweight way to let people contribute to your community in a structured
fashion. So, you're not signing a CLA and signing over copyright. So, that someone can't
license it from underneath you. So, you're going to build trust by knowing that the licensing
is not good change. It's been very effective for the Linux kernel. And so, it was one of the things
that I really wanted to see we did. You know, a lot of lawyers wanted to see CLAs, and I think
DSEO is a more equitable thing for developers on a personal level. And we've used it in Zephyr.
Public code reviews. Yep, we're doing that. Everything's up on GitHub. You can watch the history,
as things change. You can put in issues, you can track things. They're to follow the consensus
oriented decisions. We try to pretty much work by consensus. If things get escalated,
we have a TSE that can vote. There's no one to cater here. It does go to votes when things
are problematic. Or there's two different strong sides and things like that. But those sorts
of votes only happen maybe a couple of times a year. So, for the most part, it's pretty good
in consensus side. And it's been working well for us. We have a maintainer's file that lists
all of our maintainers and the hierarchical subsystems. Again, following the lesson from the
Linux kernel's maintainer's file seems to work. It's working for us here. And anyone can make a
poll request in any area. No problem there. And the distributed version of troll. The
contributing explains that. And in terms of the releases, we are doing periodic releases. We've
been doing it pretty steadily since the 2.0 release. We were figuring out what our cadence was
in the 1.0 series. And the 2.0, we've been pretty much four months. And every two years, we put out
an LTS. That's the cadence that seems to be working for this community. The Linux kernel has
like a 2-week merge window. We actually use a three month window and then a month of stabilization.
And that's working with this one. We'll revisit it if we need to down the road as things
mature more and more. But this this cadence has been working for us. And then when we come off
that with that LTS, it's every two years. And it's maintained independently for 2.5 years, so we
have an overlap. This is one of actually one of the lessons I learned from Ubuntu, which kept
that overlap on the LTS' because it made it easy for people to have a transition window. And so
when this was a doc, like I said, when we decided to try to start figuring out what the LTS policy
was going to be, this was a practice. Which is if you're going to stabilize on something for a
period of time, make sure that it's a good transition window for people who have to upgrade.
And as we move more and more towards like the CRA and longer periods of time,
I think there'll be more companies coming in with business opportunities to maintain things
for these longer periods of time. I don't think you'll see the community maintaining things longer.
Although there is discussion with Defer of extending this another year, possibly two.
But that's the discussion and we have to figure out a way to be efficient and smart about it.
And so this is working progress. So are we going to try to make more two to four weeks
stabilize and our long-term support release? Actually, you'll notice that our LTS-2 is pretty much
EOL. So I'm expecting this page to get updated very shortly and it'll all be updated by the time
we put our 4.1 release out. But I don't know if there's going to be another point release or not
with some of the security issues. I've got to talk to someone about that one. But as you can see,
we came out with the 3 LTS-3 in July and that's being maintained and 4 came out in November
and it's going to be maintained until the end of the next like until the 4.2 comes in.
So we'll be putting out the 4.1 early March, brings the cadence and working from there.
And then we'll be keeping things marked as end of life. So as you can sort of see, we have applied
the lessons from Linux to the large extent in Zephyr and it's been paying off for us. So okay,
let's happen with security. You said you wanted to be in security and we do best practices there, right?
Well, when we set up the project, we pretty much came up with this diagram and we wanted to not have
blockage on contributions upstream to tip and we wanted to release this going there. But the long-term
stable would happen periodically. And so when development is happening on tip and things can
go into stable and then any processes can be applied from that. Actually, realistically, they're
being applied into development now. But there's a subset that's going to use for audits. And that's
kind of the model we've had from the start when we're keeping to it so far. So the 4.0 releases are
latest released. We've put a lot more stuff into it after we've, you know, with the last LTS out
and the start of things they'll be showing up for the next LTS. And so the release notes are there,
everything is public and we're moving forward. Our last LTS had a new hardware model and actually had
the precision time protocol stuff supported, which is going to be good. And we actually extended the
S-bomb support to actually support the pearls and CPEs as well as SPX 2.3. So that's been there.
The S-bomb support has actually been in Zephyr since 21 and I've got a slightly
little slides in a minute about that. But as I say, we have a support window. We're probably
about it. We're transparent. How long we're going to do it as a community and we've got
buy-in from the community to do this. And the key for these are, it's product focus, there's
updates, security updates. So we actually have a P-Cert team with this project. Volunteers that is
actually monitoring the vulnerabilities and enhance the things. And we've been, you know,
we have a testing window and we're working with this. And then we put out the point releases
during the course of the support window with the security fixes. And if you actually go and
look at our charter, it was right in there from the very start that we would have a security committee
that would be focused on these types of issues. And setting up your governance so that these places
have a clear role was one of the lessons learned. Okay? The other thing was, okay, how do you do
security well? Well, back in 2016, the CI, our core infrastructure initiative project was starting
and they came up with the best practices badge. So our security team looked at it after
documenting the threat models and some of the other basics. And so I said, okay, let's look
at this and see what's like. And, you know, can we get it? And so what these criteria were for
was to effectively improve open-source projects for best practices and letting people self-certify
to meet criteria. And we actually found it a really good guide to improving our processes.
So it moved into the OpenSSF so it's there still today. But it's now OpenSSF best practices badge,
curls there, zephyrs, ends there, the curls there. And so it's possible for projects to go
through this. And you'll probably see a reasonable correlation between these projects that have
gone through it and having best practices as well as a big very popular because they can be trust,
there's an element of trust because you've got public access stations about your policies.
And so there's currently now there's a passing silver and gold when the project started,
there was just the passing. And at the higher levels, there's slightly more strict best practices
that are getting invoked. And then the criteria is, you know, there's only 23 gold at this point
still. Or there may be, I have to double check that number beforehand. But, you know, looking at
the criteria, there's 23 criteria on top of the silver. So there's 55, and so the passing
you have to go through a 67 criteria. And I test one way or the other on those criteria. And then
another 55, I'm top of that, and then another 23. And if you can do all those and have it all
probably get tested, you can be quote-and-quote gold. Some of them may come straight, pretty, you know,
some of them take different works. So we started doing this bit. Okay, yes, it's like a good
idea, let's go forth. And when it launched, and so pretty much within six months, we achieved
the passing for Zefer. So it's like, okay, good. We're passing. Yay, we're doing the right thing.
We had a good security documentation. At that point in time, and the security documentation is
something, you know, we keep up to date, keep working with the project, and we want to focus
around security development, security design, and security certifications. And so this is something
that we consider an ongoing process rather than a point in time. So our security committee
keeps looking at this type of thing over time. And price and price, we stopped passing,
in 2017. Remember me showing you that developers controlled decisions? The infrastructure
changed. The tool went in and was looking for links to keep working for the evidence and stop
working. So since we finished all our migration, we got ourselves passing again. By updating
a lot of links and things like that. And so six months later, we were interesting. And we went,
okay, and failed us for a while, we can come back. Let's try for silver. Let's see what's going to
happen there. So we did that, and the roles in response, please, had to be documented. Again,
these are all good practices for anyone to be secure software. And we had to have a longer road map,
and we had to document in certain places. When we started looking around by, we become a DNA.
So I don't know, how meant to show hands, how many people are familiar with CVD numbering
authorities or CNAs? That many. Okay. These are the ones that assign, these are the people.
So CVEs are the ones that assign the vulnerabilities. And they basically have control over the
vulnerabilities. And there's not that many open source projects that you're doing it. But it really
lets you have control as opposed to an integrator trying to guess. So if you've got a big enough
team that someone that cares about this, it's something to look at. We basically had to change
our processes a bit. We changed our processes in a useful way. You know, that we had to review
basically, we had to change some review processes a little bit. We had to create some new email lists.
So in that sense, it wasn't too hard. And then we had to interact with their system,
which is automated since then. So we defined our scope, created our value of contact.
We have a spot for contact from minor. And we have ways of saying who should be added on to lists.
So 2017, we got announced as CVEs number 30. So hey, we've got our goal passing bad. We've got
our, sorry, we've got our passing badge. You know, we're CNA. Cool. We were listed in the MBD,
which is a national vulnerability database for those who are in the last room before. And so you can find
us there still. And this is what it looks like if you actually are curious what an entry looks like.
And we have a product security, a project security, not product, incident response team.
And our security architect leads it. And we have volunteers from our committee who, you know,
basically, help with the triaging and then interacting with the maintainers to make sure we get the fixes in.
And so, so it's satisfying the program rules and requirements. So we kept going.
We were almost silver 2018 and we finally got there at towards the fall of 2018.
And part of the challenge for us was this program assumed you were writing an application.
An operating system was not what it was thinking about. So we had to write a little application
so we could catch one in the criteria, which has been a sore point for me as you can tell.
But we kept on and then, okay, Zepher has been very good since the start of putting the licensing
in each file, which is one of the things that people stumble about at the gold level.
And so we already had that. So it's a pretty easy thing for us to get to gold at this point.
And so we actually got gold in 2019. We were the fifth project to get it.
We got it before the Linux kernel did, just saying.
And then, you know, we're pretty smart. And then someone paid NCC group to audit Zepher.
And we got our first bulk vulnerability report at 26 issues. Okay, secure your team crank in.
So we actually addressed this all. You can see this in the LTS histories and so forth.
And they're all addressed. And most results is one or more TVs from what they got.
So it was a good for the project. If you ever have the chance to get your project
audited by a third party, do it. Because you'll find things that don't find things that you don't
overlook over time. And so, you know, they came up with a report after we finished with
them. Most issues were fixed in a reasonable time. It was a nice thing to see for an open source project.
One piece we decided to disable feature. That's the most sensible approach.
So we did increase our embargo window to 90 days. So that we had 30 days as a project to fix things upstream.
And then we wanted to give people doing prod ducks with our project, 60 days to fix things.
Because they've got customers to interact with sometimes. So we were trying to keep
products and customers in mind right from the start. And, you know, we had to improve some processes.
And, you know, we did this and we've documented our website, etc. And so learning from working
these things and learning from the experiences we've kept on improving our best practices.
And we wanted to make sure that we were first classes and for product makers.
And so this vulnerability registry was also created where if someone can show us evidence that
they've got a zephyr in a product, we will add them for free. They do not need to be a member of zephyr.
So they can basically say, hey, let me know, here's this product I'm shipping. Let me know
if there's any vulnerabilities. So we have a way of pushing information to them so that they have
them alert and can work in their embargo windows. And so that has been very useful.
The issue is, you know, people letting you know that they're using your product is interesting
sometimes. They don't want to. And some part of my fun, part of the fun, you know, as I have,
is put it embedded. It's trying to find where zephyr is being used. I get lots of interesting
things. I'll show you some of the products later, but we also had challenges before the concept of
a vex came in where we didn't actually use the files in fnet that had the vulnerability.
If you had just looked at the component level, you would have thought zephyr was vulnerable to
unusual 33. Because we had the same fnet version or some of the files from the fnet version,
but it really wasn't in there. None of the files were in there. And there was no way of signaling
you were not impacted. Other than writing a blog post. So when the whole concept of vex came out,
I was definitely supported because I lived through the reality of not having a way of signaling
not impacted. And if you're just looking at the components, too many false positives,
too many problems. And you don't want to fix things and upgrade things and cause perturbations
when you don't need to. So this is also an element for the s-bomg generation. And if you work with zephyr,
you'll find it can generate three or four s-bomgs depending on what you've got in your system.
And the way you do it, you can do one for the zephyr sources, a source s-bomg for the sources. Here,
a source s-bomg for your SDK, a source s-bomg for your app. And then you can have your build s-bomg
linking and referencing that to your source s-bomg as to how everything is connected. So you have a very
precise source of truth, which is what we're going to need for safety down the road just
foreshadowing here. As well as quite frankly, it gets through the whole class of false
positives for security folk. And if you want to generate these s-bomgs with zephyr today,
this is it. That's all you need to do. You just initiate some variables when you're doing your
builds with west. And these will come out automatically. And if you actually, there's one of
our members and my crew has a re-node simulator and they use zephyr as a test suite for their
simulator. And so they build 875 boards with it. And everything in there that has built past or
generated has three s-bomgs. And if you're mouse over, you'll see download s-bomg and you can download
those s-bomgs across those apps. So yes, we can do it at scale with c-programs for those who care about that.
The other place that is doing it at scale for c-programs is a s-bomgs project. They've been
doing it as well. And Joshua here is a life of implemented at all. So you may want to find
him after the call. Thanks. But this is what it looks like. Okay, we put in links, symbolic links
to the other files. And then we refer to exactly which stuff sees me to.is and which
studies made it into the alpha images. And so we can be that precise. And that just comes out of the
build information. And so our vulnerability infrastructure went to GitHub in 2021. This was one of
the decisions that our security committee made. We wanted to do that because things were more easy
to get stats and metrics. And so this gives a picture of our CV publishing history and C-WE breakdowns.
For those who understand the weakness in the iterations associated with the vulnerabilities,
this is there. And then we can look at scoring and so forth. So we have a ways now starting to
look at metrics in this area. And we decided that we wanted to branch out wider than just a committee.
And so the security team basically, the chair and the architect decided to start working group as
well. And they wanted to start doing public work on some of the security standards. So Etsy and
303645 was one of the things they were working on. And so that's open to anyone to join
whether you remember or not. And they did work at this on 223. And you can look at the public documentation
of each provision and the test station, which is sort of like what you'd see in the best practices.
It's basically public assertions about the provisions. And last year, the project decided to
actually say ask NCC group to do another audit on us, to see if things have changed, things have
regressed, et cetera. And we wanted to find out, are we better, we've got challenges, you know, we
missed things. And so they actually went and looked at our 3637, which was our LTS. And two
most of the ready issues were found and went informational. So they weren't finding things as
that initial traumatic report was for us. So in that sense, it was a good result. In the sense
that we really had not regressed and we kept the quality level high. And designing the scope of these
things is hard. But it does increase confidence if you've gone through through party audits. So
if your project has the ability to do it, I consider it. If you're lacking funding to do it,
OpenSSF's AlphaMega project is willing to probably support, if you can make a good case for it
and put an application in. So there's funding potentially there too. Because they're trying to increase
security for open source overall. And if you're interested, Flavio and David, they give a really
good talk at the Zefford and all of our summit. And I encourage you to listen to more details
from the experts, not me. But as you can see, we're doing all of these things. And the S-plum
generation is very much lining up for what we're going to need for CRA down the road. So what about
safety? Well, we've got a subset. We're going to be looking at for Zefford from the auditable.
And for Zefford, we want to try to actually get something that's going to work with a traditional
vehicle model. Because we're in our class because we're using small things that are usually
done bare metal by security experts. We want it to be able to interface with that community.
So we start our prototyping things about four years ago. And we've designed the security
committee's side of the initial certification focus is just going to be the kernel. But we're
doing the analysis out in the open. So people want to extend it. They can. And we've basically
worked through our background with the collateral we need to put together. And we have a
functional safety manager in the project now. And we've made a strategic decision to actually
document the requirements in public as well as to do the linkage in public. Now the artifacts
specifically will be for the members. But the working group is doing all of the requirement
traceability. And if there's something that you care about that's outside the scope,
even now I know the methodology. And we actually, in December, we got our concept to prove
from to sued on our methodology that we're using. So they think if we can do what we say we're
going to do, they'll be able to certify us. We just have a lot of work to do between now and then.
And so that's, we just sort of mentioned that in announcement yesterday. So what are the results
from applying all these best practices? Lots and lots of new products. They're showing up all over
the place in various things from wearable rings for biometric monitoring. Livestock trackers,
Chromebooks, the framework laptop. Some of these are actually at this effort stand. So if you want
to look at some of this stuff, just over there, you can go and talk to them and then you've got
some of the devil or person with some of their demos. They're happy to talk about that. But from
winter binds to hearing aids, things that need to ever source constraints, it's a good fit for.
We're seeing about 6.9 almost 7,000 forks in the wild. And things like winter join systems
and so forth. And in fact, when I was looking at this last night, I actually spotted Tesla there.
So it was so like, okay, Tesla's now starting to work upstream. That's cool. We're supporting
all the main arch to recharge hardware architectures. We've got a pretty complete stack. The developers
are generally referred to this as batteries included. If there's a component technology, there's
a few that are missing that I'd love to see. But a lot of the key ones are there already integrated
into the effort. There were 220 sensors, so you just can figure this information in and build it
and then you can start working with it. And there's over 700 boards in our repo at this point now.
And it continues to grow. We are seeing our average number of unique contributors per month.
See my comments about looking at over time. Our, you know, and actually you saw in the other chart
where 300. So when we start seeing 225, it'll be seeing us at 300 level. And you can sort of see
that our average commits. This is more contributors. There's more commits. You'll surprise there.
But we've got the various history and the stars. So the developers like it enough to give
a stars for what metric that's worth. But we are seeing each release. We're seeing about third,
new contributors usually happening now, which is kind of cool. And then we're seeing lots of
floating and visiting activity going on. This doesn't give us why are the products are. We have
to use other clues for that. But at least it starts to less know people are using it in certain ways.
And our ecosystem is grown, significantly in the last two years. We're getting a lot more
ideas working with SEFER. And we just got another compiler, IAR's compiler working with SEFER,
which is a safety certified compiler. So that'll be part of the whole safety story. We've got a wide range
of debugger and tracing tools, as well as emulators now. And people are giving training. AC6,
dualists is just a new one. It's just joined us. And they're doing training, as well as people
do consulting. So if you don't have SEFER skills in health, there's someone people who work with you.
We've got a fair amount of security stuff. We're coming with a trust in firmware, PSA,
and VEDTLS with SEFER. We've got some tiny ML situations going on, as well as various
language run times, including Rust. A lot of work's going on with Rust right now.
And Micah Python. And then we've got a variety of other firmware instead of a recently
that have been integrated in. A lot of people are using working with SEFER for remote management
of systems, a IoT devices. And so various companies do a lot of work with it, and those are there.
And as well as we're now starting to see graphical interfaces showing up. So we were seeing LVGL,
but we've just finished having Micah Edge and QT group join us as members, and Slintis there
now too, as well as some of the robotics. So we're continuing to grow our ecosystem around SEFER
as well, which is also helping make it more accessible. These are our platinum members. They're
the ones who are, you know, funding quite frankly the safety work, because this isn't cheap
to put them out of like. And we have a fairly wide range of silver members now, including one or two
that are not willing to have their logos up there yet, but I'm hoping as soon as they ship products
they will. So what's next? We've got focus areas. We've done a lot of best practices, and there's
still more to go. It's a journey. It's not a done deal. The growth on maintainers has been a stress
point, because as more consumers come in more reviews, maintainers are fragile resources. And so
figuring out how we can be scale things more humanely for everyone is a challenge and still get
responses and reasonable ways. We need to do our own benchmarking. We learned that when the hard
way last year, when the digital real-time performance benchmarking, and we've got work on our
testing for structure we want to rework. And then quite frankly, the CRA readiness is a big topic.
The heart of the SEFER community is here in Europe. Okay? About two, there's the developers
that respond to surveys that are coming in from Europe at this point. We are spreading out
other geographic regions, but CRA is going to be a big consideration, and making sure that
as a project we're doing everything we can to make it easy for people. And then, as we go after
safety, we require requirements. Well, developers don't know how some developers do, but most
developers you can't assume know how to formulate requirements, it's a different mental discipline.
And so getting people who know how to work with requirements, ecosystem participating with us,
is a current challenge. And we've got some outreach going on there. So getting them working here,
possibly getting them working with Linux, these are things that might see us start to do. And then
something that's near and dear to my heart is we still need to keep on improving diversity.
So, if you consider yourself a member of an underrepresented group in some way, shape or form,
it's a short time in survey, and kind of no one know what the blockers are.
So, what is the blockers? What's stopping?
And I have questions. I have kinds for any one who has questions. Any questions? Sure.
Yes, I think if the presentation was very interesting. My question is given your proximity to Linux,
how did you manage to avoid catching the email workflow?
The developers decided they wanted a workflow that would actually track and have things linked
whether rather than trying to do it afterwards. So, it was a developer decision.
So, do you think that affected the outcome your success?
I think so. I think there's a factor. A lot of the kernel developers have muscle memory.
I'm working with email flows and have filters and a bunch of other things.
The people coming in into the newbies and so forth have to get trained on this.
And some of that barriers into your now.
Okay. Thank you.
Sure.
I think you were saying that some devices and manufacturers that use effort are a
look sent to disclose that. Why?
So, I was talking to a one of the award winners at CES at the start of the month.
And you had a really interesting device. Okay. One an award.
And so, I went, this looks like an it might have it in there. So, I went and started talking to him.
And he was playing dumb initially and then he realized I was with Zephyr and said, yeah, we're using
Zephyr. I said, well, you'd be visible about it. And he went, no, it's my secret sauce.
So, as an open source project, the fact that people are feeling like it's proprietary edge for them
to be using Zephyr, it's good in some senses, but it makes it hard for people to see that.
On the other hand, I met other people at CES. They were doing some really cool things.
So, they were quite happy to be visible about Zephyr. One of which was a harvesting of motion for energy.
And they were using Zephyr for buttonflix, catching up the energy with an induction coils for
general energy. And then being able to send stats over Bluetooth to another device.
Another was putting a device on an engine and the vibrations generate power. So, you don't have the
back of the batteries. And then you're still sending a stat. So, there's, you know, there's some
people who are very interested being cool. A lot of the venture capital are happy to be visible,
but some of the others, the big manufacturers in particular sometimes are a bit reluctant to be visible.
And if you know of any products, running Zephyr, and you want to talk to me about them and let me know
how would love it. Thank you. What's we going there?
Sorry, hey. Hey, thank you for the talk. You mentioned if you, if we can show a project
using Zephyr, you can get on the list for the security firm. What's here a product?
Is my homegrown home automation system in Zephyr? Only I am using also a product or
yeah, if you've done a, like, if you've got a homegrown project that you do at home,
we would very much welcome a blog explaining how you do things and to share that knowledge with others
and point them at your code bases and whatever you want to do. So, we would welcome the blogs and if you've
got pictures of your thing, we would probably put it up on, and project sister shell people.
It's pretty as surprising. Yeah, now we want this much to decide and give it a share. Go for it.
You, sorry, just for what. In the beginning, you taught how the Zephyr community adopted a lot of
good practices from other communities. Is there anything that you say that is completely different
within the Zephyr community of the mainstream? I don't think anything is different on
of the sun. There's all variants from different places. I think we've listened, well,
the thing that's different about Zephyr's were actually embedded. And there's not that much
really, like I say, the embedded ecosystem is not your big name. The fact that we're actually
showing up on the top 30 open source projects, overall was like completely mind blowing to me,
because I've always been used to embed a piece of thing off to the side. Like I say, we're