Alright. Good morning. Welcome to the Package Management Devroom. Thanks Andrew and
Wolf for having me. I'm Adam, as it says on the screen. I'm here in a couple of capacities
today, both as a staff member of the Rust Foundation, where I'm employed as a security focus
software developer. So if somebody steals your AWS credentials through some of the
great-born malware, that's probably my fault. And I'm also here as a member of the
crates.io team. And for those of you who don't know, crates.io is the Package Registry for
Rust, basically. It's the official Package Registry. Same as PIPI, or MPM, or other similar
pictures. So since, though, a nice enough to put me up first, we're going to sit around a
very virtual campfire because of fire regulations and talk about something that happened
last year. I thought about getting marshmallows, but it just didn't seem like it was worth it.
So as I've mentioned, Crates.io is the official Package Repository for Rust. Hosts
about 220,000 crates, which is the Rust term for packages, because of course we have to invent
our own word. And is growing rapidly, much as Rust is in general across many things.
On a technical level, Crates.io uses GitHub as its sole identity provider. This was a decision
it was made back in 2014 when Crates.io was first created. It made sense at the time. There are
many good reasons to support other identity providers in the air of our ward 2026, but that
hasn't happened yet. It's an active area of discussion. If you would like to discuss that further,
Toby is somewhere over there, who is the team lead. He will absolutely hate me, don't think
him in. But it's so pretty well overall. The big benefit we've had is we've got to benefit
from GitHub needing to take active measures to protect their own platform, to control spam,
to prevent fishing attacks. And it's meant that we haven't had to think about policies around things
like account recovery, succession matters. The only real downside apart from freedom has been
the, you know, mild burning sensation of being reliant on a large American company, which is
a little bit less fun than it used to be. The growing popularity of Rust, though, has made it
a more attractive target. And there are some downsides to having a single idea as to provide
her as well. One of the big ones is you don't have to worry about standing up multiple fake
authentication flows if you're trying to fish once a fishing attack. You just have one and
everybody knows what GitHub author looks like, right? We probably all click through it like multiple
times a week for different things. And Crates, although I follow, is a pretty normal
power law distribution when it comes to package or create ownership. If you can just pop one of
the accounts of the top, you know, 50 or so crad owners, you go and control over millions upon
millions of downloads per day. Many of those are going to CI, but some of those are going to
develop a machines and developers tend to have poor security hygiene. So that's a good, that's a
lot of potential asternaticans. Now, of course, we're not the only ones in this boat,
NPM and PIPI, not among others, see at least an order of magnitude, more packaged downloads and
Crates.io. And, you know, while Rust is still making inroads in some places, JavaScript,
TypeScript, and Python are used by everyone. So last year, there were some fishing. People
try to do bad things. The first target was actually NPM. In mid July last year, a developer who owned
several popular lending packages got targeted by an email that actually was this, and generated an
API token that could then be used to publish new versions of one of their packages that contained
malware that's currently being called scavenger. If you look, it's going to be a little bit hard to
read, but basically the link here goes to NPM for November.js.com. So that's reasonably sneaky,
right? Like, if you're a half asleep and you're just looking for your emails in the morning,
I probably wouldn't have spotted it. But truthfully, this campaign really first came to my
attention when this blog post has published on the Python blog, PIPI blog a couple of weeks later
by my friend Mike, who works at PIPI and the Python software foundation.
PIPI had seen a fishing campaign about a week after the NPM attack, where the attacker was trying
to harvest credentials via the PIPJ.org domain. So in a sense of theme here, and had in fact succeeded
at fishing four user accounts, including the owner of a reasonably popular Python package,
which was about 1600 in the download charts, which considering this over 700,000 packages on
PIPI is pretty good. A new version that package was published with, as I understand, it's suspiciously
similar malware, and again, an API token was safe for further use. So this kind of a pattern here,
right? Like, you know, we've got some pretty cleverly named domains. We've got some emails. I don't
have a screenshot of the email. I don't think I don't think PIPI had one, but presumably looked
reasonably authentic, much as the previous one there did. That looks, you know, not bad.
And they obviously built a good facsimile of the login experience, and they're not completely
idiots about this, because they're stashing API tokens for later. The PIPI one actually published
a second version of the package after the first version got taken down with the same malware. So
they had an API token, and it wasn't until the author went and looked at their token settings,
that they actually figured out that they still had a token. So we fast forward into early September,
there was another instance of this, very similar pattern, which can compromise these packages
in the MPM world, chalk and debug. I'm not going to go into any great detail on this one, but
again, you can kind of see this. The other thing that was interesting about this was,
most fishing attacks against registries historically, and I'm indebted to my colleague Walter
who also works at the Rust Foundation for this. Kind of a spray and prior approach, right? Like
if you could get an email for somebody who had an account on a registry, they would just send
the email to everyone. These attacks were a bit more targeted. Like they were just targeting people who
had high end packages, you know, people who were owners of, you know, top 1% packages in their
respective ecosystems. So there's a bit more effort being put in. All right, so this was a Monday,
September the 8th was a Monday. Let's go forward to Friday. So this discussion is open,
it's a little bit carefully edges, but I'll talk through on the crates.io GitHub repo.
Basically, it is, as it says at the top, crates.io fishing, and somebody had received a fairly,
fairly decent looking email that basically said that crates.io had been compromised, and they had
access some user information. We are currently drafting a blog post to outline the timeline,
and we suggest that you rotate your login info by signing into our internal SSO, which is a temporary
fix to enjoy your attack account, modify any packages published by you. Now, you'll note something
right away in the screenshot, which is, this goes to github.rustfoundation.dev. My assumption here is that
they couldn't come up with a sufficiently confusing version of the crates.io domain name, which really
just reveals a lack of imagination, I think. But regardless, they then attempted to use the good
name of the rust foundation to basically say, it's okay, we set up our own SSO coming here.
Toby, again, sitting over there, saw this in his notifications and sent a message to the, we have a
two, a channel on Zoom up. So Zoom up is what the rust project uses for communication, and we have
basically security responses handled by the rust project has a security response working group,
and then they handled them in collaboration with whatever the effect the team is. So in this case,
the crates.io team. Toby sent the message to that a bunch of people's phones started buzzing,
including mine, because we have notifications turned on for that stuff, and away we went.
We also had about the same time, got additional confirmation from others that they were seeing the same
sort of thing. So this is, this is, for an expert sushi, he was a very prolific crates author in the
rust ecosystem. So I'm going to talk about the response, I'm going to avoid using any names
other than Toby's for the rest of this. Basically, just because every involved degree work,
I'm happy to give them public shout out if they want them. But this was a fast-moving private response
thread. I'm not going to put screenshots on top of like rural conversations here because it
moved fairly fast. I don't want to attribute things to people without full context or permission.
So I'm going to say we a lot, but in most cases it's going to be an individual.
So at this point, we collectively, plural, spring into action. Personally, I was in the back
seat of a DD in Hangzhou, China, for a conference dinner. So I was a long way from my laptop,
which was really annoying. But there were many others who were thankfully at their laptops,
or at least closer laptops and could start working right away. Initially, we had two main areas of
focus. One, communicating there was an active attack that we're aware of it, and reiterate the
crates author is never going to send you an email asking you to log in to something else, or really
anything ever, including crates.io. And two, start working to take that Rust Foundation.dev
domain name down. You know, the Rust Foundation, somewhat infamously has trademarks.
And you know, it felt like an opportunity to actually use them for something useful rather than just
scaring people with trademark policies. So as a side by just on that, I've kind of actually really
a little bit tickled that they use the Rust Foundation's name because the presentation doesn't
technically run crates.io, or they're fun to the infrastructure, doesn't own Rust.
And has a really mixed reputation in the Rust community. So invoking the name of the Rust Foundation,
there's to make it look authoritative, was definitely a choice. Maybe not a very good one.
The main Rust Foundation.dev domain, by the way. So there's a screenshot of the GitHub sub-domain,
where I had like a nice little offflow. The main Rust Foundation.dev domain at this point
contained a recoil. It was just like an embed of rigorously. Following that domain, we're going
to come back to that. Following that domain ended up being kind of fun. It was kind of a communication
channel through this process. So one of the things that we was sort of trying to figure out in
the early stages was, so at what point do we talk to GitHub and how do we talk to GitHub? Because
this fishing attack is going, obviously, being is going through GitHub infrastructure at some point,
right? Like they've got the fake part of logging flow. But these are eventually going to
turn into a real request to GitHub API. They're going to sign in. They're going to then get a
session token that they can exchange for creates.io or then presumably create a token as
what we assume is going to happen. The problem with that is, where do you report that to GitHub?
How do you report that GitHub? Now GitHub has lots of report buttons on repos and things like that.
This isn't tied to a repo. In the end, we sort of dug up an email address from I think a
who is record and went with that. This was not super successful, as it turns out.
So we're going to come back to this. I'm going to talk more about the party provider
swars in the end. But just kind of keep them in the vacuum mind that we were, you don't like having
uncertainty when you're responding to something and this was a significant source of uncertainty.
We also discussed whether we'd be able to detect anything from the crates.io
observability stack that we have. We realized pretty quickly that it was very little we can do
directly because the fishing attack wasn't actually touching crates.io in any way. They probably
weren't going to make a request to crates.io API server until they had a login, a fish login
through GitHub. There was no reason otherwise to touch the API. That made monitoring this kind of
difficult. We didn't really know if somebody had actually gotten fished or not. We do have the
ability to put crates so into read-only mode, but we agreed that doing so at that point felt
fairly premature. We didn't have a lot of evidence as being a widespread campaign and
read-only mode involves a lot more disruption than just log-ins, like people can't publish crates.
They can't. There are many things they can't do. While this was going on, part of the group
were able to get a initial blog post and some social media posts up, noting that the attack was
going on, and read-writing that crates would never actually send you an email asking you to log in
somewhere. This is about an hour into the response, and at this point we're kind of in a holding
pattern. GitHub hadn't gotten back to us, nor had the domain register, nor had the host,
the russfoundation.dev was being hosted on. We did some ad hoc brainstorming on Zoom up about
things we could do differently. I will talk about a couple of those tools again, but basically
we're sort of in a holding pattern at this point. That and hour after that, so about two hours
after the start of the campaign, the russfoundation.dev website changed, so it was no longer a
recoil. It looked like this. First things first, I don't actually think russ to help as a
smart and then JavaScript developers. Maybe better looking, but give us that. But it did give us
a laugh, so thank you, Mysterious Fisherpeople. They were not done, though. We'll come back to that.
A suggestion was made in the public security response working group channel around the same time,
was whether the russfoundation.dev domain should be submitted to the Google Safe Browsing Project.
That honestly hadn't occurred to us, and I'd be interested to know to this dad,
I don't really know if we should have done it or not. We didn't end, I don't think anything
actually happened, but I'd be interested to know if that's a standard part of anyone else's
protocol here. Because I didn't really think that we didn't really think of that.
So that's pretty much the initial few hours. As far as we can tell, and I'll explain kind of why
I think this a little bit later, nobody in the targeted set of users, and to this day we don't know
how many that was or exactly who it was, we're deceived by the Fish and Campaign.
Over the following day, also the russfoundation.dev website changed a couple of times. The next
change was to this. I will read it just for the better for the people at the back,
Crates.io database along with juicing tokens for sale, email for buying, free leak if
no offer till Sunday. So I think this is on Saturday. Security at russfoundation.dev.
Then that was followed by sold. Wow, that was quick. Who ever thinks this was sophisticated
is clearly fear mungering. Less than 15 minutes, less than 30 minutes, et cetera.
I'm going to say several months later with a fair bit of confidence that there was no compromise
of the crates.io database. We're pretty sure this is a last gas piece of the domain
trying to fish some cryptocurrency out of a sucker rather than create some fud rather than an
actual breach. I'm also guessing that apart from the mean they weren't super happy with how this
had all gone. So you know, sucks to be there. So through this process, one of the nice things was
we had a lot of cooperation. A number of people have spent time in the last few years, including
these two gentlemen, working to build a community of package registries and managers.
Alfred Omega, and especially Michael Windsor, who is also giving a talk later,
played a huge role in fostering connections between the language package registries and that's
been incredibly helpful. So this was really clutching a couple of specific ways here. One was
it's just nice to have a friendly sympathetic year. When you're going through this kind of thing,
like being able to just go to like a signal group chat or a Slack channel and just be like, hey,
this is happening and have a bunch of people go, yeah, that happened to us at sucks is kind of nice.
But secondly, having a larger group of people means you have by extension, a larger group of
contacts who you can get to. So you'll know that when I left off the story, nothing actually
being taken down. So, you know, again, spoil us for kind of later. But that was a way that we were
able to get more contact, we're at least some of the providers involved. But here's another concrete
way that helped. Remember the PIPI attack that I mentioned. Seth Larson and of PIPI and the PIPI
for Software Foundation had set up the main threat and monitoring of newly registered domains to
try and find these things before the attacks actually launched. A couple of weeks after that
Crate Solo attack, Seth was alerted to a domain that was likely to try to fish PIPI again. It was
squatting some sort of typo for PIPI. But it turned out it was actually at that point,
misconfigured to serve up a Crate Solo authentication flow on what was apparently going to be the
prod 01-crate.io domain. So, let's take a little bit more creative. Now, we can't prove that was a
same group as before, but it does feel kind of familiar, right? So, we spun the initial response
thread back up and another member of the Crate Solo team created a new GitHub and Crate Solo account
that had no connection to anything, no access to anything. And so, using that we were able to actually
monitor what would have happened had somebody fallen for the fishing attack. Basically, a Crate
Solo API token was registered with full scopes and a unique name. Like it had, it was things
like underscore underscore underscore underscore something like that. Something that we didn't
have any other examples of in the database. We were then able to find a bunch of tokens that had
been registered 11 hours earlier by another new GitHub account that was presumably testing the
flow, making sure that it worked, making sure they could create API tokens, making sure the payload
worked. So, what that also meant was we now knew what to look for in the observability setup.
So, we're able to go back to the previous attack and see whether anybody had, in fact, tried to
try to actually, or had fallen for them. We had asked people to report if they had, if they had
been fished and we did say there would be no shame, but people still going to feel shame. So,
we didn't hear from anyone, but we didn't want to take those concrete proof. This strongly hinted
that we were, in fact, like nobody in the Rust, but nobody in the Crateslayer community had,
in fact, fallen for the fishing. So, we're able to do a retrospective query and confirm that.
And with all of that, we now had enough, we had enough data that we could go get stuff taken
down and take defensive actions before anything actually got going to foil this campaign.
As far as we know, that time around, no emails ever got sent. So, that was a win. Like,
the best fishing attack is one where the attack never happens. So, that's all the nice warm
fuzzy bits. The Rust project worked really well together. The teams and working groups involved.
The Rust Foundation security team. I provided support. The broader package registry
and security communities, especially PIPI and our for Omega, provided really good support and back
up for us, and tipped us off to the abortive second attempt. So, now, let's talk about saying
it was not quite as good. Commercial, third party providers. There were quite a few providers in the
mix here. I'm only a name one in my name, but if you catch me after some beers later, maybe I'll
name some of the others. The one on my name is obviously GitHub because they have to be involved
they're the ideas to provide. So, when we went to GitHub in the first place, as I said,
we emailed an email address, we pulled out of who is because we just had no idea who else to contact,
didn't get a response. Eventually, we were able to get a lot higher level contact direct to someone
at GitHub. This is several hours after the attack started via a colleague in Alpha Omega,
and at that point, all of a sudden, the wheels started turning, unsurprisingly. And, you know,
sort of, you know, put some monitoring in, tried to prevent authentication through that flow
and so on and so forth. So, that's good. That's, you know, like, perfection is too much to ask.
Honestly, once we had the right contact, that was great. It's a little unfortunate that we
had to wait for that right contact, but, you know, that is what it is. There are also two hosting
providers involved, both of whom were presumably chosen for their acceptance of cryptocurrency,
and general just air of sketchiness, two domain name providers, and in the second attempt in
campaign, a well-known CDN company, and yes, it's probably the one you're thinking of.
The CDN provider took down the site in a pretty good turnaround times. About two hours, that's
fair enough. We don't pay them. Like, that's, that's pretty reasonable.
Um, nobody else ever responded. We sent emails. We sent social media, like, we tried. We really
did try. I think, I think Wall Street even actually called someone on the phone, which is terrifying
from a millennial. Um, and just nothing. Never heard back. Nothing ever got taken down, and that's
really just disappointing. So, you know, that's kind of a cautionary tale that you are,
as much as you can cooperate with other people, you may not get much cooperation from
commercial providers. So, if that's a no and no, and then you can factor that into your
internet response, right? We did make a couple of little changes out of this. Uh, Toby opened
a PR to disable just API token creation rather than going to full read only modes. That's a useful
thing. Toby looks really surprised. I think he forgot he did that. Um, that's really helpful.
There's still some downstream effects, trusted publishing doesn't work, but it's better than nothing.
Another thing we realized was that the rust security response working group didn't actually
have a way of paging the on-call stuff for crates.io. So, we fixed that. Um, in this case it worked out
because the attack was during the European Workday, the North Americans were just waking up,
and even me in East Asia was still awake. So, everybody was basically up and in the loop.
But we fixed that. So, that's a real lesson that we learned out of that. Make sure people can
wake the right people up when this kind of thing gets noticed. But the main lesson really is
we're better working together. We're more likely to get the outcomes that we need when we do so.
So, with that, I'm glad that we're all in the room. Make sure you make friends with people
that you don't already know. Um, and let's have a great day. Thank you very much.
I'm going to start. I don't know if we actually have time for questions. So, that's okay.
Do you have one question, but you can have to. If anybody has I'm burning, go for it.
Yes, sorry.
Okay, so the question is the original fishing attacks. How did they circumvent the two-factor authentication?
That is an excellent question that I don't actually fully know the answer to.
As far as I know, nobody did that level of analysis on the on the fake size that they were
putting up to front them. They obviously were doing something to act as a proxy between the user
and the actual and the actual identity provider. If we never got far enough into this to figure that out
to be honest with you, if somebody MPM or PIP did that analysis, it's not in the blog post and I
don't have any further details. I'm sorry. But yes, absolutely. That is absolutely a problem.
And I wish I had a better answer. All right. Thank you very much. And enjoy your day.