So please welcome Devgang Lu, he's a software engineer, and it's first time at first
them, so it's a big occasion, it's not his first time reverse engineering, but it's the first
time doing something that big, so a big one of applause for him.
Thank you very much.
So today we will be talking about reverse engineering in the world largest music streaming
platform, which you may guess is Spotify, so I want to ask if there's anyone from Spotify
in the room.
Okay, actually no one, so I shouldn't get arrested for doing this.
So a bit on who I am, I'm the kind of software developer, I'm a big contributor for open
source, I'm going from the more projects to a little bit more stuff, and also my own projects.
I enjoy, I will see reverse engineering and a bit of software security.
For that reason, I'm also interested in IoT stuff, I have a couple of CVs to my name and I have fun with
other stuff that isn't necessarily public.
I'm also a CTF player, so for those that don't know, CTFs are capture the flag competitions,
which are cybersecurity related.
I am a former Tmitary member, which is the team that competes in the European cybersecurity challenge.
With my old team, I was an organizer for CC-2024, which is the European cybersecurity competition,
and also a platform provider for ICC, which is the international cybersecurity competition.
And I've also been a deaf confinalist with macaroni.
Before we start, there's a big disclaimer.
This talk is present early in my opinion, not that of my current previous future employer,
and not that of my friends, my colleagues or people I met online.
For the same reason, this talk includes only things that I learned and discovered personally,
and no work from other people.
The work presented in this slide is to enhance Spotify, not undermining.
We don't do piracy, and I have nothing to do with another archive.
This talk was decided well before that happened.
And to Spotify, please don't mind me again, please don't sue me.
Let's have a chat first.
So now we begin.
We will talk about Spotify, but we will also talk about Libre's Pot,
which is the actual open-source projects that we and me and other people work on.
When I talk about Libre's Pot, actually I'm talking about many projects,
because there are many projects within different languages,
but all of them share the fact that they are client library for Spotify,
capable of playback through various decades.
It is completely headless, so it does not require a desktop environment,
it's entirely based or a version engineering.
It is fully capable as a Spotify connect endpoint,
and it's an alternative to all the time they prepared the Libs Potify,
which I never got to know, even.
This project is not a bookdownloader, don't use it for that, please.
A way to keep our bypass heads, we do not support free accounts,
it's not a wrapper around public APIs, it's not a control library,
it's not an alternative, go into it or clean.
So why Libre's Pot exists?
You can find your own reasons, but these are some of the most common.
You can turn any device into a Spotify connect endpoint.
It is edless, it does not require a desktop environment,
it has very low resource users,
who use it for that reason.
If you're into really into open source,
you may want to know exactly the code that you're running,
so all the Libres for projects allow you to do that.
And also you may be an audio nerd,
so you may want to build your own audio pipeline,
your multi-room setup, your DIY doc,
and streamer setup, all those kind of things.
So I was saying the Libre's Pot is a family,
because there are many projects, depending on the language,
mainly on the support.
We have Python version, we have a C version,
which runs on ESP32, for example,
then we have what I'm currently working on,
which is a Go Libre's Pot, we can go obviously.
It's an old version in Go,
which is not a developer anymore.
Then there's another project of mine,
which was the original Libre's Pot in not the original,
but my original project in Java.
And then we have the original Libre's Pot,
which is in Rust, and is the longest standing of them all.
The Java version is actually deprecated,
because, yeah.
So we'll have a look into the Spotify infrastructure,
a little bit, or at least what we need to get,
as Spotify connect and point working.
This is the state of things after 2019,
because there was major change in the infrastructure.
And also this is what we,
as a reverse engineering,
in a reverse engineering, we discovered being the infrastructure.
If there was someone from Spotify,
they could probably tell me I'm wrong.
So piece by piece, we have the AP resolve,
which is a service that returns at least
of endpoints for other services.
Then we have the access point,
which was very used, which was the thing
that was very used up until 2019.
It's a custom protocol,
of a TCP connection,
that has some dfial monkey exchange,
and then encrypts using some strange cipher,
called Shannon,
and transports simple data packets,
which are composed of just a packet type,
packet length and packet data.
From this connection,
we get store credentials,
so we log in with the access point,
and we get a set of store credentials,
so we can use them to authenticate other sessions,
of when you close and reopen the decline.
Then we have the log in five service,
which is a new authentication service
that is used for the newer services in infrastructure,
and we have to authenticate that using the credentials
from the access point,
and the most useful thing that we still need,
the access point for is retrieving the encryption keys
for the audio files,
because Spotify serves only encrypted audio files,
and you will need the key to the crypt.
On to the next piece, this is new stuff.
As I said, the login file provides authentication
for other services.
It provides a beer token to authenticate the SP client,
which is just a bunch of rest APIs.
The most interesting ones are those related to the school
of the connect state,
that makes the connect cluster.
So every Spotify connect endpoint,
publishes its state to the server,
which builds the connect cluster
and orchestrates all the endpoints.
This part of the API,
it's just a rest API,
and Spotify uses Protobuff for all the communications
on that API.
Lastly, we have the event part of things,
the dealer is called the web socket API is called,
it is located with login five,
and it uses Protobuff messages wrapped in JSON,
don't ask me why.
And it publishes all the events required
for the client to work to the client.
And also contains a special token,
which is used to synchronize the dealer and the SP client,
so that they work on the same stuff.
Now, we're going to have a look
at some of the technical challenges,
involved with the reverse engineering Spotify.
We'll start with the easiest one,
which is intercepting HTTPS traffic.
Traffic is encrypted,
so there's no really way to intercept it passively.
We will need to do many the middle.
Luckily, Spotify doesn't do setvigate pinning,
so it's even easier.
We can just pull up something like a meeting proxy,
get the setvigate authority,
install it for other trusted certificate authority
for this system, also for Chrome,
because Spotify uses the Chromium and Bedet framework,
which it reads DCA from Chrome.
We set the proxy URL in the desktop client,
and voila, we can see all the traffic unencrypted.
You can see that the client holds with all the services
I was telling you about,
so we have for the APR solve,
but then it does.
Again, five authentication,
it connects to the access point,
it pushes its state to the server,
and then connects to the editor.
We can see the access point connection here,
but it's not actually decrypted.
If we look into it,
it would be still encrypted,
and we've solved this problem later.
This was easy.
Now we get onto the interesting stuff.
So we are going to recover C++ protobuff classes in Gidra.
What does that mean?
C++.
We don't know that protobuff and Gidra.
What are those?
Protobuff is a mechanism for serializing structural data.
It's just like XML or JSON,
but weighs molar.
It's maintained by protobuff.
It has a bunch of other features.
But the interesting part about it is that it's very small.
And for that reason, it requires code generation.
So you cannot simply like to do with JSON decoded,
because there's something missing in the serialized format
that you have in the generated code for your messages.
It's put if I was using it for a while,
at least a single free spot was born.
So how does protobuff works?
As I said, the wire format, the serialization is entirely binary.
As you can see, in the example,
if we have a message called user,
which has three fields, name, favorite number,
and obvious, which are respectively one, two, and three.
As field numbers, on the right,
you see the serialized version of that message.
And you see there's never mentioned of the field name.
So there's never the name, favorite number, or obvious field name.
You only see the field number, which is called field tag inside the slide.
That allows it to be small.
But the other problem is that you cannot recover the name,
unless you have the message type definition.
Spotify uses a lot of protobuff.
So we may want to recover those.
Luckily, there's a special message called the file descriptor proto,
which describes a protophile.
And that message is serialized,
and included in the C++ source code that is generated for your messages.
So you can use a tool like a protobuff toolkit.
We run it against the desktop client binary,
and we get 900 files and 2400 messages, which is quite a lot.
And it's also quite fun, because sometimes you see stuff that hasn't been published yet,
because apparently the ship,
non-production protobuff definitions inside the binary.
This is what it looks like.
On the top left, you see the file descriptor protomessage,
I've removed the stuff that we are not interesting in.
In the bottom left, you see an example message, I've taken,
and I will use for other examples, which is the any message,
which contains only two fields, which are the type URL, and the value.
On the right, you see what this serializing defile descriptor proto
for the any protophile looks like.
And you can see there's many similarities,
and you can see how you could reconstruct the original protophile
from what you see on the right.
And that is what the protobuff toolkit tool I showed you does.
Next piece of tooling we will need is Gidram,
many of you may know it.
It's a version engineering framework created,
and maintained by the NSA search directorate.
We will need it mainly for its compilation scripting,
but it does a bunch of other things.
It helps us understand what is going on inside this Spotify,
desktop client, which is written in C++,
and try to transform it into actual readable code for us humans.
But if we do that, we get 6 million lines of code.
Of course, the binary does not have the back symbols,
so we get no variable names, no function names, anything like that.
So it's simply 6 million of meaningless code.
So how do we find what we are interested in?
There are multiple ways.
You can look for strings.
You can do some fancy code flow analysis.
What I've did and has turned out to be useful for me
is figure out where those protobuff classes generated
by protobuff are being used in the compile code.
For that, we will make use of the fact that those classes
are C++ classes, which extend the virtual
global global protobuff message class.
It's very important that class is virtual,
because virtual classes are virtual tables,
which are basically tables of addresses inside the binary,
and we can use those to trace back to the constructors
and the structures of set classes.
We do that.
We need to do it deterministically and automatically,
because there are a bunch of messages inside the Spotify,
so we can do it by end.
So we'll have a look at an example,
how we can inside the generated C++ code,
trace back the file descriptor protobuff message,
which I showed you earlier,
and we know that works because other tools use it.
How we can trace back this message back to the C++ class.
To do that, we look in generated code.
We see that this message is referenced inside
the internal structure called descriptor table.
Then we see that this is referenced in strange class,
which I don't really know what it does,
but it's sure that it was compilation time.
Luckily for us, this class is finally,
this function is finally used inside the actual C++ class.
We are interested in, which is, for example,
say it's any.
This get-metadata method is also virtual.
And as I said, virtual classes, virtual methods,
and up in the V table.
So we can go from the file descriptor protoback
to the V table of that class,
and from there we can get the constructors and the structures.
Can we automate this course?
All the code for the,
scrolled Gita script is available on my GitHub.
It's quite old, but it still works again,
I guess, Spotify.
It will not work with the latest version of the protobuff generator,
because they changed the generated code quite a bit.
The end result is this.
On the left, you can see that the script
has recovered the where the V tables for some messages are.
I feel the just for one.
You can see it has recognized where the V tables are.
It has renamed the structures.
And on the right, you see what the internal descriptor,
protobter descriptor table structure looks like.
You can see it has, for example, the file name,
and the, the point animation to the descriptor.
So we're done with that.
We did that.
Now onto the last part.
I mentioned, we have the HTTPS traffic.
We know what classes are being used to generate that HTTPS traffic,
because the, the API uses mainly protobuff.
If we see an API call, we can look into Gidra,
find where the message for that call is being created,
and look what the code does and try to figure out
how to do the same in our code.
The last piece we're missing is the access point I mentioned
in the beginning.
We still cannot see what traffic happens there.
And there might be some interesting stuff.
So now we're going to see how we can log the traffic.
And for that, we will need some more tools.
One of those is GDB.
You may know GDB if you've written any C or C++ code.
It's very, it's the standard tool for dynamic analysis.
And bug finding in C and C++ programs,
as far we have done only static analysis.
But you may be used to using GDB with the bug symbols,
because you may use it on the code you are written yourself.
Here we don't have that, because Spotify doesn't ship the bug symbols,
obviously.
So we will use an extension to GDB called the point DBG,
which facilitates a lot of the reverse engineering process.
You can see the staggering difference between running GDB
on an yellow word program with and without and with point DBG.
There's a lot more info, and also there's a lot more commands.
You can use to move faster.
Another tool that we'll use is freedom.
Freedize the dynamic code instrumentation toolkit.
It's real cool because it works on basically any platform.
And it allows you to write some JavaScript code to inject inside the process you want to look into.
So what we will typically do is use GDB to manually verify what you are looking at
if your assumptions are correct.
And then switch to something that is more automated.
You can also automate GDB, but we will do it with freedom.
So you write some JavaScript code and you basically have a script for our use case that logs all the traffic.
Last piece, what is the Cypher?
I was talking about in the beginning.
It's called the Shannon Cypher.
It's part of the sober family, sober family.
It's been developed by Cole Comostralia in 1997.
The original implementation is available only through way back machine.
I have no idea why Spotify use that in the first place.
The only advantage is that it does encryption and message authentication simultaneously, which is, I guess, Andy.
The reference implementation is very simple.
It has a method to set the key, a method to set the nonsense, a method to encrypt, to decrypt and to finish the encryption or encryption process,
and generate the message authentication code for what we have encrypted so far.
How do we find those functions inside the idra because we want to log what goes through the encryption and decryption functions?
Well, one very useful trick is to look for Constance.
For example, this constant is used in the original source code for the Shannon Cypher.
We look for the same constant inside the Spotify binary.
You guys get just reads, just like in the original source code.
So we are looking at the writing with some trial and error.
We will figure out where the functions are defined, in the idra and the compile code.
At that point, we can verify our assumption with GDB.
We've done so far only static analysis by finding the function.
Now we do the dynamic analysis.
So for example, if we look at the Shannon Encryp function, we can set a break point where we think the function is.
It has just three parameters.
We are not interested in the first one.
But if we look at the second one, which is the data buffer, which should contain the data that is being encrypted,
that is in the RSI register, the content of RSI looks like a byte pointer.
So we probably in the right place.
We look at the other parameter, which is EDX.
It contains a reasonable number.
And if we print the content of RSI for 398 bytes, which is the content of EDX,
we get something that looks like an access point packet.
So we have the packet type, which is AB.
We have the packet length and then the data.
The packet length is exactly 395.
So it's 398 minus the three initial bytes.
So we are definitely in the right place.
Now that we know the address of the Encryp function, we can set up freedom.
I run all my tests on virtual machine, because I can restore it.
I can go back and stop it and do whatever.
Luckily, free data supports this use case, because it has a free server,
which I can launch on the VM and then connect it from the host through TCP connection.
So you can see here, for example, in the screenshot that I'm connecting to my VM and launching
this spot 5 binary and getting the base address of where the binary code has been loaded.
We can do better.
This is still very manual, because we are into the free data replica.
And so we finally write the free discrete that I was talking about all along.
We get the base address, and then we use the interceptor module to look into the two functions,
which are the encryption and the encryption functions.
For the encryption function, we can do exactly what I just mentioned,
so we just get the value of the register.
We read some memory from, we read some memory for the length that is set in the register.
And we just log it.
For the decryption function, it's not that simple, because when we enter the decryption function,
obviously the data is still encrypted.
We want to look at the data when we exit from the decryption function.
And for that, we'll use a built in free data function, which is on enter and on leave.
So we save the value of the register when we enter.
And then when we leave, we will dump the content of the memory.
And we'll have the decrypted content.
This is what the discrete looks like.
So we see that the client sends the AB packet, which is the login packet.
Then it receives an AP welcome, an access point, welcome packet, which
significates that the the login was successful.
Then it receives a counter code packet, a product, a team for a packet.
And then it sends a mercury request packet, which is another thing.
The mercury protocol is something that they are using to build essentially HTTP over their own custom protocol,
which no idea what they did in the first place.
Then we receive a mercury event, and we receive another mercury request packet, which is actually the response to the original mercury request packet.
With that, we are done with the technical challenges.
Actually, there are many more technical challenges, but those are, I think, the most interesting ones.
And this was the fun part.
This is, for me, at least, the fun part of reverse engineering, the fun part of maintaining such a project.
But also come the legal challenges involved with maintaining reverse engineering projects.
Because if I suppose many of you have an open source project, and you probably don't have to deal with legal problems,
because it's your own code, it's stuff you have written yourself.
Now, no one can tell you that you cannot have that code.
That is not true for Spotify and in general, reverse engineering projects.
So what happens is that we have a constant problem, which is not making Spotify angry.
Because if we do, they have the legal power, the legal strength to take us down, and we don't.
We, as personal contributors to the open source projects, don't have the power or the willingness of answerback is Spotify was ever to, and they did times.
We don't have to, the power to answer back, and so we have to comply essentially.
I said they did, but fire as many, the MCAs on GitHub, it has emailed many people, even me and other maintainers about things.
I've not ever received an actual legal, legal inquiry from them, but some, for example, have.
For this reason, we will not implement many features that some are either requests, heavily requested, and others are kind of essential and said not to have.
Those are listener reporting, which we will talk about losses playback, which is a recent one, and adds playback and support for free accounts.
So what is listener reporting? When you use Spotify, all your listens are being reported to the server, that is to build your recently listened pool of tracks, that is also to influence the algorithm.
And it is also to credit the artists.
One downside of using the LibreSource projects is that this does not happen.
So your listens will not be accounted for, and effectively artists you listen to will not be credited.
That is very sad, but we really can do it.
Not because of the reverse engineering process, but because they rely on an entirely different system for logging playback.
So it does anything to do with the connect state, it does not have anything to do with downloading from the CDN.
It's purely an entirely separate system, and reverse engineering that and making it public.
To bring a lot of risks to the project, because that is something that many people look into to create so-called the list and bots.
So these are services that you can buy as an artist, and get not a free, but a paid boost to your tracks, and essentially get into the algorithm.
This is something that Spotify tries to combat.
If we make that code open source, then suddenly there will be a lot more of those list and bots, and Spotify will come to us and say, what the fact don't do that.
And that is not something that we want to happen.
Another one is Loser's payback. This is a recent one because Spotify has launched support for flag files quite recently, a couple of months ago.
After two or three years that the protobuff for flag playback started appearing, they actually published it, which is the thing I was talking about that in the protobuff files you see things before they are released.
They joined the Game of Ice 5 streaming providers, because the quality they usually serve is quite low.
And for that reason, Loser's payback became very quickly, an heavily requested feature, because many has had in the beginning have DIY setups, where they want to get the best out of their DIY audio pipeline,
or something like that, or like me, which I'm not an audio nerd, I want to get the best out of my extensive subscription.
I pay for it, I want to use all the features essentially, but sadly we have received quite explicitly a message from Spotify that told us don't do that.
If you keep going, you will be in trouble, because those flag files are protected by a new DRM, which we will call stop stop.
It's not actually called stop stop, but it's kind of a meme. You can probably guess what it's real name is.
And for context, other eye-ficing providers do not have DRMs to protect the eye quality flag files.
So what is this DRM, which is the big problem of supporting flag files?
I said it in the beginning, right now, and originally we can just get the encryption keys for the audio files from the access point.
So there's some back-end service that returns the key, and it returns the key as is. So you can take the key, the key, the data, everything is fine.
But recently, recently as in the past year, but also recently as in the Anas Damp thing, they started cracking down on the usage of this API, first for free accounts.
So free accounts started to become heavily limited in what they could do through this API, all the API.
The fun thing is that they started killing their own products. So many Spotify partners were essentially broken. You could not use them anymore, and that is still the case for many accounts.
There's all, there's an old blog post, not really a blog post, but a forum post with people that are really, really angry because they cannot use their iFi streamers for some reason.
And Spotify doesn't really seem to care, honestly. That also breaks some of the labor spot users because they are targeting these old API use for the encryption keys.
The new DRM, it's something entirely different. I call it the new DRM, but essentially there was no DRM before. The new DRM does not serve the decryption key as is, but it serves an obfuscated decryption key.
And you need the to the obfuscated, please stay forward, and luckily the defuscation code contains some constants and some procedures that they can claim for intellectual property infringement.
So we cannot include that code in our public repositories, or they will finally have a reason to take us down.
Last thing is ads playback and free accounts. This is another good choice to not support those for many reasons. One of them is that, but if I doesn't care as long as you touch them in the pockets.
Supporting free accounts would mean that potentially you are stealing revenue. Even if we implement all the limitations that come with free accounts being ads and you cannot listen to your playlists in order, but you have to shuffle it.
Even if we did that correctly, they would still have a lever to say, well, yeah, correctly, but there was one thing wrong, so it's all broken, you have stolen some revenue.
Then it's not hard to reverse engineer, but they tried to hide it, so it's essentially harder than the other stuff, so we would have to waste a little bit more time on it.
The logic changes frequently, depending on the business side of things, and this is what mothers want to know about.
For advanced or simple mods to the Spotify Android app, starting to support those kinds of things would give mothers an insight on how things work, and then we would get contribution from mothers, which is people that Spotify tries to combat nothing wrong with them, but Spotify surely tries to combat mothers.
And we not want to proceed with them, like you.
So, our got temporarily banned from Spotify, that was meant to happen, honestly. On the 29th of October, we created a private but not so private discourse over with some of the other contributors to work on stop stop.
The third on November, we already had a working implementation of stop stop.
So, on the fourth of November, let's go, we have the implementation, let's do this.
So, in GoLiversPot, I first implemented a flag decoder, which I did not have before, with one pull request, and with another pull request, I implemented support for stop stop.
That did not include any of the updated code, the application code. That only included the API calls required to get the application key, and then you would have to provide your own the application code to plug into the project at compile time.
So, that you could effectively use it.
And I started using, because the code was not public, the code was running on my machine. I was happy with it, the flag files were working flawlessly.
I could not hear the difference, but that doesn't matter. I wanted to use it anyway.
The day after, the fifth of November, we receive, we as the main containers of the LibbersPot projects, we receive an email from Spotify.
Actually, a support ticket they opened towards us, which is quite funny for another reason.
Saying that they have seen what we are doing with the DRM, and we should stop doing it, because they needed to preserve the integrity of their platform, the shareholders value and stuff like that.
So, essentially, we actually stopped talking about it, some containers deleted, some public information, but it all ended there.
Nothing in that email said that I could not use it myself, so I kept using.
The fifth of November, or the 15th of November, I recap, I am logged out of all my devices, and I quickly understood that I was banned.
Luckily, I could appeal to my suspension, and just 48 hours later, I got my account back, but I have not used stopstop since.
How to interpret what happened, most likely, it was a warning. I'm pretty sure it was not an automated system, because, yeah.
And that was what we think was a warning towards me, which I was the only one as far as I know, that was using stopstop on my setup and to stop doing it.
As I said, the fact that they opened a support ticket towards us was quite fun, because when we tried to email them back, when we tried to email them back, they never responded to us.
It's not the first time they write us an email, but they never respond when we asked for clarification.
But some days after, when they apparently closed the ticket on their side, we received the original title, the ticket, which was a time sensitive email to send from supportatspotify.com.
And we were asked to rate our interaction with the customer service.
So, to close, how can you help? Of course, with all of the open source projects, if they not try, use it, maybe it's not for you, but use it, Spotify will know if you use it, that doesn't mean you will get banned.
There's many people across the world that are using deliberate spot clients and no one ever reported to have been banned, but Spotify will know, because we do not try to hide.
We explicitly say that we are, we are, we are,
go lieber spot, we are, lieber spot rust and stuff like that.
So that they can, for example, filter us out from,
from their analytics.
Contribute, bug reports, feature requests are always welcome,
which are our best to make those things happen.
But you may want to make things happen on your own.
So write some code, fix a bug, implement some new feature.
If you work as Spotify, as no one here apparently,
get in touch with us, with the lieber spot maintainers,
you already have our emails.
So you can, you can clearly get in touch.
We know you can, you just don't want, and please don't buy me again,
because I actually use it.
Thank you.
Thank you very much for a very interesting presentation,
which is obviously very popular.
We literally have three minutes for questions,
so you're happy to take them.
Yep, okay.
Questions?
Yes, I'm a little bit concerned about the reporting back
and the artists not getting revenue for it.
Is there any changes or communications?
Because I want to support your artist, not support Spotify.
The limitation is getting the code public.
I have it.
I have the code that does that.
I use it, and it works.
The problem with that, I can share it with the people I know.
I know personally, I have contributed on the project,
and I trust, but I cannot share it publicly,
and I cannot share it with people I don't know.
That's the main problem, because I would get in trouble, essentially.
So yeah, that's the downside.
And that's the reason why we continuously try to get in touch with Spotify
and tell them, give us a way to do things correctly
and not with, like, right now.
I have a question, yeah.
I wonder, how did you know that encryption is shanan?
You can reverse engineer some of the original code
from way back in time, which is a lot simpler,
because it's a lot less code.
And what you do, essentially, when you reverse engineer cryptographic stuff,
is that you look for constants.
So you find some magic numbers in the code.
You Google it, and you find what cipher it is,
and that essentially what led us to understand that it was shanan.
We have one more question here, but are you willing to take more questions
outside?
Yeah, yeah, of course.
Yeah, so one more question here, and then if just outside,
you're kindly carry on the conversation here again.
Hi, have you looked into Deezer?
Oh, no, no, no, no, no, no.
I only use Spotify.