And I'm giving this presentation with Eric, and we're going to talk about Puma, which is our tool to automate
abactions in Android.
We work at the Netherlands Forensic Institute, located in the Netherlands in the Hague, and we're part
of the Ministry of Justice, and within the Institute we conduct forensic research in various
disciplines.
So, for instance DNA research, all the motive, and also digital forensics, and we're in the
landscape department, and we write parsers for file formats.
And in our work, we came across the challenge of generating reference data for our parsers
to test them, and manually it's a big challenge, and we're moving towards automating
it.
So, what is Hanskun?
It's good to note that Hanskun is not open source.
Yes, Hanskun is our digital forensic tool to investigate digital traces, and Hanskun
can process multiple devices at the same time, multiple types of devices, such as phones,
and laptops.
And Hanskun parsers, the data from these devices, and also the output of other forensic
tools, like cellar bright.
Hanskun parsers this, the data and converts it to traces like contacts and locations, messages,
and presented for instance to the users in a searchable way.
Hanskun's used nationally and internationally, and it's used in big cases, so we support a
lot of formats, and we need to make sure to keep up-to-date with updates of those formats,
and we need to validate if Hanskun parsers them correctly.
So that's why we make reference data.
We need to verify if our parsers work as we expect, so we can both generate a data ourselves
or acquire it from the internet or from other people, and we actually prefer to generate
it ourselves.
The reason for this is the ground truth, we want to know exactly what is in it, because
only then we know what should come out of it, we want it to be complete, so if we generate
reference data for applications, we want all features to be represented in it, so like
what's up, we want to send a message of photo, etc, and we need to comply with GDPR, we want
to make sure there's no personal information in there.
Why wouldn't we generate it ourselves in practice, we do a combination of the two, because
we cannot foresee all edge cases, and real-life data, we always come across some situations
we did not think of ourselves, it's not real-life, so what's that data base of a victim
spends over like 15 years, and it does not really compare with the database we made in one day,
and it's very laborious, it's not scalable, so it takes as much time to do it a second time,
it's the first time, and app updates are a real problem, we're to illustrate this, we have
our fictional owl app, let's say version one comes out, we support it in Honske,
and then an update comes out, version two, and it breaks Honske, the database scheme,
and I'd be updated, and Honske does not expect this, so what happens then,
well, nothing, because we don't know about it, at least not until some time,
then there's a crime, the police sees the device of the victim, version two is on it,
that version three comes out, version four comes out, and it's still broken, and then
it's noticed by an investigation start, and it's noticed, and the back is felt, and of course we
will try to fix it as soon as possible, so we do this for version four, because that's the
version we can get from the place to all right, we cannot really go back and test all versions,
so we still don't know if version two and three are supported, because there might have been
another change in the meantime, and then version five comes out, and we might support it,
but we don't know, applications are very volatile, because for what's up, we know from our own data
that two versions a week come out on average, and we can just not keep up with that,
and the old version might not be available anymore, and even if it's available, if we save the
APK, the server might force us to update to the newest version, so I think I've made my case that we
really need to automate this, so that's what we did with Puma, this is open source, we found out
we talked to people, we find out that a lot of people in the forensic stone main,
yeah, run into this problem, we also by version of APK on it, and Puma aims to
do to automate writing these scenarios before we did it by hand, with all scenarios who we knew exactly
when we sent a message, now we can just do it automatically, it's written in Python, so it's easy
to integrate, and it's built on top of APK, APK is a testing framework, it's used by the developers,
and we use it to automate the actions, so this is basically how it works, Puma talks to the
APK server, APK installs an application on the device, which hooks into the accessibility framework,
and that actually runs the actions, so bit more on APK, it works with production code, which is really nice,
you don't need root access, and it works both on Android and iOS, and with APK you can see everything
that's on the screen, you can interact with elements, so tap an app to open it or send the send button
to send a message and perform actions like tilting the screen and spoofing a location,
so if we have APK and why would we need to, why would we need Puma, this is APK code
to just send one telegram message, so it's several steps, we need to open the app,
then open the chat with Alice, insert the message into the text box, which we first need to click
and then click the send button, and this is fine for just sending one message, but if you have
all scenario, it's too for both, and it's not really readable, so if Puma, you might actually
wraps around this, and it's just one line, it's one initialization and one line, so you can just say
Alice send message to Bob, so the features Puma now only works on Android, but we aim to also
support Alice, you can use multiple applications so you can install an applications and messages
with WhatsApp telegram, also drive routes with Google Maps, we have ground truth locking,
generic for all applications with locks when an action is performed,
location spoofing, we will see more of that in a minute, generic pop-up handling, so system pop-ups
like you need to give this app permissions, we can just, and let all them medically, so it doesn't
break or work flow, and we have some spend some time into making the UI navigation robust, and we'll
also tell more about that. So now that we know what Puma can do, we will actually
like to show you a bit what it looks like, so this is a full scenario, and it contains of
it uses multiple phones, and also the phone can use multiple applications at once, so at the top,
we actually, in the initialized telegram, and we actually will first send some messages,
then just start around, send some more messages, but well, why tell you when you can just show
you, so these were just two physical devices which were laying on our desk, I promised we did not
touch them, and we just set them off, just if you start using it, just lay it on your desk,
it's very fun to look at it. We start off with just sending some messages, and as you can see,
we're also not typing the actual messages, we're just sending the text directly into the text box,
and so so messages pop us going to work, he landed just on Skipple, I don't know why you would go
directly to work after that, but let's go for it. So we start around from the location, he is at,
just me, we were just at the NFI, we were not on Skipple, at the at the time, and we'll start the
route, and of course, when you're driving, you should always adhere to the speed limit, so
drive 50 kilometers an hour is fast enough, but Bob deployed something to production just before
he left, and Charlie noticed something, the servers are a bit on fire, so there might be a
problem in the office, well, what do you do? Well, you don't tell somebody, you just take a picture,
that's the work, the way the world works nowadays, so when we send the picture, and Bob
receives it, he takes, oh no, I made a mistake, maybe we should start driving a little bit faster,
so he starts driving about 160 kilometers an hour to just get back to the office a little bit faster,
and if you don't believe me that we were not actually driving this fast, you cannot drive this corner
at 160 kilometers an hour, which also when you start to speed up, look when I was at the
bit of a trouble because it tries to interpolate how fast you're actually going, if we were to stop
the spoofing, immediately it will still keep driving for a kilometer or so, and suddenly
jump back to the beginning, Google Maps is not that smart. Now we'd like to talk you through a
couple of use cases, so you can probably think of many more use cases than we can, but we will talk
through three use cases that we actually use, and the first one is to initialize a application from
scratch, so when do we actually do this? Well first off, when we start researching a new application,
so we don't know what the application does yet, so we just want to figure it out, and then we
met all the features that we can think of, like setting a message, setting a picture, sharing a
location, we implement the diploma, and then we can actually run a sketch to make sure that we
have all the actions linked to the trace that come out of it, and the second one is we want to
create a large data set, because repeating things is not something we would like to do, so just
write it in one line, that says, I send a message, then send 500 messages, and you have a nice
full database. If you would do this manually, well it takes a lot of time, trust me, we've done it in
the past, and well you can also make some mistakes, and then you have to write down that you made
a mistake, and then you just look stupid. With Bouma, it's very simple, you just run the script,
and then supervise, but one way, with the script, my question with no where it happens, and you
also meant to make sure it does every step that you set, it should do, but also, as again, it's just
very fun to watch a phone, do things when you're not actually touching it. Of course, always some
caveats, well it does take time to implement something in Bouma, unfortunately, every development takes
time, UI change can break Bouma, where your heavily reliance on actually where the button might
be, or what the button is called, so when something changes, we also might need to change something,
and we don't support actually creating new accounts. Most of the time, which is create one
account manually, and we use that account when running Bouma. Second use case, monitoring app updates,
in this case, you already have a application that is populated with data, and then you can just
update the application to see if there are changes in the database structure, for example.
So we actually have a pipeline for this, which we start always up with, no, downladying the new update,
you can just down it from the website in case of what's set, but we can also use Bouma to
get it from the app store. We installed on the populated device, ready, and we just start the app.
So we just still Bouma update the app, start it, done, then we extract the database, give it to
Hansgen, just see what it's a bit out. We validate that the traces, but we found are actually still
the traces that we expected to be in there, and then we can report the results back to ourselves.
Hey, we still support this new version, everything is still okidoki. This is useful for applications
where the database actually changes when you update the application, which we know is the case for
what's set, where this is not the case, is for telegram, here you actually need to perform
some UI actions before the new database structure is used. So the first step is still the same,
you just update it on all the device, but now we actually need to execute some actions with Bouma,
and also here, make sure that you log, which is actually happening, which is because you need this
in a few minutes, and the same steps, you just extract the database again, feel it to Hansgen,
validate the fundraising report result, but the validation here is a bit more difficult,
because you just execute new actions, so there are new traces in your database, but Bouma can
help you with that, because it knows what it did, so you can validate that it still did. What it did,
and this is useful when the database structure actually changes when you use it,
which brings us to the next point, there are some things we noticed when we started
automating this pipeline, and the first one was, well, telegram likes to update this blocks,
it just stores all the data in the database, but it used a tiny large object,
and what we could see here is, well, there are two messages, which still look to be in the same
spot, but the magic of the block has changed, which might not be a problem with every block
was a message, but some blocks might be a friend request, and if you start parsing a friend request
as a message, things might break. The next thing we noticed is that the WhatsApp scheme has
actually quite stable, so in the three years we are monitoring it, there were two changes,
only one was very minors, some apps and timestamps have changed, and the second one was
actually a major scheme update were also the message stable was changed, well, what we also noticed
there was that was a staged rollout, so when the new version came out, there were new tables
in there, but when you send a message, it was still in the old table, but when you created the new
account, the new tables would be used, which makes it a bit more difficult, but at some point
the data was, what's my credit to get a table, but it was not finished by your version,
it actually was linked to the account of the user, in conclusion, and I think I need to speed
a bit, so you all know this, well, what is this? Well, it's just very useful for us, yes, I see,
five minutes, as I said, need to need to speed a bit, so the initial app just saves a lot of time,
we don't like to manually enter text messages, there are computers for that, but there's also
means that we know when something happened, we know within 24 hours if we still support the new
app version not, and we have, well, these are a nice reporting way of doing it, it's just a read me,
if it works, it works, and just a fun thing for us, we actually now support telegram faster than
celebrate us, this is a high potential, not just for us as a fully color person, but also for
researchers, educators, and we hope also for some hobbyists, maybe you can automate your grocery
shopping with this. There are, of course, the caveat, well, it has more time,
it's not as if the government is looking in your phone, but if you want a complete forensic
image, you will actually find that Puma has some traces on there, because it does use any
of the commands to actually connect to these applications, and be mindful of the stage well
at now to the fun part. How does it actually work? Well, a Puma makes use of a state machine,
just to make sure that we know where we are in the application at all times, so we have split
every application into three parts, there are states, transitions, and actions. So every screen
that you can see is most likely a state, so in this case you have the conversation state,
and in this case this is the initial state. The universal state is just the first screen that
that opens when you open the application for the first time. Then you can use transition by pressing
the new message button, for example, when you go to another screen, well, that's just not a state,
and you can have a transition to the new message state, and you can have a transition back to
the new message state, but if you actually create a new jet, well, then you go to a jet state.
And from that jet state, you can off, of course, all the transition to other states,
like you can call somebody, you can have some jet settings, you can send media or send media
from a gallery, and Puma can just transition between all these states automatically. So each
action that you have, so what you actually saw in the script is linked to a specific state.
So if you are in the conversation state, you can just say, I want to send a message from my gallery
to Bob, and we'll automatically transition through these states, and send the media from there.
If something goes wrong during those transitions, it actually checks,
am I actually in the state? I'm now expected to be in, if not, it will try to detect which state
it's actually in and continue from there. And if it cannot find out in which state it was,
it will just reset the application, then we know we are in the initial state, and it will try again.
This will, it will only do that once, just to prevent some infinite loops.
But the nice part is, well, when you open the gallery, there might be some
permission, Bob perhaps, it can handle those automatically as well.
Some applications spoofing always nice. So you can just set your location at a specific point,
which is very useful if you don't want your friends to know you're at. Your home, if you ask,
oh, okay, loud no, I'm in the middle of the event Ocean. Very nice to prevent some of social
in the interaction, but you can also navigate as we just saw. So you can either provide a GPX
yourself or you can just call open street maps, which is we've actually built this in. You can just
say, I want to drive from Skipple to identify if we will request a route from the internet and
will just start updating automatically. You can also just adjust the speed, but also just give
it some variance to give it a bit more realism. You say, I want to play 50 kilometers an hour,
I want the deviants of 500 hours an hour, and it will just fluctuate a bit between that.
Well, now for the slide, you never saw coming, please contribute. Welcome to open source.
The first of all, if you use it and you notice that anything isn't working, just please report a bug.
It helps us a lot. We can keep up the date with it. If you want anything just at a future request
or if you're a bit more technically involved, please add something yourself. It can be found
in the difference in suitors, Puma. And if you want a bit more technical in the information,
we have also written that paper about it is on the slide here below, and the QR card will take you
to the GitHub repository. Thank you very much.
What minute? We will be here to hold it.