Alright, welcome everyone. I'm back. If you were here in the room before as well, I just
had a talk like an hour ago. But now I'm going to give a talk again. This time about my
summer vacation of last year, I fell into a deep rabbit hole of a reverse engineering
theme of FES and ESXI kernel. And I thought it'd be fun to show you my vacation pictures,
so to say. Quick introduction. I have a lot of slides. I didn't learn anything from
views, but it's a day. So I would go a little bit quickly. Very quickly. I work for
Steve. For 20 years now, I'm a security researcher. I do R&D, a version nearing development,
author of DICEX, which to talk an hour ago was about. And what FES is something we support
and DICEX, and something I spent quite a bit. Quick glossary. So DICEX, I'm not going
to go into DICEX. So I'm going to summarize the talk from hour ago in one sentence. It's
a system investigation network and it's impure Python and it's pretty good. You try it.
A choir. It's a tool that's built on DICEX that allows us to collect artifacts from all kinds
of different systems. EZXI, if you're not aware, hypervisor runs on the on bare metal. And
FEMFS is kind of like the proprietary file system that EZXI uses to do it like, like,
multi-host file systems. And you know, I'll file a file locking for that and stuff like that.
FEMFS 3, this will be important later, introduced directories, which is very exciting. FEMFS
5 and 6 are largely incremental improvements on FEMFS 3. And why it is important, you'll
see later. So I think this is like a few years ago, I think like 2018 or something like
that, we were already kind of playing around with hypervisor based data acquisition, but
it was very cumbersome. So basically what we would do, only in specific configurations. So
if there was a hypervisor that was connected to a storage network, we would slide in an
additional server, basically, attached to the storage network, mount the storage that
the data store was on for the hypervisor. Use the open source available FEMFS or FEMFS
6 tools to mount the data store and then run the acquired to against the virtual machines
on that data store. However, there is a slide problem with that, which is that FEMFS
tools is pretty broken. There's a lot of bugs in there, which means we got a lot of data
corruption, missing data, just straight up for the machine, so we couldn't acquire, which
is pretty bad if you're trying to investigate an entire network. And it's on my
tank, which is not good either. So armed with item, I thought like while how hard
could it be. Few like a year later, so we had like a job that was like, hey, we have
this rate set up. It's trashed, please recover my data. And it was of rate of FEMFS
data store and to be able to kind of aid my job a little bit easier on recovering the
rate configuration. I just built some tools to interact kind of like peak at FEMFS structures
and to kind of like calculate some offsets, so I can see if I got the rate configuration
and the striping right and stuff like that. So I built some special tools and kind of my
high level understanding of FEMFS at that point, looked a little bit like this, so you have
a disk and then you have your regular position table and then you have like FEMFS specific
LVM implementation that's on some offset or some position. And then within that you kind
of like have FEMFS or you have your you know your metadata files kind of similar to how
NTFS has like MFT and use engine or stuff like that. FEMFS takes kind of like a page out
of that book instead of being more like other unit file systems which all have like just
data structures on disk. FEMFS takes a approach of data structures are in files or the metadata
structures are files, so to say. And then if you interpret those, you can just refer to machines
and read your very exciting directory structure. So I wrote a implementation is the along the
very short summary of that. I was like, well, I wrote all these inspection tools, might as well
pull through and make my own interpretation. I took a little bit of FEMFS tools a little bit
of Ida because what I didn't have that much time to rob the implementation, so I was like
just do it quickly. I fixed some of the most obvious FEMFS tools bugs like they were just like
unit for the 64 and I was like unaligned of like some offset, so sometimes the like a few
bytes would be out of bounds of the field structure and then it would calculate the entirely wrong
offset for some file. And I'd have some to do this for the future of course because you know
whoever finishes all the products right away in first time. And at a benefit that we actually got
with this is I integrated FEMFS into disk target, which got like the big framework and because
acquire is built on a disk target, this means that acquire also automatically gets support for
FEMFS. And it just happens to be that if you're on an ESXI shell like on SSH or just like the physical
shell, you can actually read the block devices that the data store is stored on. The data is
local storage. The design is a whole different beast. We're not going to go into that, but it was like
local or I suppose the storage, you can actually just access the block device, read it, you can
even write to it if you want exciting things there. But this means that if we just run an executable
on the system, we can just open a block device as part of the FEMFS tools structure as
and just start reading files straight from the block device, kind of like how things like roll
copy work on a Windows as well. And I was doing the world until you open source it and someone
comes along and says your shit broken and then you're pretty sad because that means you have to
start looking into it again. Okay and I got a pretty nice you know I got a stack trace so that's
pretty good. But someone came along and was like you know if I just create a big data store
and I create two directions in there and I try to list the second directory it breaks. So that's
pretty big problem because you know 65 terabytes is not that big in the sense of of big
music side deployments. So you know we do have to take it seriously because we ourselves
will probably encounter this pretty soon as well. So I started investigating. We talked on this
word. I spent some time like setting up trying to get a reproducible case. Luckily this guy
just like really shares some of his his workflow and like what he thinks is wrong as well.
Rich aids in my investigation as well. I have some suspicion of what might be wrong at this point
but not quite sure. But I am able to reproduce the case like just a very bare bones
reproduction and it turns out if there's any file system that's just bigger than 16 terabytes
then it just starts breaking for some reason. So I started noticing this specific symptom.
Well just like that for everything over 16 terabytes it just breaks and reading like the small
block system resource file. It just starts giving me the wrong data and in FEMFS like
file blocks are allocated either large block or small block most of the things are small blocks
like directory information is small blocks but then you know like FEMDK data would be a large block
and you know I'm trying to read a directory content which would be in the small block file
and if that doesn't work or is it gives me the wrong data I'm starting to parse the wrong directory
information and the cause is who knows what I don't know yet. So but at least I can reproduce
you know I called hide between or behind you know works on my machine. So I just start inspecting
my own code read start combing through which and you know trying to find anything that might be wrong
here and you know adding debug frames you know debug frames trying to see if I can find out
what's what's wrong. I did find a description see but I can't explain it. So I start
for version engineering FEMFS again still can't explain it so I'm like starting to go a little
with deeper version engineering again and like at this point I'm basically re-implementing my
own implementation from scratch again but I still can't explain actually what's wrong. Like at
this point I've just re-version engineering and read create a whole new implementation scratch
up until the point where I can start reading directories again and it's still still wrong and I'm
I'm pretty sure I got it right you know this time looking at comparing it against item but still
I'm just I'm just way off. So what does bug actually look like? Well you know just this hypothetical
hack stump of the small block file. This is what I would expect from reading from these offsets
and this is what I get. So spot the differences. I'm just like 0X1, 6 years off for some reason
and I have no idea why. I do report back to the initial bug report I like hey I have to
determine this. I start to be re-implementing it. I still really configure out why I'm
depth many by itself. It's pretty big. So it's not like off by one. It's off by like a megabyte
really a bit more even. And I even found out that like there's some debug logging you can enable
on EZXI host as well and you can log like what the physical offset would be of a file read
and it's the same. So I'm like I'm pretty sure my calculation is correct. I'm pretty sure
I got all the math right. I got all the offset's right stuff like that. But for some reason
if I just compare my data, if I just hex dump it I'm still just so many by itself.
So I'm like I want to re-reverting near this further because I just like okay well I'm just
going to you know not stop until I 100% lay where everything in Ida. I didn't do that. I'm
not that crazy quite. But I do want to make my re-reverting near my life a lot easier. And having
anything with symbols or debug info is called when you're doing re-reverting nearing. So I often
just try to find you know scrap something together for public sources. And one of my favorite
things to do is use five-stotal retrohands and I just copy pay some strings, run in strings,
I find in binaries and I just run a retrohand and see if I can find anything that has any
reference to these symbols or these these functions. I got a few matches. Very exciting.
Unfortunately the first one is myself. The second one is myself as well. But the third one
the third one is interesting because that says it's an elf. So it's a shared library.
So what is this file? This magical file which has a reference to this string. It's an elf
32 binary compiled with gc2 which is old and while I was making these slides I was pretty curious
how old is that. It's old I mean. And I noticed so on the bottom right you see like some strings
that were in this binary and they were in sections named dot-step and dot-step. And I was like
what is this? And I was already half way implementing my own parser for this. First try
throwing into idah or vidra unfortunately they didn't recognize it. And I was like already
making some Python code to parse this into some some some readable format turns out
this is like a stabs debug format which to my knowledge is pretty old like three days
12 but you can still dump it with object dump minus g and you get like a nice sea header file
as output that you can just you know inspect or throw through do whatever you want.
The binary itself turns out to be some random binary from e63 installation so it wasn't called
e6i back then it was called e6 and it was like an old rpm based version of e6
before they had their own fancy kernel and they were actually like Linux based and they just
shipped a whole bunch of rpm that they installed on the Linux based and it turns out that some
of those old versions actually shipped with debug builds. So the most same thing to do after that of course
is to start hoarding e6 installation media and start figuring out what is the range of versions
that actually contain debug symbols. So this is my target and I with my three greatest friends
five total instant archive and Google good working I tried to collect as many
installation media as possible. I get quite a lot quite a lot of all versions as well.
This screenshot is like it doesn't quite do just as how much time this actually cost
probably took me like a week maybe like we can have to actually collect all of this like find all
of these things. So many of your rails just broken don't work anymore but I got quite a lot.
I also while doing all this searching for all these symbols I stumbled upon
three interesting well two blog posts and one interesting presentation which specifically talk
about something called VMware Workbench and I mean the title says gets symbols like I want this.
I want to get symbols as well but you know then it also says authenticate which VMware use
credentials you know download the VMware Workbench okay okay okay and then you can just have an
RPM with with with symbols and like this is great I want this as well but then you know there's
a screenshot as well what is what look like and you can see like VM kernel like all the builds
like 620 debug symbols and fade away it is great but unfortunately reality caught up to me
brought home you know acquired VMware and well everything's down you can't get any of this stuff
anymore ever it's unfortunately lost and to devoid so this got me pretty sad but I tried emailing
everyone involved in these in these blogs as well and fortunately the author of the ZDI blog
actually had a copy of one of these files at least or like 203 and he was able to pride them to me so
I'm very very very gracious for that so the TLTLDR turns out that also ESX3 all binaries included
debug symbols so like to stab debug format and it turns out that VMFS3 was introduced in ESX3
and because VMFS3 is the version that well like supports directories like it's the pretty
big rewrite of VMFS but fortunately like VMFS5 and 6 are largely incremental improvements on
a VMFS3 so I can still use these symbols to get a pretty good base understanding of how
VMFS works onto ESX3.5 there are debug versions of all binaries so not quite symbols but still we
get debug assertions a lot of debug prints not really helps out in a version game as well
the data that was lost on the VM workbench kind of like fortunately I found out in the end
it only contains like the kernel so like the base kernel binary and you don't get any symbols
or debug versions of any of the kernel modules and VMFS implemented in a kernel module so I
wouldn't even really had that much at a benefit from having these but from the one I did get from
from Reno I was able to get like a few slightly more modern like constants or structured
definitions but only like one or two instead of you know the gold mine that turned out to be the
audit debug symbols so you actually get quite decent symbols from this you get like four
unions as well and this just made refer to engineering so so much easier because instead of having
to guess you know like what the layout is or or an idea you know how how some instructors are
are referenced I just know that you know a disk block is based the union of all these different things
and a file descriptor you know has like a lock block a mena block and then a union of whatever comes
after that and that just makes all the very engineering so much easier and making your own implementation
so much easier as well so armed with this new knowledge turns out so I kind of like tried to
look how much symbols I can actually use so my approach here was to just take a more recent version
of the binaries so I got a recent version of LVM, recent version of the MFS and then I just try
to kind of like match the symbols from the old versions and like have a kind of like side by side
version of the engineering so I do reverse engineer like the most modern version of the MFS but
with the aid of like having a debug symbols of an older version but I still still do a lot of
reversioning from stretch of course but luckily there is this file if you open it there is a Python
file in which there are like there's like a Python file which has some structures copy-pasted in
there from like the the file system header it's not allowed but it does help a little bit with
finding out some newer fields on newer version of the MFS. Long story to you know I thought
this talk was about fixing a bug so I in my completely implementation I was only focused on
the MFS and when I started you know when I finally had all the symbols I started with a version
of engineering everything using the symbols I was like hmm I haven't really looked at LVM yet
maybe the problem is there and so I started reversionering LVM and suddenly I noticed something
in that implementation and I took so many shortcuts in the LVM implementation because that's kind
of the part where in the beginning I said I took some from theme festivals and I basically took the
LVM implementation of theme festivals and that's I I shouldn't have done that I should have
looked a little bit closer into how LVM works because I would have saved me a lot of time here
so I kind of like rewrite LVM as well and as a bonus I actually support like great
configurations of of theme festivals now too so what's the bug this was like my previous understanding
of theme festivals turns out in LVM you actually have a bunch of extents with wood and
you know a gap in between and then theme festivals the file system layer is kind of like overlaid on
top of that so it is immediately explained like the gap or the offset I was wrong with I'm reading
off as files so what I'm still doing in either this point I fixed the bug you know I could have just
made a quick fix in my old implementation but I was like I would just see this through I'll just
finish the entire thing and I rewrite everything from scratch take me about a month a little bit
more maybe but at least this time I know I have everything verified across my own
version engineering instead of you know like basing it off of other people's research not to
disrespect that of course like it's so it helps so much in getting your initial understanding of theme
festivals but you do see like a lot of shortcuts taken in the other implementations
so the fruits of my labor is that in the end I wrote a bunch of code and it got merged
that's good some poor soul had to review this unfortunately and so what's next I'm finally
done not quite so I'm kind of like working on a blog that's report on theme festivals because
I think it's a waste to keep like the knowledge to internal knowledge of actually how theme
festivals really works on at a hoot to keep that the log in a pie file and you know some
either databases it's not doing it much justice so there will be like a bigger report on like
exactly what are like the forensic implications of the MFS exactly what are the data structures
how are they how do they relate to each other stuff like that in the future and that's it there's
no next slide
the question was I use item could I have done this with any like other open source
the compilation tool yes I just personally prefer item is what I used a lot before for example
Keter I was open source and before like primary ninja I was quite as good as it is these days
and I've kind of like just I have my muscle memory in item so whenever I need to get something done
quickly I just were like big I'm just more comfortable in that item
no there's stickers here if you want to buy a way okay thank you