Good morning everybody, thank you for waking up early.
First topic is how to get back traces.
How to get back traces?
Yes.
Okay.
Hello.
Test test.
Yes.
Okay.
So back traces for C and C++ programs.
So most of the time when we are running C or C++ program, the Toneoctor or anywhere else,
and we have an issue, then we get a sickhold, that's about it.
Can we do better than that for sure?
We can do something like that.
So with some work, we could have a back trace right in the RTDR,
or Cslog, whatever, displaying the call stack at the moment of the crash.
That's not the only option to debug an embedded system we could use GDB,
but when you have thousands of devices on the field using GDB is not really an option,
we could use columns, but columns, they can be tricky, they can be heavy,
they are dumping huge parts of the memory, which could include secrets.
So they are also kind of tricky.
And many times having a back trace can be very handy.
So today's work, we will be mostly focusing on the omg7 architecture,
or text 8.
And if you want to reproduce some of the examples that I'm giving,
then you can compile a base, the octo image for QNU,
and run it using run QNU.
Now, some simple C program, so we have a main function,
calling the F1 function, calling the F2 function,
calling the F3 function, calling the F3 function,
so the F3 function is doing some simple variable.
And if we stop there, we will get the segmentation for that we had in the first place,
and that will be it.
Another option that we have is that we can catch the segmentation for signal
to be sent by the kernel, and do some stuff over there.
And what would that be?
So before we dive in, small reminder about the Rm assembly,
so we have 16 registers 0 to 15, 0 to 12 are generic.
13 is the stack pointers, or positioning the stack.
15 is the program counter, or instruction pointer,
or position of execution.
And the link register of 14 is the place where we will jump to,
if we are entering function, and we need to go back to return.
And how does that work with the same example as before?
So let's say we are in the F1 function,
and we are about to call the F2 function,
so we will use the branch instruction, and that instruction will update the R15,
and the register with the next address that will be that one,
that will be executed once we get back from that function.
And if we go to the F2 function at the top of the function,
we are pushing the link register to the stack,
so that it's saved, then we are doing some stuff.
And when it's time to return to go back,
then we are popping the link register into the program counter,
and turns out that we assume execution over there.
And it's specific to MV7, but it's always more or less the same.
And what that means is that in the stack, at the moment of the crash,
we will have a few stuff that is pushed onto the stack,
then we have the link register, the return address in yellow,
same for F2, same for F1, and if we want to get a back trace,
at that point it involves getting more or less
the yellow boxes, but to get the yellow boxes,
you need to know the size of the blue boxes, and that size can vary.
And how do we know the size of the blue boxes,
so that we can get all of those link registers,
turns out that out of the box, we have something to do that.
So that's the F, the binary that we are running,
and it's a binary, it's not just machine instructions,
it has some sections, and in those sections,
we have the red sections over there,
which are MV7 specific, and those sections,
they contain some instructions on how to unwind the stack,
so how to get the sizes of the blue boxes.
That's specific, and then we have some stuff in green,
debug frame, section 32, which is somehow generic,
which is called do-off, do-off because of L, I guess.
It also contains unwinding instructions,
but that time, motion-agnostic.
And there is also the EH frame, which is empty on Rm,
but which cannot be, which is somehow not empty on those architectures.
We can print those ARM and winding instructions,
which is with LF, and what we have is that,
for every section of our program,
we get some kind of virtual motion instructions
that are telling us how to unwind the stack,
and they are giving us instructions on how to get our 14,
which is the leaf register, the precious our 14.
For the debug frame, do-off instructions,
it's more or less the same.
We have some kind of unwinding instructions to get our 14 as well.
If we go back to the program that I showed earlier,
and we try to unwind by ourselves,
then we can use GDB to put a breakpoints
into the segmentation for the handler,
and try to unwind the stack manually.
What will that be?
First, we can check for the program content
to see where we are at.
We see that at that point,
we have 20 bytes of stuff in the stack,
and then we have the link register,
but we can deduce that by passing the assembly,
but that's not the point.
The point is to use do-off to do that.
For do-off, we need to check our position,
which is 65E,
so that would be in that section,
and do-off says if you are past 658,
then the top of the stack,
the top of the next stack,
is at 24,
an obvious over there,
of 14, leaves at CFA minus 4,
so at 24 minus 4, so at 20.
We have the same stuff,
but that one is machine-reliable.
If we have a look to the stack,
in GDV, so that would be 0,
for 8, 12, 16, 20,
and that's the link register.
Because we have a segmentation for handler,
the handling is a bit specific,
because we have the kernels,
and in the signal,
and then it does not get to the program,
it gets to the GDVC,
which has a trampoline,
that will call that Cisco,
that will in turn get back to our program.
That's a bit specific,
but what we need to know is that,
right after that instruction,
in the stack,
we have some stuff that the kernel has put in there for us,
and that stuff contains the registers at the time of the,
at the time of the crash,
and so if we have the stack pointer value,
and the right offset,
we can have the PC and the SP value,
at the time of the crash,
and so the PC value,
that will be in the instruction,
we are going to load three at the address zero,
and that's six, seven, eight over there,
and there we can again use dwarf,
that will tell us that the link register,
that time is at 16 minus four,
and so if we do stack position,
plus 16 minus four,
then we get the link register,
and the link register is in F2,
when we are about to call F3.
If we repeat that operation again,
then we will get the position of F1,
do that again,
name, do that again,
and we have the custom backtrace.
So that backtrace,
it looks like very much the one in GDB,
so we had to work half of that one,
and that one is for F3 with scholars and everything,
but they are done in the same way,
GDB is also using dwarf instruction,
more likely to unwind the stack,
and to print that nice backtrace.
Now there is another way to unwind the stack,
which is quite popular,
it's called frame pointers,
and the concept is that the compiler,
it will use a specific register,
so it could be R7 in some mode,
which GCRN could be R11,
could be something else,
and that register,
it is meant to contain at any given time,
the top of the current stack,
so the position of the top of the current stack,
so that's the stuff that we get using dwarf.
And so whenever we are entering a function,
the frame pointer is saved in the stack,
and if we want to unwind with that,
it's fairly easy.
We need to read R7,
which will give us the position of the top of the stack.
From there, the link register is that the constant,
is meant to be at the constant offset,
so we get the link register,
then we can read the previous R7,
go back to the previous stack,
do that operation again, go back to previous stack,
and then we have a back trace,
without using dwarf in an easy way.
So if you try to do that at home,
with GCRN, in some mode with optimization,
then you get no frame pointer.
If you do that without optimization,
then you get a frame pointer in R7,
but it doesn't work.
R7 does not contain the right stuff,
and you can get about that.
It would be nice if it was sold at some point.
If we are not in some mode,
and we instruct GCRN to remove the frame pointer,
or we use O0,
then we will get a frame pointer in R11.
There, I will spare you the demonstration,
but it works as I described,
and you can get a back trace
by going from frame to frame.
Do we need to do that manually?
So it turns out not,
there are some libraries to do that.
One of them is even wine.
It has a not-so-nice API,
but it does the stuff.
For Iran specifically,
if we have a look to the backends that are supported,
we have dwarf,
so the stuff that we just saw.
We have EX and EX,
which is the arms specific stuff.
And we have the frame pointer.
For Iran EX,
it has some limitations,
so I do not consider that one in that talk.
The frame pointer,
as you saw,
it's broken in some mode with GCC,
it's working with LLVM,
and it's also broken in ribbon wine.
So how to use,
and the dwarf stuff,
a ribbon wine,
has a generic dwarf and a winder
that can be used on any given architecture.
But it was somehow broken on on V7,
and I have patched that.
So now,
if you use ribbon wine on the master with dwarf mode,
then you will get back traces.
Now,
if we take our binary and try to run it on Yachtau,
most likely with ribbon wine wine,
most likely we will not have anything.
And the reason why is that,
out of the box,
Yachtau is treating every single binary and library.
And it means that we have tiny binary,
but which are empty of any dwarf instructions.
And if we are relying on dwarf to do the unwhite then,
we are screwed.
And to keep those dwarf sections,
we can instruct strip to keep the debug frame section.
And there,
we will have a tiny bit bigger binary,
but we have the opportunity to get a back trace.
And that back trace there,
it has no symbols,
except for one,
but we will see why later on.
But it's already something.
And if you know perfectly all the positions
in your program then,
you can figure out what does that mean.
Otherwise,
you can try to get symbols.
And how do we get symbols?
They gain lead in several locations of the ELF.
So, we have the dynamic symbols in green.
Those are needed for runtime linking,
because it's done with function names.
So, if your library is,
or if you executable this library,
and most likely the symbols,
we live over there.
That's the reason why we had some symbol previously.
Then, we have the debug info section.
So, that one is again a dwarf section,
but with the symbols.
And then, we have the static symbol tables at the bottom.
So, if we strip,
what's the result?
The dynamic is still there because it's needed,
but the static section is gone,
and the dwarf section is gone.
And that's the reason why we had no symbols.
And we can work around that and keep the symbols
by instructing strip once again to keep the debug stuff.
So, keep the dwarf stuff, keep the symbols.
And there, we get that time a nice back trace,
and we'll have the tiny bit bigger binary.
Another option is to use what's called mini debugging info.
And sorry for the back entrance.
So, mini debugging info,
the concept is that we take that big binary.
We only keep, we make another ELF,
which only contains the dwarf stuff.
We compress that ELF,
and we re-inject that ELF to the first ELF,
which has been stripped.
So, we have the initial stripped stuff,
plus this compressed dwarf section.
That's mini debugging info.
And that mini debugging info mechanism,
it's supported by GDP,
supported by Lebanon,
it's supported by several libraries.
And we can do that by hand by doing what I exposed.
So, doing the OBGCOP,
the strip, the compressing, and the re-injecting.
And then, we will have slightly bigger binary again,
but if we compile with even one mini debugging support,
then once again, we get the back trace.
Taking a step back,
so if we keep everything,
then most likely we will be able to unwind.
If we strip everything,
we will most likely not be able to unwind,
unless we rely on the own specific stuff.
If we keep the debug frame,
then we will have the possibility to unwind the stack,
and then we can either keep the static symbols
or use mini debugging info to get symbols.
Let's take another example.
In that one, we are using open SSL,
and we are doing a use after free,
just before returning.
So, there, if we run that one in the opto,
we have a back trace,
and that one has at least two flows,
so first it's partial,
because it stops at the digested data,
and we come from the main,
so we should have main as number three.
That's the first flow,
and the second flow is that it has no symbols.
The thing is,
we still have this EVP digested data,
and the reason why is that open SSL is a shadow library,
that one lives in the dynamic section that I exposed,
and that's why we have that one in the back trace.
Now, the reason why we have a partial back trace,
that's because, by default, as I said,
the opto is tripping everything.
So, if we tell the opto,
please keep the drops for my binary,
we need to do that for all the libraries
that our C program might ever use,
and that's not possible in the opto out of the box,
so I have a patch in the opto
to add a local.conforption,
which is called package keep sections,
and that option is instructing strict
to keep some sections, as we saw.
That patch has never made it way upstream,
because people were confused
between mini debugging for,
or for everything,
and I did not get that one upstream,
but I see some Miyakto shots in the audience,
so now that you know about all of that,
it would be nice to reconsider,
and I take that patch or find a way
to instruct your tutorial to keep some specific sections
that we want to preserve.
On the other hand,
Yoko has support for mini debugging for,
so what we can do is that we can combine the dispatch
and the mini debugging for feature of Yoko,
and for,
so the sizes are the sizes of the core image,
the compressed core image,
and for small overhead,
we can have the debug frame and the symbols.
Of course, we can keep everything
by setting mini bit package to one,
and in that case, Yoko won't strip anything,
and then we will have everything we need.
But it depends on your case,
it could be that 200 megabytes is perfect fine,
and then you don't need to bother with street pan everything,
and you can just keep the images it is,
and do you unwind, and it will be fine.
But if you are operating with a constraints storage,
then it could be interesting to find
what street is doing,
and keep the unwind, keep the symbols,
and have back traces.
So now we have a full back trace.
We have the main function as we saw
and the rest of the back trace.
Yes.
So in conclusion,
if you want to do some unwinding in your C
and C++ program,
then you need to have some unwind information,
could be dwarf,
could be some architectural specific information,
then you need the symbols,
they can be the static symbols,
they can be dwarf, they can be mini debugging for.
Then you need an unwinder,
so it could be unwind,
it could be that you write that one by hand,
as we saw doing that manually,
it could be something else.
If you need to preserve some sections of the ELF,
then you might need to patch your tool.
And if you do all of that,
then you can get to have back traces
or C++ programs in the C++ or in the logs.
And I found that very handy to have directly
the crashes at the customers,
pushed from the logs without having to do
GDB code and anything.
I think it's a nice addition.
So that's about it for that talk.
I guess we have done for some questions.
Thank you very much.
Thank you for your talk.
If I understand it correctly,
it should also work on CortexN,
and also ARMVAs.
So this should be pretty universal.
So there's no hardware limitation,
nothing is missing,
except of this bad case with thumb and O0.
So some specific to ARMV7,
as I know,
so if you want to rely on frame pointer to do the
and winding on ARMV7,
then you need to frame pointers.
So it can involve not choosing some mode,
or you can involve using anything else
than GCC or patching GCC.
And if you are using ARMV8,
then you won't get this ARMV7 specific,
exact sections,
but you will get for sure the debug frame stuff,
which is in there by default on ARM64,
and on X86, as far as I know.
And the reason it's there by default on those architecture,
is that C++,
the catching mechanism of C++,
it also relies on that stack,
and winding mechanism.
And so it needs to have some sections in there.
And so on every architecture,
there is at least one unwinding section of the ELF,
that is not stripped,
so that C++ can do it stuff,
and an unwind in case of an exception.
And so on ARMV7,
C++ is relying on X86,
which is architecture specific,
on ARMV8,
it's relying on debug frame.
So they won't be stripped by default,
even on your window,
so that patching is not necessary.
And on X86,
I guess we have frame pointers by default,
and if not,
we have that debug frame section as well,
which is in there.
And all the architecture like I don't know.
You mentioned X86,
if I understood correctly,
you said that you had issues,
or you did not consider it.
Is it because it is too complex,
like in support, or anything?
So two issues with that one.
The first is that,
as I mentioned,
it's architecture specific,
so if you have ribbon-wine patch for that one,
find you in everything,
then you need to port your mechanism to another architecture,
then we'll depend on a completely different stack,
and that's somehow problematic,
or to me at least.
The second issue that I have is something
that is also known by GCC,
with those EXIDX.
In some frame,
they are issuing a canton-wine instruction,
and that one is preventing an winding to go further,
and in some specific C++ stuff,
with exceptions,
sometimes you have partial back traces,
because of that canton-wine instruction,
that's known to GCC,
and they will maybe do something about that,
but for now,
I have experienced issues in printing back traces,
in C++,
with around exceptions,
because of the EXIDX,
and there,
instead of trying to make things right,
GCC,
the smart move for me was to switch completely
to do our fun,
to get rid of the EXIDX stuff.
That's the reason I did not consider
much of those sections.
Okay, thanks.
Thank you.
We are out of time.
Thank you for the talk.
Thank you.