We are going to go to a couple of bugs we found while working on this
5, how we went through looking into the bugs, how we fix them.
Some of the techniques we use aren't specifically risked 5, but we thought that would be relevant
to the topic, so this is a bit about me, so I'm a senior engineer and further work code
to think since 2012, I work on mostly Linux, QMU type things, so nice of a context,
so there's an introduction, this is going to be a tale of two bugs and I sort of threw
a bonus one in, which is similar to one of the other ones, while I said in the description
we were going to do is out the others, we are going to use the bugger for one and I promise
no life denies because we know how all those go.
We also learn, hopefully, a bit about the risk of our architecture, so the first one, I think
this is chronologically the first one we found, is a kernel user access bug, so we'll start with
the bug report, so if you know that there is a testing system called XYZ box that goes through
a number of different architectures, it's an automated system, which looks for bugs, so we've
like things like stress testing, and we got this bug report on the mailing list about a user space
process causing a kernel option, obviously user space process shouldn't be causing kernel
option, so this is a kernel stress test, I have a little bit of the first part of the bug
report of the ups there, and I'll show some more about this, so the kernel actually provides
quite a lot of information about what happens when things go wrong, this is not the old days
where you'd be lucky to get the registers and then possibly some weird guru meditation fault
and you really didn't know what was going on, at least have now some good inbuilt the next
debugging, so this is output, so it starts with at the top, well it's telling you what happened,
so, and this is a very specific error report, so it's not like it tried to access the memory
that didn't exist, this is the kernel trying to get access to user memory in properly,
we get the information about what CPU it was on, which process was running,
we get some information about what hardware, so this information is better easily passed around to
people to say, if you're looking into what was going on here, we get an information about what the build
was, and then after that we can see information about where in the kernel the code was running,
so the EPC is basically where the process of the process of core was executing, so we know it's
in schedule tail, there's a return address that tells you where it came from, so you can look at
how it was called, and then we get registered dump which I've sort of curtailed for space reasons,
and we've got some very nice projectors here, but even that I think people will be strained
to feed anything lower, and we have information about the process of core status, so that tells
you a bit about what the hardware was reporting, such as status bits, the actual bad address,
and a cause info, so you can go and buy it into the hardware, I say manual and go,
this is actually what it meant, if you would like to, you get a call trace, assuming that information
wasn't manual, because occasionally bugs, if you call into bad code, not always, you would
get that info, so you get quite a lot of information, so actually can we go from this information
to actually go through and work out what happened, so the basics of this, well this is a very
specific type of bug, so if people know written device drivers, the curve is not meant to go
randomly wondering through the user memory, this is obviously usually a sign of some problem happening,
so viscify like some other architectures has protection for that, so the kernel has specific
copying functions to access user memory, there's checks in to check that that's properly used,
so for example the kernel has a copy to use a user get user, those are where the kernel should be
getting its memory, so one thing we can think about for this bug is that somebody used a bad
pointer somewhere, so for example just to go into this feature a bit more each CPU call has a
status control status register with a flag called SUM in it, SUM stands for permission to
provide a user memory, when this flag is set, obviously it allows it, if it isn't set then you get
a specific fault, so we can also say that this fault isn't looking at a bad memory page,
because that memory page must exist to be able to notice a user page, so it's not a
suddenly passed a pad pointer in and the kernel can see it at all, so we have the basics, we know
that how this bug is happening, the report also gives us the kernel getref and config used to build
it, so we can go away, build the same binary, so we can either look into the binary code,
make a test system around and rerun it, because one thing we haven't really known here is
whether this is actually reproducible on real hardware, could it be a bug in QM, you it's self,
I mean we haven't really said looked into whether that could be the case, however we were going
to do a reproduction case on say I think it's going to be a Visc5, one of the sci-fi freedom
machines, but I don't think we ever actually got around to it, because when we started looking into
the code, we decided to look into the code, we can read the assembly output, it's not always
possible to reproduce exactly the same binary, but we can fairly hopefully get something fairly close,
so we being of old school type of debuggers, think that you know what, that's not even try and
on this case the bug was so intimate, which is another thing that we can say is information into
our bug hunting, is that we're not really going to get a lot out of attaching a debug at this,
because it's happening every few hours, we're going to be looking for a lot of data, so we're
going to go firstly, we can disassemble the code because we obviously love our Visc5 assembly stuff
and we can go and look at the code and say actually what's happened here, so we can take the bug
with port and go back to the code, so we're going to use the new option, we can dump the code out,
look at roughly where it was, so we've used the output from the upst to go and look at the
schedule tail function here, obviously there's a lot of information we move just to make this clear,
so I've highlighted in the bold area the UDAC access code, just as no the K config allows you to add
a lot more debug information into the objects, so if I default I don't think it turns that on,
but it allows you to do things like cross reference, these instructions to roughly where they were
produced in the source file, so in here we have a schedule tail is very simple function and the only
place it actually looks to go into the user is if this current pointer is set, so at this point of
set it's doing something when it schedules and on the code here we can show that that's the area
of code roughly relating to the put user, so we've read through that and go that's fairly simple,
so actually what is this code doing, so I'll go through a little bit of actually what the code is doing,
so we can actually work out whether the code is right or not and actually then
okay is where we crashed, where we think it is, so first thing we load the s2 register with
the sum bit which we've got very, very, very far from the iSeries correct, I mean if this wasn't correct
really we wouldn't have a working system at all, well so going back if the putman get user code
was not correct in the first place it would be very unlikely we ever get a working system,
so much of the kernel relies on this that you'd be lucky to even get to a in-it process,
so we'll go through a little bit about this segment, so as to the sum flag the CSRS instruction
we're just going to set into the status register to ensure this bit is set, so we know fairly
much this bit should have been set when the code happens and this at store word is where it crashed
now, obviously that correlates with it's storing something, we know that it was storing a value,
doesn't really matter what the value was because s1 is the point to where it's going to store and we
can always we can verify that as well with the crash saying that's what we look like and then at the end
it's clearing it's clearing that flag again, wax fairly simple it should have worked,
if s1 was wrong we should have did it got on a different page fault, the page fault we're getting
is an access problem, so but that shouldn't have happened because that bit should have been set,
but what's the bit in the middle? Now I didn't put the actual code mapping in here but there is
definitely something else going on here, this is not a simple set a bit, store a value, clear a bit
that really should have happened for the put user, so what we got here, well actually what seems
to be happening is that the compiler has showed our input, our put user code, the call to it actually
get the task process ID, so it's the call to task pit number and s, which you think be fairly
simple, right? So that shouldn't mean a problem, but there's clearly this is the only opportunity
we have unless our call has magically gone wrong that we could add a bug, so we can read through that code
and I'm not going to go through that code, but it turns out it actually does quite a bit and we already
discounted the bug for being the set childhood, and we have only the task pit of v in our
comment call, which calls into the kernel's pit code to go what is the process ID of this
task pointer because the kernel has these task pointers that say this is each representation of a thread,
so it needs the actual number to give to user space, but there's no sign in there that there should
have been a change to this flag, in fact there's no, no call to that, so this way it reflects
so what we could do next, well the fairly much easiest thing, we'll be just just wrap this call
in something that says on exit to this call is the SUM flag still set and you know what occasionally
it's not, so what's going on here, well we did two things firstly, really the only thing that
we can probably assume it's happening here is that it's a task it might get scheduled out,
scheduling, it doesn't happen on, so with the preemptible kernel and various things, scheduling
places, so we can make a very good bet that it's task which, but task which you shouldn't
un-set that sort of stuff really, so actually the first thing which was really easy to do,
this was about 40 lines of code, was we started another kernel thread which went, I'm going to just
sit here, sleep a little bit, check for some flag being set because it shouldn't have been and
every so often you get this little thread would inherit some flag from some way, so we've gone through
we've written a test case or two, we've worked out this, so we're fairly sure now that the problem
is with task switch, so the risk 5 switching from one thread to another somehow has completely failed
to consider that this flag ever needed to be saved, now we're fairly small kernels, you don't have
a lot of code for these segments of say doing a process look up, but things like the S-wise
edbot use kernels that run with a lot of checking involved, it turns out the more things you add
into the say the RCU etc, parts of the kernel, the bigger the code becomes and therefore
the highly likely that it will get scheduled, so this was something that hadn't been seen
hadn't been looked at, so how actually would we go about to fix this because really it's not
going to be very interesting if we just leave this bug around, firstly this is stopping tests running
because obviously if your test is getting with an ups that's not very good test and it then fails,
so first idea, well thus switch code is available, just save the sum flag into the task info,
that was actually I think the best way of sorting this out, turns out everything that's arguing
about this sort of code and it got sort of sideline and I think it still hasn't been fixed,
this could actually be a problem down the line but no be has run into it yet,
there are places in the kernel which use quite wide access to users, so they say explicitly
I'm going to do a lot of movement of memory to use a space or from enable it, do things
disable it explicitly, whereas in these macros, put user is really just enable user access,
put the thing which is probably going to be a word, byte or whatever, disable the access,
so the quickest fix here is just to reorder the macro, so this is the patch that actually got
accepted for this, it's fragment of it, so the first thing you do is move explicitly say a temporary
code vowel, it could be value but it's just as good as the x being the argument, really people,
anyway that's forced it to be evaluated before and then we use that value in the actual
putting a user and we can verify that by going to look at the code again, I'm going to show you
the code because it reordered it, it moved that outside the area of checking, this actually
generally fix the problem pretty much because we now only have a very small window and occasionally
these code get called with the preemption disabled so it can't switch anyway, it's also pretty good
for everywhere else because we could have also fixed a number of other issues where code code
could have been called without the user space access being called because who knows where else this
was, I was too lazy to go and find out any of the places but obviously if we find one there may
or may not be others, this is sort of these are sort of bugs that happen because people don't
think exactly how things get ordered and the compiler still may well be within its rights to be
ordered this, I'm not a compiler expert and having had attempts or to looking into things like
GCC that hurts my head and I would rather other people sort that out for me thank you very much
so a quite little bonus side investigation I did and this is a project we're doing at the moment
because we're extremely strange in weird people we've been trying running risk five big
end-ion I submitted that thought but wasn't accepted so I'm gonna get it in here
so we actually have a project on our website I think we're doing a blog post about it
we have pretty much a working system running risk five big end-ion but as part of that we went
for a few issues we weren't talking too much more about it but the in it code was failing for
certain things were failing we've stopped frames going weird now we could have gone for this
idea bugger again it wasn't too early in the system but we can again go through the code we know
that the thing is failing it takes a bit more time on this one so I haven't included the code
like two or three pages so what was happening is because risk five runs instructions always
little end-ion because who cares what instruction end-ion is you have you're only really
carrying out data it turns out people like hand-assembling instructions because who doesn't so apparently
the pause instruction went by swapped by data it turns out to be a valid stack of
movement wow that was not fun anyway that was a complete matter side quest on things
as you can see the sometimes you say you can go do things with or without a debugger
we will talk a little bit about tracing but again know that's sort of things that are really useful
for long-term problems so our second one a user in legal instruction so I'm told FDSDA is actually
meant to be called free desktop SDK we put in some effort on free desktop SDK help things like
build we have some runners build fails with sigil now this is repeatable so
this is a case where actually you know what let's go with debugger gdb is actually sort of something
especially when you can get a favor of people problem so I've seen the log got the log
last bits of log from the bug report you say bin bash well why bin bash is a dying but illegal
instruction gets we run help build of stops repeatable on most architectures however some of
the runners are not showing this problem but we can repeat it we know we can repeat it and we've
repeated it on our own hardware so we run that job under gdb backtrace is at least useful in this
case we actually get a backtrace we know what the problem is sigil okay so it's in something called
ABSL which is live absile we can dump the really intensely unuseful function name thank you for
C++ for mangling the hell out of that but it's actually a speed cycle that's an instruction
that gets into the register A0 the number of cycles the core has run since I think it was reset
obviously if it can disassemble the instruction and it's known to have been good
why are we getting the illegal instruction from it well this is where my colleague would let me
ping that for you this is why I don't try and talk to him very much but your favourite search engine is
actually one of your friends here because actually this is an ownership searching for it
actually finds a really good bug report as well so the problem here is that read cycle
like on a few other architectures where information like this is a security problem well
since it's 6.6 this has been disabled now security reasons I won't go into all the
sort of security problem leaks that happened from this sort of thing but it's been disabled we can't
use it now why live ABSL is using it to read the time is another matter the number of comments in
the thread about the bug report may show a few slightly less favorable feelings about this
but we have get time of day and we have plot get time they're fairly good these have been officially
available since probably the time of Linux 4 via the kernels quick fiscal interface so
why have the baby settled does this is another question anyway new in the baby settled stop this
because obviously they've realised this was fairly silly thing to do I mean I'm all for efficient coding
but that isn't in great example of it so this is a really easy thing to fix there's two things
we can do here and since F3 desktop SDK we can control the build because it all declaratively built
just replace the tank thing use the new version if we couldn't have done this we can use the
SPI PMU driver that has assist us control which was not intuitive so I didn't actually have to work
to be unable to use a space because if you don't like security and you can control your machines who cares
right so we fixed that good everything runs again until it crashes
right so further discussions on this problem go what we must have another bug
suddenly posted this is what my runner showed in the demessage
okay so that's Warhex so as humans I don't like Warhex very much
and this it basically is the program data so the program instruction data showing hex the current
program counter it was there I can't be asked to run the two-hour job to find out what the hell is
going on you can turn this into some nice little text and because I don't like decoding code
from hex and we think we have better things to do just feed that into canoes so we can use
use the same upstum before and it's there again so I want my back with have it it was the same bug
I don't think I've got any pain for it but no it's the same bug in the same library
because GRPC includes it's own copy of liberally yourself she's really guys
anyway the fix there was to go and change the build again to tell GRPC to use the user
the commonly local copy so after a day of shouting at people we actually got the builds going
again which is great so we had some builds obviously the actual course of the issue was upgrading from
the runners from 6.6 at some point some of the runners may not have ever been updated properly
I failed to check what exactly all of them were maybe some of our architectures don't
obviously one of the disabled mental read cycle not sure
however because we are absolutely insane people I went you know what this should be too
difficult to fix the hard way because we're in slightly weird we can actually add a check into the
illegal instruction handler so the kernel the low level code when it gets the exception calls into
some C code that obviously in visc5 kernel is called due trap in illegal it gets past some
registers about the state of the code and obviously this is code either goes through and goes
what you did was incredibly silly and prints your instruction dump and oops or
delivers a signal or whatever to your process and but we know what we could actually check
whether that bug was read time and emulate it only issue nothing natural is being done this before
so we'll do a little bit of a quick run through what I would do it also seem to be a bit of a hack
so I did actually put this on the Linux viscv kernel list and people were either shocked or
went what an earth if you'd done that for that is probably a reasonable response so if you don't
know how instructions are specified risk life like most of us has some sort of way of packing
bits out say these are bits certain bits do this so each subset of bits say you have some sort of
operation code and then what that operation is doing for a data obviously people generally don't
do much for this it's usually compilers for you and so just as a done we we're looking at the
system group which is the said ICR extension which gives you a whole pile of ways of getting
up control the status registers which reads cycle is one of them as you can do this CSR
read gets your register so if anybody's interested this is what I did roughly there's a bit
more because there are things that we were missing but it goes well is this instruction that
was fetched a system instruction if so we can then go and decode some more information from it
so there's helpers in header files because obviously some people need to look at instructions
for other things like if you're trapped so you can go through extract we can say is this our CSR
read and set if so we can look for access to the cycle register and because the instruction
the instruction traps a design to allow you to restart if it had to fix something up we can just
basically say into the registers that we were passed from our task set Rd from the CSR time obviously
we don't want to use we use CSR cycle because whilst we could get that from the kernel that
would be just leaking the instruction like leaking information we didn't want to lead
CSR time is obviously apparently a better way so we can just move the EPC register on to where
it was return so just say this we ended up introducing this a bit of a support maybe someone
will get merged there was no actual set register there was a get there was no a way of extracting
the one of the subcoats and that and obviously we had to do it a bit of messing about with instruction
so but it worked we did like some test code for this as well there was obviously you would like to test
so we do a whole pile of test code the result was as I said we sent that to the list
nobody liked it now that wasn't entirely unsurprising it was more of a proof that it could be done
I was bored and it would be you maybe useful for somebody in the future who might actually need to
do this I bet to fix up some legacy code maybe something in the future need sorting out
anyway it's going to be left out there people could have looked at if they like it is entirely
to the thing it was about I think with the test code there's a couple of days of work
so the conclusions right so did say about that so in really in conclusion some of this
great five minutes yeah padding this out of the extra bug we got that like put
so conclusions this may not be as easy as just using a debugger but for cases where
you have some really difficult bugs it can help as a different way it's very old school this is
how I would do it back in the ages when say the old acorn stuff let me go
but it's really useful to look at sometimes it's just useful to look at the code and I'm trying
to understand what it's doing in this case in the case of the user access bug it was very clear
that the code looked fine it was just that it was ordering wrong and the kernel also had a lot of
useful logging systems nowadays for instance if your bug isn't causing a crash but is definitely
repeatable there's the there's a lot of kernel logging you can use it to efficiently export data
from the kernel and docket elsewhere for instance I had a problem with a take rate
DMA controller now every few hours you'd get a blip in the audio no you could quite understand
why so really just add a couple of trace points the DMA driver where it was looking at things like
moving on to buffers reporting position I think things out in the DMA driver that were probably
where it was able to be export that data via the trace and just export it off the system you
could look at it later and analyze it so running it overnight just showed that there was a bug
in the positional parameters when there was an occasional buffer swap and parameters were not
often updated in the way we expected so there's basically just a check needed and you could then
do that so let's say this is probably old school but hopefully was interesting to people
I haven't got ran to releasing the slides I didn't get time to get these formatted out
I've designed it decided to change a lot of our formatting or last minute so
yeah hopefully we'll be putting them up I don't know if you know be somewhere we can put a url for that
anyway hard question I'll make a blog post about this the same but we have a couple of slides
of references these are like the issues some kernel code testing blog posts we had a number of
amazing discussions it's always nice to have people weigh in on stuff and then go well you know
what actually you might have did I start talking about this so thank you and hopefully
it might have been at least something interesting and may have given you an idea about
alternative ways obviously there are many ways to look at bugs and sometimes just talking to your
rubber duck might help and if you could explain it to a rubber duck you probably got to the point
where you actually understand it so there's a few minutes left don't know if anybody has questions
idea I meant thank you really enjoy your presentation thank you so much can you describe
how your kernel testing works so the question about how the kernel testing works so this is not
our kernel testing this is the the s boys edbot project has I'm not sure if it's still running
at the moment but that has information I think online somewhere google will probably give you an
idea it's found a number of issues if you search through the kernel logs there's a number of things
it's reported and people have gone it's testing around a wide range of kernels and architectures so
hopefully that answers your bug your question that's not bug
any further so great thank you for attending thank you for I should say thank you to co-think
so oh there's one more quick one yeah what motivated you to do big mian risk five
uh what motivated us to do risk and big nian risk five well I was just about to say that my company
thank you for my co-think to sponsor me to be here so I haven't had to pay for my own transport
what motivated us to risk so I did an amort of big nian about 10 years ago
because obviously we're consulting group we do things for people suddenly when that will cost you
five million dollars we'll have it done next year and I went um I'll do you for do you that for about
half a million and we've already two months now and obviously they went we'll take you we'll do it
great so we did that we got actually quite a bit done got the customer system going and then they
went hmm now not good do that but next thing is that this five haven't announced that big nian
data is that it's going to be available in new course um I needed something to do I need to
talk about because things to talk about don't come along too often and more importantly
no you'd done it before so we went and did it and went you know what this is going to be just a
quick project we'll see if it can be done at the worst the company might get some dog posts and
some talks out of it and at the best we can try and go to people like Ryzen go
my wallet is looking for a bit to finish the moment would you like to buy some consultancy services
but yeah yeah it was more of a let's try just try doing this it will be we'll spend about
four or five weeks of engineering time on it and um it turns out it's actually quite doable
however risk anybody actually takes on who knows but we'll we'll try and get some talks
so that's done so the one if we did the big engine and I think time is about up
great thank you