For whatever reason, this is not full of screen, but you know, all right, hey, welcome
everybody to our little presentation about how to break things.
My name is Marco Salbe, I'm fairly technical.
Let's just keep all this, we don't have too much time.
Let's go short agenda, why break things and how to.
Why it's fun and it helps you reproduce bugs.
It helps you learn new software, you know, like you get a new database, you need to learn,
you can start breaking it and see what it shows in the looks and that gives you a very
good experience, right?
And then I think what everybody is interested here, it helps you test software.
And you may tell me, reproduce bugs, but if the disk is full, it's not a bug, right?
Thing is, what is nice to have is this, right?
Proper handling of errors and elegant retry or elegant abortion that gives us information
and that we know it's going to be handled in a consistent way in production.
So you know, this run out of space on this and it says I'm going to retry for minutes.
So it knows that space can come back.
So how you get to have this is, you need to have a test where the disk space is fully
consumed.
So what we're going to be looking at is precisely this.
This was a very long presentation, it was planted for 90 minutes, so I have to cut it down
quite substantially.
Or for that, I can give you all the slides later.
And we're going to go through the slides, but I'm going to be doing mostly demo.
So we're going to switch to terminal in a minute.
So we're going to see TCQ disk, toxic proxy, carried the FS and C groups, PR limit,
tasks set, C surface, and hopefully we'll get time to a trace.
So let's get started.
TCQ disk is the transport control Q discipline.
And it helps you inject latency, corruption, reorder the packets in your TCP stream.
So essentially, it's a really, really useful tool.
It's the Swiss Army Knife of networking is poking.
So, correct packets, again duplicate packets, limit transfer rate, etc.
Let's go very quickly and make that demo.
So I have a few, let's clear this out.
Quick, just one second.
I was keeping those alive.
So I have some Docker machines here, Docker, C, T, Mark, one, oops, I'm already there.
Oh, I was already there.
Oh, that was, Jesus, I'm sorry, Docker, C, T, Mark, C, T, Mark, C, T, Mark, C,
so I'm just going to do very simple, let me go copy, paste my stuff.
So let's start a listener here, good Lord, why are you not copying things?
Ah, God's of demos are not with me today.
Jesus, what's wrong with you?
Say that again?
You're already in the container.
Oh, Jesus, yeah, I got nervous, I'm sorry, I'm new to this.
So I have a listener, you know, and then let's set up a stream that is going to push bites in there.
And you can see it's running hundreds of bits.
And what we're going to do is find out the interface for those Dockers.
And okay, that's not going to be the whole thing, perfect.
And then the bridge AD is going to give us, oops, something didn't work.
Okay, that's expected, Jesus, what a day.
Okay, come on.
I promise I practice.
Okay, that's good, and then bridge.
Okay, that's good.
And now I can do, get the IP address, okay.
And now I can actually use that bridge AD, oops, with this EQ disk.
And I will add a QDC playing to that device, and I'm going to say delay, right?
I'm going to add some delay between 500 microseconds to 10 milliseconds to 25% of the packets.
And, yep, there you go.
And, you know, my rate drop is substantially right.
And it's one megabyte, and we can show, oops, oops, oops.
This one?
Yeah, it's here.
Ah, I'm not sure if I can do this.
This, come on.
Okay, yeah, sorry.
Okay, so again, essentially root is the root Q.
Netem is network emulator, which is the QDC playing you are injecting.
And then the Netem emulator delay is the function, and then the parameters.
There are, you know, you can, again, instead of adding delay, you can
retry or read, sorry, reorder or you could add bandwidth limit.
So, the different commands for the same netem, which is again, the type of net work you're adding.
And you can see here, it is showing me that there is a QDC playing netem on the root,
and it's between 400 milliseconds, 400 microseconds, and 10 milliseconds.
And then we can remove it.
And there we go, and it goes back to 100 megabits.
Okay, so again, this is one way to do it.
You could also do it at the host level.
Let me go out of here, and then I need the V4.
And these guys are using host network.
So, will you see it?
Oh no, I did that one for toxic proxy, I'm sorry.
I thought that I did for this EQ disk.
Okay, then let's go to the next tool, which is, again, toxic proxy, and, okay, many, many.
Okay, toxic proxy, this was developed by the guys at Shopify, and again, it also allows you to do
net working errors, latency, limit bandwidth, trigger timeouts, and it has a rest API.
So, you can actually reach it through curl or whatever HTTP library you have.
And again, very simple, you create a configuration with a given name, oops, sorry.
And then, what port the proxy will listen, and what port will your application connect to go through
the proxy? And then you simply start toxic proxy like that.
Yes, I already have it running here. Let's see. Let's do it local, the host here.
So, I had to run my container using net host, and I'm exposing this port, and I'm running
just the Shopify, oops, so do I make this bigger, you know, using the Shopify toxic proxy container.
And then, I will set up my container here, and just set up my list in it, what's one with you,
and oops, oops, oops, I already use, okay, why? Oh, come on, oh, let's try it before.
I was also testing with net server, but it was just we complicated. Okay, so we have a list
and on my other Docker, I will do the pushing, and I'm not sure if I already have my proxy running.
No, okay, I have my proxy running, so I have to go there and start it.
Then I will have to, this will actually create the configuration instead of having in a JSON file,
I will just have a command through the toxic proxy CLI, and let's see how that goes.
Okay, and you can see on the left that it created a new proxy,
and now it's listening on the 666, on 999, sorry, so I can do this, and you can see this guy is on 666,
and this guy is on 999, and it's pushing by through the proxy. So now we can go ahead and add what it's called
toxicity and a toxic, and so again using the proxy proxy CLI, I do toxic add, and I want to add latency,
and I want to add 1000 microseconds of latency, and I'm going to give that an aim,
and it's going to work on the NCT stream, which if you notice is the one I use it here,
so it corresponds with that name, and oops, not there, here,
and you can see that the rate has plummeted, and we can go ahead and then remove the toxicity,
the toxic, oops, again, but I'm sorry, and you can see that the rate is at full speed again.
Okay, next, cherry DFS, it's hard to pronounce, has very fancy and poetic license,
and it can inject latency errors, and it can affect the specific IO C schools, so there is a
large list of C schools, I'll show you in a second, and this works on top of PUSFS, so you will need to
have PUSFS installed, and let's go, let's go, let's go, let's go,
so I did more profuse, I created my oops, sorry, I created my application directory,
and I carried the back end directory, and then I ran carried D, and I said on application that
I didn't even mount the carried D back end, and then I can do C D slash, won't application,
and I would run a C smash, I guess, no, let me just copy from here,
okay, I already run the C smash prepaid, okay, so I'm just going to execute the C smash,
and I'm inside the application database, so C smash is going to run on the fake file system we have,
and there you have, it's showing us some weight, and then I can go here, and I can say,
I will do net estate, and I'm going to check it's running, and it should be running on 1990,
so, and it's there, it's established it, so we have a working file system, and I will,
it has a set of examples, for recipes, very simple,
there we go, so, and it has like, these are examples, but very easy to extend, so I'm just
going to do a recipe, perhaps, delay, and we should see, why it's not working,
that should be working, I'm going to write this, why is that not working,
well, that's quite unexpected, this is benchmark,
well, that should be fine, I don't know why it's refusing to delay it, that's very weird,
and actually it should bring me back, you know, when I do recipes delay, it should bring me,
it should give me my prompt back, but this is quite embarrassing, okay, I'll skip this one,
I'll come back to it, if time allows, this is really cumbersome, oh, mode, oh, shift, oh, there's
V, oh, I'm not quite sure, yeah, multiplication that I did, am I there, yeah, okay,
let me, okay, I know, perhaps, let's just clear,
no, something, so it looks like a bug in Chariah that, really, it should work just like that,
I'm super sorry, promise I tested all these, have no clue what's not working, let me just try
one more, what if I do full, let's try this, all the recipes, who, no, that should make it fail instantly,
okay, again, it does look like a bug, I'm not sure why, but I'll look it up later, let's move on
because we don't have much time, okay, Cgroups, you know, is what you do containers with,
and you can also limit a number of resources using Cgroups, and what we're going to do is,
we have our C, C, C group and container, okay, oh, and here I created a slow disk
directory, you just create, you know, like let's do it again, slow, and once you create the
directory, it will create a number of virtual files that are actually an interface to the C group
configuration, so now what we can do is, we run that C's bench again,
six-manage, okay, okay, it's just not,
and, okay, and now I can, um, digrib, the X, X-s bench, and I can do Echo 5, 6, 7, 6,
into our C group blocks, so I'm adding C's bench to this C group, and now I'm going to flottle,
and I can do that,
what is it here? So, I will, um, find the idea of the device
is 259C, and then you do, oh, come on, and this is not a slow disk, it's just a slow, so I'm just
going to send, uh, I'm going to limit to 15 megabytes on the right BPS device, so, and once you do that,
oh, good, why is not for working?
I, you hear my word, I tested this a hundred times, I got bored,
five, five, six, eight, five, three,
okay, the guts of them are not with me today, okay, it's not going as I expected, I give you my word,
no, no, I know, I have absolutely no clue why this will not work,
no, the device, it's fine, I just see it right there, and, um, no, that was,
wow, okay, I don't know what to say, truly, truly embarrassed,
um, oh, you say that, let me try with my original C group and say if that works,
okay, no, not here, uh, here, and if I do, so I added the PAD and now I add the latency,
wow, I can fully tell you it's how it works, it's how it should work, okay, and I don't,
if you're a limit, let's see if this one works, one minute, okay, I'm so sorry,
I truly don't know why it didn't work, like, but I can tell you they work, I tested a hundred times,
and I guess I left something broken in my, all my previous testing, I'll be glad to sit down
outside and show you that how they actually do work, I'm truly sorry for the miss help,
and I will, okay, I will also answer any questions you have, but I'm truly truly sorry
that it didn't went as expected, I can't share the slides, I don't have a,
okay, I will make sure they are there, yeah, and I will also add the notes for all the examples,
I am truly, truly apologetic, but