Hello everyone, my name is also, it's my first time on fuzzdom and first time presenting
on a conference, so I'm a bit nervous, so don't judge me, and today I'm going to be talking
about testing support for multiple syndication methods and clickhouse using combinatorics
and behavior models.
I work as a QA engineer at Altinity and also pursuing my degree, master degree in data science,
LMU in Munich, and I got my Bachelor degree in Applied Mathematics and Computer Science
from Moscow State University.
A little bit about our company, we provide managed services and support for clickhouse, develop
features for clickhouse, and run other open source projects that are listed here, and feel
free to join us like community where we help each other discuss new topics and do such
things.
The feature that we're going to test today will be from the clickhouse, it's an open source
called a database, design from real-time analytics, it's super fast, super efficient, and
the feature is called multiple syndication methods, it's a recent addition to clickhouse
by Altinity developer for better security and flexibility, and it's just simply allows
a user to have multiple syndication methods either of the same type or different types.
So let's get familiar with this feature.
In clickhouse, in order to create user, we use create user query, or statement, and before
we could only have one authentication method for user, and we use create user, some
username, in this case it's name one identified with the authentication type, here it's
plain text password, the bike loss, and the password itself, here it's stringed my password.
But now you can specify multiple syndication methods, separated by Cuomo.
Key was specified three methods, and user will be able to again with all this free password
one, two, and three, and the internal presentation of the user also changed before authentication
type was stored as inam eight, but now it's stored in the form of array of inam's,
and the authentication parameters are also stored in the form of the array.
In order to change user, we're using Altay user statement, and before we could change
one authentication method with another, but now we can set one set of multiple syndication
methods with another set of multiple authentication method, and by using Altay user
identified with statement, we will override previous authentication methods with new ones,
and we will only be able to plug into clickhouse server with new authentication methods.
But what if we don't want override methods?
Now clickhouse supports Altay user identified statement, which will basically add new authentication
methods to the user. So by executing this query, user will be able to again with previous
passwords and also with new password 6 and 7. Also, one more statement, Altay user
data authentication method to new was introduced. It was inspired by my SQL discard all passwords,
and basically do it all the passwords except the most recently added one. So the user will
be able again with new password only. And also, Valetantilk was improved. Now you can have
a separate expiration type for each authentication method, but if you will use Valetantilk
was without identified with course, you will apply this of an expiration date to all authentication
method that user has by the moment you execute this query. Now let's check the example to
better and send this feature. So the first query, we create a user block identified with plain
text password by one, we could pass it by two, plain text password by three. He we created a user
with free authentication method and use a block can login to clickhouse password one to and free.
After that, we change links, box authentication method to plain text password by four,
we could pass it by five. Now user can use a block and I will login with password four and five
because we overwritten his authentication method. Now we are adding to a new authentication method
to block and he is able to login with password four, five, six and seven because we added six
and seven. And the last query we were resating authentication methods to new. And in this case,
Bob will be able to login to clickhouse only with password seven because big repository by seven
is the last authentication method that was added to him. So imagine how hard it will be to test
this feature by hand because you have so many permutation of this authentication method and also
a lot of different actions could be done. So this is why we're using combinatorial testing to test
this feature. So what is combinatorial testing? It's a way of testing software by checking different
combinations of input parameters. Here is a small example, we have free variables ABC and each
variable has two values. So in this case, we're going to get a test cases. But in real
world examples, number of variables is not just a handful, it stands 100 or even 1000 and the
range of values that this variable could take is also pretty huge. And this tree will grow exponentially
and you end up with a lot of different combinations that you want to test. And we do that to catch
any potential issues that interaction between variable could cause. So and this way we can
expand our test coverage and not focus on the just like one scenario tests. So but like generating
this combination is not enough. We also need to know the expected result for all these combinations.
And for that, we're going to use Oracle's. And when we're talking about combination test
Oracle problem arises, it's a problem when we cannot say the expected result for a given test case.
And the Oracle is just some function model or just something that can tell us the expected
result for a given test case. How are we going to use this Oracle? So we have a test case,
we're going to put it on the software that we're testing and also we're going to ask the Oracle,
what expected result for a given test case? Then we're going to compare this to how comes if they
are the same, then we're saying that this past otherwise test failed. So and there are different types
of Oracle's first one is automated Oracle's, it's just some algorithm that automatically can tell us
the expected result for a given test case. Also there are human-based Oracle's which rely on
human judgment or domain experts and there are hybrid approaches that combine automation with
human expertise. Today we're going to do this on creating automated Oracle's or behavior
models to automatically tell the expected result for generated combinations.
So the first thing that we want to do to test our multiple syndication method teacher
is to define the input parameters for the feature. And we're going to think of them in terms of
actions because it's most intuitive approach and we're going to answer the question, what are
the possible actions that user can perform with this feature? So basically a short list,
we can create user with multiple syndication methods, we can change user syndication methods,
we can add new syndication methods to user and we can object as syndication methods to the most recently
added method. And of course we can just drop a user. But testing this action separately does not
make much more sense because in real world people that use software do the sequence of actions
and we want to make sure that this sequence works correctly and that's why we will test the
sequence of these actions. Let's do some math and let's focus on one action create user
and calculate how many combinations, how many ways do we have to create user. But first we're
going to introduce some assumptions for the sake of simplicity. We will say that a user can
have no more than two syndication methods assigned or change direction and the two syndication methods
can only be selected from the following five types. In total in clickhouse we have 13 different
authentication types but for simplicity we will be working with five. So in order to create user
with one authentication method we obviously have five ways like choosing one of each of this and
what happened? Okay everything worked and let's calculate how many ways do we have to create
user with two authentication methods. So it's just be the combinations of two from five and we use
this formula and we're getting 10 ways. So in total we have 15 ways to create a user with each
user having no more than two authentication methods and this authentication method has selected
from five available types. The same math will be applied to auto user identified with statement
the same for auto identified with and for result authentication method to new we only have one
way because this query has not any parameters. So in total we will have 15 plus 15 plus one,
31 different ways of changing users authentication method with auto user statement. Now let's
check from create and then we will see free changing user queries. So for the first query we will
have 13 different ways to create user as we calculated before and for the second third and fourth
we will have very one ways. Then we take the Cartesian project of all these numbers and we
getting almost half a million of different sequences. That's a huge number and that will be so painful
to do that by hand. So that's why we need some ore code to tell the expected result for all
these combinations and why it decided to stick with four actions. So basically we have four states
user was created, user was changed, new authentication method were added to user and authentication
method were reset and by having the sequence of length four we will have all these states in the
sequence and we will also check all the transition from one state to another because we're taking
the Cartesian project. We take all 15 ways here or 31 here and we will check all the transition
from one state to another and we want to have efficient coverage without unnecessary complexity
because it's already too complex. So let's catch our test. So we have four queries, one Greek query,
three auto queries and after executing each query we will check that the result of execution is
correct and we will try to again took a quick house with authentication method seen in the
all previous query. So in this case we will try to log in with every authentication method
seen in Greek query after executing first auto we will try to log in with every authentication
in Greek query in auto query and in the end we will try to log in with every authentication method
that's seen in this sequence. Now let's check the so-called architecture of the test. So we have two
constructors, query user query construct and auto user query constructed that just basically
construct the different queries for us and we have two actions, actually three actions,
query user action and auto user action. They call the constructors, they construct the return
them, the queries, they execute the queries and after that they build the behavior state.
This is just the object that stores query user name identification methods and other parameters
but the most important the stores exit code and output message that we received from Greek house.
After that this behavior state is appended to the global behavior and after that this action
goes to the model and model computes the expected exit code and output message based on the current
state and based on the behavior and then it compares the expected exit code and message with one
stored in behavior state and if they are the same then test passed otherwise test failed.
Auto user action does the same, it's create state, append state to the behavior, goes to the model,
ask is everything okay and so on. For again to quick calls action we execute select current user
which returns the user that is currently executing the query and we check that this user is the
same as in state like if it tried to again with a valid user. And now let's check code but before
I will take up a words about test flows, we're using this framework for writing test programs,
it supports behavior, parallel combinatorial and requirement driven testing and if you want to know
more information about test flows you can visit the website. So let's check the code.
It's probably too small so I'm going to just explain what's going on here. It's a create user constructor.
It's just a super basic Python class that has some attributes and methods and each method just
generate a query step by step so first we create an instance of a class we have this query.
After we add username, we're getting this query, we're adding identified basically adding
just a string and then we call method to add authentication method to the query here we call
set with no password and we're adding identified with no password and in the end we're getting
perfectly valid click-out query we can execute. Here how model looks like that gives us an expected result
for every test case. So the main method of the model is the expect method which goes through
other expect method and each expect method is responsible to catch in the certain error,
certain exception. So the first one catches that no password cannot be used with
add identified statement because it does not make sense to have no password with another authentication
method it's not secure and that's why it's forbidden by design. So if we're not using
identified we will go to the next expect method which is another distinct user cannot be altered.
So if user is not we don't have user we cannot change his authentication method or we cannot do
anything with this user. So if we're not doing that we will go to the next expect method which is
no password cannot be used with another authentication method. It's just similar to the first one but for
example you cannot create the user. The query rate uses a user name identified with no password
and plain text password because if you have user identified with no password you can login with
every password you like. And so one takes for like all possible errors and if everything is
okay we will go to the last method it's expected okay which expects that we will not see
exceptions and execut will be zero. This is how test looks like first I could read to list
ways to create user ways to alter so this list will have 15 ways as we calculated after
adding all the change user method we will have 31 ways to change user here we take the
Cartesian product and finishing up with half million combinations and then we execute
combinations in parallel using a pull of threads to make the test faster and we call this scenario
we just execute an actions and then tries to login after every action. Let's check how the model works.
We first will check the simple expect method which checks that no password cannot be used with
add keyword. We basically checking that the current state is altered then we checking that add
identified is not empty and we checking that in the list of authentication method we don't have
no password. If we do have no password in this list we will expect to see this syntax error
from that click house and here are the examples of the bad query because we use no password with
add identified and here is the example with a good query. Here's more complex expect method that
goes through whole behavior and it checks that no password yeah no password cannot
exist with other authentication method. Basically we go through all the behavior here and we create
the authentication method that are currently available for a user we go state by state we create
this authentication method list and in the end we check that no password is not in this authentication
method list and if it is we expect the error otherwise everything is correct and we will go to the
next expect method. So this again the examples this query is not valid because we try to create
user with two methods and one of them is no password and here again the same example we try to
change user and one of the authentication methods is no password the query is not valid. We find
a couple of issues by doing this communal testing with behavior model and we also running our
test every other day in the ACT so if everything is changing in new quick calls releases we will
fix everything quickly reveal a test or review the feature itself and fix it and the full test code is
available here is a bit harder because we not have this assumptions in the code so if you want to check
can do that it's open source and thank you for your time I'm happy to answer any questions.
So are you running all of the tests every possible combination you need to continue to
defend or you try to run the subsets and hope that's enough to cover all of this workplaces.
Without assumptions that we had millions of combinations and we tried to run as much as we can but
we didn't run all of them because it's too much combinations and we'll take weeks or months to run
all of them. No we don't use any code coverage so you do one done combinations.
Yes sometimes we do just random and sometimes we use covering arrays so we say that we want to test
like every parameter or certain amount of times and we use covering arrays which is also the part
of test flows yes and we testing like each sequence is tested like each parameter to
test the certain amount of times yes for your test that you use the value of them until
that parameter that you use the one of the time passing in your test so that you can check
the four of the combinations duration and order changes that's time to pass it in the future.
Okay the question was I'm repeating questions so it will be available on the recording
about a while until and if I just specified time in the past and in the future or I like
wait in the test yes something like that. I have these problems without like changing time
and the quick holes so I use just sleeves so I said some expiration date and I sleep for some
time and they check that I cannot log in after that time and yes I just said like the previous
time and the future time. Talking about the topics I don't use loops just setting time in the
past and time in the future. The worst one I think it was the last one within cluster
close so like I created user one one note with certain authentication methods and I was not able
to log in with this user on other notes. Yes. How did you cover the authentication or
their full load use date and check how you see where these types of the combination methods were used.
The question was about if this technique can be used on another click hole features yes we use
combinatorial testing when we test attach partition with different partition keys and also I use
this feature to test gson webtock and authorization to click holes.
Okay then you can obviously reach this peak after the session. Yeah. Thanks very much. Thank you.