So, let us get started with this discussion. So, I will be discussing regarding package
testing and how molecule can be used to streamline your process along with Jenkins. So,
let me introduce myself. I am I am my name is Yosh. I work at Perkona in a QATM. So, we
basically test database packages and distributions before we you know release them. So,
the agent of for this talk will be I will be introducing what packages are what do we
mean by package testing and the release process where exactly do we fit then problems encountered
by the QAs like which are the major problems and issues that they encounter and what
molecule is how it can solve your issues. Then, we will go into the briefs of molecule
what it is and how to install it some key concepts regarding it. Then, we will give you some
example and also in the end there will be a demo regarding a certain scenario that
might be helpful. So, let us get started with the packages what do we mean by package.
So, a package can be a collection of a software that can be distributed. So, it can be an
RPM file or depth file, it can be tar balls, it can be Docker image and so on. So, anything
that can help in distribution of your software is a package and depending on the support
of that product that you have it differs based on the OS and architecture. So, you can
have a product but it can have multiple packages based on the architecture and OS. So,
once such example is of Perkona server that we have we basically have that product released
for different architectures and operating systems. So, that is what we will be dealing with
Debian and RPM packages in this example. Now, what do we mean by package testing? So,
package testing usually involves installing the packages then verifying the packages that
are installed. So, it is like version tests and all then feature execution. So, you test those
features and then there are upgrade and upgrade as depending on your product. Then, if you
have a clustering setup, you need to check whether or not the clusters work properly or not
an execute test accordingly. So, each package needs to be tested individually on the supported
infrastructure. So, that creates a need for having infrastructure is also one key issue when you
are doing things manually. So, lack of automation is one problem. Another problem is delays in
the infrastructure provisioning due to reliance on the other teams. So, let us say if you have a
QA team and you need a support for a new particular product or you want to test your product
for a particular OS. If you have a central Jenkins and you need to rely on the other teams for
adding over your node that creates delays. So, the back and forth communication is the key
issue in this case. So, having a lack of control causes this kind of problems. So, now, we will
see what molecule is. So, molecule is primarily designed to test sensible roles, playbooks and
collections. So, the core idea behind Ansible is to create infrastructure provision your
infrastructure and execute your roles collections on it and then destroy it once you are done with it.
So, it is an ideal tool for configuration validation. Now, there might be a question like why not
use Terraform and all. But, the thing is Terraform is good for infrastructure validation rather than
configuration validation which molecule provides out of the box due to Ansible support.
Now, it has a support for different set of environments. So, let us say if you have AWS,
you want to test your products on AWS, then heads nerve or any other custom provider,
you can utilize this set up easily and it is easy to learn. So, why molecule on a broader
term? So, it gives you flexibility based on your requirement. So, you can there is no strict rule
that you should follow a particular process when you are using molecule. So, it gives you that
flexibility when you are building things. Then, there is this speed. You can build things in such
a way that you can execute test in parallel for different environments. So, that saves time and
resources are optimally utilized because once you create environment and they are tested things
and the destruction of this environment is also ensured. So, you can be sure that you are not
wasting resources in this case and then the key thing is focus. So, the QA guys are not the
guys they need to deal with QA stuff. So, they need to check the product efficiently rather
than learning new tools and you know, relearning those things and wasting time on that.
Better to focus on the issues rather than you know learning the infrastructure related stuff.
So, having a config centric tool is good thing in this case and it has a wide support. Like,
you know you have testing frameworks like test in fra and bad test that you can utilize.
Also cloud providers are there. So, you have AWS, GCE, Azure and Hedsner providers available
which you can utilize. So, that makes it a good candidate for using. Now, let us see how
can we install molecule. So, the primary way to install molecule is using the capability of Python.
So, you can do that to install by molecule. But, there are certain dependencies that you need to
ensure for your Python environment. You can also use it to install from source using the
people installed command along with the repo demand the branch that you need to specify.
So, now let us see the concepts what are the concepts that you need to be aware of when you are
using molecule. So, there are key six concepts the first is of stage. Then there are drivers,
then there are platforms. Then there are provisioners, verifiers and scenarios. So, on the
broader terms, stages are the flow of the test. So, each step that you are executing is a stage,
that you are executing. Then the driver is basically where you are executing your stuff. So,
is it on AWS, is it on vagrant or Hedsner? So, drivers basically do the back end stuff that you use.
And then there are platforms. So, platforms basically are the properties of the drivers that you selected.
So, let us say if you selected EC2, it has its own set of configurations that are supported.
So, you follow those particular syntax. And then there are provisioners. So, provisioners basically
are responsible for setting up your environment and configuring them using your playbooks.
And there are verifiers, verifiers basically mean that you are verifying your existing setup
with the testing setup that you want to be verified with. It can be test in fra or it can be an
n civil playbook. And scenario is the entirety of this molecule setup. So, you run molecule test using
scenarios. So, yeah. So, this is a sample molecule dot YML file. Now, we will go into the depths
of each and every stage. So, first of all, what do we mean by stage? It is execution of each
and every thing that we are executing YML molecule. So, molecule has a life cycle. So, ideal
life cycle involves these steps. So, first of all, you installed role dependencies using molecule.
Like, let us say if you have a particular role that you want to use from Ansible Galaxy,
you utilize the dependency module to do that. Then there are linting checks. If you want your
Ansible playbooks to be checked while linting and all. So, you can do that. Then there is this
destroy stage where you actually destroy the infrastructure. So, ideally what folks do is before
creating their infras, they run the destroy stage to ensure that nothing remaining is done again.
So, for example, if in some cases you have some instances running and it is not ideal to run things
on that instance which was old. So, this just for the sake of insurance people run it before
creation. Then there is prepare stage. So, it is kind of like provisioning your infrastructure
with a set of tasks or playbooks that you have. Then converge. Convert is a key thing. So,
basically you write all your tests and all in the convert stage. Convert stage. It is basically
Ansible playbook. It can be a role or a collection. So, and then there is this hydropotums. So,
this seventh stage is like re-learning the converge just to ensure that things are running properly
or not. It verifies the converge again to see if there is any difference on the execution.
And then there is this side effect. So, side effect is like running converge again,
but with your own set of scripts. So, one use case might be let us say you set up a cluster using
converge stage. Side effect can help you out in once the cluster is set up. You can execute
test using those side effect steps. Then there is this verify. So, verify stage runs your verification
test. So, for example, let us say if you want to verify for configuration file exists or a
service file exists, you can do those checks using the verify stage. It can be an Ansible playbook,
but usually it is love testing tool like testing for where you define the Python scripts to check
the services existence. And then the clean up stage is like destroying your infrastructure.
So, that you are done with the tests. So, now let us see what are drivers. So, drivers are the
key thing that actually does the heavy lifting. So, these drivers are like plugins. They are not
by default available except for a default driver that Ansible that molecule has. So, you install
drivers using PIP command. So, let us say if you want to install Docker driver, you install this
using PIP installed molecule dash Docker. So, you run your test on Docker container. So, if you
want using the Docker driver, the same thing goes for very grant EC2 heads network cloud GCE. So,
these are the providers that you can utilize. So, it satisfies your three needs. The first one is
local testing. So, let us say if you want to test things locally, you can do so using this Docker
and very grant. If you want to run on cloud, you can utilize EC2 molecule heads network or GCE.
And then the deadly gated is something which is installed by default molecule. It has its own
set of configurations. So, usually it is looking for custom test that you want to execute on a
custom infra. So, it has a bit more complexity to it. So, let us see what do we mean by
platforms. So, we saw drivers. Now, platforms are the configurations of those drivers. So,
you can see for example, in this example, we are using AWS EC2 as a driver. It has its own
set of syntax like you know instance tag SSH user instance type. So, platform defines drivers.
So, that is the key concept behind platforms. And then there are provisioners. So, provisioner
is a step where you define the playbooks. So, by default they are stored in the molecule directory.
But, let us say if you have a custom requirement like you need to execute a specific playbook
for each and every step, you can define them in the provisioner. It can also have inventory details
like if you have a particular path where the inventories are configured, you can use them in your
setup. And yeah, let us see very fires. So, very fires are responsible for verification of your
configuration. It can be by default it is sensible playbook. But, you can change that and utilize
test in frog or etc. In this case, default verified or via mail playbook is used.
But, let us say if you use test in fry. In this case, the configuration is a bit different.
You need to ensure that you have a test folder present within the molecule directory. So,
this is dependent on the verified that you are selecting. So, yeah. And then there is this concept
of scenarios. Scenario is where the things actually start. So, you run molecule test using
scenarios. So, you can have a scenario that can have a single molecule file. And many such scenarios
can exist for a particular set of test that you have. So, in this case, you can see
molecule.yml file exists for you run to jami, you run to noble. So, there are different
scenarios based on OS in this example. And they are running for a single task.
You execute your scenarios using molecule test hyphenase command and the scenario name.
Now, based on your requirement scenarios can be a bit different. So, let us say in this example
that we were saying, you have scenarios for each OS executing for a single and simple playbook.
In this case, it is an uncivil rule. So, yeah. So, based on your requirement, things can be
a bit different, which can help you in having more flexibility in selection. So, it can be like
a single scenario based setup for a single rule where you define all the supported operating
systems in this particular setup. So, the role is configured in such a way that all operating
systems are covered in this scenario. You can have multiple scenarios for each OS for role or
a playbook as seen before. You can also have multiple scenarios for each OS for a different set of
playbooks. Let us say if you have a custom requirement where you want to ensure that certain
scenarios have a different set of runs. You can do so using the third setup. So, now, we will see
a demo where there are multiple scenarios for each OS within the same role. So, we will be testing
a role called PS. So, under the PS folder, you have three key folders. The first one is molecule,
second is playbooks and the last one is tasks. The molecule folder has three scenarios.
They been 12, you went to Jamie, you went to noble. Each having its own molecule.yml file.
And there are set of tests. For the sake of example, I have kept all the tests to be same
and they are located in the test folder. Then you have playbooks. Playbooks are very defined things.
So, cleanup.yml create destroy playbook and prepare the playbooks and tasks is basically what
is being executed. So, you can see a molecule.yml file for a particular scenario of debut and 12.
You can see first of all the dependency name galaxies been used. So, we are using Ansible Galaxy.
If you want to run a particular role in your setup, pre-existing in the galaxy library, you can
do so. Then you have a driver. It is an EC2 instance that we are using. Then we have a platform
that defines or EC2 instance. What are the things included in that instance? Including tags,
device and all. Then there is a provision or we are using Ansible Provisioner because we have
Ansible Playbooks that we want to execute things with. So, create has its own set of
yml file which we are referencing in this example. Then verify we are using test in
fries of verify and scenarios are sequences where we want to execute things one by one. So,
in the playbook.yml we are executing a role. We have defined it under the roles and this will be
executed and for the test script we are checking two tests first does the service exist named
mySQL. If this test is executed as true, this means that a service exists named mySQL and then
we are checking is the service running. So, these are the two checks that we are doing.
So, now I will show you the code for the Jenkins setup that we have. So, we are running Jenkins using
a code. So, the job is defined using this yml file using Jenkins job plugin of Python. So,
we define a job, we give it a name, we give project type, it is a pipeline project that we have.
We are also referencing the URL of the repository where you can find the grue file. We are also
giving it explicit script path of the grue file. So, this is the same repository that we are using.
So, here is the grue file that we will be used. So, first of all we will be using an agent
on a Jenkins worker node, it is a bookworm agent, then it has its options for credentials purposes.
So, let us say we are accessing AWS on the backend molecule needs to have access to AWS.
So, that is what we are defining over here using with credentials. It is a function which is defined
at the bottom. So, you can see right. These are the set of things that you need. You need a secret key
and access key ID. Then we have stages. So, we have 4 key stages in this example. The first one
is the name set up what name do we want this job to have. Then there is this checkout phase where
we are checking out this particular repository to execute our molecule tests. Because molecule
tests is present within this particular repository on GitHub. Then there is this prepared stage
where we are actually installing molecule before we run things. By default the worker is
plain bookworm. We need to ensure that we have molecule running on the worker. So, yeah, we install
molecule and then we run things in parallel. Now, how are we running things in parallel? So, we have
a function defined that has a rest of operating systems that we want to run tests on and then we
have a folder defined where our molecule scenarios are located. So, this is a function named
molecule parallel test which has two arguments operating systems and the path where our scenarios
are located. Let us see the function. So, the operating function system function has three operating
systems. You are going to knowable jammy and the bin 12. Then install molecule has a set of
batch commands that we used to install molecule for this particular bookworm worker.
And yeah, the molecule parallel test is defined in this way. So, we are mapping the inputs that
we are getting and then we are executing them in parallel as a stage in Jenkins. So, each scenario
is executed in parallel. So, this is how we execute the test. Now, let us see the locks. So,
you can see this job executed and all the stages are running in parallel after the based on the
operating system name. Let us open one particular example. So, we are using the example of the
novel. So, initially destroyed things if it existed then it created the instances.
Once the creation stage completed, we prepare by installing certain things that are needed for
the percolas server to install. Once the prepared stage was completed, we went to the
convert stage where we are actually installing the repo and enabling them. So, all packages are
installed during this particular stage. And then we are executing a verify test. So, we had a
set of tests that we are executing. That is this one. If server service is existing and if service
is running, you can see those tests passed because it was installed properly. And then we destroy
the instance. So, yeah, let us take a look at what is present in the particular playbook. So,
we are using EC2 instance. So, it has its own set of definitions that you need to follow. But,
they can be changed based on your requirement. So, in our use case, the permissions that we had.
So, we created this particular setup because we wanted to use this for other things. So, we are
following a particular structure of the create stage. The way in which you created your infrastructure
needs to be destroyed in a similar way. So, we are using the information that we got from the
create stage and we are using it to destroy it using this destroyed by ML file. Then there is this
playbook where we are actually executing the role. So, the role is defined in main dot by ML file.
As you can see, we are first of all installing percone release package. So, we want to enable
a particular repo. So, we have this tool called percone release that lets us do that. So, once
it is installed, we are basically enabling the repose that we need. So, for example, let us
see if we want to set test for percone server 8 4, we enable this particular repo. And then we
enable the extra db cluster repo. Then we install the packages. And once the packages are done,
we start those services. And then there is this utility which is used during testing. So, we install
that utility. So, this is how the setup is there. Yeah. So, that is it.
Anyone has any questions or queries? Yeah.
Can you speak louder, I didn't get the last step.
Can you speak louder? Actually, don't know the chef part that you are referring to. So,
I do know about cookbooks, but if chef has infrastructure provisioning setup, then it is similar
to chef in a way. But if it does not have a particular you know support for creating in
front destroying them, then I believe it is not similar to the thing that you are referring.
So, the molecule helps you in creating and destroying after your test. So, that is the thing.
There are folks that are doing that using their custom providers. So, it does have that support.
But you will have to look into the reports for those particular providers and install them using
those setup that they are doing. So, it is a community driven project. It might have a support for
the needs that you have. Yeah.
Yeah. There are multiple ways in which we do that. So, let us say you have a packaging
the testing repo and you want to test that particular rep after upgrading from mean repo.
So, you install things in the mean repo, then you reinstall your package from the testing
repo. But you need to check if that upgrade work properly or not. So, you run testing
but you cross reference them using the versions file. So, there are parameters involved.
But you use a particular config file with which you can double check your inputs. So, you have
parameters for installing the particular testing repo in the Jenkins. But if you want to test
and validate the versions, you need a external file. It can be a filing the repo named versions
file where you define the test versions in that. So, it is kind of manual work but you can automate
that when the packages are released in the testing repo. So, once packages are released, you can
rename and modify that particular thing in an automated command commit. So, that can solve that issue.
We are doing this using a versions file in case of per corner. Thank you.