All right, we have, so now we are scaling to two speakers in parallel, so very well
welcome to Manu and Carson from Highlight.
I think they will say something by themselves and, yeah, stages yours, here we go.
Second, hi, I'm Carson from Highlight and we are presenting today to talk about our work with
Ansible. I think we'll be fun with one microphone and two speakers, yeah, we want to present
something about our Ansible and we like to say we still like complexity, so we are still with
Doughgård and Postfix, maybe, yeah, maybe there are very, very nice new technologies coming
up, maybe there's one day time also to switch. Yeah, so we were very happy to see all these
developments, also we had a very good day yesterday meeting in the official mail stuff beforehand
and we're very much looking forward to see how StarWord will be able to scale in these bigger
setups that we are doing and we're also looking forward to Apache James as another option.
So what we usually do is that we have, we have these complex systems and we have to find ways
how to manage that and our choice of the tool was Ansible and Carson started with one project,
one playbook here and another one there and suddenly things moved on and we have created quite a
few roads. Yeah, over the time, so when the project is good, God complex, more complex and more
complex, we have added much more tools for that. Also, we have customers who are asking for
could you also deploy a web server, could you also deploy our database, our editor server,
also for all groupar and everything. Also, we have very, very different needs on our customer
there's still the customer who comes to us they okay, I want to have a single server with some special
things. There's more companies, they maybe want to have load balancing and high availability
but they only have two servers, it's okay, but they are also big infrastructures. There may be
governmental authorities, there may be universities, NGOs and so they want to be on premise
still and they have special needs for high availability for their databases, they still maintain.
So we have really different needs here. So look here, for the small service, we may be just
need to post-figs, the staff got us to be for main security and the lightweight web interface.
With the bigger things like here, this is typical where we do mostly the main security part
and the main routing in front of something. Something is mostly in our case exchange or then
CRM systems, ERP, there's also main processing. When you look here, it's quickly get complicated.
So we have databases in it, we have special routes for all main security, we have monitoring needs,
we have all the logging, all the firewall things that they're inside. They also want to do
maybe DNS with us. And one of the other needs often is that there has to be a segregation
of the network so that we have something that is in the peripheral and there nothing should have
any access direct access to some databases or anything like that. So this is also one of the needs
that why we still have this kind of more elaborate setup or as we were saying before with a
big shopping list to refer to the earlier comment. The other thing is that also our setup works well
for ISPs and universities where the big change to the previous is that we're also having the
backends to ourselves so that the central component of the off-cut postfix also then will be able
to store the mails in our own backends. And here also the groupback comes in where we also have
then components like there's a web server in front that the groupback itself was often many many
possible options you can set. And there also the every customer has his own needs.
Yeah, why we choose Ansible. We are at handling, we are doing mostly consulting.
So we do the projects with our customers, we enable them to to deploy such systems to
configure and administrators or systems. So we also tell them how all the things work so they can do
so they can maintain the infrastructure alone afterwards. As consultants it's the best thing is
we become in deployed something and go out. And Ansible is perfect for that because it does
not need any local service just needs SSH to the system. Also Ansible is very good together
facts from network so like live IP addresses like local certificates and so on. So we can
gather all the information from all the VMs from all the containers we have and put them together
and write them back. Think of a list of IP addresses. So you have a front and server which needs
all the addresses of the of the back and server to put it in the list. So they can be
they can be reached if possible so we can have an easy load balancing maybe.
Also we found it easy to maybe create passwords in the rollout or yeah also he's certificates
also on the other hand we have wrote some some roads which have some features other roads
maybe not have like here. The example when I have a roll for the Google so go it can
it can include the roll for for an SQL server creates a database creates a user for itself
and maybe also put in the other data for the tables.
Yeah next thing as consultants we we don't say okay we just work with dbn or with zuzu you have to
you have to work with our with our favorite distribution so we we go to the to our customers and
they have dbn Ubuntu redhead in all flavors sometimes zuzu so we have to all cover that.
Yeah and so the way we have set up the whole thing is that actually these specific roads are
available and continue to be developed and they are added as submodules into the bigger project
and by that we can continue to develop and their customers are able to and the users are
able to get the latest features from these roads. So what we do is we we bring this toolbox
become to a project and we configure and talk with the customers on what actually their exact
needs are and then we configure the setup and basically in the end once we have that we just have to
add our different servers into the different groups and by that we're defining their exact
what they need to do and we roll out so by doing this we easily can scale and add servers
just at the go. Also what this means is that disaster recovery all we need is a new machine
with where we can log in by SSH and and the moment we can roll out the Ansible whole completely
again and everything is back to the previous state. Yes so another one of the things that we often
have a need is that in these bigger setups we have to have different roads so we have a development
part where early adopting features can be tried we have a testing and then we have the production
so that everything that goes into production has gone through thorough Q&A and so the customers
actually have a very very flexible and easy thing. We wanted to the onboarding itself is just
by forking our initial template project and with this all of the submoves are already available
and then we do this configuration and we have maybe some custom settings and some custom files
and here we wanted to show something so let's see if that we get logged in.
Okay then the question. We just the question was where the Ansible say roads now for now
a published we just tried to publish it today on open code it's a German repository for
software for the German government thing yeah it didn't went so well because we had several
errors while uploading them so they are not completely there but we will fill it up in the next days
and so we will have to continue with the application like this. Okay so no don't you're
today sorry yeah yeah so the roads and in this Ansible this is the thing we we're all
are yeah how to say how our work was going in we've counted again I think there we had
now 44 roads and other roads are then like shown in the first slide they're typically we have
something like post weeks we have Daphcaut but when you have maybe post weeks you want to have
a small extra service like publish it wait also in the end we didn't even have that but
as an example so this is all covered by one roll the one roll and you will we will be always
created out of a template so they have basically all the same default options and default us so
it's when you create a new one you just edit the list of packages that should be installed
there's a default task to add files remove files also for creating new passwords or asking for
for the passwords like when you have secrets license files and so on it will be
will be encrypted locally and put also on the on the server side we have things like setting
up the firewall IP tables well if you want so you roll has the information which which ports
are needed so it can open the the specific port and just for all the systems which need to access it
yeah so the previous saying templating configuration files for for the service also activate
deactivating we we love to work with ETC keeper as it's making get repository out of the ETC
directory on on Linux systems so you have the whole history in in a local get and we also started
to do something like configuration dumps locally so you always have the latest
config dump from from the system and also try to start to write out in
write out documentation automatically so in in the end when you roll out the whole cluster you
we could create a big manual how to manage and how to manage the cluster later and how it is installed
yeah i do ask a keyboard i always use the us keyboard and i put a stupid password
okay so for our here we wanted to show you what so our basically this template looks like so we have
various locations that can be done so in the import and the roles and modules is all of these other
roles that we are just importing as get sub modules and then for what we actually only have to do
is in the inventory and the hosts we have this any style configuration file and here we can define
very simply what which server will actually have which kind of roles so the post fixed one
mail example is in the mx and then if we wanted to have another one it would be in the mail out
and so on so it's very very simple and easy and also horizontal scalable just by adding more to the
same group same as if you just want to have a single server just put in every group the same
server name you rest the standard automatically by Ansible okay but that's a bunch of
photos yeah five minutes and also you see here we have the files for the files for
is for custom files so we bring default configuration files with us they will be template it was
very events and so on and if you want to have a custom configuration extra or override any template
file we bring in you can put it under files and they will be automatically overruled our yeah
this will overrule our our default templates yeah a set specific for different distributions
and we decided to to take our template also for very small routes just wrote one for a
memcage maybe eight options also eight variables to set also it's nothing more general then
installing one package and drop one file on the other other side but we find it very easy to
still still use our template as it's look like the same in any case so it's more easy to
to understand in the end if all rules follow the same rules yeah this is when when we wrote out
to come complete cluster in the in the end was with Ansible this takes some some time maybe
30 minutes maybe 60 depends more on the or on the latency on your SSH then then about Ansible
yeah here's a link maybe you can click on it as said we we started the it's a morning to
to push them and then there was an error at open code those roads are not perfectly fine
in always as always they're missing much of the documentation we try to to have like
like a template version for the for the roads there they're not all in with the same skeleton
let's say but we will update them in the next days also we will put all the missing
all the missing roads online and what what we still have to decide where we want to go with
where we will have our issue tracker and so on but we will publish that maybe here
but I think also in our blocks of fair and so on yeah thank you
yes so I very much like maybe you can repeat the question then so okay so my question is
when I work with Ansible I somehow get the feeling that I'm always not sure how to avoid
configuration drift over time so an example would be you create a follow-up Ansible with some
files in it then you do version two of your roll and do you then put the deletion of the
folder in version two so are you taking all the stuff that you did in the past with you are
do you just change the flag from to absent or how do you handle this section everything
the question was how to work with config files that are missing in the next version so
how to delete something like that so Ansible is about how to what to do on the system
and it does not care about fire sets that are not needed in the in the new version let's say
this is why we have created a default list in every roll where you can put the paths in
which file or which directory should be deleted
everything we're going to ask about meet again if you have used it for connecting or if you have an
experience because we are currently put screen to implement it in our Ansible and
like know how to answer all the experience from the project
yeah question was about our experience with my to gen my to gen is a
tool framework to which can speed up Ansible very much
when it's working it's fun it speeds up really up to five times so I'm switching
constantly so currently it's somehow not able or was my to gen somehow not able at least in my
installation to install packages all other stuff is working so when yeah depends sometimes
I switch to the GitHub version and go back afterwards it's not for a stable environment
so the question was yeah so the question was if open code will be our central
available with the community to also accept merge requests and so on we haven't made up our
mind completely about this we wanted to go for the event of first and have it out so that's why
and we open code will be available for so our playbooks will be there it is not 100% sure that
we will use this as the main form with interaction with the community but we will publish it on our
blog and then also update it here okay yes then one more question
so the question was regarding updates if it and then if it would then kill the server and
bring it back up so yes we have changes that require a restart where we have a handler
and this handler wouldn't be called so when we use we then go to production we have
Ansible to go in serial so that this doesn't happen to everything at the same time
okay any further questions yeah okay this
so the question was how we test our roles after a change for example adding a new operating
system version not automatically yet so in the future we will have pipelines that will
run against those things to see that these basic functionalities will be there
yep so the idea is so you can update the roles and follow the change lock and nothing will
break in the end and we need a bit much bigger testing platform before that yep