Okay so hello everyone nice to see a bigger room than last year was more convenient
last year because it almost happened so basically shortly about me I'm pushing some
stuff to get up you can find some repos in the presentation with some links to the
things that they do basically I want to discuss topic how we got here and from the point of
you of how the whole ecosystem looks at least from my perspective how I feel it looks like
and then I would like to show a little bit about story how I came from Java to closure
you're actually moving glass of developers from the Java to move to closure so they all go into
the same story so I want to tell you what was my motivation before the things that I'm doing
then I'll talk about a little bit of architecture patterns this comes back to the library from
the I presented last year and so last year we were almost at basically entering to production and
now we're only one year in production and we learn lots of stuff and we figure out lots of
stuff that how things can be done that works for us and I'll talk about you know I will
experience in production with with our library that we have built and then I'll hopefully write some
conclusion I said most of the stuff is based my experience if you if you think something I'm telling
you will not fit your environment I would have be happy to discuss with you later on so we are
basically building application which has around 10,000 active users and about 15 to 30 maybe up to
100 million records so basically this is the scope where this is built so what I have figured out
that lots of people are confusing simply these you know like we have all this nice framework
hello world they look so beautiful everything is nice in on presentation but I don't think the
people until they actually start working on something feel the complexity behind it so and I see
everywhere people I just designing systems to which I put from check boxes and it reminds me you know
on academia they are usually doing these papers where they are basically have to fit pick the
topic that is kind of popular but not too too strange so they can actually get a funding and I see
we have the same thing in software happening over and over again that people there are so much
architectural patterns in the company they have to put them check boxes to get the project approved
and then they do stuff and then start to in the end figure out how to actually implement a
business logic because they actually never even took it to consideration before and from from my
perspective at least we have so much library so much stuff happening and it's all like horrifying
when I see what people are pulling into the project just to complete some task which looks like
super simple to complete so from me this faulty looks very very horrifying and the other consideration
which came very in focus at least over last year especially now with door are coming in place
is the security and here what at least my fear is but also I mean I also the Uncle Bob was talking
about it basically that you know we will come to the point where something horrible will happen
and we will be blamed as a developers and then government will start regulating what we are
actually using for development and recently I was article that basically US government is saying
engineers to stop using CNZ plus plus for development which is for me I mean it's not like we are
using this stuff because it's fun we obviously have use cases we are using it at least I hope so
but yeah coming back to the first point it is also questionable so when I started coding
I was always like this open source and the decision it uses Java E was perfect everything
I love it so I use lots of spring boot at that time and what was when it designed
nice system we have these three layers in multiplayer architecture and the testing was kind of always
this weird thing that would come last and it was very very hard to actually squeeze it into you
had to mock all this stuff you have this spring boot data thing you have some functions there
you are mocking implementation some H2 database weird things you have never really felt that
I have actually tested the functionality I was only basically testing and writing to test the code actually
and usually what's noticeable when I would do some refactoring which didn't change the business
logic I would have to always throw away half of the test because they were completely used
because they were basically following the code so then I started to think how can I organize the
code to actually be able to test to my test actually testing this logic because if I do
or internal refactoring they should not actually change the implementation itself and I stumbled
at that point on the SQL S pattern and basic even sourcing so in Java I brought a small
library that will handle this to me and based on this already I could see the pattern
where I could actually have a small percentage layer which was generic I would have some
handlers and small routing code upfront and basically I could test my business logic completely
from outside so my internal service would completely be isolated I could test it from outside
and whatever I changed it side it would not affect the business logic from the outside
and in 2019 I got opportunity to work a new green field project and I said okay let's start
something different so we need some dynamic language because we need to be related to data driven as well
so we decided to the closure and it was amazing so the difference between the closure and most
of that language it's all about data and all there are a couple of functions but whatever you do
you always use map produce you are putting some stuff into map getting some stuff about it
to map and filtering so it was super easy to make very complex things with this basic acronyms
and this was basically the for me very very nice revelation after that and yeah I figured
out that architecture actually is important because it is actually affecting how we putting stuff
in the code so you know people read some business requirements they see I know we have some
graphs let's use graph database and they put a graph database well it is very easy to put it there
but you know two years later to maintain it is when when the pain comes or some times
years database and we did the same thing we started with postgres as a data storage and use
force because we had some search of course we had use open search and then figured out this is
not so cheap or easy to maintain and basically then after why we started thinking okay do we really
need anything more than for example postgres so postgres like can store noise build data some
JSON this structure data it can store also it has a very powerful query capabilities and so on so
it is very important to think what this how much data you will have and what are actual things
that you require and start small from up from down to up and then add new stuff afterwards not
upfront then there's always this topic whether to go microservices small it and there we
started microservices because everybody is doing it so it was fun to add stuff we had long
free to services and then you figured out that this thing does work so and then we started
config things together and we came in the end we have now around 20 services and we have
basic 3 criteria by something should be isolated so first thing is scalability so there's some
surface service some part of code that means huge amount of resources then you extracted outside
because it's helped with scale basically independently and it's cheaper to scale with small part
the other reason why we extract something is because it's using some weird library and this is
mostly coming to updating and security so for example I know we use PDF box and microservices
trying some documents or we use Chrome to render some HTML documents we extracted this
is separate service because then when there's a bug coming in or security vulnerability we just
replay this one service and this was also the second criteria we used for extracting something out
and the third but this is less important is when we see two teams using working with the same code
then we kind of tried to split it together because it resolves less conflict and it's less
less in the negotiation everything that remains outside of this we put together in kind of a
monolithic service and this is worked fast really best so first you have you can really test
huge scope in one of these service and you still have this capability of updating stuff independently
and what helped us and basically this is how we built our whole library is to design
things about more about state transition then actually mutation so we are building a big aggregate
so if you have a group of six and and if we put it together at one station because it is in memory
it is easy to update you can easily test it instead of you know having some slacks then joining
data in memory trying to figure out what is what and trying to connect this together so it was easier
you just want things together one big station you can store now and post this two gigabytes
station in the field so if you really need all the data to make any decision then just put it put it together
so yeah it is the core this is our library so this is the only thing that used for
almost entire system is that there are some outliers somewhere in the edge but most of the
stuff this is the whole thing we use the most important features we have is decorative dependencies
solutions so you in your handler you need to define what things you need and then you are picking
the data from the from the other services so you declare what you need and then the system is
pulling this for you this is very important because it makes API very simple everything will be the same
you you hide this resolution code into one small place and you just declare what you need instead of
everybody somewhere in the middle of the code pulling some dependency and and then trying to figure out
later on what was pulled there and how can I actually reflect on this thing so then this came
with nice combination of the colors API so we have our own front and client basically it's 300
lines of code it can create absolutely all of our services so we've wrote it four years ago it's
close to the same it works till now perfectly working and I'm in a small graph on the right side
basically all of our service is the same thing so we resolve the dependencies we will command handler
command handler is because of that very pure so you have all these data coming from the dependencies
you fetch your current state of your aggregate you process it and then your command handler
basically speeds out a couple of units which we are then storing back to the to the database and as
last step we are using then the outbox pattern to store things that need to be distributed to other
services so for example you create a user and you need to send the email so you would create a user
and you will store in database okay I need to send the email then this will be then string to the other
service of course this email service is this 2% for example where you don't have a pure command handler
and it would as far as possible from this code because you know this thing is dangerous and hard to
test so we then isolate these things out and usually use some words you know libraries to actually send
emails and so and what we discovered now in production if I have a problem in the system everything
is the same I can just go to database I can figure out okay what is my flow what was my
what was the other service doing and I can pinpoint I don't even know what the service is doing
but I can already figure out where the code might be that the second handling from just observing
from the database I don't even go to logs when I have a problem I take a request ID I can trace
in the database back to the service handler that actually handled this thing I pulled the data
and I write a unit test and I can figure out what the problem is and yeah so we are actually working
on the huge banking system so we have the workflow for the long processing we have then the
calculation engine which is calculating with multiple dimensions the consolidation tree we have
a document rendering so we managed to fit this whole system into the with with this same library
with this same set of principles and the other thing that was you know benefit as well was the
testing as said most of the stuff is pure we really have huge amount we have tests we have services
that have 20 commands and have 4,000 unit tests because we can really test everything that is
inside and we are deploying now 10 times per day to production throughout the day because we have
so much that people just don't even care and they they make a poor request they send it for approval
we get the proof goes to production nobody's brain actually about this anymore so what was how
biggest pain point that basically came like middle of last year was dependencies so we had our
library we had a full control what is using nobody's actually pulling dependencies and we try to
really minimize this as much as possible we are still working on to drop some things mostly things are
like that we are using like a key recipe for the connection pool because we didn't try to
write this on our own but all of the dependencies we have from outside we have for them we have
build them at least on our own because if the problem comes we need to be able to to fix them on
our own so basically that's why we that's main motivation why we're trying to minimize and
other dependencies are like a small closure wrapper we have a small wrapper around a Java HTTP client
before we were using some weird HTTP client but now as Java nicely implemented the client there
is no need to do that so we just use a Java HTTP client and we are doing basically a white listen library
so when we talk with a Java spring boot white listening was no go so we had a topic security
person always there it is impossible to white list libraries that we are going to use with this
approach it was we can do quite listing without problems we know what is running in our system we
know everything what is where and yeah we have also built some pipeline jobs where I can
pick a button up the entire system it goes over all the system fetches all the new dependencies
set up automatically pull requests and in one hour I can up the entire system so some critical
moment really it is like lock project and so on are basically not a problem anymore so what I can
what can I recommend use as much as possible what language offer and I think we should all go
in this direction to move you know these basic things for example for me the Java doesn't have a
JSON implementation it's a horrible thing I go to Python I go to no JSON I have a JSON there I mean
everything is working with JSON this is like for me they should all they should make it extendable
of course we can make better implementation but at least if I open a Java language I should be able
to serialization without adding some external dependencies so now Java has HTTP client I think
this was lots of misery I remember this moving HTTP client from a passage to be client from version
4.4 to 4.5 plus MS and these kind of things are now gone away and really try to stay to the basic
things and higher complexity be higher complexity because between a small layer and focus on
writing a pure business logic that can actually be testing yeah I said the architecture is very
important it doesn't influence all in your components you're going to use you have to really
think about how the implementation behind you be all to the story details this I mentioned and yes
excellent it's also interesting that you're doing a for bank which a notorious for having
crappy software right yes so have you considered using geeks for dependencies I know I know people
I'm just going to squeeze it in yes so we had to implement last year kind of exit strategy
we were sighted already actually try geeks but then it took us two days to implement it we didn't
have more time so unfortunately not to talk with us yes I will go so the next speaker can
set up who is the next Peter picker yeah he's not a speaker where's the speaker for the next one
okay you can set up and please repeat the questions and I also you know we'll
this is just back to back I'm going to ask you to because it's getting more busy to start squeezing in
okay so we don't have empty chairs in between anyway next question sorry
we have some property testing in some some especially in calculation because it's nicely
actually but most of the stuff is actually standard you didn't test but also property
is comes to an especially in calculation but you need to figure out what combination of
you're going to spit out what and this is helpful there as well
so first we try to figure out do we can we fix it somehow on our own without actually
employees especially in front and you know it sounds small things you can write a function
at five point you only to include dependencies we do the review of course within the team we
discuss what can be done we overchecked the pennies that later there and then we try to see
we've taken causation security so for example there's lots of this PDF rendering libraries
which you can use but we figure out okay we can render HTML with the Chrome right so we then
rely on Chrome and so but it's mostly discussion within the team and of course for code review
where everybody will then review if there's somebody at the some dependence in song
most of the studies that we use we have forked and we have our own builds and all I mean we
use polyclone but at least we need to prove that we can actually build it so to be able to
write on a fixed stuff and there was some bugs in like a Hikari CP which we had to fix this song
sorry the question what the question is how we analyze how we decide which dependencies we use
within the team and whether or not we build all of our dependencies the store plan
oh yeah
other questions
yes
so I've seen that basically how this is that we try to use more or less full system
dusting more and the system is like getting big and you cannot really do this that is a strategy
so we have more of most of the logic is tested with the unit is because the whole thing is
basically more so the service is the purest I can develop the whole service however it
will still be running locally we have then of course integration tests and the UITests which
cover the whole system the what was the important as the scalability of testing infrastructure and
infrastructure itself so we use serverless for this case and basically we are running thousands of
integration and unison and UITests basically to test the whole thing in the case when you
have a system then you need to define some boundaries of testing we didn't come to that for us
I think we have built quite a medium-sized system and we still need to do it but then we would really
have to define the boundaries of the system and how they communicate with each other and test it there
and